Work recognition device, work recognition system, work recognition method, and program

Abstract

Provided is a work recognition device (12) comprising: an acquisition unit (121) that acquires sensor data including acceleration and angular velocity as measured in accordance with the movement of a worker in order to accurately recognize the work performed by the worker; an operation estimation unit (123) that uses time series data of the sensor data to estimate a series of operations performed by the worker; a work information generation unit (125) that uses the series of operations estimated with regard to the worker to generate work information indicating the work content performed by the worker; and an output unit (127) that outputs the generated work information.

Description

Task recognition device, task recognition system, task recognition method, and recording medium

[0001] The present disclosure relates to an activity recognition device, an activity recognition system, an activity recognition method, and a recording medium.

[0002] In healthcare settings, accurate recognition of the work being performed by workers such as nurses is required for them to perform their work efficiently. Non-Patent Document 1 reports a technology for recognizing activities in nursing care using an acceleration sensor. The method of Non-Patent Document 1 uses an acceleration sensor mounted on a mobile device carried by a nurse working at a healthcare facility. The method of Non-Patent Document 1 acquires acceleration data measured by the acceleration sensor according to the nurse's activities. The method of Non-Patent Document 1 prepares a dataset to be used for machine learning by labeling the acceleration data measured according to the activity. The method of Non-Patent Document 1 trains a model that outputs an activity according to input acceleration data through machine learning using the prepared dataset.

[0003] FR Sayem, et.al. “Feature-based Method for Nurse Care Complex Activity Recognition from Accelerometer Sensor,” UbiComp-ISWC '21 Adjunct, September 21-26, 2021, Virtual, USA, pp.446-451, 2021.

[0004] According to the method of Non-Patent Document 1, the activities of nurses working in healthcare facilities can be estimated using a model trained by the above-mentioned method. However, the method of Non-Patent Document 1 cannot recognize tasks that include multiple actions. Therefore, the method of Non-Patent Document 1 cannot accurately recognize the tasks performed by workers.

[0005] An object of the present disclosure is to provide an activity recognition device, an activity recognition system, an activity recognition method, and a recording medium that can accurately recognize activities performed by a worker.

[0006] A task recognition device according to one aspect of the present disclosure includes an acquisition unit that acquires sensor data including acceleration and angular velocity measured in response to the movements of a worker; a motion estimation unit that estimates a series of motions performed by the worker using time-series data of the sensor data; a task information generation unit that generates task information indicating the content of the task performed by the worker using the estimated series of motions of the worker; and an output unit that outputs the generated task information.

[0007] In one aspect of the task recognition method of the present disclosure, sensor data including acceleration and angular velocity measured in accordance with the movements of a worker is acquired, a series of actions performed by the worker is estimated using time-series data of the sensor data, task information indicating the content of the task performed by the worker is generated using the estimated series of actions of the worker, and the generated task information is output.

[0008] A program according to one aspect of the present disclosure causes a computer to perform the following processes: acquiring sensor data including acceleration and angular velocity measured in accordance with the movements of a worker; estimating a series of actions performed by the worker using time-series data of the sensor data; generating work information indicating the work content performed by the worker using the estimated series of actions of the worker; and outputting the generated work information.

[0009] According to the present disclosure, it is possible to provide an activity recognition device, an activity recognition system, an activity recognition method, and a recording medium that can accurately recognize activities performed by a worker.

[0010] FIG. 1 is a block diagram showing an example of the configuration of a task recognition system according to the present disclosure. FIG. 1 is a conceptual diagram showing an example of the configuration of a task recognition system according to the present disclosure. FIG. 2 is a conceptual diagram showing an example of the configuration of a task recognition system according to the present disclosure. FIG. 3 is a block diagram showing an example of the configuration of a measurement device included in the task recognition system according to the present disclosure. FIG. 4 is a block diagram showing an example of the configuration of a task recognition device included in the task recognition system according to the present disclosure. FIG. 5 is a conceptual diagram showing an example of motion estimation using a motion estimation model according to the present disclosure. FIG. 6 is a conceptual diagram showing an example of motion estimation using a motion estimation model according to the present disclosure. FIG. 7 is a conceptual diagram showing an example of task recognition using a task recognition model according to the present disclosure. FIG. 8 is a conceptual diagram showing an example of task recognition using a task recognition model according to the present disclosure. FIG. 9 is a conceptual diagram showing an example of task recognition using a task recognition model according to the present disclosure. FIG. 10 is a flowchart showing an example of the operation of the task recognition device included in the task recognition system according to the present disclosure. FIG. 11 is a block diagram showing an example of the configuration of a task recognition system according to the present disclosure. FIG. 12 is a conceptual diagram showing an example of the configuration of a task recognition system according to the present disclosure. FIG. 13 is a conceptual diagram showing an example of the configuration of a task recognition system according to the present disclosure. FIG. 1 is a conceptual diagram showing an example of the configuration of a task recognition device included in the task recognition system according to the present disclosure. FIG. 2 is a conceptual diagram showing an example of motion estimation using a motion estimation model according to the present disclosure. FIG. 3 is a conceptual diagram showing an example of task recognition using a task recognition model according to the present disclosure. FIG. 4 is a conceptual diagram showing an example of task recognition using a task recognition model according to the present disclosure. FIG. 5 is a conceptual diagram showing an example of display of task information output from the task recognition system according to the present disclosure. FIG. 6 is a flowchart showing an example of operation of the task recognition device included in the task recognition system according to the present disclosure. FIG. 7 is a block diagram showing an example of the configuration of the task recognition system according to the present disclosure. FIG. 8 is a conceptual diagram showing an example of the configuration of the task recognition device included in the task recognition system according to the present disclosure. FIG. 9 is a flowchart showing an example of operation of the task recognition device included in the task recognition system according to the present disclosure.FIG. 1 is a block diagram showing an example of the configuration of an activity recognition system according to the present disclosure. FIG. 2 is a flowchart showing an example of the operation of a measurement device provided in the activity recognition system according to the present disclosure. FIG. 3 is a flowchart showing an example of the operation of the activity recognition device provided in the activity recognition system according to the present disclosure. FIG. 4 is a conceptual diagram showing an example of activity estimation using an activity estimation model according to the present disclosure. FIG. 5 is a flowchart showing an example of activity estimation using an activity estimation model according to the present disclosure. FIG. 6 is a flowchart showing an example of the operation of a measurement device provided in the activity recognition system according to the present disclosure. FIG. 7 is a block diagram showing an example of the configuration of an activity recognition device according to the present disclosure. FIG. 8 is a flowchart showing an example of the operation of an activity recognition device according to the present disclosure. FIG. 9 is a block diagram showing an example of a hardware configuration for executing control and processing according to each disclosure.

[0011] Below, embodiments for implementing the present disclosure will be described using the drawings. In this disclosure, drawings used in the description of each embodiment relate to one or more embodiments. Furthermore, elements included in each drawing may apply to one or more embodiments. The embodiments described below are limited in a way that is technically preferable for implementing the present disclosure, but this does not limit the scope of the disclosure to the following. In all drawings used to describe the following embodiments, similar parts are designated by the same reference numerals unless otherwise stated. In the following embodiments, repeated description of similar configurations and operations may be omitted. The direction of arrows in the drawings is an example and does not limit the direction of data, information, signals, etc.

[0012] First Embodiment First, an activity recognition system according to the first embodiment will be described with reference to the drawings. The activity recognition system of this embodiment measures sensor data corresponding to the actions of a worker using a sensor mounted on a mobile device carried by the worker. The activity recognition system of this embodiment estimates the worker's actions using the sensor data measured in accordance with the worker's actions. The activity recognition system of this embodiment recognizes the activity performed by the worker by combining a series of actions estimated for a single worker. In the following, an environment in which a worker such as a nurse works at a work site such as a hospital is assumed. The method of this embodiment can be applied to work at any work site, not just at a work site such as a hospital.

[0013] (Configuration) FIG. 1 is a block diagram showing an example of an activity recognition system according to the present disclosure. The activity recognition system 1 includes a measurement device 10 and an activity recognition device 12. For example, the measurement device 10 and the activity recognition device 12 are implemented in a mobile device carried by a worker. For example, the measurement device 10 may be implemented in a mobile device carried by the worker, and the activity recognition device 12 may be implemented in a server or terminal device connected to the mobile device via a network. For example, the activity recognition system 1 may be realized as a wearable device including the activity recognition device 12 and the measurement device 10. Below, the measurement device 10 and the activity recognition device 12 will be described briefly, followed by individual descriptions of these devices.

[0014] The measuring device 10 is a wearable device with a communication function. For example, the measuring device 10 is realized by a wristband-type wearable device worn on the wrist. For example, the measuring device 10 may be realized by a wearable device that can be worn on the upper arm, head, ear, waist, ankle, foot, etc. Furthermore, the measuring device 10 may be realized by the functions of a sensor, processor, and communication device implemented in a mobile terminal carried by the worker.

[0015] The measurement device 10 includes a sensor capable of detecting the movement of a worker. For example, the measurement device 10 includes sensors such as an acceleration sensor and an angular velocity sensor. The sensor is not limited to an acceleration sensor and an angular velocity sensor as long as it can recognize the movement of the worker. The measurement device 10 measures sensor data. The sensor data is data in which measurement values ​​measured by a sensor are associated with the time at which the measurement values ​​were measured. The measurement device 10 outputs the measured sensor data to the task recognition device 12.

[0016] The task recognition device 12 acquires sensor data from the measurement device 10. The task recognition device 12 estimates the worker's actions using time-series data of the acquired sensor data. The task recognition device 12 estimates a series of actions included in the task to be recognized. For example, the task recognition device 12 estimates the worker's actions using a machine learning technique. The task recognition device 12 combines the estimated series of actions to recognize the content of the task performed by the worker. The task recognition device 12 generates task information including the recognized task content. The task recognition device 12 outputs the generated task information. For example, the task recognition device 12 outputs the task information to a management terminal (not shown) used by a manager who manages the workers.

[0017] FIG. 2 illustrates an example in which the task recognition device 12 is implemented in a mobile terminal 160 carried by a worker. The measurement device 10 is connected to the task recognition device 12 via wireless communication. The functions of the measurement device 10 may be implemented by a sensor, processor, or communication device mounted on the mobile terminal 160. Alternatively, the task recognition device 12 may be provided as an application installed on the mobile terminal 160. In the configuration example of FIG. 2 , the mobile terminal 160 is connected to the network NW via a wireless relay device 190. For example, the wireless relay device 190 may be implemented by an access point, a router, or the like. The mobile terminal 160 is connected to a management terminal 180 via the wireless relay device 190 and the network NW. The management terminal 180 is a terminal device used by a manager who manages the tasks of the workers. For example, the management terminal 180 may be implemented by a terminal device such as a stationary personal computer. For example, the management terminal 180 may be implemented by a mobile terminal device such as a tablet or smartphone.

[0018] FIG. 3 shows an example in which the task recognition device 12 is implemented in a server 170. The measurement device 10 is connected to a mobile device 160 via wireless communication. The functions of the measurement device 10 may be realized by a sensor, processor, or communication device mounted on the mobile device 160. In the configuration example of FIG. 3 , the mobile device 160 is connected to the task recognition device 12 implemented in the server 170 via a wireless relay device 190 and a network NW. The server 170 is also connected to a management terminal 180 via the network NW. The management terminal 180 may be connected to the mobile device 160 via the network NW. In this case, the functions of the task recognition device 12 may be implemented in the management terminal 180.

[0019] [Measurement Apparatus] FIG. 4 is a block diagram showing an example of the configuration of a measurement apparatus according to the present disclosure. The measurement apparatus 10 includes a sensor 110, a control unit 115, and a communication unit 117. The sensor 110 includes an acceleration sensor 111 and an angular velocity sensor 112. The sensor 110 may include sensors other than the acceleration sensor 111 and the angular velocity sensor 112. For example, the sensor 110 may be implemented in a mobile terminal (not shown). In this case, the control unit 115 and the communication unit 117 are realized by the control function and communication function of the mobile terminal. Furthermore, the measurement apparatus 10 may be hardware separate from the mobile terminal carried by the operator. In this case, the measurement apparatus 10 is connected to the mobile terminal so as to be able to communicate with it via short-range communication or the like.

[0020] The acceleration sensor 111 is a sensor that measures acceleration in three axial directions (also called spatial acceleration). The acceleration sensor 111 measures acceleration as a physical quantity corresponding to the movement of the worker. The acceleration sensor 111 outputs the measured acceleration to the control unit 115. For example, a piezoelectric, piezo-resistive, or capacitance type sensor can be used as the acceleration sensor 111. There are no limitations on the sensor used as the acceleration sensor 111 as long as it can measure acceleration.

[0021] The angular velocity sensor 112 is a sensor that measures angular velocity around three axes (also called spatial angular velocity). The angular velocity sensor 112 measures the angular velocity as a physical quantity corresponding to the movement of the worker. The angular velocity sensor 112 outputs the measured angular velocity to the control unit 115. For example, a vibration type or capacitance type sensor can be used as the angular velocity sensor 112. There are no limitations on the sensor used as the angular velocity sensor 112 as long as it can measure angular velocity.

[0022] The sensor 110 may be realized by, for example, an inertial measurement unit (IMU) that measures acceleration and angular velocity. An example of an inertial measurement unit is an IMU (Inertial Measurement Unit). The IMU includes an acceleration sensor 111 that measures acceleration in three axial directions and an angular velocity sensor 112 that measures angular velocity around three axes. The sensor 110 may be realized by an inertial measurement unit such as a VG (Vertical Gyro) or an AHRS (Attitude Heading Reference System). The sensor 110 may also be realized by a GPS / INS (Global Positioning System / Inertial Navigation System). The sensor 110 may be realized by a device other than an inertial measurement unit as long as it can measure physical quantities corresponding to the movement of the worker.

[0023] The control unit 115 causes the acceleration sensor 111 and the angular velocity sensor 112 to measure sensor data. For example, the control unit 115 causes the acceleration sensor 111 and the angular velocity sensor 112 to start measurement in response to a measurement start signal transmitted from the task recognition device 12. For example, the control unit 115 may cause the acceleration sensor 111 and the angular velocity sensor 112 to start measurement in response to detection of the worker's movement. Furthermore, the control unit 115 may be configured to cause the acceleration sensor 111 and the angular velocity sensor 112 to start measurement at a predetermined timing that is set in advance.

[0024] The control unit 115 acquires acceleration in three axial directions from the acceleration sensor 111. The control unit 115 also acquires angular velocities around three axes from the angular velocity sensor 112. For example, the control unit 115 performs analog-to-digital conversion (AD) on the acquired physical quantities (analog data), such as angular velocity and acceleration. The physical quantities (analog data) measured by the acceleration sensor 111 and the angular velocity sensor 112 may be converted to digital data by each of the acceleration sensor 111 and the angular velocity sensor 112. For example, an AD conversion circuit that AD converts the physical quantities (analog data), such as angular velocity and acceleration, may be provided. The control unit 115 outputs the converted digital data (also referred to as sensor data) to the communication unit 117. For example, the control unit 115 may temporarily store the sensor data in a storage unit (not shown). For example, the sensor data stored in the storage unit is transmitted from the communication unit 117 at a predetermined timing.

[0025] The sensor data includes at least acceleration data converted into digital data and angular velocity data converted into digital data. The acceleration data includes acceleration vectors in three axial directions. The angular velocity data includes angular velocity vectors around three axes. The acceleration data and angular velocity data are associated with the measurement times of the data. The control unit 115 may also apply corrections such as corrections for mounting errors, temperature corrections, and linearity corrections to the acceleration data and angular velocity data.

[0026] For example, the control unit 115 is realized by a microcomputer or microcontroller that performs overall control and data processing of the measuring device 10. For example, the control unit 115 includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), flash memory, etc. If the sensor 110 is implemented in a mobile terminal (not shown), the control unit 115 is realized by the control function of the mobile terminal.

[0027] The communication unit 117 acquires sensor data from the control unit 115. The communication unit 117 transmits the acquired sensor data to the mobile terminal 160 in which the task recognition device 12 is implemented. The communication unit 117 transmits time-series data of the sensor data. For example, the communication unit 117 transmits the sensor data to the mobile terminal 160 via wireless communication conforming to standards such as Bluetooth (registered trademark) or Wi-Fi (registered trademark). The communication function of the communication unit 117 may be conforming to standards other than Bluetooth (registered trademark) or Wi-Fi (registered trademark).

[0028] The timing of transmitting the sensor data is not particularly limited. For example, the communication unit 117 transmits the sensor data at a predetermined transmission timing. For example, the communication unit 117 may transmit the sensor data in real time in response to the measurement of the sensor data. For example, the communication unit 117 may store sensor data measured over a predetermined period in a storage device (not shown). In this case, the communication unit 117 transmits the sensor data stored in the storage device at a predetermined timing. The sensor data transmitted from the communication unit 117 is received by the mobile terminal 160 in which the task recognition device 12 is implemented. For example, the communication unit 117 may be configured to receive a measurement start signal from the mobile terminal 160 in which the task recognition device 12 is implemented. In this case, the communication unit 117 outputs the received measurement start signal to the control unit 115. If the task recognition device 12 is implemented in the server 170 or the management terminal 180, the sensor data transmitted from the mobile terminal 160 is transmitted to the server 170 or the management terminal 180 via a network NW such as the Internet or an intranet.

[0029] 5 is a block diagram showing an example of the configuration of an activity recognition device according to the present disclosure. The activity recognition device 12 includes an acquisition unit 121, a movement estimation unit 123, a storage unit 124, an activity information generation unit 125, and an output unit 127. The storage unit 124 stores an activity estimation model 143 and an activity recognition model 145.

[0030] The acquisition unit 121 acquires time-series data of sensor data transmitted from the measurement device 10. The time-series data of sensor data acquired by the acquisition unit 121 is used to recognize the worker's actions. For example, the acquisition unit 121 receives the sensor data transmitted from the measurement device 10 via wireless communication conforming to standards such as Bluetooth (registered trademark) or Wi-Fi (registered trademark). The communication function of the acquisition unit 121 may conform to standards other than Bluetooth (registered trademark) or Wi-Fi (registered trademark). When the task recognition device 12 is not implemented in the mobile terminal 160, the acquisition unit 121 acquires the sensor data via the network NW.

[0031] The movement estimation unit 123 estimates the movement of the worker using time-series data of the sensor data. The movement estimation unit 123 inputs the time-series data of the sensor data to the movement estimation model 143. In response to the input of the time-series data of the sensor data, the movement estimation model 143 outputs movement data indicating the movement of the worker. The movement estimation unit 123 estimates the movement of the worker using the movement data output from the movement estimation model 143. The movement estimation unit 123 estimates a series of movements included in the work performed by the worker. The series of movements estimated by the movement estimation unit 123 is used for work recognition by the work information generation unit 125.

[0032] For example, the motion estimation unit 123 may be configured to convert sensor data measured according to the worker's movements into normalized sensor data according to the worker's skeleton. For example, human digital twin technology can be used to estimate normalized sensor data according to the worker's skeleton. For example, skeletal data related to the worker's skeleton includes the three-dimensional positions of the worker's joints. The three-dimensional positions of the worker's joints can be measured using motion capture or the like. Using the three-dimensional positions of the joints, the lengths of the worker's parts, such as the arms, legs, torso, and shoulder width, can be calculated. The waveform pattern of the sensor data differs depending on the wearing position of the measurement device 10. Using the skeletal data, the movement of the measurement device 10 worn at any position can be converted into the movement of a reference position (e.g., the wrist). For example, using the skeletal data, the movement of the upper arm can be converted into the movement of the wrist. In other words, using the skeletal data, the movement of the measurement device 10 worn at any position can be normalized to the movement of the reference position. Using normalized sensor data normalized to the movement of the reference position, the worker's movements can be more accurately estimated.

[0033] The movement estimation model 143 outputs movement data indicating the movement of a worker in response to input of time-series data of sensor data. The movement data indicates one of a plurality of movements included in the task to be recognized. The movement estimation model 143 is a machine learning model. The movement estimation model 143 is generated by learning using a dataset in which time-series data of sensor data measured according to movements included in the task to be recognized is associated with labels indicating those movements. For example, the movement estimation model 143 is a model trained using, as training data, a dataset in which time-series data of sensor data measured according to movements included in the task to be recognized, measured for a plurality of workers, is associated with labels indicating those movements.

[0034] For example, the motion estimation model 143 may be configured to output motion data corresponding to the worker's skeleton. For example, human digital twin technology can be used to estimate motion data corresponding to the worker's skeleton. For example, skeletal data related to the worker's skeleton includes the three-dimensional positions of the worker's joints. The three-dimensional positions of the worker's joints can be measured using motion capture or the like. Using the three-dimensional positions of the joints, the lengths of the worker's parts, such as the arms, legs, torso, and shoulder width, can be calculated. The waveform pattern of the sensor data differs depending on the position at which the measurement device 10 is worn. Using skeletal data, the movement of the position at which the measurement device 10 is worn can be converted into the movement of a reference position (e.g., the wrist). For example, using skeletal data, the movement of the upper arm can be converted into the movement of the wrist. In other words, using skeletal data, the movement of the measurement device 10 worn at any position can be normalized to the movement of the reference position.

[0035] The algorithm for training the movement estimation model 143 is not particularly limited. For example, the movement estimation model 143 is generated by learning using a linear regression algorithm. For example, the movement estimation model 143 is generated by learning using a support vector machine (SVM) algorithm. For example, the movement estimation model 143 is generated by learning using a Gaussian process regression (GPR) algorithm. For example, the movement estimation model 143 is generated by learning using a random forest (RF) algorithm. For example, the movement estimation model 143 may be a machine learning model such as an incomplete heterogeneous variational autoencoder or a random forest RF. If the movement estimation model 143 is an incomplete heterogeneous variational autoencoder, it can estimate the movement included in the recognition task performed by the worker even if there is some loss in the time-series data of the sensor data.

[0036] 6 is a conceptual diagram showing an example of motion data estimation using a motion estimation model according to the present disclosure. Sensor data (time-series data) measured regarding a worker is input to the motion estimation model 143. In the example of FIG. 6, sensor data C1, sensor data C2, ..., and sensor data C n are input to the motion estimation model 143 (n is a natural number). The motion estimation model 143 outputs motion data M1 indicating motion 1 performed by the worker in response to the input of sensor data C1. The motion estimation model 143 outputs motion data M2 indicating motion 2 performed by the worker in response to the input of sensor data C2. The motion estimation model 143 outputs the motion data M2 indicating motion 2 performed by the worker in response to the input of sensor data C ｎ In response to the input of ｎ Output.

[0037] For example, tasks to be recognized include injections, intravenous drips, administering medication, assisting with meals, assisting with excretion, taking temperatures, measuring pulses, measuring blood pressure, transporting patients, sitting training, and posture training. If the task to be recognized is an injection, the action data includes actions of disinfecting hands and fingers, applying a tourniquet to a patient's arm, and operating a syringe. The actions estimated by the action estimation model 143 are not limited to the above examples, as long as they are included in the tasks to be recognized.

[0038] FIG. 7 is a conceptual diagram illustrating an example of motion data estimation using a motion estimation model according to the present disclosure. In the example of FIG. 7 , the task to be recognized is an injection. The injection, which is the task to be recognized, includes actions related to disinfecting hands, putting on gloves, applying a tourniquet, disinfecting the puncture site, operating the syringe, stopping bleeding at the puncture site, and removing the tourniquet. In the example of FIG. 7 , sensor data C1, sensor data C2, sensor data C3, sensor data C4, sensor data C5, sensor data C6, and sensor data C7 are input to the motion estimation model 143. In response to the input of sensor data C1, the motion estimation model 143 outputs motion data M1 representing the motion of the operator disinfecting hands. In response to the input of sensor data C2, the motion estimation model 143 outputs motion data M2 representing the motion of the operator putting on gloves. In response to the input of sensor data C3, the motion estimation model 143 outputs motion data M3 representing the motion of the operator applying a tourniquet to a patient's arm. In response to input of sensor data C4, motion estimation model 143 outputs motion data M4 representing the motion of the operator disinfecting the patient's puncture site. In response to input of sensor data C5, motion estimation model 143 outputs motion data M5 representing the motion of the operator operating a syringe. In response to input of sensor data C6, motion estimation model 143 outputs motion data M7 representing the motion of the operator stopping bleeding at the patient's puncture site. In response to input of sensor data C7, motion estimation model 143 outputs motion data M6 representing the motion of the operator removing a tourniquet from the patient's arm. The motion data M1, motion data M2, motion data M3, motion data M4, motion data M5, motion data M6, and motion data M7 output from motion estimation model 143 are used to recognize the tasks performed by the operator. Note that the example in FIG. 7 is merely an example and does not limit the tasks included in the injection task to be recognized.

[0039] The storage unit 124 stores a movement estimation model 143 and an activity recognition model 145. The storage unit 124 may also store data other than the movement estimation model 143 and the activity recognition model 145. For example, the movement estimation model 143 and the activity recognition model 145 may be stored in the storage unit 124 when the product is shipped from a factory. The movement estimation model 143 and the activity recognition model 145 may also be stored in the storage unit 124 at the time of calibration performed before use by a user. The movement estimation model 143 and the activity recognition model 145 may also be stored in an external storage device (not shown) accessible from the activity recognition device 12. In this case, the activity recognition device 12 may access the movement estimation model 143 and the disease risk estimation model via an interface (not shown) connected to the storage device. For example, the movement estimation model 143 and the activity recognition model 145 may be models available via an API (Application Programming Interface).

[0040] The work information generation unit 125 estimates the work content of the worker using a plurality of pieces of motion data estimated using time-series data of sensor data. The work information generation unit 125 estimates the work content of the worker using a series of motion data. The work information generation unit 125 inputs the plurality of pieces of motion data estimated using the time-series data of sensor data to the work recognition model 145. In response to the input of the plurality of pieces of motion data, the work recognition model 145 outputs work data indicating the work content of the worker. The work information generation unit 125 generates work information including the work content of the worker using the work data output from the work recognition model 145. For example, the work information generation unit 125 generates work information including the name or identifier of the worker, work time, work location, work target, work content, work status, etc.

[0041] The task recognition model 145 outputs task data indicating the task content of a worker in response to input of multiple pieces of task data. The task recognition model 145 is a machine learning model. The task recognition model 145 is generated by learning using a dataset in which a series of task data indicating tasks to be recognized is associated with labels indicating the task content. For example, the task recognition model 145 is a model trained using, as training data, a dataset in which a series of task data indicating tasks to be recognized, measured for multiple workers, is associated with labels indicating the task content.

[0042] There are no particular limitations on the algorithm used to train the task recognition model 145. For example, the task recognition model 145 may be generated by training using a linear regression algorithm. For example, the task recognition model 145 may be generated by training using a support vector machine (SVM) algorithm. For example, the task recognition model 145 may be generated by training using a Gaussian process regression (GPR) algorithm. For example, the task recognition model 145 may be a machine learning model such as an incomplete heterogeneous variational autoencoder or a random forest (RF). If the task recognition model 145 is an incomplete heterogeneous variational autoencoder, it is possible to estimate the task to be recognized performed by the worker even if there is some loss in the action data.

[0043] The task information generation unit 125 may be configured to estimate the task content of the worker based on a rule that is registered in advance. In this case, instead of the task recognition model 145, a table (not shown) in which task content associated with a combination of multiple actions is registered may be stored in the storage unit 124.

[0044] 8 is a conceptual diagram showing an example of task recognition using the task recognition model according to the present disclosure. A plurality of pieces of motion data estimated using time-series data of sensor data measured on a worker are input to the task recognition model 145. In the example of FIG. 8, task recognition model 145 receives task recognition data M1, task recognition data M2, ..., and task recognition data M3. nare input to the activity recognition model 145 (n is a natural number). The activity recognition model 145 receives the action data M1, action data M2, and action data M ｎ In response to the input, task data W indicating the task h performed by the worker is generated. h (h is a natural number).

[0045] FIG. 9 is a conceptual diagram illustrating an example of task recognition using a task recognition model according to the present disclosure. In the example of FIG. 9 , the task to be recognized is an injection. The task to be recognized, an injection, includes actions related to disinfecting hands, putting on gloves, applying a tourniquet, disinfecting the puncture site, operating a syringe, stopping bleeding at the puncture site, and removing the tourniquet. In the example of FIG. 9 , action data M1, action data M2, action data M3, action data M4, action data M5, action data M6, and action data M7 are input to the action estimation model 143. The action data M1, action data M2, action data M3, action data M4, action data M5, action data M6, and action data M7 are the same as those in the example of FIG. 7 . In response to the input of the action data M1, action data M2, action data M3, action data M4, action data M5, action data M6, and action data M7, the task recognition model 145 outputs task data W1 indicating that the task performed by the operator is an injection.

[0046] FIG. 10 is a conceptual diagram showing an example of task recognition using a task recognition model according to the present disclosure. In the example of FIG. 10, the task to be recognized is an injection. FIG. 10 shows an example in which multiple pieces of action data included in the task to be recognized are missing. In the example of FIG. 10, actions related to disinfecting hands, putting on a tourniquet, and removing the tourniquet are missing. In the example of FIG. 10, the task to be recognized, an injection, includes actions related to putting on gloves, disinfecting the puncture site, operating a syringe, and stopping bleeding at the puncture site. In the example of FIG. 10, action data M2, action data M4, action data M5, and action data M6 are input to the action estimation model 143. Action data M2, action data M4, action data M5, and action data M ６is the same as the example in Figure 7. In response to input of action data M2, action data M4, action data M5, and action data M6, task recognition model 145 outputs task data W1 indicating that the task performed by the worker is an injection. In this way, task recognition model 145 may be configured to estimate the task to be recognized even if any of the series of actions included in the task to be recognized is missing. If task recognition model 145 is configured in this way, it can estimate the task to be recognized even if any of the series of actions included in the task to be recognized is missing.

[0047] FIG. 11 is a conceptual diagram illustrating an example of task recognition using a task recognition model according to the present disclosure. In the example of FIG. 11, the task to be recognized is an injection. FIG. 11 illustrates an example in which the task to be recognized contains missing action data. In the example of FIG. 11, action data M4 indicating the action of disinfecting the patient's puncture site is missing. Assume that disinfecting the patient's puncture site is a required action in the task to be recognized, which is an injection. In the example of FIG. 11, action data M1, action data M2, action data M3, action data M5, action data M6, and action data M7 are input to the action estimation model 143. The action data M1, action data M2, action data M3, action data M5, action data M6, and action data M7 are the same as those in the example of FIG. 7. In response to the input of the action data M1, action data M2, action data M3, action data M5, action data M6, and action data M7, the task recognition model 145 outputs task data W1 indicating that the task performed by the operator is an injection. That is, the task recognition model 145 estimates that the task to be recognized is an injection, even though the task data M4 indicating the action of disinfecting the patient's puncture site, which is essential for the injection task, is missing. In the example of FIG. 11 , the task recognition model 145 outputs an identifier NG, indicating that the injection task, which is the task to be recognized, was not performed correctly, in association with information indicating that the estimated task to be recognized is an injection. In this way, the task recognition model 145 may be configured to output a task status indicating that the task to be recognized was not performed correctly if any of the series of actions included in the task to be recognized is missing. Furthermore, the task recognition model 145 may be configured to output a task status indicating that the task to be recognized was performed correctly if there is no missing action in the series of actions included in the task to be recognized. With the task recognition model 145 configured in this way, the administrator can recognize the task status indicating whether the task to be recognized was performed correctly.

[0048] The output unit 127 outputs the work information generated by the work information generation unit 125. There are no particular limitations on the destination to which the work information is output. For example, the output unit 127 outputs the work information via the network NW to a management terminal 180 used by an administrator who manages the work of workers. The administrator can understand the work content of the workers by viewing the work information displayed on the screen of the management terminal 180. For example, the output unit 127 may output the work information to a mobile terminal 160 carried by the worker. The worker can reconfirm the work content of the work he or she has performed by viewing the work information displayed on the screen of the mobile terminal 160. For example, the output unit 127 may output the work information to an external system that uses the work information. There are no particular limitations on the use of the output work information.

[0049] FIG. 12 is a conceptual diagram illustrating an example of a display of work information according to the present disclosure. The management terminal 180 is a terminal device used by a manager who manages the work of workers. The screen of the management terminal 180 displays work information output from the work recognition device 12. The screen of the management terminal 180 displays work information including the work content recognized for the worker in a display format that encourages decision-making by the manager who manages the work of the worker. The screen of the management terminal 180 displays the following work information: "Nurse: A, Time: 10:30, Location: Room B, Patient: C, Work Content: Injection, Work Status: OK, ...." The work information displayed on the screen of the management terminal 180 includes an identifier indicating the worker's name, work time, work location, work target, work content, and work status. By visually checking the work information displayed on the screen of the management terminal 180, the manager can accurately understand the work content of the worker.

[0050] (Operation) Next, the operation of the task recognition system 1 will be described with reference to the drawings. The operation of the task recognition device 12 included in the task recognition system 1 will be described below. FIG. 13 is a flowchart for explaining an example of the operation of the task recognition device 12. In describing the processing according to the flowchart of FIG. 13, the components of the task recognition device 12 will be described as the subject of the operations. The subject of the processing according to the flowchart of FIG. 13 may be the task recognition device 12.

[0051] 13, first, the acquiring unit 121 acquires sensor data measured in accordance with the movement of the worker (step S11). The acquiring unit 121 acquires time-series data of the sensor data from the measuring device 10.

[0052] Next, the movement estimating unit 123 estimates the movement of the worker using the time-series data of the acquired sensor data (step S12). For example, the movement estimating unit 123 inputs the time-series data of the sensor data to the movement estimation model 143. The movement estimating unit 123 estimates the movement of the worker using the movement data output from the movement estimation model 143. The movement estimating unit 123 estimates a series of movements included in the work performed by the worker.

[0053] Next, the work information generation unit 125 combines the estimated multiple actions to estimate the work content of the worker (step S13). For example, the work information generation unit 125 inputs multiple action data estimated using time-series data of sensor data into the work recognition model 145. The work information generation unit 125 uses the work data output from the work recognition model 145 to generate work information including the work content of the worker.

[0054] Next, the work information generating unit 125 generates work information including the work content of the estimated worker (step S14). For example, the work information generating unit 125 generates work information including the name and identifier of the worker, the work time, the work location, the work target person, the work content, and an identifier indicating the work status.

[0055] Next, the output unit 127 outputs the generated work information (step S15). For example, the output unit 127 outputs the work information via the network NW to the management terminal 180 used by the manager who manages the work of the workers. For example, the output unit 127 outputs the work information to the mobile terminal 160 carried by the worker. For example, the output unit 127 outputs the work information to an external system or the like that uses the work information.

[0056] As described above, the task recognition system of this embodiment includes a measuring device and a task recognition device. The measuring device is worn by a worker. The measuring device includes an acceleration sensor and an angular velocity sensor. The measuring device measures sensor data including acceleration and angular velocity in response to the worker's movements. The measuring device outputs the measured sensor data to the task recognition device. The task recognition device includes an acquisition unit, a memory unit, a motion estimation unit, a task information generation unit, and an output unit. The acquisition unit acquires sensor data including acceleration and angular velocity measured in response to the worker's movements. The memory unit stores a motion estimation model and a task recognition model. The motion estimation model and the task recognition model are machine models trained using machine learning techniques. The motion estimation model outputs motion data indicating the worker's movements in response to input of time-series data of the sensor data. The motion estimation unit uses the motion estimation model to estimate a series of movements performed by the worker. The task recognition model outputs task data indicating the task content of the worker in response to input of a series of motion data related to the task to be recognized. The task information generation unit uses the task recognition model to recognize the task performed by the worker. The task information generating unit generates task information indicating the details of the task performed by the worker based on the recognized task, and the output unit outputs the generated task information.

[0057] The task recognition system of this embodiment estimates a series of actions performed by a worker using time-series data of sensor data measured in accordance with the worker's movements. The task recognition system of this embodiment generates task information indicating the content of the task performed by the worker using the series of actions estimated for the worker. Therefore, this embodiment can accurately recognize the task performed by the worker.

[0058] In one aspect of this embodiment, the motion estimation unit converts sensor data measured by a measurement device attached to an arbitrary attachment position into normalized sensor data that would be measured if the measurement device were attached to a reference position, in accordance with the worker's skeletal data. The motion estimation unit estimates multiple motions performed by the worker using motion data output from a motion estimation model in response to input of time-series data of the normalized sensor data. The task information generation unit generates task information indicating the content of the task performed by the worker using task data output from a task recognition model in response to input of motion data estimated using the time-series data of the normalized sensor data. In this aspect, regardless of the attachment position of the measurement device, the sensor data can be converted into sensor data that would be measured if the measurement device were attached to a reference position. Therefore, according to this aspect, the worker's task data can be recognized with high accuracy regardless of the attachment position of the measurement device.

[0059] In one aspect of this embodiment, the task recognition device displays task information, including the task details recognized for the worker, on the screen of a management terminal used by the manager in a display format that encourages the manager to make decisions about the tasks of the workers. According to this aspect, the manager can accurately understand the task details performed by the worker.

[0060] Second Embodiment Next, a task recognition system according to a second embodiment will be described with reference to the drawings. The task recognition system of this embodiment estimates the task content of a collaborative task performed by multiple workers. In this embodiment, it is assumed that the multiple workers who will perform the collaborative task are determined in advance. There are no particular limitations on the collaborative task estimated by the task recognition system of this embodiment, as long as it involves multiple workers. For example, the task recognition system of this embodiment recognizes the collaborative task of two nurses transferring a patient lying in a bed to a stretcher or wheelchair. In the following, an environment is assumed in which workers such as nurses perform collaborative tasks at a work site such as a hospital. The method of this embodiment can be applied to collaborative tasks at any work site, not just at a work site such as a hospital.

[0061] (Configuration) Fig. 14 is a block diagram showing an example of an activity recognition system according to the present disclosure. The activity recognition system 2 includes multiple measurement devices 20 and at least one activity recognition device 22. Although Fig. 14 illustrates only one activity recognition device 22, the activity recognition system 2 may include multiple activity recognition devices 22. For example, the activity recognition device 22 is implemented in a mobile device carried by the worker. For example, the activity recognition device 22 may be implemented in a server or a terminal device. For example, the activity recognition system 2 may be realized as a wearable device including the activity recognition device 22 and the measurement device 20.

[0062] Each of the multiple measuring devices 20 has the same configuration as the measuring device 10 of the first embodiment. For example, the measuring device 20 is realized by a wristband-type wearable device worn on the wrist. For example, the measuring device 20 may be realized by a wearable device that can be worn on the upper arm, head, ear, waist, ankle, foot, etc. Furthermore, the measuring device 20 may be realized by the functions of a sensor, processor, and communication device implemented in a mobile terminal carried by the worker.

[0063] The task recognition device 22 acquires sensor data from multiple measurement devices 20. The task recognition device 22 estimates the actions of each worker using time-series data of the acquired sensor data. The task recognition device 22 estimates a series of actions included in the collaborative task to be recognized. For example, the task recognition device 22 estimates the actions of each worker using a machine learning technique. The task recognition device 22 recognizes the task content of the collaborative task by multiple workers by combining the series of actions estimated for each worker. The task recognition device 22 generates task information including the recognized task content of the collaborative task. The task recognition device 22 outputs the generated task information. For example, the task recognition device 22 outputs the task information to a management terminal (not shown) used by a manager who manages the workers.

[0064] FIG. 15 shows an example in which an activity recognition device 22 is implemented in a mobile device 260 carried by each of multiple workers 1 to m (m is a natural number). In the configuration example of FIG. 15 , the measurement device 20 worn by each of the multiple workers 1 to m is connected via wireless communication to the mobile device 260 carried by each of the multiple workers 1 to m. The functions of the measurement device 20 may be realized by a sensor, processor, or communication device mounted on the mobile device 260. Furthermore, the activity recognition device 22 may be provided as an application installed on each of the multiple mobile devices 260. In the configuration example of FIG. 15 , each of the multiple mobile devices 260 is connected to a network NW via a wireless relay device 290. For example, the wireless relay device 290 may be realized by an access point, a router, or the like. The mobile device 260 is connected to a management terminal 280 via the wireless relay device 290 and the network NW. The management terminal 280 is a terminal device used by a manager who manages the work of the workers. For example, the management terminal 280 is realized by a terminal device such as a desktop personal computer. For example, the management terminal 280 may also be realized by a portable terminal device such as a tablet or smartphone.

[0065] FIG. 16 shows an example in which the task recognition device 22 is implemented in a server 270. The measurement device 20 is connected to a mobile device 260 carried by each of multiple workers 1 to m. The functions of the measurement device 20 may be realized by a sensor, processor, or communication device mounted on the mobile device 260. In the configuration example of FIG. 16 , the measurement device 20 connected to each of the multiple mobile devices 260 is connected to a network NW via a wireless relay device 290. Each mobile device 260 is connected to the task recognition device 22 via the network NW. The server 270 is also connected to a management terminal 280 via the network NW. The management terminal 280 may be connected to each of the multiple mobile devices 260 via the network NW. In this case, the functions of the task recognition device 22 may be implemented in the management terminal 280.

[0066] FIG. 17 illustrates an example in which a mobile terminal 260 carried by worker 1 and a mobile terminal 260 carried by worker 2 are connected via short-range communication. In the configuration example of FIG. 17 , the task recognition device 22 is implemented in both the mobile terminal 260 carried by worker 1 and worker 2. The mobile terminal 260 includes the task recognition device 22 and a short-range communication device 27. The short-range communication device 27 is a device that enables the mobile terminal 260 to communicate with other nearby mobile terminals 260. For example, wireless communication technologies such as Bluetooth (registered trademark), ZigBee (registered trademark), and Wi-Fi_DIRECT (registered trademark) can be applied to the short-range communication device 27. Optical wireless communication technology may also be applied to the short-range communication device 27. A combination of multiple wireless communication technologies and optical communication technologies may also be applied to the short-range communication device 27.

[0067] 17 , the measurement device 20 worn by worker 1 measures sensor data corresponding to the movements of worker 1. The measurement device 20 worn by worker 1 transmits the sensor data to a portable device 260 carried by worker 1. In addition, the short-range communication device 27 of the portable device 260 carried by worker 1 receives the sensor data measured by the measurement device 20 worn by worker 2. The task recognition device 22 implemented in the portable device 260 carried by worker 1 estimates the task content of the collaborative task performed by worker 1 and worker 2, using the sensor data measured according to the movements of worker 1 and worker 2.

[0068] 17 , the measurement device 20 worn by worker 2 measures sensor data corresponding to the movements of worker 2. The portable measurement device 20 worn by worker 2 transmits the sensor data to a mobile device 260 carried by worker 2. In addition, the short-range communication device 27 of the mobile device 260 carried by worker 2 receives the sensor data measured by the measurement device 20 worn by worker 1. The task recognition device 22 implemented in the mobile device 260 carried by worker 2 estimates the task content of the collaborative task performed by worker 1 and worker 2, using the sensor data measured according to the movements of worker 1 and worker 2.

[0069] For example, the mobile terminal 260 transmits sensor data carried on strong radio waves at regular intervals in accordance with a predetermined short-range communication standard. The mobile terminal 260 receives the radio waves carrying the sensor data using a function for scanning strong radio waves transmitted at regular intervals from other mobile terminals 260. By using the proximity profile, the mobile terminal 260 can estimate the distance to the other mobile terminal 260 that is the source of the radio waves, based on the attenuation rate of the radio wave strength in accordance with the predetermined short-range communication standard.

[0070] FIG. 18 shows an example in which a mobile terminal 260 carried by worker 1 and a mobile terminal 260 carried by worker k are connected via short-range communication (k is a natural number smaller than n). In the configuration example of FIG. 18 , the mobile terminal 260 carried by worker 1 has an activity recognition device 22 and a short-range communication device 27. That is, the activity recognition device 22 is implemented in the mobile terminal 260 carried by worker 1. The mobile terminal 260 has a short-range communication device 27. That is, the activity recognition device 22 is not implemented in the mobile terminal 260 carried by worker k. The short-range communication device 27 is a device that enables the mobile terminal 260 to communicate with other nearby mobile terminals 260. The short-range communication device 27 has the same configuration as in FIG. 18 .

[0071] 18 , the measurement device 20 worn by worker 1 measures sensor data corresponding to the movements of worker 1. The measurement device 20 worn by worker 1 transmits the sensor data to a portable device 260 carried by worker 1. In addition, the short-range communication device 27 of the portable device 260 carried by worker 1 receives the sensor data measured by the measurement device 20 worn by worker k. The task recognition device 22 implemented in the portable device 260 carried by worker 1 estimates the task content of the collaborative task performed by worker 1 and worker k, using the sensor data measured according to the movements of worker 1 and worker k.

[0072] In the example of FIG. 18 , the measurement device 20 worn by worker k measures sensor data corresponding to the movements of worker k. The measurement device 20 worn by worker k transmits the sensor data to a portable device 260 carried by worker k. In addition, the short-range communication device 27 of the portable device 260 carried by worker k receives the sensor data measured by the measurement device 20 worn by worker 1. The task recognition device 22 implemented in the portable device 260 carried by worker k estimates the task content of the collaborative task between worker 1 and worker k, using the sensor data measured according to the movements of worker 1 and worker k. In the configuration example of FIG. 18 , the processing required for task recognition in the portable device 260 carried by worker k is unnecessary.

[0073] 19 is a block diagram showing an example of the configuration of an activity recognition device according to the present disclosure. The activity recognition device 22 includes an acquisition unit 221, a movement estimation unit 223, a storage unit 224, an activity information generation unit 225, and an output unit 227. The storage unit 224 stores an activity estimation model 243 and an activity recognition model 245.

[0074] The acquisition unit 221 acquires time-series data of sensor data measured for multiple workers from the measurement device 20 connected to mobile devices carried by the multiple workers. In the example of Fig. 19, the acquisition unit 221 acquires time-series data of sensor data measured for workers 1 to m (m is a natural number). The time-series data of sensor data acquired by the acquisition unit 221 is used to recognize the actions of each worker.

[0075] The movement estimation unit 223 estimates the movement of each worker using time-series data of sensor data measured for multiple workers. The movement estimation unit 223 inputs the time-series data of sensor data measured for multiple workers to the movement estimation model 243. In response to the input of the time-series data of sensor data measured for multiple workers, the movement estimation model 243 outputs movement data indicating the movement of each worker. The movement estimation unit 223 estimates the movement of each worker using the movement data output from the movement estimation model 243. The movement estimation unit 223 estimates, for each worker, a series of movements included in the work performed by the worker. The series of movements estimated for each worker by the movement estimation unit 223 is used for work recognition by the work information generation unit 225.

[0076] The movement estimation model 243 outputs movement data indicating the movement of each worker in response to input of time-series data of sensor data measured for multiple workers. The movement data indicates one of multiple movements included in the collaborative work to be recognized. The movement estimation model 243 is a machine learning model. The movement estimation model 243 is generated by learning using a dataset in which time-series data of sensor data measured according to movements included in the collaborative work to be recognized is associated with labels indicating the movements. For example, the movement estimation model 243 is a model trained using, as training data, a dataset in which time-series data of sensor data measured according to movements included in the collaborative work to be recognized, measured for multiple subjects, is associated with labels indicating the movements.

[0077] The algorithm for training the movement estimation model 243 is not particularly limited. For example, the movement estimation model 243 is generated by training using a linear regression algorithm. For example, the movement estimation model 243 is generated by training using a support vector machine (SVM) algorithm. For example, the movement estimation model 243 is generated by training using a Gaussian process regression (GPR) algorithm. For example, the movement estimation model 243 is generated by training using a random forest (RF) algorithm. For example, the movement estimation model 243 may be a machine learning model such as an incomplete heterogeneous variational autoencoder or a random forest RF. If the movement estimation model 243 is an incomplete heterogeneous variational autoencoder, it can estimate the movement included in the task to be recognized performed by the worker even if there is some loss in the time-series data of the sensor data.

[0078] 20 is a conceptual diagram showing an example of motion data estimation using a motion estimation model according to the present disclosure. Time-series data of sensor data measured for multiple workers 1 to m is input to the motion estimation model 243. In the example of FIG. 20, time-series data of sensor data measured for multiple workers 1 to m (m is a natural number) is input to the motion estimation model 243.

[0079] 20 illustrates how time-series data of sensor data is collected for each worker and input to the movement estimation model 243. The time-series data of sensor data may be input to the movement estimation model 243 in the order of measurement time without being collected for each worker. The sensor data includes an identifier indicating the worker. Therefore, even if the time-series data of sensor data is input to the movement estimation model 243 in the order of measurement, it is possible to determine the subject (worker) of the movement corresponding to the movement data output from the movement estimation model 243. Furthermore, the sensor data includes the time at which the sensor data was measured. Therefore, even if the time-series data of sensor data is input randomly to the movement estimation model 243, it is possible to determine the time at which the movement corresponding to the movement data output from the movement estimation model 243 was performed.

[0080] For worker 1, the sensor data C 11 , sensor data C 12 , ..., and sensor data C 1n is input to the motion estimation model 243 (n is a natural number). 11 In response to the input of 11 The motion estimation model 243 outputs the sensor data C 12 In response to the input of 12 The motion estimation model 243 outputs the sensor data C 1n In response to the input of 1n Output.

[0081] For worker m, the sensor data C m1 , sensor data C m2 , ..., and sensor data C mp is input to the motion estimation model 243 (p is a natural number). m1 In response to the input of m1 The motion estimation model 243 outputs the sensor data C m2In response to the input of m2 The motion estimation model 243 outputs the sensor data C mp In response to the input of mp Output.

[0082] The collaborative work to be recognized is not particularly limited as long as it is work performed collaboratively by multiple workers. For example, the collaborative work to be recognized includes transfer assistance, such as transferring a patient from a bed to a stretcher or wheelchair. For example, the collaborative work to be recognized includes positioning, such as changing the position of a patient who has difficulty moving autonomously. For example, the collaborative work to be recognized includes the task of carrying items by multiple workers. For example, the collaborative work to be recognized includes positioning, such as changing the position of a patient who has difficulty moving autonomously. If the task to be recognized is transfer assistance performed by two workers, the work content (transfer assistance) performed by those workers can be estimated by combining motion data corresponding to the multiple motions in the transfer assistance performed collaboratively by the two workers.

[0083] The storage unit 224 stores a movement estimation model 243 and an activity recognition model 245. The storage unit 224 may also store data other than the movement estimation model 243 and the activity recognition model 245. For example, the movement estimation model 243 and the activity recognition model 245 may be stored in the storage unit 224 when the product is shipped from a factory. The movement estimation model 243 and the activity recognition model 245 may also be stored in the storage unit 224 at the timing of calibration performed before use by a user. The movement estimation model 243 and the activity recognition model 245 may also be stored in an external storage device (not shown) accessible from the activity recognition device 22. In this case, the activity recognition device 22 may access the movement estimation model 243 and the activity recognition model 245 via an interface (not shown) connected to the storage device. For example, the movement estimation model 243 and the activity recognition model 245 may be models available via an API (Application Programming Interface).

[0084] The task information generation unit 225 estimates a collaborative task performed by multiple workers using multiple pieces of motion data estimated using time-series sensor data related to these workers. The task information generation unit 225 inputs the multiple pieces of motion data estimated related to the multiple workers to the task recognition model 245. In response to the input of the multiple pieces of motion data, the task recognition model 245 outputs collaborative task data indicating the task content of the collaborative task performed by the multiple workers. The task information generation unit 225 uses the collaborative task data output from the task recognition model 245 to generate task information including the task content of the collaborative task performed by the multiple workers. For example, the task information generation unit 225 generates task information including the names and identifiers of the workers, task time, task location, task target, task content, task status, etc.

[0085] The activity recognition model 245 outputs collaborative activity data indicating the activity content of a collaborative activity performed by multiple workers in response to input of multiple pieces of action data related to multiple workers. The activity recognition model 245 is a machine learning model. The activity recognition model 245 is generated by learning using a dataset in which a series of action data indicating actions included in the collaborative activity to be recognized is associated with labels indicating the content of the collaborative activity. For example, the activity recognition model 245 is a model trained using, as training data, a dataset in which a series of action data indicating actions included in the collaborative activity to be recognized, measured for multiple subjects, is associated with labels indicating the content of the collaborative activity.

[0086] There are no particular limitations on the algorithm used to train the activity recognition model 245. For example, the activity recognition model 245 may be generated by training using a linear regression algorithm. For example, the activity recognition model 245 may be generated by training using a support vector machine (SVM) algorithm. For example, the activity recognition model 245 may be generated by training using a Gaussian process regression (GPR) algorithm. For example, the activity recognition model 245 may be a machine learning model such as an incomplete heterogeneous variational autoencoder or a random forest (RF). If the activity recognition model 245 is an incomplete heterogeneous variational autoencoder, it can estimate a collaborative activity, which is a recognition target performed by multiple workers, even if there is some loss in the action data.

[0087] The task information generation unit 225 may be configured to estimate the task content of a collaborative task by multiple workers based on a rule-based rule registered in advance. In this case, instead of the task recognition model 245, a table (not shown) in which task content of a collaborative task associated with a combination of multiple actions is registered may be stored in the storage unit 224.

[0088] 21 is a conceptual diagram illustrating an example of task recognition using the task recognition model according to the present disclosure. A plurality of pieces of motion data estimated for a plurality of workers are input to the task recognition model 245. In the example of FIG. 21 , for worker 1, motion data M 11 , operation data M 12 , ..., and the motion data M 1n is input to the task recognition model 245 (n is a natural number). m1 , operation data M m2 , ..., and the motion data M mp is input to the task recognition model 245 (p is a natural number). 11 , operation data M 12 , ..., operation data M 1n , operation data M m1 , operation data M m2 , ..., operation data M mpIn response to the input of the motion data, the task recognition model 245 generates collaborative task data W g (g is a natural number).

[0089] Fig. 22 is a conceptual diagram showing an example of task recognition using task recognition models according to the present disclosure. Fig. 22 shows an example of estimating collaborative task data indicating the task content of collaborative tasks performed by multiple workers using task data estimated for each worker. Task recognition model 245-1 and task recognition model 245-2 are machine learning models.

[0090] Similar to the task recognition model 145 in the first embodiment, the task recognition model 245-1 outputs task data indicating the task content of each worker in response to input of multiple pieces of task data estimated for each worker. In the example of FIG. 22 , task recognition model 245-1 outputs task data M 11 , operation data M 12 , ..., and the motion data M 1n is input to the task recognition model 245-1 (n is a natural number). 11 , operation data M 12 , ..., operation data M 1n In response to the input of the task data W indicating the task s performed by the worker 1, s (s is a natural number). For worker m, the action data M m1 , operation data M m2 , ..., and the motion data M mp is input to the task recognition model 245-1 (p is a natural number). m1 , operation data M m2 , ..., operation data M mp In response to the input of t (t is a natural number).

[0091] The task recognition model 245-2 outputs collaborative task data indicating the task content of a collaborative task by multiple workers in response to the input of task data estimated for each worker. In the example of FIG. 22, the task data W estimated for worker 1 is s, ..., Work data W t is input to the task recognition model 245-2. The task recognition model 245-2 s , ..., Work data W t p In response to the input of the data, the collaborative work data W indicating the collaborative work g performed by the workers 1 to m is generated. g (g is a natural number) The configuration in FIG. 22 can be realized by adding an activity recognition model 245-2 after the activity recognition model 145 in the first embodiment.

[0092] The output unit 227 outputs the work information generated by the work information generation unit 225. There are no particular limitations on the destination to which the work information is output. For example, the output unit 227 outputs the work information via the network NW to a management terminal 280 used by an administrator who manages work by multiple workers. The administrator can understand the work content of the collaborative work by multiple workers by viewing the work information displayed on the screen of the management terminal 280. For example, the output unit 227 may output the work information to mobile terminals 260 carried by multiple workers. Each of the multiple workers can reconfirm the work content of the collaborative work they have performed by viewing the work information displayed on the screen of the mobile terminal 260. For example, the output unit 227 may output the work information to an external system that uses the work information. There are no particular limitations on the use of the output work information.

[0093] FIG. 23 is a conceptual diagram illustrating an example of a display of task information according to the present disclosure. The management terminal 280 is a terminal device used by an administrator who manages tasks performed by multiple workers. The screen of the management terminal 280 displays task information output from the task recognition device 22. The screen of the management terminal 280 displays the following task information: "Nurses: A and B, Time: 10:30, Location: Room B, Patient: C, Task Content: Transfer Assistance, Task Status: OK, ...." The task information displayed on the screen of the management terminal 280 includes the names of multiple workers, task time, task location, task target, collaborative task content, and an identifier indicating the task status. The administrator can grasp the tasks performed by multiple workers by visually checking the task information displayed on the screen of the management terminal 280.

[0094] (Operation) Next, the operation of the task recognition system 2 will be described with reference to the drawings. The operation of the task recognition device 22 included in the task recognition system 2 will be described below. FIG. 24 is a flowchart for explaining an example of the operation of the task recognition device according to the present disclosure. In describing the processing according to the flowchart of FIG. 24, the components of the task recognition device 22 will be described as the subject of the operations. The subject of the processing according to the flowchart of FIG. 24 may be the task recognition device 22.

[0095] 24, first, the acquisition unit 221 acquires sensor data measured in accordance with the movements of a plurality of workers (step S21). The acquisition unit 221 acquires time-series data of the sensor data from the mobile terminal 260.

[0096] Next, the movement estimation unit 223 estimates the movement of each worker using the time-series data of the sensor data related to the multiple workers (step S22). For example, the movement estimation unit 223 inputs the time-series data of the sensor data related to the multiple workers to the movement estimation model 243. The movement estimation unit 223 estimates the movement of each worker using the movement data output from the movement estimation model 243. The movement estimation unit 223 estimates a series of movements included in a collaborative work performed by the multiple workers.

[0097] Next, the task information generation unit 225 combines the multiple actions of each worker to estimate the content of the collaborative task by the multiple workers (step S23). For example, the task information generation unit 225 inputs the multiple action data estimated for the multiple workers into the task recognition model 245. The task information generation unit 225 uses the collaborative task data output from the task recognition model 245 to generate task information including the task content of the collaborative task by the multiple workers.

[0098] Next, the work information generation unit 225 generates collaborative work information including the work content of the collaborative work by multiple workers (step S24). For example, the work information generation unit 225 generates work information including the names and identifiers of the multiple workers, the work time, the work location, the work target person, the collaborative work content, and an identifier indicating the work status.

[0099] Next, the output unit 227 outputs the generated work information (step S25). For example, the output unit 227 outputs the work information via the network NW to the management terminal 280 used by the manager who manages the work of the workers. For example, the output unit 227 outputs the work information to the mobile terminal 260 carried by the worker. For example, the output unit 227 outputs the work information to an external system or the like that uses the work information.

[0100] As described above, the task recognition system of this embodiment includes a measuring device and an task recognition device. The measuring device is worn by a worker. The measuring device includes an acceleration sensor and an angular velocity sensor. The measuring device measures sensor data including acceleration and angular velocity in response to the worker's movements. The measuring device outputs the measured sensor data to the task recognition device. The task recognition device includes an acquisition unit, a memory unit, a motion estimation unit, a task information generation unit, and an output unit. The acquisition unit acquires sensor data including acceleration and angular velocity measured in response to the movements of multiple workers. The memory unit stores a motion estimation model and a task recognition model. The motion estimation model and the task recognition model are machine models trained using machine learning techniques. The motion estimation model outputs motion data indicating the movements of multiple workers in response to input of time-series data of sensor data. The motion estimation unit uses the motion estimation model to estimate a series of movements performed by the multiple workers. The task recognition model outputs task data indicating the content of a collaborative task by multiple workers in response to input of a series of motion data related to the collaborative task to be recognized. The task information generation unit uses the task recognition model to recognize a collaborative task performed by multiple workers. The task information generation unit generates task information indicating the content of the collaborative task performed by the multiple workers based on the recognized collaborative task. The output unit outputs the generated task information.

[0101] The task recognition system of this embodiment estimates a series of actions performed by multiple workers using time-series data of sensor data measured in response to the movements of the multiple workers. The task recognition system of this embodiment generates task information indicating the content of a collaborative task performed by the multiple workers using the series of actions estimated for the multiple workers. Therefore, this embodiment makes it possible to recognize a collaborative task performed by multiple workers.

[0102] Third Embodiment Next, an activity recognition system according to a third embodiment will be described with reference to the drawings. The activity recognition system of this embodiment determines which of multiple workers are performing a collaborative task based on the locations of the workers. The activity recognition system of this embodiment estimates the content of the collaborative task performed by the multiple workers determined to be performing the collaborative task. The following discussion assumes an environment in which workers such as nurses perform collaborative tasks at a work site such as a hospital. The method of this embodiment can be applied to collaborative tasks at any work site, not just at a work site such as a hospital. (Configuration) FIG. 25 is a block diagram illustrating an example of an activity recognition system according to the present disclosure. The activity recognition system 3 includes multiple measurement devices 30 and at least one activity recognition device 32. While FIG. 25 illustrates only one activity recognition device 32, the activity recognition system 3 may include multiple activity recognition devices 32. For example, the activity recognition device 32 is implemented in a mobile device carried by the worker. For example, the activity recognition device 32 may be implemented in a server or a terminal device. For example, the activity recognition system 3 may be realized as a wearable device including the activity recognition device 32 and the measurement device 30.

[0103] Each of the multiple measuring devices 30 has the same configuration as the measuring device 10 of the first embodiment. For example, the measuring device 30 is realized by a wristband-type wearable device worn on the wrist. For example, the measuring device 30 may be realized by a wearable device that can be worn on the upper arm, head, ear, waist, ankle, foot, etc. Furthermore, the measuring device 30 may be realized by the functions of a sensor, processor, and communication device implemented in a mobile terminal carried by the worker. Sensor data measured by the measuring device 30 is transmitted to the mobile terminal carried by the worker wearing the measuring device 30. The mobile terminal adds location information of the mobile device to the received sensor data. The mobile terminal transmits the sensor data with the added location information.

[0104] The task recognition device 32 acquires sensor data measured according to the movements of the workers carrying their mobile devices (described later) from mobile devices carried by each of the multiple workers. The sensor data includes location information of the mobile device that is the source of the sensor data. The task recognition device 32 uses the location information of the multiple mobile devices to determine the multiple workers performing a collaborative task. For example, the task recognition device 32 determines the multiple workers performing a collaborative task according to the distance between the mobile devices. For example, the task recognition device 32 determines the multiple workers performing a collaborative task according to the locations of the multiple mobile devices.

[0105] The task recognition device 32 estimates the actions of each worker using time-series data of sensor data measured according to the movements of multiple workers determined to be performing a collaborative task. The task recognition device 32 estimates a series of actions included in the collaborative task to be recognized. For example, the task recognition device 32 estimates the actions of each worker using a machine learning technique. The task recognition device 32 recognizes the task content of the collaborative task by multiple workers by combining the series of actions estimated for each worker. The task recognition device 32 generates task information including the task content of the recognized collaborative task. The task recognition device 32 outputs the generated task information. For example, the task recognition device 32 outputs the task information to a management terminal (not shown) used by a manager who manages the workers.

[0106] 26 shows an example in which the task recognition device 32 is implemented in a server 370. The measurement device 30 measures the task in a collaborative task area A where a collaborative task is being performed. c The area is connected to a mobile terminal 360 carried by each of multiple workers 1 to m working within the area. For example, c The workers who are within the range of the collaborative work area A are determined using the position information measured by the GPS (Global Positioning System) function installed in the mobile terminals 360 carried by the workers. c The plurality of workers within the range of the collaborative work area A may be determined by a common wireless relay device 390 to which the mobile terminals 360 carried by the workers are wirelessly connected. c The plurality of workers within the range are determined by the task recognition device 32 implemented in the server 370.

[0107] The functions of the measurement device 30 may be realized by a sensor, processor, or communication device mounted on the mobile terminal 360. In the configuration example of FIG. 26 , the measurement device 30 connected to each of the multiple mobile terminals 360 is connected to the network NW via a wireless relay device 390. Each mobile terminal 360 is connected to the task recognition device 32 via the network NW. Furthermore, the server 370 is connected to the management terminal 380 via the network NW. The management terminal 380 may be connected to each of the multiple mobile terminals 360 via the network NW. In this case, the functions of the task recognition device 32 may be implemented in the management terminal 380.

[0108] [Task Recognition Device] FIG. 27 is a block diagram showing an example of the configuration of a task recognition device according to the present disclosure. The task recognition device 32 includes an acquisition unit 321, a determination unit 322, a movement estimation unit 323, a storage unit 324, a task information generation unit 325, and an output unit 327. A movement estimation model 343 and a task recognition model 345 are stored in the storage unit 324. The acquisition unit 321, the movement estimation unit 323, the storage unit 324, the task information generation unit 325, and the output unit 327 are similar to the configurations included in the task recognition device 32 of the second embodiment. The movement estimation model 343 and the task recognition model 345 are similar to the configurations included in the task recognition device 32 of the second embodiment. Therefore, in the following, descriptions of the acquisition unit 321, the movement estimation unit 323, the storage unit 324, the task information generation unit 325, and the output unit 327 will be simplified or omitted.

[0109] The acquisition unit 321 acquires time-series data of sensor data measured for multiple workers from the measurement device 30 connected to portable devices carried by the multiple workers. The sensor data includes location information of the portable devices that are the sender of the sensor data. In the example of FIG. 27 , the acquisition unit 321 acquires time-series data of sensor data measured for workers 1 to m (m is a natural number). The time-series data of sensor data acquired by the acquisition unit 321 is used to recognize the actions of each worker.

[0110] The determination unit 322 acquires location information included in sensor data measured for multiple workers. The determination unit 322 uses the acquired location information to determine multiple workers performing a collaborative task. For example, the determination unit 322 determines multiple workers performing a collaborative task based on the distance between the mobile devices 360. For example, the determination unit 322 determines that multiple workers carrying mobile devices 360 that are located at a predetermined distance from each other are performing a collaborative task. For example, the determination unit 322 determines multiple workers performing a collaborative task based on the locations of the multiple mobile devices. For example, the determination unit 322 determines that multiple workers carrying mobile devices 360 located within a predetermined collaborative task range are performing a collaborative task. The determination unit 322 outputs sensor data measured for the multiple workers determined to be performing a collaborative task to the movement estimation unit 323. For example, the determination unit 322 may instruct the acquisition unit 321 to output sensor data measured for the multiple workers determined to be performing a collaborative task to the movement estimation unit 323.

[0111] The movement estimation unit 323 has the same configuration as the movement estimation unit 223 in the second embodiment. The movement estimation unit 323 estimates the movement of each worker using time-series data of sensor data measured for multiple workers determined to be working together. The movement estimation unit 323 inputs the time-series data of sensor data measured for the multiple workers to the movement estimation model 343. In response to the input of the time-series data of sensor data measured for the multiple workers, the movement estimation model 343 outputs movement data indicating the movement of each worker. The movement estimation unit 323 estimates the movement of each worker using the movement data output from the movement estimation model 343. The movement estimation unit 323 estimates, for each worker, a series of movements included in the work performed by the worker. The series of movements of each worker estimated by the movement estimation unit 323 is used for work recognition by the work information generation unit 325.

[0112] The movement estimation model 343 has the same configuration as the movement estimation model 243 of the second embodiment. The movement estimation model 343 outputs movement data indicating the movement of each worker in response to input of time-series sensor data measured for multiple workers. The movement data indicates any one of multiple movements included in the collaborative work to be recognized.

[0113] The storage unit 324 stores a movement estimation model 343 and an activity recognition model 345. The storage unit 324 may also store data other than the movement estimation model 343 and the activity recognition model 345. For example, the movement estimation model 343 and the activity recognition model 345 may be stored in the storage unit 324 when the product is shipped from a factory. The movement estimation model 343 and the activity recognition model 345 may also be stored in the storage unit 324 at the timing of calibration performed before use by a user. The movement estimation model 343 and the activity recognition model 345 may also be stored in an external storage device (not shown) accessible from the activity recognition device 32. In this case, the activity recognition device 32 may access the movement estimation model 343 and the activity recognition model 345 via an interface (not shown) connected to the storage device. For example, the movement estimation model 343 and the activity recognition model 345 may be models available via an API (Application Programming Interface).

[0114] The task information generation unit 325 has a configuration similar to that of the task information generation unit 225 of the second embodiment. The task information generation unit 325 estimates a collaborative task performed by multiple workers using multiple pieces of motion data estimated using time-series sensor data related to the multiple workers. The task information generation unit 325 inputs the multiple pieces of motion data estimated for the multiple workers to the task recognition model 345. In response to the input of the multiple pieces of motion data, the task recognition model 345 outputs collaborative task data indicating the task content of the collaborative task performed by the multiple workers. The task information generation unit 325 uses the collaborative task data output from the task recognition model 345 to generate task information including the task content of the collaborative task performed by the multiple workers. For example, the task information generation unit 325 generates task information including the names and identifiers of the workers, task time, task location, task target, task content, task status, etc.

[0115] The task recognition model 345 has the same configuration as the task information generation unit 225 of the second embodiment. The task recognition model 345 outputs collaborative task data indicating the task content of a collaborative task performed by a plurality of workers in response to input of a plurality of pieces of action data related to a plurality of workers.

[0116] The output unit 327 has the same configuration as the output unit 227 in the second embodiment. The output unit 327 outputs the work information generated by the work information generation unit 325. There are no particular limitations on the destination to which the work information is output. For example, the output unit 327 outputs the work information via the network NW to a management terminal 380 used by an administrator who manages work by multiple workers. The administrator can understand the work content of the collaborative work by multiple workers by viewing the work information displayed on the screen of the management terminal 380. For example, the output unit 327 may output the work information to mobile terminals 360 carried by multiple workers. Each of the multiple workers can reconfirm the work content of the collaborative work they performed by viewing the work information displayed on the screen of the mobile terminal 360. For example, the output unit 327 may output the work information to an external system that uses the work information. There are no particular limitations on the use of the output work information.

[0117] (Operation) Next, the operation of the task recognition system 3 will be described with reference to the drawings. The operation of the task recognition device 32 included in the task recognition system 3 will be described below. FIG. 28 is a flowchart for explaining an example of the operation of the task recognition device according to the present disclosure. In describing the processing according to the flowchart of FIG. 28, the components of the task recognition device 32 will be described as the subject of operations. The subject of operations according to the flowchart of FIG. 28 may be the task recognition device 32.

[0118] 28 , first, the acquisition unit 321 acquires sensor data measured in response to the movements of multiple workers (step S31). The acquisition unit 321 acquires time-series data of the sensor data from the mobile terminal 360. The sensor data includes location information of the mobile terminal 360. The location information of the mobile terminal 360 indicates the location of the worker carrying the mobile terminal 360.

[0119] Next, the determination unit 322 determines the multiple workers performing the joint work using the position information included in the sensor data measured for the multiple workers (step S32). For example, the determination unit 322 determines the multiple workers performing the joint work based on the distance between the mobile terminals 360. For example, the determination unit 322 determines the multiple workers performing the joint work based on the positions of the multiple mobile terminals.

[0120] Next, the movement estimation unit 323 estimates the movement of each worker using the time-series data of the sensor data related to the multiple workers (step S33). For example, the movement estimation unit 323 inputs the time-series data of the sensor data related to the multiple workers to the movement estimation model 343. The movement estimation unit 323 estimates the movement of each worker using the movement data output from the movement estimation model 343. The movement estimation unit 323 estimates a series of movements included in a collaborative work performed by the multiple workers.

[0121] Next, the task information generation unit 325 combines the multiple actions of each worker to estimate the content of the collaborative task by the multiple workers (step S34). For example, the task information generation unit 325 inputs the multiple action data estimated for the multiple workers into the task recognition model 345. The task information generation unit 325 uses the collaborative task data output from the task recognition model 345 to generate task information including the task content of the collaborative task by the multiple workers.

[0122] Next, the work information generation unit 325 generates collaborative work information including the work content of the collaborative work by multiple workers (step S35). For example, the work information generation unit 325 generates work information including the names and identifiers of the multiple workers, the work time, the work location, the work target person, the collaborative work content, and an identifier indicating the work status.

[0123] Next, the output unit 327 outputs the generated work information (step S36). For example, the output unit 327 outputs the work information via the network NW to a management terminal 380 used by a manager who manages the work of the workers. For example, the output unit 327 outputs the work information to a mobile terminal 360 carried by the worker. For example, the output unit 327 outputs the work information to an external system or the like that uses the work information.

[0124] As described above, the task recognition system of this embodiment includes a measuring device and an task recognition device. The measuring device is worn by a worker. The measuring device includes an acceleration sensor and an angular velocity sensor. The measuring device measures sensor data including acceleration and angular velocity in response to the worker's movements. The measuring device outputs the measured sensor data to the task recognition device. The task recognition device includes an acquisition unit, a determination unit, a memory unit, a motion estimation unit, a task information generation unit, and an output unit. The acquisition unit acquires sensor data including acceleration and angular velocity measured in response to the movements of multiple workers. The acquisition unit also acquires position information of the multiple workers. The determination unit uses the position information of the multiple workers to determine whether multiple workers are performing a collaborative task. The memory unit stores a motion estimation model and a task recognition model. The motion estimation model and the task recognition model are machine models trained using machine learning techniques. The motion estimation model outputs motion data indicating the movements of multiple workers in response to input of time-series sensor data related to multiple workers determined to be performing a collaborative task. The motion estimation unit uses a motion estimation model to estimate a series of motions performed by multiple workers determined to be performing a collaborative task. The task recognition model outputs task data indicating the content of the collaborative task performed by the multiple workers in response to input of a series of motion data related to the collaborative task to be recognized. The task information generation unit uses the task recognition model to recognize the collaborative task performed by the multiple workers. The task information generation unit generates task information indicating the content of the collaborative task performed by the multiple workers based on the recognized collaborative task. The output unit outputs the generated task information.

[0125] The task recognition system of this embodiment determines whether multiple workers are performing a collaborative task based on their locations. The task recognition system of this embodiment estimates a series of actions performed by the multiple workers using time-series data of sensor data measured according to the movements of the multiple workers determined to be performing a collaborative task. The task recognition system of this embodiment generates task information indicating the content of the collaborative task performed by the multiple workers using the series of actions estimated for the multiple workers. Therefore, this embodiment makes it possible to recognize a collaborative task performed by multiple workers in any situation.

[0126] Fourth Embodiment Next, an activity recognition system according to a fourth embodiment will be described with reference to the drawings. The activity recognition system of this embodiment uses sound data in addition to sensor data to estimate the activity content of a worker. Below, an example will be given in which an activity recognition device that recognizes activities using sound data is added to the activity recognition system of the first embodiment. The technique of this embodiment may also be applied to the activity recognition systems of the second and third embodiments. Below, an environment in which a worker such as a nurse works at a work site such as a hospital is assumed. The technique of this embodiment can be applied to activities at any work site, not just at work sites such as hospitals.

[0127] (Configuration) FIG. 29 is a block diagram showing an example of an activity recognition system according to the present disclosure. The activity recognition system 4 includes a measurement device 40 and an activity recognition device 42. For example, the measurement device 40 and the activity recognition device 42 are implemented in a mobile device carried by a worker. For example, the measurement device 40 may be implemented in a mobile device (not shown) carried by the worker, and the activity recognition device 42 may be implemented in a server (not shown) or terminal device connected to the mobile device via a network. For example, the activity recognition system 4 may be realized as a wearable device including the activity recognition device 42 and the measurement device 40. Below, the measurement device 40 and the activity recognition device 42 will be described briefly, followed by individual descriptions of these devices.

[0128] The measuring device 40 is a wearable device with a communication function. For example, the measuring device 40 is realized by a wristband-type wearable device worn on the wrist. For example, the measuring device 40 may be realized by a wearable device that can be worn on the upper arm, head, ear, waist, ankle, foot, etc. Furthermore, the measuring device 40 may be realized by the functions of a sensor, processor, and communication device implemented in a mobile terminal carried by the worker.

[0129] The measurement device 40 includes a sensor capable of detecting the movement of the worker. For example, the measurement device 40 includes sensors such as an acceleration sensor and an angular velocity sensor. The sensor is not limited to an acceleration sensor and an angular velocity sensor as long as it can recognize the movement of the worker. The measurement device 40 measures sensor data. The sensor data is data in which measurement values ​​measured by a sensor are associated with the time at which the measurement values ​​were measured. The measurement device 40 outputs the measured sensor data to the task recognition device 42.

[0130] The measurement device 40 also includes a microphone (described below). The microphone converts sounds (analog data) generated around the measurement device 40 into sound data (digital data) and collects the sound. The sound data is data in which the sound data collected by the microphone is associated with the time at which the sound data was measured. The measurement device 40 outputs the collected sound data to the task recognition device 42. The microphone function may be realized by a microphone mounted on a mobile terminal carried by the worker.

[0131] The task recognition device 42 acquires sensor data and sound data from the measurement device 40. The task recognition device 42 estimates the worker's movements and voice using the acquired time-series sensor data and sound data. The task recognition device 42 estimates a series of movements and voices included in the task to be recognized. For example, the task recognition device 42 uses a machine learning technique to estimate the worker's movements and voices. The task recognition device 42 combines the estimated series of movements and voices to recognize the content of the task performed by the worker. The task recognition device 42 generates task information including the recognized task content. The task recognition device 42 outputs the generated task information. For example, the task recognition device 42 outputs the task information to a management terminal (not shown) used by a manager who manages the workers.

[0132] [Measurement Device] FIG. 30 is a block diagram showing an example of the configuration of a measurement device according to the present disclosure. The measurement device 40 includes a sensor 410, a microphone 414, a control unit 415, and a communication unit 417. The sensor 410 includes an acceleration sensor 411 and an angular velocity sensor 412. The sensor 410 may include sensors other than the acceleration sensor 411 and the angular velocity sensor 412. For example, the sensor 410 may be implemented in a mobile terminal (not shown). In this case, the control unit 415 and the communication unit 417 are realized by the control function and communication function of the mobile terminal. Furthermore, the measurement device 40 may be hardware separate from the mobile terminal carried by the worker. In this case, the measurement device 40 is connected to the mobile terminal so as to be able to communicate with it via short-range communication or the like.

[0133] The sensor 410 has the same configuration as the sensor 110 of the first embodiment. The acceleration sensor 411 is a sensor that measures acceleration in three axial directions (also called spatial acceleration). The acceleration sensor 411 measures acceleration as a physical quantity corresponding to the movement of the worker. The acceleration sensor 411 outputs the measured acceleration to the control unit 415. The angular velocity sensor 412 is a sensor that measures angular velocity around three axes (also called spatial angular velocity). The angular velocity sensor 412 measures angular velocity as a physical quantity corresponding to the movement of the worker. The angular velocity sensor 412 outputs the measured angular velocity to the control unit 415.

[0134] The control unit 415 has the same configuration as the control unit 115 of the first embodiment. The control unit 415 acquires acceleration in three axial directions from the acceleration sensor 411. The control unit 415 also acquires angular velocities around three axes from the angular velocity sensor 412. For example, the control unit 415 performs analog-to-digital conversion (AD conversion) on physical quantities (analog data) such as the acquired angular velocities and accelerations to generate sensor data. The control unit 415 also acquires sound data from the microphone 414. The control unit 415 adds the acquired sound data to the sensor data. The control unit 415 outputs the sensor data with the added sound data to the communication unit 417.

[0135] The communication unit 417 has the same configuration as the communication unit 117 in the first embodiment. The communication unit 417 acquires sensor data from the control unit 415. The communication unit 417 transmits the acquired sensor data to a mobile terminal on which the task recognition device 42 is implemented. Note that the communication unit 417 may transmit the sensor data and the sound data separately. There are no particular limitations on the method of transmitting the sensor data and the sound data.

[0136] The sensor data transmitted from the communication unit 417 is received by a mobile device equipped with the task recognition device 42. For example, the communication unit 417 may be configured to receive a measurement start signal from a mobile device equipped with the task recognition device 42. In this case, the communication unit 417 outputs the received measurement start signal to the control unit 415. If the task recognition device 42 is equipped in a server or a management terminal, the sensor data transmitted from the mobile device is transmitted to the server or management terminal via a network NW such as the Internet or an intranet.

[0137] 31 is a block diagram showing an example of the configuration of a task recognition device according to the present disclosure. The task recognition device 42 includes an acquisition unit 421, a movement estimation unit 423, a storage unit 424, a task information generation unit 425, and an output unit 427. The storage unit 424 stores a movement estimation model 443 and a task recognition model 445.

[0138] The acquisition unit 421 has the same configuration as the acquisition unit 121 of the first embodiment. The acquisition unit 421 acquires time-series data of sensor data transmitted from the measurement device 40. The sensor data includes sound data. The time-series data of the sensor data and sound data acquired by the acquisition unit 421 is used to recognize the actions of the worker.

[0139] The movement estimation unit 423 estimates the movements and voice of the worker using time-series data of sensor data and sound data. The movement estimation unit 423 inputs the time-series data of sensor data and sound data to the movement estimation model 443. In response to the input of the time-series data of sensor data, the movement estimation model 443 outputs movement data indicating the movements of the worker. The movement estimation unit 423 estimates the movements of the worker using the movement data output from the movement estimation model 443. In addition, in response to the input of the time-series data of sound data, the movement estimation model 443 outputs voice data indicating the voice of the worker. The movement estimation unit 423 estimates a series of movements and voices included in the work performed by the worker. The series of movements and voices estimated by the movement estimation unit 423 are used for work recognition by the work information generation unit 425.

[0140] The movement estimation model 443 outputs movement data indicating the movement of a worker in response to input time-series data of sensor data. The movement data indicates one of multiple movements included in the task to be recognized. Furthermore, the movement estimation model 443 outputs voice data indicating the voice of the worker in response to input time-series data of sound data. The voice data indicates one of multiple movements included in the task to be recognized. The movement estimation model 443 is a machine learning model. The movement estimation model 443 is generated by learning using a dataset in which time-series data of sensor data and sound data measured according to the movements and movements included in the task to be recognized are associated with labels indicating those movements and movements. For example, the movement estimation model 443 is a model trained using, as training data, a dataset in which time-series data of sensor data and sound data are associated with labels indicating those movements and movements. The sensor data and sound data are measured according to the movements and movements included in the task to be recognized, measured for multiple workers.

[0141] The motion estimation model 443 may be configured to output motion data corresponding to the worker's skeleton. For example, human digital twin technology can be used to estimate motion data corresponding to the worker's skeleton. For example, skeletal data related to the worker's skeleton includes the three-dimensional positions of the worker's joints. The three-dimensional positions of the worker's joints can be measured using motion capture or the like. Using the three-dimensional positions of the joints, the lengths of the worker's parts, such as the arms, legs, torso, and shoulder width, can be calculated. The waveform pattern of the sensor data differs depending on the position at which the measurement device 40 is worn. Using the skeletal data, the movement of the position at which the measurement device 40 is worn can be converted into the movement of a reference position (e.g., the wrist). For example, using the skeletal data, the movement of the upper arm can be converted into the movement of the wrist. In other words, using the skeletal data, the movement of the measurement device 40 worn at any position can be normalized to the movement of the reference position.

[0142] The algorithm for training the movement estimation model 443 is not particularly limited. For example, the movement estimation model 443 is generated by training using a linear regression algorithm. For example, the movement estimation model 443 is generated by training using a support vector machine (SVM) algorithm. For example, the movement estimation model 443 is generated by training using a Gaussian process regression (GPR) algorithm. For example, the movement estimation model 443 is generated by training using a random forest (RF) algorithm. For example, the movement estimation model 443 may be a machine learning model such as an incomplete heterogeneous variational autoencoder or a random forest RF. If the movement estimation model 443 is an incomplete heterogeneous variational autoencoder, it can estimate the movement and voice included in the recognition target task performed by the worker even if there are some gaps in the time-series data of the sensor data and sound data.

[0143] FIG. 32 is a conceptual diagram illustrating an example of motion data estimation using a motion estimation model according to the present disclosure. Time-series sensor data and sound data measured regarding a worker are input to the motion estimation model 443. In the example of FIG. 32 , the task to be recognized is an injection. The injection task to be recognized includes actions related to disinfecting hands, putting on gloves, applying a tourniquet, disinfecting the puncture site, operating the syringe, stopping bleeding at the puncture site, and removing the tourniquet. The injection task to be recognized also includes voices related to patient confirmation, explanation of precautions, instructions to grasp the thumb, and instructions to open the hand. Note that FIG. 32 is merely an example and does not limit the motions and voices included in the injection task to be recognized. FIG. 32 illustrates an example in which motion data and voice data are estimated using a single motion estimation model 443. The model for estimating motion data and the model for estimating voice data may be configured separately.

[0144] In the example of FIG. 32 , sound data S1, sound data S2, sound data S3, sound data S4, sensor data C1, sensor data C2, sensor data C3, sensor data C4, sensor data C5, sensor data C6, and sensor data C7 are input to the movement estimation model 443. In response to the input of sound data S1, the movement estimation model 443 outputs voice data A1 representing a voice confirming a patient. In response to the input of sound data S2, the movement estimation model 443 outputs voice data A2 representing a voice explaining precautions. In response to the input of sound data S3, the movement estimation model 443 outputs voice data A3 representing a voice instructing the operator to clench their thumb. In response to the input of sound data S4, the movement estimation model 443 outputs voice data A4 representing a voice instructing the operator to open their hand. In response to the input of sensor data C1, the movement estimation model 443 outputs movement data M1 representing a movement of the operator disinfecting their hands. In response to input of sensor data C2, the motion estimation model 443 outputs motion data M2 indicating the motion of the operator donning gloves. In response to input of sensor data C3, the motion estimation model 443 outputs motion data M3 indicating the motion of the operator putting on a tourniquet on the patient's arm. In response to input of sensor data C4, the motion estimation model 443 outputs motion data M4 indicating the motion of the operator disinfecting the patient's puncture site. In response to input of sensor data C5, the motion estimation model 443 outputs motion data M5 indicating the motion of the operator operating a syringe. In response to input of sensor data C6, the motion estimation model 443 outputs motion data M7 indicating the motion of the operator stopping bleeding at the patient's puncture site. In response to input of sensor data C7, the motion estimation model 443 outputs motion data M6 indicating the motion of the operator removing the tourniquet from the patient's arm. The voice data A1, voice data A2, voice data A3, voice data A4, action data M1, action data M2, action data M3, action data M4, action data M5, action data M6, and action data M7 are used to recognize the work performed by the worker.

[0145] The storage unit 424 stores a movement estimation model 443 and an activity recognition model 445. The storage unit 424 may also store data other than the movement estimation model 443 and the activity recognition model 445. For example, the movement estimation model 443 and the activity recognition model 445 may be stored in the storage unit 424 when the product is shipped from a factory. The movement estimation model 443 and the activity recognition model 445 may also be stored in the storage unit 424 at the time of calibration performed before use by a user. The movement estimation model 443 and the activity recognition model 445 may also be stored in an external storage device (not shown) accessible from the activity recognition device 42. In this case, the activity recognition device 42 may access the movement estimation model 443 and the disease risk estimation model via an interface (not shown) connected to the storage device. For example, the movement estimation model 443 and the activity recognition model 445 may be models available via an API (Application Programming Interface).

[0146] The task information generation unit 425 estimates the task content of the worker using a plurality of pieces of motion data and voice data estimated using the time-series data of sensor data and sound data. The task information generation unit 425 estimates the task content of the worker using the time-series data of sensor data and sound data. The task information generation unit 425 inputs the plurality of pieces of motion data estimated using the time-series data of sensor data to the task recognition model 445. The task information generation unit 425 also inputs the plurality of pieces of voice data estimated using the time-series data of sound data to the task recognition model 445. In response to the input of the plurality of pieces of motion data and sound data, the task recognition model 445 outputs task data indicating the task content of the worker. The task information generation unit 425 generates task information including the task content of the worker using the task data output from the task recognition model 445. For example, the task information generation unit 425 generates task information including the worker's name or identifier, task time, task location, task target, task content, task status, etc.

[0147] The task recognition model 445 outputs task data indicating the task content of a worker in response to input of multiple pieces of task data and voice data. The task recognition model 445 is a machine learning model. The task recognition model 445 is generated by learning using a dataset in which a series of task data and voice data indicating the tasks to be recognized are associated with labels indicating the task content. For example, the task recognition model 445 is a model trained using, as training data, a dataset in which a series of task data and voice data indicating the tasks to be recognized, measured for multiple workers, are associated with labels indicating the task content.

[0148] There are no particular limitations on the algorithm used to train the task recognition model 445. For example, the task recognition model 445 may be generated by training using a linear regression algorithm. For example, the task recognition model 445 may be generated by training using a support vector machine (SVM) algorithm. For example, the task recognition model 445 may be generated by training using a Gaussian process regression (GPR) algorithm. For example, the task recognition model 445 may be a machine learning model such as an incomplete heterogeneous variational autoencoder or a random forest (RF). If the task recognition model 445 is an incomplete heterogeneous variational autoencoder, it is possible to estimate the task to be recognized performed by the worker even if there are some gaps in the action data and voice data.

[0149] The task information generation unit 425 may be configured to estimate the task content of the worker based on a rule that is registered in advance. In this case, instead of the task recognition model 445, a table (not shown) in which task content associated with a combination of a plurality of actions and voices is registered may be stored in the storage unit 424.

[0150] FIG. 33 is a conceptual diagram illustrating an example of task recognition using a task recognition model according to the present disclosure. In the example of FIG. 33 , the task to be recognized is an injection. The injection, which is the task to be recognized, includes actions related to disinfecting hands, putting on gloves, applying a tourniquet, disinfecting the puncture site, operating the syringe, stopping bleeding at the puncture site, and removing the tourniquet. The injection, which is the task to be recognized, also includes voices related to patient identification, explanation of precautions, instructions to clench the thumb, and instructions to open the hand. In the example of FIG. 33 , voice data A1, voice data A2, voice data A3, voice data A4, action data M1, action data M2, action data M3, action data M4, action data M5, action data M6, and action data M7 are input to the action estimation model 443. The voice data A1, voice data A2, voice data A3, voice data A4, action data M1, action data M2, action data M3, action data M4, action data M5, action data M6, and action data M7 are the same as those in the example of Fig. 32. The task recognition model 445 outputs task data W1 in response to input of the voice data A1, voice data A2, voice data A3, voice data A4, action data M1, action data M2, action data M3, action data M4, action data M5, action data M6, and action data M7. The task data W1 indicates that the task is an injection.

[0151] The output unit 427 outputs the work information generated by the work information generation unit 425. There are no particular limitations on the destination to which the work information is output. For example, the output unit 427 outputs the work information via the network NW to a management terminal used by an administrator who manages the work of the workers. The administrator can understand the work content of the workers by viewing the work information displayed on the screen of the management terminal. For example, the output unit 427 may output the work information to a mobile terminal carried by the worker. The worker can reconfirm the work content of the work he or she performed by viewing the work information displayed on the screen of the mobile terminal. For example, the output unit 427 may output the work information to an external system that uses the work information. There are no particular limitations on the use of the output work information.

[0152] The technique of this embodiment can also be applied to the second and third embodiments. For example, the task recognition system may be configured to identify individual workers using sound data or voice data of the workers. For example, the task recognition system may be configured to recognize the content of a collaborative task by multiple workers by performing voice recognition on the content of a conversation between multiple workers. For example, the task recognition system may be configured to determine that two workers are working collaboratively while conversing if they are alternating vocalizations. The task recognition system may also be configured to authenticate workers using sound data. For example, the task recognition system may be configured to issue a warning if an unauthenticated worker is performing a task that requires special skills.

[0153] (Operation) Next, the operation of the task recognition system 4 will be described with reference to the drawings. The operation of the task recognition device 42 included in the task recognition system 4 will be described below. FIG. 34 is a flowchart for explaining an example of the operation of the task recognition device 42. In describing the processing according to the flowchart of FIG. 34, the components of the task recognition device 42 will be described as the subject of operations. The subject of operations in the processing according to the flowchart of FIG. 34 may be the task recognition device 42.

[0154] 34 , first, the acquisition unit 421 acquires sensor data measured in accordance with the worker's movements and sound data resulting from the worker's voice (step S41). The acquisition unit 421 acquires the sensor data and sound data from the measurement device 40.

[0155] Next, the movement estimation unit 423 estimates the movement and voice of the worker using the acquired sensor data and sound data (step S42). For example, the movement estimation unit 423 inputs the sensor data and sound data to the movement estimation model 443. The movement estimation unit 423 estimates the movement and voice of the worker using the movement data and voice data output from the movement estimation model 443. The movement estimation unit 423 estimates a series of movements and voices included in the work performed by the worker.

[0156] Next, the task information generation unit 425 combines the estimated motions and voices to estimate the task content of the worker (step S43). For example, the task information generation unit 425 inputs the estimated motion data and voice data of the worker into the task recognition model 445. The task information generation unit 425 uses the task data output from the task recognition model 445 to generate task information including the task content of the worker.

[0157] Next, the work information generation unit 425 generates work information including the work content of the estimated worker (step S44). For example, the work information generation unit 425 generates work information including the name and identifier of the worker, the work time, the work location, the work target person, the work content, the utterance content, an identifier indicating the work status, etc.

[0158] Next, the output unit 427 outputs the generated work information (step S45). For example, the output unit 427 outputs the work information via the network NW to a management terminal used by a manager who manages the work of the workers. For example, the output unit 427 outputs the work information to a mobile terminal carried by the worker. For example, the output unit 427 outputs the work information to an external system that uses the work information.

[0159] As described above, the task recognition system of this embodiment includes a measuring device and an task recognition device. The measuring device is worn by a worker. The measuring device includes an acceleration sensor and an angular velocity sensor. The measuring device measures sensor data including acceleration and angular velocity in response to the worker's movements. The measuring device also includes a microphone. The measuring device measures sound data based on voice uttered by the worker. The measuring device outputs the measured sensor data and sound data to the task recognition device. The task recognition device includes an acquisition unit, a storage unit, a motion estimation unit, a task information generation unit, and an output unit. The acquisition unit acquires sensor data including acceleration and angular velocity measured in response to the worker's movements. The acquisition unit also acquires sound data based on the voice uttered by the worker. The storage unit stores a motion estimation model and a task recognition model. The motion estimation model and task recognition model are machine models trained using machine learning techniques. The motion estimation model outputs motion data indicating the worker's movements in response to input time-series data of sensor data. The motion estimation model also outputs sound data indicating the worker's voice in response to input sound data. The action estimation unit uses the action estimation model to estimate a series of actions performed by the worker and a series of sounds uttered by the worker. The task recognition model outputs task data indicating the content of the worker's task in response to input of a series of action data and sound data related to the task to be recognized. The task information generation unit uses the task recognition model to recognize the task performed by the worker. The task information generation unit generates task information indicating the content of the task performed by the worker based on the recognized task. The output unit outputs the generated task information.

[0160] The task recognition system of this embodiment estimates a series of actions performed by a worker using time-series data of sensor data measured in accordance with the worker's movements. The task recognition system of this embodiment also estimates a series of sounds uttered by the worker using sound data based on the sounds uttered by the worker. The task recognition system of this embodiment generates task information indicating the content of the task performed by the worker using the series of actions and sounds estimated for the worker. According to this embodiment, by referring to the worker's voice in addition to the worker's movements, the task performed by the worker can be recognized with greater accuracy.

[0161] Fifth Embodiment Next, an activity recognition device according to a fifth embodiment will be described with reference to the drawings. The activity recognition device of this embodiment has a simplified configuration of the activity recognition device included in the activity recognition systems according to the first to fourth embodiments.

[0162] (Configuration) FIG. 35 is a block diagram showing an example configuration of a task recognition device according to the present disclosure. The task recognition device 52 includes an acquisition unit 521, a movement estimation unit 523, a task information generation unit 525, and an output unit 527. The acquisition unit 521 acquires sensor data including acceleration and angular velocity measured in accordance with the worker's movements. The movement estimation unit 523 estimates a series of movements performed by the worker using time-series data of the sensor data. The task information generation unit 525 generates task information indicating the content of the task performed by the worker using the series of movements estimated for the worker. The output unit 527 outputs the generated task information.

[0163] (Operation) Figure 36 is a flowchart showing an example of the operation of the task recognition device according to the present disclosure. In describing the processing according to the flowchart of Figure 36, the components of the task recognition device 52 will be described as the subject of the operations. The subject of the operations according to the flowchart of Figure 36 may be the task recognition device 52.

[0164] In FIG. 36, first, the acquisition unit 521 acquires sensor data including acceleration and angular velocity measured in accordance with the movement of the worker (step S51).

[0165] Next, the movement estimation unit 523 estimates a series of movements made by the worker using the time-series data of the sensor data (step S52).

[0166] Next, the work information generating unit 525 generates work information indicating the work content performed by the worker, using the series of actions estimated for the worker (step S53).

[0167] Next, the output unit 527 outputs the generated work information (step S54).

[0168] The task recognition device of this embodiment estimates a series of actions performed by a worker using time-series data of sensor data measured in accordance with the worker's movements. The task recognition device of this embodiment generates task information indicating the content of the task performed by the worker using the series of actions estimated for the worker. Therefore, this embodiment can accurately recognize the tasks performed by the worker.

[0169] The present disclosure assumes an environment in which workers, such as nurses, perform healthcare-related tasks at a work site such as a hospital. For example, the method of the present disclosure can be applied to managing the work of workers in real time. For example, the method of the present disclosure can be applied to managing the progress of tasks assigned to workers. For example, the method of the present disclosure can be applied to accumulating the work history of tasks performed by workers. The method of the present disclosure is not limited to healthcare-related tasks at work sites such as hospitals, but can be applied to any task at any work site. For example, the method of the present disclosure can be applied to managing field workers at construction sites, manufacturing sites, etc. For example, the method of the present disclosure can be applied to managing teachers and students in an educational setting. For example, the method of the present disclosure can be applied to managing employees in a business environment.

[0170] (Hardware) Next, a hardware configuration for executing control and processing according to the present disclosure will be described with reference to the drawings. Here, an information processing device 90 (computer) shown in Fig. 37 is given as an example of such a hardware configuration. The information processing device 90 in Fig. 37 is an example configuration for executing control and processing according to the present disclosure and does not limit the scope of the present disclosure.

[0171] As shown in Fig. 37 , an information processing device 90 includes a processor 91, a memory 92, an auxiliary storage device 93, an input / output interface 95, and a communication interface 96. In Fig. 37 , interface is abbreviated as I / F (Interface). The processor 91, memory 92, auxiliary storage device 93, input / output interface 95, and communication interface 96 are connected to each other via a bus 98 so as to be able to communicate data with each other. The processor 91, memory 92, auxiliary storage device 93, and input / output interface 95 are also connected to a network such as the Internet or an intranet via the communication interface 96.

[0172] The processor 91 loads a program (instructions) stored in an auxiliary storage device 93 or the like into the memory 92. For example, the program is a software program for executing control and processing according to the present disclosure. The processor 91 executes the program loaded into the memory 92. The processor 91 executes the program to execute control and processing according to the present disclosure.

[0173] The memory 92 is a storage device having an area in which a program is loaded. The processor 91 loads a program stored in an auxiliary storage device 93 or the like into the memory 92. The memory 92 is realized by a volatile memory such as a dynamic random access memory (DRAM). Alternatively, a non-volatile memory such as a magnetoresistive random access memory (MRAM) may be used as the memory 92.

[0174] The auxiliary storage device 93 stores various data such as programs. For example, the auxiliary storage device 93 is realized by a local disk such as a hard disk or flash memory. Note that it is also possible to configure the system so that various data is stored in the memory 92, thereby omitting the auxiliary storage device 93.

[0175] The input / output interface 95 is an interface for connecting the information processing device 90 to peripheral devices based on standards and specifications. The communication interface 96 is an interface for connecting to external systems and devices via a network such as the Internet or an intranet based on standards and specifications. The input / output interface 95 and the communication interface 96 may be a common interface for connecting to external devices.

[0176] Input devices such as a keyboard, mouse, and touch panel may be connected to the information processing device 90 as needed. These input devices are used to input information and settings. When a touch panel is used as the input device, a screen having the function of the touch panel serves as the interface. The processor 91 and the input devices are connected via an input / output interface 95.

[0177] The information processing device 90 may be equipped with a display device for displaying information. When the display device is equipped, the information processing device 90 is equipped with a display control device (not shown) for controlling the display of the display device. The information processing device 90 and the display device are connected via an input / output interface 95.

[0178] The information processing device 90 may be equipped with a drive device. The drive device acts as an intermediary between the processor 91 and a recording medium (program recording medium) to read data and programs stored on the recording medium and to write processing results of the information processing device 90 to the recording medium. The information processing device 90 and the drive device are connected via an input / output interface 95.

[0179] The above is an example of a hardware configuration for enabling the control and processing according to the present disclosure. The hardware configuration of Figure 37 is an example of a hardware configuration for executing the control and processing according to the present disclosure, and does not limit the scope of the present disclosure. A program that causes a computer to execute the control and processing according to the present disclosure is also included in the scope of the present disclosure.

[0180] A program recording medium on which a program for executing the processing of this embodiment is recorded is also included within the scope of the present invention. For example, the program recording medium is a computer-readable, non-transitory recording medium. The recording medium can be, for example, an optical recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc). The recording medium may also be a semiconductor recording medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) card. The recording medium may also be a magnetic recording medium such as a flexible disk, or other recording medium.

[0181] The components according to the present disclosure may be combined in any manner. The components according to the present disclosure may be realized by software. The components according to the present disclosure may be realized by circuits.

[0182] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0183] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. The descriptions included in the supplementary notes below have significance as grounds for correction. (Supplementary Note 1) A task recognition device comprising: an acquisition unit that acquires sensor data including accelerations and angular velocities measured in accordance with the movements of a worker; a motion estimation unit that estimates a series of motions performed by the worker using time-series data of the sensor data; a task information generation unit that generates task information indicating the content of the task performed by the worker using the estimated series of motions of the worker; and an output unit that outputs the generated task information. (Supplementary Note 2) The task recognition device according to Supplementary Note 1, wherein the acquisition unit acquires the sensor data measured in accordance with movements of the plurality of workers, the movement estimation unit estimates a series of movements performed by the plurality of workers using time-series data of the sensor data, the task information generation unit generates the task information indicating a content of a collaborative task performed by the plurality of workers using the series of movements estimated for the plurality of workers, and the output unit outputs the generated task information indicating a content of the collaborative task performed by the plurality of workers. (Supplementary Note 3) The task recognition device according to Supplementary Note 2, further comprising a determination unit that determines the plurality of workers performing a collaborative task according to positions of the plurality of workers, wherein the acquisition unit acquires position information of the plurality of workers, the determination unit determines the plurality of workers performing a collaborative task using the position information of the plurality of workers, and the movement estimation unit estimates a series of movements performed by the plurality of workers using time-series data of the sensor data for the plurality of workers determined to be performing a collaborative task. (Supplementary Note 4) The task recognition device according to any one of Supplementary Notes 1 to 3, wherein the acquisition unit acquires sound data based on a voice uttered by the worker; the action estimation unit uses the sound data to estimate a series of voices uttered by the worker; and the task information generation unit uses the series of actions estimated for the worker and the series of voices estimated for the worker to generate the task information indicating the content of the task performed by the worker.(Supplementary Note 5) The task recognition device according to any one of Supplementary Notes 1 to 4, wherein the motion estimation unit estimates a plurality of motions performed by the worker using a motion estimation model that outputs motion data indicating the motions of the worker in response to input of time-series data of the sensor data, and the task information generation unit recognizes the task performed by the worker using a task recognition model that outputs task data indicating task details of the worker in response to input of a plurality of motion data related to a task to be recognized. (Supplementary Note 6) The task recognition device according to Supplementary Note 5, wherein the motion estimation model and the task recognition model are machine learning models trained using a machine learning technique. (Supplementary Note 7) The task recognition device according to Supplementary Note 5 or 6, wherein the motion estimation unit converts the sensor data measured by a measurement device worn at an arbitrary attachment position in accordance with skeletal data of the worker into normalized sensor data that would be measured if the measurement device were worn at a reference position, and estimates a plurality of motions performed by the worker using the motion data output from the motion estimation model in response to input of time-series data of the normalized sensor data, and the task information generation unit generates the task information indicating a task content performed by the worker using the task data output from the task recognition model in response to input of the motion data estimated using the time-series data of the normalized sensor data. (Supplementary Note 8) A task recognition system comprising: the task recognition device according to any one of Supplements 1 to 7; and a measurement device that is worn by the worker, measures the sensor data including acceleration and angular velocity, and outputs the measured sensor data to the task recognition device. (Supplementary Note 9) The task recognition system according to Supplementary Note 8, wherein the task recognition device displays the task information, including the task content recognized for the worker, on a screen of a management terminal used by a manager who manages the task of the worker in a display format that encourages decision-making by the manager.(Supplementary Note 10) A task recognition method in which a computer acquires sensor data including accelerations and angular velocities measured in accordance with the movements of a worker, estimates a series of actions performed by the worker using time-series data of the sensor data, generates task information indicating the content of the task performed by the worker using the series of actions estimated for the worker, and outputs the generated task information. (Supplementary Note 11) The task recognition method according to Supplementary Note 10, in which a computer acquires the sensor data measured in accordance with the movements of a plurality of the workers, estimates a series of actions performed by the plurality of the workers using the time-series data of the sensor data, generates the task information indicating the content of a collaborative task performed by the plurality of the workers using the series of actions estimated for the plurality of the workers, and outputs the generated task information indicating the content of the collaborative task performed by the plurality of the workers. (Supplementary Note 12) The task recognition method according to Supplementary Note 11, comprising: acquiring location information of a plurality of the workers; determining the plurality of the workers performing a collaborative task using the location information of the plurality of the workers; and estimating a series of actions performed by the plurality of the workers using time-series data of the sensor data related to the plurality of the workers determined to be performing a collaborative task. (Supplementary Note 13) The task recognition method according to any one of Supplementary Notes 10 to 12, comprising: acquiring sound data based on voices uttered by the workers; estimating a series of voices uttered by the workers using the sound data; and generating the task information indicating the content of the task performed by the workers using the series of actions estimated for the workers and the series of voices estimated for the workers. (Supplementary Note 14) A task recognition method according to any one of Supplementary Notes 10 to 13, which estimates a plurality of tasks performed by the worker using a task estimation model that outputs task data indicating the tasks of the worker in response to input of time-series data of the sensor data, and recognizes tasks performed by the worker using a task recognition model that outputs task data indicating the task content of the worker in response to input of a plurality of task data related to a task to be recognized.(Supplementary Note 15) The task recognition method according to Supplementary Note 14, wherein the sensor data measured by a measuring device attached at an arbitrary attachment position in accordance with skeletal data of the worker is converted into normalized sensor data that would be measured if the measuring device were attached at a reference position, the task recognition method comprising: estimating a plurality of tasks performed by the worker using the task data output from the task recognition model in response to input of the task data estimated using the time-series data of the normalized sensor data; and generating the task information indicating task details performed by the worker using the task data output from the task recognition model in response to input of the task data estimated using the time-series data of the normalized sensor data. (Supplementary Note 16) The task recognition method according to any one of Supplementary Notes 10 to 15, wherein the task information including the task details recognized for the worker is displayed on a screen of a management terminal used by a manager who manages the work of the worker in a display format that prompts the manager to make a decision. (Supplementary Note 17) A non-transitory recording medium having recorded thereon a program that causes a computer to execute the following processes: acquiring sensor data including accelerations and angular velocities measured in accordance with the movements of a worker, estimating a series of actions performed by the worker using time-series data of the sensor data, generating work information indicating work content performed by the worker using the series of actions estimated for the worker, and outputting the generated work information. (Supplementary Note 18) The non-transitory recording medium according to Supplementary Note 17, having recorded thereon a program that causes a computer to execute the following processes: acquiring the sensor data measured in accordance with the movements of a plurality of the workers, estimating a series of actions performed by the plurality of workers using time-series data of the sensor data, generating the work information indicating work content performed by the plurality of workers using the series of actions estimated for the plurality of workers, and outputting the generated work information indicating work content performed by the plurality of workers.(Supplementary Note 19) The non-transitory recording medium according to Supplementary Note 18, having recorded thereon a program that causes a computer to execute the following processes: acquiring position information of the plurality of workers, determining the plurality of workers performing a collaborative task using the position information of the plurality of workers, and estimating a series of actions performed by the plurality of workers using time-series data of the sensor data related to the plurality of workers determined to be performing a collaborative task. (Supplementary Note 20) The non-transitory recording medium according to any one of Supplementary Notes 17 to 19, having recorded thereon a program that causes a computer to execute the following processes: acquiring sound data based on voices uttered by the workers, estimating a series of voices uttered by the workers using the sound data, and generating the task information indicating the content of the task performed by the workers using the series of actions estimated for the workers and the series of voices estimated for the workers. (Supplementary Note 21) A non-transitory recording medium according to any one of Supplementary Notes 17 to 20, having recorded thereon a program that causes a computer to execute the following steps: a process of estimating a plurality of actions performed by the worker using an action estimation model that outputs action data indicating the actions of the worker in response to input of time-series data of the sensor data; and a process of recognizing the work performed by the worker using an action recognition model that outputs action data indicating the work content of the worker in response to input of a plurality of pieces of action data related to the work to be recognized. (Supplementary Note 22) A non-transitory recording medium according to Supplementary Note 21, having recorded thereon a program that causes a computer to execute the following processes: a process of converting the sensor data measured by a measuring device attached at an arbitrary attachment position in accordance with skeletal data of the worker into standardized sensor data that would be measured if the measuring device were attached at a reference position; a process of estimating multiple actions performed by the worker using the action data output from the action estimation model in response to input of time-series data of the standardized sensor data; and a process of generating the work information indicating the content of the work performed by the worker using the work data output from the work recognition model in response to input of the action data estimated using the time-series data of the standardized sensor data.(Appendix 23) A non-transitory recording medium described in any one of Appendices 17 to 22, having recorded thereon a program that causes a computer to execute a process of displaying the work information, including the work content recognized for the worker, on the screen of a management terminal used by an administrator managing the worker's work in a display format that encourages decision-making by the administrator.

[0184] 1, 2, 3, 4 Task recognition system 10, 20, 30, 40 Measurement device 12, 22, 32, 42, 52 Task recognition device 27 Short-range communication device 110, 410 Sensor 111, 411 Acceleration sensor 112, 412 Angular velocity sensor 115, 415 Control unit 117, 417 Communication unit 121, 221, 321, 421, 521 Acquisition unit 123, 223, 323, 423, 523 Motion estimation unit 124, 224, 324, 424 Storage unit 125, 225, 325, 425, 525 Task information generation unit 127, 227, 327, 427, 527 Output unit 143, 243, 343, 443 Motion estimation model 145, 245, 345, 445 Task recognition model 160, 260, 360 Mobile terminal 170, 270, 370 Server 180, 280, 380 Management terminal 190, 290, 390 Wireless relay device 322 Determination unit 414 Microphone

Claims

1. An acquisition unit that acquires sensor data including acceleration and angular velocity measured in accordance with the worker's movements, An action estimation unit that estimates a series of actions performed by the worker using the time-series data of the sensor data, A work information generation unit generates work information indicating the content of work performed by the worker using a series of actions estimated with respect to the worker, A work recognition device comprising: an output unit that outputs the generated work information.

2. The acquisition unit is, The sensor data measured according to the movements of multiple workers is acquired, The aforementioned motion estimation unit, Using the time-series data of the aforementioned sensor data, a series of actions performed by multiple operators is estimated. The aforementioned work information generation unit, Using a series of actions estimated for multiple workers, work information is generated that shows the content of collaborative work performed by the multiple workers. The output unit is, The work recognition device according to claim 1, which outputs work information indicating the content of collaborative work performed by a plurality of the aforementioned workers.

3. The system further includes a determination unit that determines which of the aforementioned workers are performing collaborative work, based on the positions of the aforementioned workers. The acquisition unit is, The location information of multiple workers is acquired, The determination unit, Using the location information of multiple workers, the system identifies the multiple workers who are working together. The aforementioned motion estimation unit, The work recognition device according to claim 2, which estimates a series of actions performed by multiple workers using time-series data of sensor data relating to multiple workers who are determined to be performing collaborative work.

4. The acquisition unit is, Acquire sound data based on the voice emitted by the aforementioned worker, The aforementioned motion estimation unit, Using the aforementioned sound data, a series of sounds emitted by the worker are estimated. The aforementioned work information generation unit, The work recognition device according to claim 1, which generates work information indicating the content of work performed by the worker using a series of actions estimated with respect to the worker and a series of sounds estimated with respect to the worker.

5. The aforementioned motion estimation unit, Using a motion estimation model that outputs motion data indicating the worker's actions in response to the input of time-series data of the sensor data, multiple actions performed by the worker are estimated. The aforementioned work information generation unit, The work recognition device according to claim 1, which recognizes work performed by an operator using a work recognition model that outputs work data indicating the operator's work content in response to the input of a plurality of operation data related to the work to be recognized.

6. The work recognition device according to claim 5, wherein the motion estimation model and the work recognition model are machine learning models learned using machine learning techniques.

7. The aforementioned motion estimation unit, The sensor data measured by the measuring device attached to an arbitrary mounting position, in accordance with the worker's skeletal data, is converted into normalized sensor data that would be measured if the measuring device were attached to a reference position. Using the motion data output from the motion estimation model in response to the input of time-series data of the standardized sensor data, multiple actions performed by the worker are estimated. The aforementioned work information generation unit, The work recognition device according to claim 5, which generates work information indicating the content of work performed by the worker using the work data output from the work recognition model in response to the input of the operation data estimated using the time-series data of the standardized sensor data.

8. A work recognition device according to any one of claims 1 to 7, A work recognition system comprising: a measuring device attached to the worker, which measures the sensor data including acceleration and angular velocity, and outputs the measured sensor data to the work recognition device.

9. Computers Sensor data, including acceleration and angular velocity measured in accordance with the worker's movements, is acquired. Using the time-series data of the aforementioned sensor data, a series of actions performed by the worker is estimated. Using the estimated sequence of actions performed by the worker, work information indicating the content of the work performed by the worker is generated. A work recognition method that outputs the generated work information.

10. A process to acquire sensor data including acceleration and angular velocity measured in accordance with the worker's movements, A process to estimate a series of actions performed by the worker using the time-series data of the aforementioned sensor data, A process that generates work information indicating the content of work performed by the worker using a series of actions estimated with respect to the worker, A program that causes a computer to execute a process that outputs the generated work information.