Collaborative work recognition system and collaborative work recognition method

Abstract

To identify whether the work is a cooperative work of multiple workers, in consideration of property of the cooperative work.SOLUTION: A cooperative work recognition system causes a computer having a processor and a memory to execute processing. The processor reads out, for each of a plurality of workers, a work log formed by accumulating work types and work periods from a work start time to a work end time, the work type being identified by learning feature quantities of the work included in data received from a sensor in a predetermined method, reads out a cooperative work definition database storing a type representing property of cooperative work performed by the plurality of workers in association with work of the workers constituting the cooperative work of the type, and uses the read-out work log and the read-out cooperative work definition database to determine whether work of a worker is cooperative work performed by the plurality of workers.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a collaborative activity recognition system and a collaborative activity recognition method. [Background technology]

[0002] Generally, when there are multiple people, in order to identify which group a certain person belongs to, the position information and behavior of each person are recognized using a sensor such as a camera, and the criteria for judgment are the positional relationship of the target person to others and whether there is a relationship between the behavior of the target person and others. For example, there is a technology described in Patent Document 1 that makes such judgments. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2018 / 138803 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, groups are divided into groups based on the proximity between people and the similarity of their movement directions in situations where multiple people are present. However, collaborative work performed on-site cannot necessarily be divided into groups based on the similarity of the members' locations and their individual tasks. Just because workers are close to each other does not necessarily mean they are engaged in the same collaborative work. Furthermore, collaborative work may also be comprised of workers engaged in different individual tasks. For example, "surveying" is a collaborative task in which workers operate equipment at a distance from each other. Here, the work involved in surveying consists of the paired tasks of one worker peering into the measuring equipment and another worker fixing the measurement target and piercing it into the ground. Thus, whether an individual's work is a solo or collaborative task must be defined and identified based on the nature of the collaborative task.

[0005] An object of the present invention is to provide a technique that can identify whether a task is a collaborative task performed by multiple workers, taking into account the nature of the collaborative task. [Means for solving the problem]

[0006] The collaborative activity recognition system of the present invention is a collaborative activity recognition system in which processing is performed by a computer having a processor and a memory, wherein the processor reads a work log for each of a plurality of workers, which log accumulates the type of work identified by learning, using a predetermined method, task features contained in data received from a sensor, and the work period from the start time to the end time of the work; reads a collaborative activity definition database which stores a correspondence between types indicating the nature of collaborative activities by a plurality of workers and the tasks of the workers that make up that type of collaborative activity; and uses the read work log and the collaborative activity definition database to determine whether an activity by a worker is a collaborative activity by a plurality of workers. [Effects of the Invention]

[0007] According to the present invention, it is possible to identify whether a task is a collaborative task performed by a plurality of workers, taking into consideration the nature of the collaborative task. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram illustrating an example of a system configuration of a collaborative activity recognition system according to an embodiment of the present invention. [Figure 2] FIG. 10 is a schematic diagram showing an example in which a worker wears clothing to which a sensor unit is attached. [Figure 3] 10 shows a flowchart for outputting a personal work log from data acquired from the sensor unit. [Figure 4] This is an example of visualizing a worker's personal work log by time. [Figure 5] This is an example of a database for collaborative work defined for a certain site. [Figure 6]FIG. 8 is a diagram showing the scene corresponding to FIG. 7. [Figure 7] This is an example of visualizing the work logs of multiple workers at a certain site. [Figure 8] Figure 8 is a flowchart for identifying collaborative work from individual work logs. [Figure 9] 9 is a diagram showing the intermediate process of converting FIG. 6 according to the flowchart of FIG. 8. FIG. [Figure 10] This is a diagram showing the collaborative work identification results of a certain site visualized as a log. [Figure 11] FIG. 10 is a diagram illustrating an example showing overlapping work areas. [Figure 12] FIG. 1 is a diagram illustrating an example of a schematic configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0009] Examples of the present disclosure will be described below with reference to the drawings. However, the following examples are merely illustrative of the present disclosure and are not intended to limit the scope of the present disclosure to these examples. Those skilled in the art can implement the present disclosure in various other forms without departing from the scope of the present disclosure. Furthermore, in the configurations of the invention described below, the same reference numerals may be used in different drawings for identical parts or parts having similar functions, and redundant description may be omitted. Furthermore, when there are multiple elements having the same or similar functions, the same reference numerals may be used with different subscripts. However, when there is no need to distinguish between multiple elements, the subscripts may be omitted. Furthermore, the position, size, shape, and scope of each component shown in the drawings may not represent the actual position, size, shape, and scope in order to facilitate understanding of the present disclosure. Therefore, the present disclosure is not limited to the position, size, shape, and scope disclosed in the drawings. Furthermore, in this specification, elements referred to in the singular may be plural unless otherwise clearly indicated in the context.

[0010] In the following explanation, various types of information may be described using expressions such as "database," "table," and "list," but the various types of information may also be expressed in data structures other than these. To indicate that the information is not dependent on the data structure, "XX table," "XX list," etc. may be referred to as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, and these are interchangeable.

[0011] When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. However, when there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0012] Furthermore, in the following description, processing performed by executing a program may be described, but the program is executed by a processor (e.g., a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit)) to perform the specified processing while appropriately using storage resources (e.g., memory) and / or interface devices (e.g., communication ports), and therefore the processor may be the subject of the processing. Similarly, the subject of the processing performed by executing a program may be a controller, device, system, computer, or node having a processor. The subject of the processing performed by executing a program may be any computing unit, and may include a dedicated circuit (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)) that performs specific processing.

[0013] A program may be installed on a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. If the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Also, in the following description, two or more programs may be realized as one program, and one program may be realized as two or more programs.

[0014] As will be described in detail later, the information processing device used in this system can be realized, for example, by a general computer 1600, as shown in Figure 12 (schematic computer diagram), which includes a CPU 1601, memory 1602, an external storage device 1603 such as an HDD (Hard Disk Drive), a reading / writing device 1607 for reading and writing information from a portable storage medium 1608 such as a CD (Compact Disk) or USB memory, an input device 1606 such as a keyboard or mouse, an output device 1605 such as a display, a communication device 1604 such as a NIC (Network Interface Card) for connecting to a communication network, and an internal communication line (referred to as a system bus) 1609 such as a system bus that connects these.

[0015] Furthermore, various data stored in the information processing device or used for processing can be realized by the CPU 1601 reading and using the data from the memory 1602 or the external storage device 1603. Furthermore, each functional unit of the information processing device (for example, the posture estimation unit 122, the feature extraction unit 123, the personal task identification unit 124, the collaborative task determination unit 127, and the inference result display unit 128) can be realized by the CPU 1601 loading a predetermined program stored in the external storage device 1603 into the memory 1602 and executing it.

[0016] The above-mentioned predetermined program may be stored (downloaded) into the external storage device 1603 from the storage medium 1608 via the reading / writing device 1607 or from a network via the communication device 1604, and then loaded onto the memory 1602 and executed by the CPU 1601. Alternatively, the program may be directly loaded onto the memory 1602 from the storage medium 1608 via the reading / writing device 1607 or from a network via the communication device 1604, and then executed by the CPU 1601.

[0017] In the following, we will explain an example in which this system is configured using a single computer, but all or part of these functions may be distributed across one or more computers, such as a cloud, and similar functions may be realized by communicating with each other via a network.

[0018] <1. Overall system configuration> FIG. 1 is a block diagram illustrating a collaborative activity recognition system according to a first embodiment of the present disclosure. The collaborative activity recognition system 1 illustrated in FIG. 1 is a system for estimating the posture of a user 210, and includes a sensor unit 110, an information processing device 120, and a sensor hub 220. The sensor unit 110 and the information processing device 120 are connected to each other so that they can communicate with each other via a wired or wireless connection. The sensor unit 110 and the information processing device 120 may also be connected to each other via a communication network with the sensor hub 220. While the following illustrates a configuration in which the sensor hub 220 is included in the sensor unit 110, the sensor hub 220 may be configured independently of the sensor unit 110, and the two may also be connected to each other via a communication network.

[0019] The sensor unit 110 is a wearable unit that is attached to clothing 111 of the user 210. The sensor unit 110 may be attached to the clothing 111 using, for example, adhesive tape or may be embedded in the clothing 111 in advance. The clothing 111 is, for example, ordinary clothing that does not fit tightly against the body of the user 210, such as work clothes. The sensor unit 110 has a posture sensor 112, a sensor hub 220, and a communication unit 113. The posture sensor 112 is a movement measurement sensor that measures the movement of the user 210. The movement is a general movement that includes movements associated with work in industry and agriculture, as well as movements for a specific purpose such as dance, gymnastics, and playing a musical instrument.

[0020] 2 is a schematic diagram showing an example in which a user 210 wears clothing 111 to which a sensor unit 110 is attached. In FIG. 2, the sensor unit 110 includes a plurality of posture sensors 112 and a sensor hub 220 that aggregates motion data 140 (see FIG. 1), which is sensor data measured by the plurality of posture sensors 112. Each posture sensor 112 is connected to the sensor hub 220 directly or via another posture sensor 112, and transmits the motion data 140 to the sensor hub 220. Note that the posture sensors 112 and the sensor hub 220 may be connected by wire or wirelessly.

[0021] In the example of FIG. 2 , the posture sensors 112 are attached to the clothing 111 so as to be positioned on various parts of the user 210's entire body. However, the placement of the posture sensors 112 is not limited to this example. For example, the posture sensors 112 may be attached to the clothing 111 so as to be positioned on only a part of the user's body, such as the upper body or the lower body, depending on the posture to be estimated. Furthermore, the type of posture sensor 112 may be one or more types, and can be appropriately selected depending on the posture to be estimated. For example, when directly estimating the posture of the user 210, a sensor capable of measuring the position or movement of each part of the user's body, such as an acceleration sensor and a position sensor, is desirable. The type of posture sensor 112 is not limited to this example, and may be, for example, a gyro sensor, a geomagnetic sensor, an electromyographic sensor, an acceleration sensor, a video sensor, or an audio sensor. The sensor hub 220 transmits the movement data 140 received from each posture sensor 112 to the information processing device 120 via the communication unit 113. 2, the communication unit 113 is, for example, a communication device 1604 built into the sensor hub 220. The attitude sensor 112 and the sensor hub 220 may be capable of receiving power from a power supply device (not shown) such as a battery.

[0022] Returning to the explanation of Figure 1, the information processing device 120 has a first function (posture estimation unit 121) for estimating the posture of the user 210 based on the action data 140 from the sensor unit 110, and a function (individual action identification unit 124) for determining the action identification and each action time of the user 210 from the action data 140 or the estimated posture, and creating an action log. Secondly, it has a processing function (joint action determination unit 127) for performing the same processing as for the user 210 on multiple other users equipped with the sensor unit 110, and determining collaborative work from the work logs of all users.

[0023] As described above, the information processing device 120 can be configured, for example, by a general server, and the neural network described below may be implemented by an FPGA or the like.

[0024] The information processing device 120 includes, as functional components, a communication unit 121, a posture estimation unit 122, a feature extraction unit 123, a personal task identification unit 124, a log storage unit 125, a collaborative task definition DB 126, a collaborative task determination unit 127, and an inference result display unit 128.

[0025] The communication unit 121 communicates with external devices such as the sensor unit 110. For example, the communication unit 121 receives motion data 140 from the communication unit 113 of the sensor unit 110. The posture estimation unit 122 estimates the posture of the user 210 based on the motion data 140 received by the communication unit 121 and calculates posture data indicating the estimated posture. The function of the posture estimation unit 122 may be implemented in the sensor unit 110. In this case, the posture data is included in the motion data 140 and transmitted to the communication unit 121. The posture data is data indicating the positions of each joint of a human being and visually represents the posture of the user 210 wearing the sensor unit 110. The posture data may have a general configuration and therefore will not be illustrated. For example, the posture data may include momentary values ​​detected by posture sensors 112 provided at each part of the human body, such as the elbows and wrists. The posture data may also be displayed on a screen by the inference result display unit 128.

[0026] The feature extraction unit 123 periodically extracts task features 151 from the motion data 140 or posture data to identify the individual task in which the user 210 is engaged. The individual task identification unit 124 infers the type of individual task the user 210 has engaged in and the duration of that task from the feature 151, and stores this as an activity log 152 in the log storage unit 125, which is a memory unit. The collaborative task definition DB 126 is a database that stores definitions of collaborative tasks in which users engage as prior knowledge; a specific configuration will be described later. The collaborative task determination unit 127 determines, from the activity logs 152 of each user 220 stored in the log storage unit 125, whether each user's activity was a solo activity or a collaborative activity with another user, in accordance with the collaborative task definition DB. The inference result display unit 128 has a function of visualizing and displaying, on a screen such as a monitor, the posture data obtained by the posture estimation unit 122 and the work logs of multiple users obtained by the collaborative work determination unit 127.

[0027] <2. Example of creating a personal work log> Here, we will describe an example of a specific procedure for creating an individual worker's work log 152 from the action data 140 acquired from the user 210. Fig. 3 is a flowchart showing this example, and this process may be executed by the information processing device 120.

[0028] First, in S301, the communication unit 113 receives the motion data 140 from the attitude sensor 112, which includes, for example, an acceleration sensor, and is worn by the user 210, and stores the motion data 140 in, for example, a RAM or the like provided in the information processing device 120.

[0029] In S302, the feature extraction unit 123 extracts the feature 151 from the motion data 140 using, for example, a trained long short-term memory (LSTM) or a convolutional neural network (CNN). The LSTM or CNN may have parameters obtained in the process of learning the correspondence between the motion data 140 over a certain period of time and the individual work, position, or posture of the corresponding user 210, and the feature 151 is an output when the motion data 140 is input to the LSTM or CNN, and may be, for example, a vector structure. The feature 151 is information representing the characteristics of the motion of a part of the worker detected by the posture sensor 112 including, for example, an acceleration sensor.

[0030] In S303, the personal task identification unit 124 identifies the personal task of the user 210, for example, at regular intervals, from the feature 151 obtained in S302. For example, a linear layer used in a deep neural network (DNN) may be used for the identification, and the parameters of the linear layer may be learned correspondences between the feature 151 and the personal task. The personal task identification unit 124 performs the learning process by inputting, for example, a value a of the feature 151 obtained from a sensor A worn on the wrist at time t1, a value b of the feature 151 obtained from the sensor A at the next time t2, and a value c of the feature 151 obtained from the sensor A at the next time t3. The personal task identification unit 124 identifies that the feature 151 values ​​a and b at times t1 and t2 are feature values ​​when the worker performs the task type "carrying," and that the feature 151 value c at time t3 is a feature value when the worker performs the task type "screwdriving." The individual task identification unit 124 performs such identification for all sensors worn by the worker.

[0031] In S304, the personal task identification unit 124 may store the task identification results obtained in S303 for each fixed period in chronological order and determine the start and end times of each personal task performed by the user 210 from the stored results. For example, if the accuracy of the identification results obtained in S303 is uncertain, a smoothing process may be performed on a fixed number of temporally adjacent identification results, and the start and end times may be determined based on the smoothing process. For example, if the value c of the feature 151 at time t3 is obtained multiple times in succession, the personal task identification unit 124 may determine that the "carrying" action at times t1 and t2 has changed to the "screwdriving" action, and may determine the time up to time t2 as the end time of the "carrying" action and determine the time from time t3 onward as the start time of the "screwdriving" action. The personal task identification unit 124 makes the above determination using data (e.g., feature 151) received in real time from the posture sensor 112. This makes it possible to determine in real time what type of work is currently being performed.

[0032] The individual task identification unit 124 may also simultaneously infer the posture and position information of the user 210. Inferring the posture and position may involve introducing a linear layer that directly learns the correspondence between the feature 151 and the posture and position, and using the output of that layer. Alternatively, the position information of each sensor may be the temporal accumulation of acceleration measurements obtained from the posture sensor 112, which includes an acceleration sensor. The position and posture may be inferred by regarding each sensor position as the position of a human joint. For example, the individual task identification unit 124 may input the feature value d obtained from the posture sensor 112 attached to the worker's fingertips, the feature value e obtained from the posture sensor 112 attached to the worker's wrist, the feature value f obtained from the posture sensor 112 attached to the worker's elbow, and the feature value g obtained from the posture sensor 112 attached to the worker's knee, and perform the learning process. If only the feature value d changes, the unit infers that the worker is performing the action of "driving a screw" while standing still at a certain position. The posture and position may be inferred by the posture estimation unit 122.

[0033] In S305, the personal task identification unit 124 may store in the log storage unit 125 the start time, end time, inferred position and posture information of the user 210 for each personal task in which the user 210 was engaged, obtained in S304.

[0034] FIG. 4 illustrates an example of the work log 152 for user 210 obtained by creating the individual work log. Table 401 shows an example of the work log 152 for worker A, corresponding to user 210, at each time. In the example of table 401, worker A is shown as carrying luggage and performing installation work using tools. However, this is merely an example of an implementation example, and the target of implementation is not limited thereto. Table 401 also lists the type of work performed at each time, and the position and posture information of worker A at that time may also be included in the work log. For example, FIG. 4 shows that a worker identified by worker ID "A" performed "carrying" work at times "1" and "2" and "hammer" work at times "3" and "4." Similarly, for subsequent times, data corresponding to the individual worker's work identified by learning and associated with each time is accumulated as work log 152.

[0035] <3. Example of collaborative work definition DB configuration> 5 is a diagram showing an example of the joint work definition DB 126 as a table 501. As will be explained below, in the joint work definition DB 126, joint works are defined based on prior knowledge based on the work content, scope, and period of each worker, so that workers who are close to each other or who perform similar work can be recognized as separate groups.

[0036] The collaborative work DB 126 may define in advance collaborative works that can be expected for each business operator, and in this embodiment, table 501 defines transportation and installation as examples of types of collaborative work, and shows an example of the collaborative work DB 126 defined for businesses involved in these collaborative works. The collaborative work definition DB 126 may be definable for any business operator, and its configuration may be learned by a DNN, or may be described by rules such as those in table 501.

[0037] In table 501, for example, "joint installation" work and "joint transportation" work are defined as collaborative work, but other definable collaborative work may be defined depending on the usage form. Table 501 may describe, for each collaborative work, at least a set 502 of individual work that may constitute the collaborative work, a rule 503 regarding temporal overlap with the work period of another worker (hereinafter referred to as time overlap rule 503) for determining whether the individual work of a certain worker constitutes a collaborative work, a rule 504 regarding spatial overlap with the work range of another worker (hereinafter referred to as spatial overlap rule 504), and a paired work 505.

[0038] For example, if a "joint transportation" task is defined as multiple workers transporting a heavy load, the set of individual tasks 502 performed by each worker constituting the joint transportation task may be, for example, individual transport actions and transport guide actions. Furthermore, when defining the rule 503 regarding temporal overlap, for example, due to the nature of the joint task of multiple workers transporting a heavy load while maintaining balance, the start and end times of the transport actions of all workers involved in the transport, including the guide, may be defined to be exactly the same. Furthermore, to more strictly limit the workers engaged in the same joint transportation task, the rule 504 regarding spatial overlap may be set. For example, if the size of the object to be transported is known, a work range may be defined based on that size. Furthermore, only workers who participated in the transport within that work range at each time may be considered to have engaged in the joint transportation task. The compliant work range may be, for example, an area within a predetermined width of the object to be transported. Furthermore, the compliant work range may be determined based on the number of workers working on the object. For example, in order to ensure sufficient space per worker for work, the work area may be set wider as the number of workers working on an object increases. Furthermore, the work area may be set based on the worker's posture and joint positions estimated from data obtained from the sensor units 112 worn by the workers. For example, when a worker is working with a hammer in a seated position, this is different from work that involves movement such as carrying something, and therefore the work area per worker may be set based on data obtained from the sensor units 112 worn on both elbows.

[0039] Next, for example, if a "collaborative installation" task is defined as the task of assembling and installing an object by multiple people, the set of individual tasks 502 constituting this collaborative task may be, for example, hammering, screwdriving, and holding down. Furthermore, as a rule 503 regarding the temporal overlap of each individual task constituting the collaborative installation task, for example, a strict coincidence of the task periods may not be required, and a temporal offset may be allowed. Furthermore, as a rule 504 regarding the spatial overlap of individual tasks, for example, the size of the object to be installed may be known in advance, and an area conforming to the object size may be defined as the work range. For example, workers who participated in installing the object within this area may be considered to be engaged in collaborative work. Furthermore, as for paired tasks 505, for example, in a collaborative installation task, if the task of hammering an object requires both a worker who hammers and a worker who holds down the object at the same time, it may be described that screwing in a screw requires simultaneously tightening the screw and holding down the object.

[0040] <4. Specific steps for identifying collaborative work> Here, as an example, a specific procedure for identifying collaborative work will be described using an example of an object transport and installation work site. Table 701 in FIG. 7 is an example of the work log 152 stored in the log storage unit 125, and records the individual work performed by each worker for each work time from work time 1 to work time 10 in a work in which, for example, five workers A, B, C, D, and E transport and install multiple objects. The contents of Table 701 are similar to those of Table 501, and therefore a description thereof will be omitted here. For the purpose of explanation, this embodiment assumes a situation in which there are five workers and three objects, but the number of workers and the number of objects are not limited to this.

[0041] FIG. 6 visually illustrates the work site of Table 701, depicting the work of workers A, B, C, D, and E, for example, carrying and installing three objects X, Y, and Z, over time. For example, Situation 601 shows each worker carrying an object from task time 1 to task time 2. Situations 602, 603, and 604 show each worker installing an object at task times 5-6, 7-8, and 9-10, respectively. Estimating each user's location may be achieved by recognizing the user's location information from a sensor unit 112 attached to clothing 111 worn by user 210. If the sensor unit 112 worn by user 210 includes, for example, a camera, it may be possible to recognize objects X, Y, and Z and their sizes. A work area for the spatial overlap rule 504 may be defined for each object. This site is an example of collaborative work by multiple workers, but the present system can be applied to various other sites.

[0042] FIG. 8 is an example of a specific flowchart for the joint work determination unit 127 to identify a joint work from the personal work log 152 stored in the log storage unit 125 and the joint work definition DB 126.

[0043] In S801, in preparation for identifying a collaborative task, the collaborative task determination unit 127 reads the task logs 152 of all workers stored in the task log storage unit 125 into a memory such as a RAM of the information processing device 120. Here, the task logs 152 of all workers are records of individual tasks at each time for each individual, as represented in table 701, for example.

[0044] In S802, the collaborative work determination unit 127 replaces each individual work with a related collaborative work based on the individual worker's work log 152 in table 701 and the collaborative work definition DB 126. In the example of table 701, "transport" and "guiding action" are replaced with the related "collaborative transport work" from column 502 of table 501, and "hammer," "screwdriver," and "hold" are replaced with the related "collaborative installation" work from column 503 of table 501. For example, the collaborative work determination unit 127 leaves undefined tasks such as "moving" and "resting" as individual tasks without replacing them because they do not constitute collaborative work. As a result of the processing of S802, table 701 is converted, for example, to table 901 shown in FIG. 9. In the subsequent steps (S803 to S807), the process determines whether the individual work of each worker listed in Table 901 can be linked to any work by other workers, and determines which person is actually performing the collaborative work.

[0045] In S803, the joint work determination unit 127 sets the joint works that exist in table 901 and are defined in the joint work definition DB 126 as a set V={joint transport, joint installation}. As explained in S802, set V does not include works that are not defined in table 501 of Fig. 5, such as "movement" and "resting," and therefore these works are not subject to processing in the subsequent steps (S803 to S807).

[0046] The processing from S804 onwards is processing aimed at identifying a joint work for each work included in the set V. In S804, the joint work determination unit 127 selects, for example, an unprocessed "joint installation" as the joint work P.

[0047] In S805, the collaborative work determination unit 127 performs processing for collaborative work identification with respect to the collaborative installation selected in S804, for example, based on each piece of position information estimated from the accumulation of acceleration information obtained from the sensor units 110 attached to the clothing 111 worn by each of workers A, B, C, D, and E, and, for example, the work logs 152 of the individual workers in table 901. Here, the set of individual works related to the collaborative installation work in table 901 is set U={A2, A3, B2, B3, C2, C3, D2, E2}.

[0048] In S806, the collaborative work determination unit 127 may determine the tasks that constitute the same collaborative work based on the rules 503 regarding temporal overlap and the rules 504 regarding spatial overlap defined in the collaborative work definition DB 126, such as Table 501. According to Table 501, for the "collaborative installation" task, the start and end times may be offset. Therefore, the collaborative work determination unit 127 refers to the rules 503 regarding temporal overlap. For example, since the set W1 = {A2, B2, C2, D2, E2} overlaps in time from time 5 to time 6, the collaborative work determination unit 127 determines that this collaborative installation task is a collaborative work by five workers, A to E. Furthermore, since the set W2 = {A3, B3, C3, D2} overlaps in time from time 10, the collaborative work determination unit 127 determines that this collaborative installation task is a collaborative work by three workers, A to C. In response to this, the joint activity determination section 127 may additionally use, for example, the rule 504 regarding spatial overlap to determine that activities that overlap spatially and temporally are the same activity.

[0049] FIG. 11 is a diagram illustrating an example of determining the spatial overlap rule 504. FIG. 11 illustrates position information 1103 acquired from the sensor unit 110 attached to the clothing 111 worn by each worker in the same situation as the situation 602 illustrated in FIG. 6 . For example, if the size of the object X to be delivered is known in advance, the installation work of object X may be defined as work within an area having a radius corresponding to the size of the object to be installed. For example, if the size (e.g., the length of one side) of object X is XA, the work range for installing object X may be defined as a circular area with a radius Rx (= XA + a predetermined buffer) described in the circular area 1101. In this case, for example, if the same work range is drawn by workers A, B, and C in the worker position information 1103, there is an overlapping range (in this example, A∩B∩C) between the work ranges of the three workers. Therefore, it can be determined that the three workers worked within the radius Rx, and may be considered to have worked collaboratively on the same object X. Similarly, a radius Ry may be introduced for the object Y to define a similar rule 504 regarding spatial overlap. The same can be considered when an object Z is also present.

[0050] In this way, by using both the rule 503 regarding temporal overlap and the rule 504 regarding spatial overlap, the object on which each worker worked at each time point may be determined based on the positional relationship between workers A, B, C, D, and E and the relationship between the predefined work ranges for installing object XYZ. For example, for the set W1, in the situation 602 corresponding to the temporal overlap of each task constituting the set W1 and the corresponding worker position information 1103, workers A, B, and C are present within a work range with a radius Rx, and according to table 901, they are engaged in a joint installation task at the same time. Therefore, the joint task determination unit 127 may determine that workers A, B, and C were engaged in the installation of object X. Similarly, since workers D and E both worked within a work range radius Ry, they may be determined to have been engaged in the installation of object Y. As a result, workers A2, B2, and C2 of the set W1 may be classified as the same joint task, and workers D2 and E2 may be classified as different joint tasks. Regarding the set W2, for example, in situation 604, since worker CD is engaged in installation work on object Y within an area of ​​radius Ry, C3 and D3 may be considered to have been engaged in the same work, and since worker AB is engaged in installation work in an area of ​​radius Rz, A3 and B3 may be considered to have been engaged in the same work.

[0051] In S806, it is also possible to determine that a joint work does not constitute a joint work even if the rule 504 regarding spatial overlap of the work ranges is satisfied. At times 7 and 8 in table 901, which corresponds to situation 603, workers A and B finish work on object X and start moving. At the time of this movement, although workers A and B are in the same work range 1101 as worker C, they have finished the joint installation work, and therefore rule 503 regarding temporal overlap of the periods is not satisfied. Therefore, the joint work determination unit 127 may determine that workers A and B are not engaged in a joint work even if they are located close to a certain extent.

[0052] In S807, the joint work determination unit 127 checks whether or not all elements of the joint work candidate set V have been processed in the operations up to S806. For example, if the "joint installation" work has been processed but the "joint transportation" work has not been processed, the process returns to S804 and processes the unprocessed joint work again.

[0053] In S804, for example, from the set V, the joint work determination unit 127 extracts "joint transportation" as the joint work P.

[0054] In S805, the joint work determination unit 127 stores a set of individual work related to joint transportation as a set U={A1, B1, C1, D1, E1}. The joint work determination unit 127 extracts, for example, joint transportation A1 from the set U of individual work.

[0055] In S806, the collaborative work determination unit 127 selects a task from the collaborative work candidate set V that is the same collaborative work as A1, based on the rule 503 regarding temporal overlap and the rule 504 regarding spatial overlap defined in the collaborative work definition DB 126. According to Table 501, the rule 503 regarding temporal overlap of the "collaborative transportation" task requires that at least the start and end times of the individual tasks closely match. Therefore, the collaborative work determination unit 127 determines B1 and C1 as candidates for the same collaborative work as A1. On the other hand, for B1 and C1 whose start and end times match, if the size of the transported object is known in advance, for example, the rule 504 regarding spatial overlap of the "collaborative transportation" task may be set to object transportation work within an area with a radius corresponding to the size of the object, as described with reference to FIG. 11. Here, based on the situation 601 in FIG. 6, since workers A, B, and C were engaged in collaborative transportation work in areas close to each other, A1, B1, and C1 may be determined to be the same collaborative transportation work. Regarding D1 and E1, according to the rule 503 regarding time overlap, since the start time and end time do not exactly match, it can be determined that they were engaged in different transportation tasks.

[0056] In S807, the joint work determination section 127 may determine that all joint works have been processed, and end the processing of this flowchart.

[0057] 10 is a diagram showing an example of a display screen on which the results of collaborative work identification based on this embodiment are output. Table 1001 shows, as an example, the work processes performed by all workers at the work sites shown in FIGS. 6 and 7, the workers who were involved in those work processes, and the processes they were responsible for. The inference result display unit 128 may display the above results on the screen. In the previous example of the work site, multiple workers carried out the work of transporting and installing multiple objects, so the workers who were involved in each work process and the processes they were responsible for are listed.

[0058] For example, by having the individual task identification unit 124 save the task log 152 for each individual, as in table 701 in FIG. 7, the worker manager can manage the work content of each individual worker. Furthermore, by having the collaborative task determination unit 127 display the collaborative task identification results as in FIG. 10, it is possible to grasp at a glance what tasks the workers performed as a team on-site. For example, it becomes possible to easily visualize what each worker transported and installed. For example, by outputting a table like that in FIG. 10, it is possible to clearly identify the workers who were involved in each task, which is expected to lead to a use case where responsibility is clearly assigned in the event of a problem such as poor construction.

[0059] 10, for example, if poor construction of object X is suspected, since the people in charge of the "installation" work of object X are workers A, B, and C, it can be determined that the problem is caused by the work of these workers (or any of them). Furthermore, if the paired work 505 is defined as work that requires a hold down ("hold down") when driving a "screwdriver" and a "hammer," then by referring to the work log 701 in FIG. 7, it can be determined that the cause of the poor construction is that when worker B performed the "screwdriver" work at time 7, worker C performed the "hammer" action at time 7 and time 8 without performing the corresponding hold down action.

[0060] In this way, according to this embodiment, it is possible to identify whether each work process performed on-site was a solo task or a collaborative task performed by multiple workers based on sensor data acquired from multiple workers, and in the latter case, it is possible to output the names of the workers who were involved in each collaborative task.

[0061] Specifically, as described with reference to FIGS. 1, 3, 8, etc., in the collaborative activity recognition system 1 in which processing is performed by a computer (e.g., information processing device 120) having a processor and a memory, the processor learns, by a predetermined method (e.g., DNN), the feature quantities (e.g., feature quantities 151) of the activities included in data received from a sensor (e.g., the posture sensor 112 of the sensor unit 110) for each of a plurality of workers, and stores the identified types of activities (e.g., "transport" or "screwdriving") and the activity duration from the start time of the activity to the end time of the activity. The system reads out a work log (for example, table 401 or table 701 showing an individual's work log), reads out a collaborative work definition database (for example, table 501 showing an example of collaborative work definition DB 126) that stores a type indicating the nature of a collaborative work by multiple workers (for example, collaborative installation work or collaborative transportation work) in association with the work of the workers that make up that type of collaborative work (for example, transportation or guiding action in the case of collaborative transportation work), and uses the read-out work log and the collaborative work definition database to determine whether the work by the worker is a collaborative work by multiple workers. This makes it possible to easily identify whether a work is a collaborative work by multiple workers, taking into account the nature of the collaborative work.

[0062] 3, the processor periodically performs the learning using the data received from the sensor, and identifies the type of work performed by the worker based on the worker's posture estimated from the worker's joint positions. As a result, when a worker is performing multiple tasks, the type of each task can be identified by performing the learning periodically.

[0063] 2 and the like, the processor identifies the type of work using the data that is the output value of the sensor worn at the joint position of the worker, and outputs the work log. This makes it possible to identify the type of work using the feature values ​​detected by the sensor worn at the joint position of the worker, and output the work log.

[0064] 5 and other figures, the collaborative work definition database defines rules (e.g., rule 504) regarding spatial overlap of work ranges for determining whether the work of each worker constitutes a collaborative work, and the processor determines whether a work is a collaborative work by multiple workers using the rules regarding spatial overlap and the work log. As a result, it is possible to determine that a collaborative work is being performed only when the work positions of the workers satisfy the spatial conditions.

[0065] 5 and other figures, the collaborative work definition database defines rules (e.g., rule 503) regarding the temporal overlap of work periods for determining whether the work of each worker constitutes a collaborative work, and the processor determines whether a work is a collaborative work by multiple workers using the rules regarding the temporal overlap and the work logs. As a result, it is possible to determine that a work is being done collaboratively only when the work time and timing of the workers satisfy the temporal conditions.

[0066] 5, 11, etc., the processor determines the working area, which is determined based on the rules regarding spatial overlap, based on the posture of the worker estimated from the joint positions of the worker learned using the data received from the sensor. This makes it possible to determine the working area in a way that reflects the arm length and other physical characteristics of each worker.

[0067] 3 and 4, the processor makes the determination using the data received in real time from the sensor, thereby making it possible to determine the type of work currently being performed in real time.

[0068] 8, the processor repeatedly determines whether or not a task performed by a worker is a collaborative task by multiple workers for all types of tasks included in the task log of each worker. This makes it possible to determine whether or not a task performed by all workers is a collaborative task.

[0069] 5 and S806 in FIG. 8, the collaborative work definition database may define rules regarding spatial overlap of work ranges for determining that the work of each worker constitutes a collaborative work, and rules regarding temporal overlap of work periods for determining that the work of each worker constitutes a collaborative work, and the processor may determine whether a work is a collaborative work by multiple workers using the rules regarding spatial overlap, the rules regarding temporal overlap, and the work logs. In this way, only work that satisfies not only the rules regarding spatial overlap of work ranges but also the rules regarding temporal overlap of work periods can be determined to be a collaborative work.

[0070] As described with reference to FIG. 10 and other figures, the processor outputs on a screen a collaborative work identification result that associates the type of collaborative work obtained as a result of the determination, a work process including an object worked on by the type of collaborative work, and a worker who performed the work process. This allows system users to easily understand the collaborative work identification result. In addition to the collaborative work identification result, the processor can also identify the workers who performed the collaborative work.

[0071] Previously, human behavior was recognized by defining nearby individuals as a group and a group of people moving in a similar direction as a group. However, whether or not a group can be considered a single group based on distance varied depending on the nature of their activities. Even if multiple individuals were nearby, they could be engaged in different tasks. In other words, determining whether a group of people exhibiting similar behaviors could be considered a group, and even if they exhibit similar behaviors, they could still be separate groups, making it difficult to accurately identify groups of workers. For example, when moving, if a group of two people (A, B) carrying a sofa is followed by a worker (C) carrying a lighter load, the former and latter tasks may be mistaken for a single group. Furthermore, when using a ladder for work at height, it is necessary to determine whether the worker climbing the ladder and the worker holding the ladder below are part of a single collaborative task. In this embodiment, the start and end of an activity are detected, and the degree of overlap in the time of the activities and the spatial location of the workers around the object are checked. Taking into account the temporal and spatial relationships of the group members' actions, the system determines whether a group is being performed by multiple workers. Therefore, it is possible to grasp collaborative work by a plurality of workers and the workers involved in the collaborative work more accurately than before.

[0072] Although the present invention has been described in detail above using the drawings, it is not limited to the various examples described above, and various modifications are possible without departing from the spirit of the present invention. For example, in the above embodiment, the collaborative work definition DB 126 shown in FIG. 5 does not define how many workers should work on an object that is the target of work by workers. However, if the object to be worked on and the number of workers who should be involved in the object are known in advance, object information that associates the object with the number of workers who should be involved in the object can be stored in the collaborative work definition DB 126, and the collaborative work determination unit 127 can read the object information and determine whether or not the work is a collaborative work. This allows for a simple method of determining whether or not the work is a collaborative work. [Explanation of symbols]

[0073] 1 Collaborative Work Recognition System 110 Sensor unit 120 Information processing equipment 220 Sensor Hub 121 Communications Department 122 Posture estimation section 123 Feature Extraction Unit 124 Personal Work Identification Unit 125 Log Storage Department 126 Collaboration definition DB 127 Collaborative Work Judgment Department 128 Inference result display section

Claims

1. A collaborative activity recognition system in which processing is performed by a computer having a processor and a memory, The processor: A work log is read out for each of a plurality of workers, which includes the type of work identified by learning the feature amount of the work included in the data received from the sensor using a predetermined method and the work period from the start time of the work to the end time of the work, and the work log is stored. A collaborative work definition database is read out, which stores a type representing the nature of a collaborative work by a plurality of workers and the work of the workers that constitutes the type of collaborative work in association with each other; When determining whether a task performed by a worker is a collaborative task performed by a plurality of workers using the read task log and the collaborative task definition database, the processor periodically performs the learning using the data received from the sensor, and identifies the type of work performed by the worker based on the worker's posture estimated from the worker's joint positions. A collaborative task recognition system characterized by:

2. the processor uses the data, which are output values ​​of the sensors worn at the joint positions of the worker, to identify the type of work and output the work log. The collaborative task recognition system according to claim 1 .

3. The collaborative work definition database defines rules regarding spatial overlap of work ranges for determining that work performed by each worker constitutes a collaborative work; the processor determines whether the work is a collaborative work by a plurality of workers by using the rules regarding spatial overlap and the work log; The collaborative task recognition system according to claim 1 .

4. The collaborative work definition database defines rules regarding the time overlap of work periods for determining that work by each worker constitutes a collaborative work; the processor determines whether the work is a collaborative work by a plurality of workers by using the rule regarding time overlap and the work log; The collaborative task recognition system according to claim 1 .

5. The processor determines the working range determined based on the rule regarding spatial overlap based on a posture of the worker estimated from joint positions of the worker learned using the data received from the sensor. The collaborative activity recognition system according to claim 3 .

6. the processor makes the determination using the data received in real time from the sensor. The collaborative task recognition system according to claim 1 .

7. The processor repeatedly determines whether a task performed by a worker is a collaborative task performed by multiple workers for all types of tasks included in the task log of each worker. The collaborative task recognition system according to claim 1 .

8. The collaborative work definition database defines rules regarding spatial overlap of work scopes for determining that work by each worker constitutes a collaborative work, and rules regarding temporal overlap of work periods for determining that work by each worker constitutes a collaborative work, the processor determines whether the work is a collaborative work by a plurality of workers by using the rules regarding spatial overlap, the rules regarding time overlap, and the work log; The collaborative task recognition system according to claim 1 .

9. the processor associates the type of collaborative work obtained as a result of the determination with a work process including an object worked on by the type of collaborative work, and a worker of the work process, and outputs the associated information to a screen. The collaborative task recognition system according to claim 1 .

10. A collaborative activity recognition method performed in a collaborative activity recognition system in which processing is performed by a computer having a processor and a memory, comprising: A work log is read out for each of a plurality of workers, which includes the type of work identified by learning the feature amount of the work included in the data received from the sensor using a predetermined method and the work period from the start time of the work to the end time of the work, and the work log is stored. A collaborative work definition database is read out, which stores a type representing the nature of a collaborative work by a plurality of workers and the work of the workers that constitutes the type of collaborative work in association with each other; When determining whether a task performed by a worker is a collaborative task performed by a plurality of workers using the read task log and the collaborative task definition database, performing the learning using the data received from the sensor at regular intervals, and identifying the type of work performed by the worker based on the worker's posture estimated from the worker's joint positions; A collaborative activity recognition method.

Images