Work recognition support system and work recognition support method

Abstract

To enable accurate discrimination between work tasks whose transition patterns of elemental actions are similar, and support efficient work recognition.SOLUTION: A work recognition support system S includes: a storage device 1 that stores an action transition table 30 defining transition patterns of elemental actions required for each work task; and an arithmetic device 3 that executes processing for acquiring motion data 10 of a target person of work recognition from a sensor, processing for recognizing actions and postures of respective body parts of the target person in time series on the basis of the motion data 10, processing for recognizing time transitions of elemental actions in the respective body parts from the recognized actions and postures, and processing of identifying work performed by the target person from among work procedures by collating the time transitions of the elemental actions with the action transition table 30.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a task recognition support system and a task recognition support method, and more particularly to a technology that can accurately distinguish tasks even if the transition patterns of elemental actions are similar, thereby enabling efficient task recognition support. [Background technology]

[0002] At manufacturing sites and the like, there have long been problems such as defective losses due to omissions or mistakes in work, delays in work processes due to the accumulation of redundancies or slow movements, and reduced efficiency and increased costs due to excessive or insufficient worker allocation.Therefore, there is a movement at such work sites to reduce the above-mentioned defective losses and optimize processes and personnel allocation by recognizing the actions of each worker and providing appropriate feedback to the site or management department. Therefore, there is a method for identifying the work content of a target person by breaking down the work into short-term elemental actions in chronological order and recognizing and determining the transitions of these elemental actions. In such a method, a behavior transition table is prepared that defines the transitions of the elemental actions for each work content, and the work content is determined by comparing the combinations of elemental actions that have actually been recognized for workers at a manufacturing site, etc., with the behavior transition table.

[0003] As a prior art related to the above-mentioned method, for example, a behavior recognition system (Patent Document 1) has been proposed that can accurately recognize daily actions that do not follow a certain pattern but that vary in order and frequency depending on the action. This prior art is a behavior recognition system that detects the movements of a target person and processes the movement data to recognize their actions, and the recognition process is configured in two steps: a first step and a second step, in which the first step recognizes a certain movement pattern performed over a short period, such as the period of a single movement of bringing something to one's mouth, based on time-series changes in information about the person's movement detected by processing the movement data, and the second step recognizes an entire movement over a long period, such as the period of eating, which is a combination of several movements performed over the short period, such as eating, based on the appearance of the certain movement pattern included in the period to be recognized.

[0004] Furthermore, a behavior recognition method, device, and program capable of grasping behavior at multiple abstraction levels (Patent Document 2) have also been proposed. This technology is a behavior recognition method in a behavior recognition device that acquires sensor data measuring the behavior of an individual and recognizes the behavior of the individual from the sensor data, and includes: a behavior model storage step of storing model data corresponding to a behavior model and a behavior recognition algorithm for each of multiple abstraction levels; a first identification step of extracting a feature vector from the sensor data and performing a recognition process on the feature vector using model data corresponding to a first abstraction level of the multiple abstraction levels, thereby recognizing and storing a behavior identifier of the first abstraction level; and a second recognition step of recognizing and storing, for each of the multiple abstraction levels, a sequence consisting of one or more behavior identifiers of a level lower than the corresponding abstraction level, by performing a recognition process using model data corresponding to the abstraction level, wherein the second recognition step is repeated for at least one abstraction level including the second abstraction level. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-215927 [Patent Document 2] Japanese Patent Application Laid-Open No. 2010-213782 Summary of the Invention [Problem to be solved by the invention]

[0006] However, with conventional technology, it is difficult to deal with cases where the transition patterns of the elemental actions are the same or very similar despite the fact that the tasks are different from each other. For example, if a transition pattern is recognized in which the combination of elemental actions "waving hand to the right" and "waving hand to the left" is repeated multiple times, the task content may be "saying goodbye" or "wiping windows." However, these task contents are the same as the combination of elemental actions, and it is difficult to accurately distinguish them even with conventional technology.

[0007] On the other hand, it is possible to represent actions at a high level of abstraction with transitions of actions at a lower level of abstraction by performing recognition at multiple levels of abstraction, as in the technology shown in Patent Document 2. However, in recognizing actions at each level of abstraction, only comparison with the action transition table at the next lower level of abstraction is performed, so this does not lead to a solution to the above problem (in the above example, to distinguish between "saying goodbye" and "wiping the window," action transition information at a level lower in abstraction than "waving one's hand to the right" and "waving one's hand to the left" is not used for comparison). Furthermore, as the number of element actions with different abstraction levels increases over time, the number of action transition tables used for matching also increases by the number of abstraction levels multiplied by the number of action types. This makes misidentification more likely to occur, and if a misidentification occurs with an action identifier with a low abstraction level, it will affect all higher-level identifiers.

[0008] The present invention has been made to solve the above-mentioned problems, and aims to provide a technology that can accurately distinguish between tasks even if the transition patterns of elemental actions are similar, and can support efficient task recognition. [Means for solving the problem]

[0009] One aspect of a representative task recognition support system of the present invention for solving the above-mentioned problems is characterized by including: a storage device that stores a behavior transition table that specifies, for each task that constitutes a work procedure, a transition pattern of component actions required for that task; a process for acquiring motion data about a subject of task recognition from a motion observation sensor; a process for recognizing the behavior and posture of each body part of the subject in chronological order based on the motion data; a process for recognizing the time transition of component actions of each body part from the recognized behavior and posture; and a calculation device that executes a process for comparing the time transition of the component actions with the behavior transition table in the storage device and identifying the task performed by the subject from the work procedure.

[0010] Furthermore, one aspect of a representative task recognition support method of the present invention is characterized in that an information processing device includes a storage device that stores a task transition table that defines, for each task that constitutes a task procedure, a transition pattern of component tasks required for that task, and executes the following processes: acquiring task data relating to a subject of task recognition from a sensor for observing tasks; recognizing the task and posture of each body part of the subject in chronological order based on the task data; recognizing the time transition of component tasks of each body part from the recognized tasks and postures; and comparing the time transition of component tasks with the task transition table in the storage device to identify the task performed by the subject from the task procedure. [Effects of the Invention]

[0011] According to the present invention, it is possible to accurately distinguish between tasks even if the transition patterns of elemental actions are similar, thereby supporting efficient task recognition. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is an overall block diagram showing a task recognition support system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating an example of the hardware configuration of an information processing device that constitutes the task recognition support system of the present embodiment. [Figure 3] FIG. 2 is a block diagram showing the correspondence between data in the present embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a sensor data DB according to the present embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of body part action recognition based on motion data according to the present embodiment. [Figure 6] 10A and 10B are diagrams illustrating an example of behavior target recognition and behavior position recognition using environmental data according to the present embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of elemental behavior recognition in the present embodiment. [Figure 8] 10 is a diagram illustrating an example of behavior transition data acquired from a work procedure manual in this embodiment. FIG. [Figure 9] FIG. 4 is a diagram illustrating an example of a behavior transition table according to the present embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a task recognition support flow (flow of elemental behavior matching and detailed behavior matching) in this embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of element behavior matching in the present embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of detailed behavior matching in the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, the embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the description of the examples shown below. Examples in which the specific configuration is modified are also included within the scope that does not deviate from the idea or purpose of the present invention. For example, the following examples are intended to explain the present invention in detail, and are not necessarily limited to those that include all of the configurations included in the description.

[0014] In the configuration of the invention described below, the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and redundant explanations may be omitted.

[0015] The designations "first," "second," "third," etc. in this specification are used to identify components and do not necessarily limit the number, order, or content thereof. Furthermore, numbers used to identify components are used in different contexts, and numbers used in one context do not necessarily indicate the same configuration in another context. Furthermore, this does not prevent a component identified by a certain number from also serving the function of a component identified by another number.

[0016] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings etc. may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings etc.

[0017] As used herein, elements referred to in the singular are intended to include the plural unless the context clearly indicates otherwise.

[0018] In the following description, the communication device may be one or more communication interface devices. The one or more communication interface devices may be one or more communication interface devices of the same type (e.g., one or more NICs (Network Interface Cards)), or two or more communication interface devices of different types (e.g., an NIC and an HBA (Host Bus Adapter)). Note that the network that the communication device accesses for communication may be the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), a mobile phone network, or the like, but is not limited to these.

[0019] In the following description, a "main storage device" refers to one or more memory devices, which are an example of one or more storage devices. At least one of the memory devices in the main storage device may be a volatile memory device or a non-volatile memory device.

[0020] In the following description, an "auxiliary storage device" may be one or more persistent storage devices, which are an example of one or more storage devices. The persistent storage device may typically be a non-volatile storage device, specifically, for example, a hard disk drive (HDD), a solid state drive (SSD), or a non-volatile memory express (NVMe) drive.

[0021] Furthermore, in the following description, a "processor" which is an arithmetic unit may be one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be other types of processor devices such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a hardware circuit that performs part or all of the processing (for example, an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0022] In the following description, information that provides an output in response to an input may be described using expressions such as "xxx table" or "xxx database." However, this information may be data of any structure (for example, structured data or unstructured data), or may be a learning model such as a neural network, genetic algorithm, or random forest that generates an output in response to an input. Therefore, "xxx table" or "xxx database" may be referred to as "xxx information." In the following description, the structure of each database or table is an example, and one database or table may be divided into two or more databases or tables, or all or part of two or more databases or tables may be one database or table.

[0023] In the following description, processing may be described using a "program" as the subject. However, because a program is executed by a processor to perform a predetermined process using a storage device and / or an interface device, etc., as appropriate, the subject of the process may also be the processor (or a device such as a controller having the processor). A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0024] Here, "elementary actions" refers to each unit element when a series of actions is divided. For example, if a "series of actions" is related to assembly work on a production line, these "elementary actions" would be "screw tightening," "part installation," "inspection," etc. Elemental actions are not limited to individual work contents or work procedures, but are composed of broad, general action contents.

[0025] Furthermore, the task recognition support technology of this embodiment can be used to ensure safety and support workers by understanding the tasks performed by workers in factories, for example. Of course, the application is not limited to factory tasks, and the technology can be applied to any task as long as each task can be composed of elemental actions.

[0026] <1. System configuration> FIG. 1 is a diagram showing an example of the overall configuration of a task recognition support system S according to this embodiment. First, an overview of the overall configuration of the task recognition support system S according to this embodiment, as well as the flow from acquiring data to be used in processing to outputting the processing results, will be described. As a premise for this description, it is assumed that a worker at a manufacturing site performs a series of tasks in accordance with a work procedure manual 11. In other words, the target of task recognition is the worker (hereinafter referred to as the task target).

[0027] Various sensors 6 are also installed at the manufacturing site. Examples of the sensors 6 include a digital still camera, an acceleration sensor, an angular velocity sensor, a temperature sensor, and a sound sensor. The sensors 6 observe the worker and measure various sensor data 10 related to the worker. This sensor data 10 includes RGB image data of the worker and his / her surrounding environment captured by a digital still camera, frequency data observed by a sound sensor, acceleration data observed by an acceleration sensor, and angular velocity data observed by an angular velocity sensor. Each piece of sensor data 10 is captured from the sensors 6 into the behavior recognition device 110 of the task recognition support system S in real time or at regular intervals.

[0028] The task recognition support system S is composed of a behavior recognition device 110 and a task recognition device 120 connected via a network N. The behavior recognition device 110 is a device that breaks down the task subject's behavior into elemental actions based on sensor data 10 related to the task subject and performs recognition. The task recognition device 120 is a device that implements various functions for determining which task in the work procedure manual 11 the task performed by the task subject corresponds to.

[0029] Of the above configurations, the behavior recognition device 110 acquires the sensor data 10 using a data acquisition unit 111. The acquired sensor data 10 may include motion data indicating the body motion of the worker and environmental data indicating the working environment of the worker. A specific description of the sensor data 10 will be given later.

[0030] As shown in FIG. 1, the behavior recognition device 110 includes a data acquisition unit 111 as well as a part behavior recognition unit 112, a behavior target recognition unit 113, a behavior position recognition unit 114, and an element behavior recognition unit 115.

[0031] Of these, the part action recognition unit 112 receives motion data from the sensor data 10 from the data acquisition unit 111, and recognizes part action information consisting of action information and posture information of each body part of the work subject. The action target recognition unit 113 receives environmental data from the sensor data 10 from the data acquisition unit 111, and recognizes object information that is the action target. The action position recognition unit 114 receives environmental data from the sensor data 10 from the data acquisition unit 111, and recognizes location information that is the action position. The element action recognition unit 115 receives part action information from the part action recognition unit 112, object information from the action target recognition unit 113, and location information from the action position recognition unit 114, and recognizes the chronological transition of the element actions included in the work.

[0032] The task recognition device 120 of this embodiment uses a work procedure manual 11 to narrow down target task candidates when performing task recognition. The work procedure manual 11 is a work site procedure manual or instruction manual that lists the behavioral transitions for each task. The behavioral transition table creation unit 121 of the task recognition device 120 acquires information on each task procedure described in the work procedure manual 11 from, for example, an appropriate device such as an administrator's terminal. The behavioral transition table creation unit 121 generates a behavioral transition table 30 from the acquired information on each task procedure.

[0033] In addition to the behavior transition table creation unit 121, the task recognition device 120 also includes an element behavior matching unit 122 and a detailed behavior matching unit 123. Of these, the element behavior matching unit 122 receives the chronological transitions of the element behaviors of the task from the element behavior recognition unit 115 of the task recognition device 110 and the behavior transition table 30 from the behavior transition table creation unit 121, and matches the chronological transitions of the element behaviors with the behavior transition table 30. The element behavior matching unit 122 also outputs the task recognition result 12 of the task obtained by this matching to, for example, a terminal of a manager.

[0034] Furthermore, if the component action matching unit 122 is unable to uniquely narrow down the matching task, the detailed action matching unit 123 recognizes the task by further detailed matching. The detailed action matching unit 123 receives part action information from the part action recognition unit 112 of the action recognition device 110, object information from the action target recognition unit 113, location information from the action position recognition unit 114, the (plurality of) task candidates narrowed down from the component action matching unit 122, and the action transition table 30 from the action transition table creation unit 121, and outputs the task recognition result 12 for the task.

[0035] The activity recognition support system S including the behavior recognition device 110 and the activity recognition device 120 can be configured with a general server device. Fig. 2 shows an example of the hardware configuration of an information processing device J that is such a server device.

[0036] <Hardware configuration of information processing device> As shown in FIG. 2, the information processing device J has a configuration in which an auxiliary storage device 1, a main storage device 2, a processor 3, an output device 4, and a communication device 5 are connected via a BUS.

[0037] Of these, the auxiliary storage device 1 is a storage means configured by non-volatile storage elements, as already described, and in this embodiment, stores at least a behavior transition table 30 and a sensor data DB 35. Specific data configuration examples of these pieces of information will be described later.

[0038] The main memory device 2 also serves as storage means for storing programs P, including an OS (Operating System) that controls the information processing device J as a whole, and various applications. The processor 3 loads and executes the programs P in the main memory device 2, implementing the required functions. In this manner, in this embodiment, functions such as calculation and control are realized by the processor 3 executing the programs P stored in the storage means, thereby implementing predetermined processing in cooperation with other hardware, and only the functional blocks thereof are shown in FIG. 1.

[0039] The output device 4 is a means for outputting the results of information processing to the information processing device J, i.e., the administrator of the task recognition support system S, and can be a UI (User Interface) device such as a display, speaker, touch panel, or printer. The communication device 5 is a communication means configured with a communication chip compatible with the protocol of the network N. The communication device 5 accesses the network N and is communicatively connected to each device constituting the task recognition support system S via this network N. The information processing device J may also be equipped with other UI (User Interface) devices such as a keyboard, mouse, or microphone as input devices for receiving input operations by the administrator.

[0040] <Functional configuration of the task recognition support system> Next, we will explain the functional configuration of the task recognition support system S. Fig. 3 is a block diagram illustrating how a data acquisition unit 111 acquires sensor data 10, and a body part action recognition unit 112, an action target recognition unit 113, and an action position recognition unit 114 execute processing based on motion data 20 and environmental data 21 including the acquired sensor data 10.

[0041] <2. Data Acquisition Section> The data acquisition unit 111 receives sensor data 10 from sensors 6 installed at manufacturing sites or the like via a network N, and stores the received sensor data in a sensor data DB 35. The sensor data DB 35 is a collection of records linking together values ​​such as the observation date and time of the data, the manufacturing site, the sensor type, and the sensor data (see FIG. 4). The data acquisition unit 111 also outputs motion data 20 and environmental data 21 from the sensor data 10. As the sensors 6 that observe the sensor data 10, it is preferable to provide one or more types of sensors 6, such as accelerometers and gyroscopes, that can acquire information on the movements of workers, and one or more types of sensors 6, such as cameras and audio sensors, that can acquire information on the work environment.

[0042] The movement data 20 output from the data acquisition unit 111 represents the transition of the movement of the worker over time, and is, for example, time-series data on the coordinates and angles of each joint of the body. The movement data 20 is sent to the part action recognition unit 112. On the other hand, the environmental data 21 output from the data acquisition unit 111 represents the environment during work, and is, for example, data such as a target object image showing the object to be worked on, a target position image showing the work position, and target action sound emitted when performing the target action. The environmental data 21 is sent to the action target recognition unit 113 and the action position recognition unit 114.

[0043] <3. Part behavior recognition unit> Fig. 5 is a schematic diagram showing an example of part action recognition in the part action recognition unit 112. In the example shown in Fig. 5, the body of the work subject is divided into a total of four parts: the left arm, the right arm, the trunk, and the lower limbs, and action information and posture information for each part (body part) is recognized. The following figures will also show an example in which the body of the work subject is divided into a total of four parts: the left arm, the right arm, the trunk, and the lower limbs and recognized.

[0044] As the motion data 20, the motion measurement values ​​of each body part obtained by the sensor 6 may be used as the feature values ​​for that body part directly, or the feature values ​​may be extracted separately from a feature extractor that has been trained by supervised learning. In this case, the feature extractor is a model obtained by providing a deep learning engine with training data that is a set of motion measurement values ​​(e.g., values ​​of acceleration, angular velocity, etc.) observed by the sensor 6 for each of the four body parts (left arm, right arm, trunk, and lower limbs) and the correct motion of each body part (e.g., left arm: holding, left arm posture: lateral, etc.), and proceeding with learning.

[0045] In this embodiment, a deep neural network trained by supervised learning, for example, is used to output a part-action identifier consisting of the action and posture of each body part at each time from the motion feature amounts of each body part. The part-action identifier is the value of the action and posture of each body part in FIG. 5, such as "hold" for the left arm and "sideways" for the posture of the left arm. Even when trained by a deep neural network, for example, the motion features of each body part are composed of movements between several joints, which is less complex than movements composed of the entire body of the worker, and therefore less difficult to identify.

[0046] By dividing the worker's entire body into several body parts and recognizing the actions and postures of each body part, it is possible to retain the differences between elemental actions in a form that can be expressed by comparing the detailed movements and postures of the body (for example, one posture of the body part is "sideways" and the other is "downwards"), even if the elemental action name is the same (for example, actions of different body parts are both "holding"). Therefore, even if the chronological transition patterns of elemental actions are the same or very similar, it is possible to distinguish between them with high accuracy. For example, at the final time in Figure 5, while "crouching," the system recognizes the granularity of "holding" with the left hand in a "downwards" direction and "hitting" with the right hand, and can grasp the details of the action of "hitting" and clearly distinguish it from a "hitting" action in an "upwards" direction.

[0047] The recognition result by the part action recognition unit 112 , that is, the part action identifier of each body part, is sent to the element action recognition unit 115 and the detailed action matching unit 123 .

[0048] <4. Action target recognition unit and action location recognition unit> 6 is a schematic diagram showing an example of behavior target recognition in the behavior target recognition unit 113 and an example of behavior location recognition in the behavior location recognition unit 114. The behavior target recognition unit 113 extracts object features at hand for detecting an object of behavior as one of the environmental data 21 in the sensor data 10. Furthermore, the behavior location recognition unit 114 extracts location features for detecting a behavior location as one of the environmental data 21 in the sensor data 10.

[0049] FIG. 6 shows an example in which the action target recognition unit 113 extracts hand-periphery image features, which are data around the worker's hands cut out from image data captured by a first-person camera, for each of the left and right hands as object features. Alternatively, object information may be obtained by frequency analysis using a glove-type sound sensor, or feature extraction may be performed by integrating feature features around both hands without separating the features for each hand. Also shown is an example in which the action location recognition unit 114 directly uses image data captured by a first-person camera as location image features. Alternatively, the workspace may be mapped in advance, and a tracking sensor that tracks the worker's position information may be compared with the pre-mapping information.

[0050] By recognizing object information and location information of the action target, it is possible to retain a form that can express the differences between elemental actions by comparing the action target and action location, even if the elemental action names are the same. As a result, it is possible to distinguish with high accuracy between tasks with the same or very similar transition patterns in the chronological order of elemental actions. For example, at the median time in Figure 6, it is recognized at a granular level that the action is being performed in "Workplace A" using an "electric screwdriver," and it is possible to clearly distinguish between the action of tightening screws from tightening screws with a manual screwdriver and tightening screws on different objects in different workplaces.

[0051] The activity position recognition unit 114 may output a position identifier that is the activity position from a feature extractor that has learned the feature amounts of images taken around the worker using supervised learning. In this case, the feature extractor is a model obtained by providing training data, which is a set of image data (or its feature amounts) taken around the worker and the correct answer for the activity position (for example, a position identifier such as workplace A or workplace B), to a deep learning engine and proceeding with learning.

[0052] The behavior location recognition unit 114 outputs a location identifier, which is a behavior location, from the feature extractor. The output location identifier is sent to the element behavior recognition unit 115 and the detailed behavior matching unit 123.

[0053] <5. Elemental Action Recognition Unit> 7 is a schematic diagram showing an example of element action recognition in the element action recognition unit 115. The element action recognition unit 115 outputs an element action identifier (e.g., screw tightening) at each time from a part action identifier including action information and posture information of each body part transmitted from the part action recognition unit 112 (e.g., left arm action: pressing, left arm posture: downward, ...), an object identifier including object information transmitted from the action target recognition unit 113 (e.g., left arm object: screw, right arm object: electric screwdriver), and a position identifier including location information transmitted from the action position recognition unit 114 (e.g., workshop A). ​​The output element action identifier is transmitted to the element action matching unit 122.

[0054] One possible method for identifying element behavior identifiers is to prepare an behavior transition table 30 (see FIG. 9 ) that indicates the correspondence between each element behavior and a part behavior identifier, object identifier, and location identifier, and then match plausible element behaviors from the table using data clustering (described in detail below). Alternatively, a configuration using a deep neural network trained by supervised learning may be used for identification. Comparing the method using the behavior transition table 30 with the method using the deep neural network, the behavior transition table 30 can more simply identify each element included in an action by combining the part behavior identifier, object identifier, and location identifier identified for each body part of the worker, which is expected to further improve the identification accuracy of behavior recognition.

[0055] <6. Behavior transition table creation section> Fig. 8 is a schematic diagram showing an example of behavior transition data acquisition from the work procedure manual 11 in the behavior transition table creation unit 121. Fig. 9 is a schematic diagram of an example of a behavior transition table 30 created in the behavior transition table creation unit 121. The behavior transition table creation unit 121 receives the work procedure manual 11 as input and outputs the behavior transition table 30 shown in Fig. 9.

[0056] For example, a work procedure manual or instruction manual available at the work site can be used as is as the work procedure manual 11. In the work procedure manual 11, the work procedure is often explained in advance using transitions at the element action level, as in the action transition data of Fig. 8, but there are also cases where each element action is described in sentences, as in the example of the work procedure manual 11 shown in Fig. 8. Even in this case, the action transition data can be created in the action transition table creation unit 121.

[0057] In this case, the behavior transition table creation unit 121 performs natural language processing on the work procedure manual 11 using a deep neural network trained by supervised learning, and identifies the component behavior identifiers of each behavior from the explanation of each component behavior in the work procedure manual 11. For example, a model is prepared that has undergone deep learning using a set of text data of numerous work procedure manuals 11 and correct answers for the component behavior identifiers contained in the text data as training data, and the component behavior identifiers are identified by providing the text data of the work procedure manual 11 to be processed to the model. Note that if the explanation in the work procedure manual 11 contains detailed descriptions of specific actions that specify body parts or target positions, detailed behaviors consisting of each body part behavior identifier, object identifier, and position identifier are also output using a model similar to the above model.

[0058] The behavior transition data for each task identified by the behavior transition table creation unit 121 is added to the behavior transition table 30 of Fig. 9. In the example of Fig. 9, for each transition behavior of each task in the behavior transition table 30, the element behavior is always included as a unique identifier. On the other hand, for detailed behavior, only those output in the behavior transition data of Fig. 8 are added. The output behavior transition table 30 is sent to the element behavior matching unit 122 and the detailed behavior matching unit 123.

[0059] <7. Behavior Verification Unit> 10 is a flowchart showing the flow of behavior matching in the behavior matching unit 125, which is composed of the element behavior matching unit 122 and the detailed behavior matching unit 123. The behavior matching unit 125 compares the time-series transition patterns of the part behavior identifier including the behavior information and posture information of each body part transmitted from the part behavior recognition unit 112, the object identifier including the object information transmitted from the behavior target recognition unit 113, the position identifier including the location information transmitted from the behavior position recognition unit 114, and the element behavior identifier transmitted from the element behavior recognition unit 115 with the behavior transition table 30 created in the behavior transition table creation unit 121.

[0060] The behavior matching unit 125 narrows down the type of task through the matching and outputs a task recognition result. As shown in the flowchart of FIG. 10, the component behavior matching unit 122 performs matching (S1) in the first step. Only if there are multiple narrowed-down task candidate results in the matching result (S2: YES), the detailed behavior matching unit 123 performs matching in the second step (S3). The behavior matching unit 125 outputs the unique task candidate thus narrowed down as the task recognition result 12 (S4). If the narrowed-down task candidate is uniquely determined in the first step (S2: NO), the task recognition result 12 is output in the second step without going through the detailed behavior matching unit 123 (S5). In this way, by performing matching separately for the first and second steps, the amount of calculation and calculation time required to output the task recognition result can be reduced in cases where the task recognition result 12 can be output based only on the transition pattern of the component behaviors.

[0061] <8. Elemental Behavior Matching Unit> 11 is a schematic diagram showing an example of behavior matching performed by the element behavior matching unit 122. The element behavior matching unit 122 matches the element behavior transitions 40 configured by the time-series transitions of the element behavior identifiers sent from the element behavior recognition unit 115 with the behavior transition table 30 created by the behavior transition table creation unit 121, and narrows down the tasks represented by the element behavior transitions 40.

[0062] A method of comparing the element action transitions 40 with the action transition table 30 may be, for example, time-series data clustering, which calculates the chronological distance relationship between the element action transitions 40 and the transition patterns of the element actions of each task in the action transition table 30, and narrows down the tasks with the closest distance relationship. Such a method may, for example, calculate a chronological distance relationship of 1.0 for a set of element actions "transport" → "insertion" → "hammer slam" in the three stages of transitions 1 to 3 indicated by the element action transitions 40, and for each record in the action transition table 30, for which the element actions in the three stages of transitions 1 to 3 are in the order of "transport" → "insertion" → "hammer slam"; for which two of the three stages are in the order, the chronological distance relationship is 0.66; for which only one of the three stages is in the order, the chronological distance relationship is 0.33. Although the behavior transition table 30 also includes detailed behaviors for each task, the element behavior matching unit 122 only performs matching based on the transition patterns of element behaviors and does not handle information on detailed behaviors.

[0063] The element action matching unit 122 outputs the candidate actions extracted by narrowing down the tasks through the element action matching. For example, if there are multiple candidate actions extracted through the time-series data clustering, all of them are output. In the example of Figure 11, as in element action transition 40, the only task in the action transition table where the element action transitions from "transport" to "fitting" to "hammering" is "attaching XX below △△." Therefore, through time-series data clustering or the like, only one candidate action, "attaching XX below △△," is output as the candidate action through matching.

[0064] <9. Detailed Behavior Verification Unit> 12 is a schematic diagram showing an example of behavior matching performed by the detailed behavior matching unit 123. The detailed behavior matching unit 123 matches detailed behavior transitions 50, which are time-series transition patterns, with the behavior transition table 30 created in the behavior transition table creation unit 121 in detailed behaviors configured by part behavior identifiers including behavior information and posture information of each body part transmitted from the part behavior recognition unit 112, object identifiers including object information transmitted from the behavior target recognition unit 113, and location identifiers including location information transmitted from the behavior location recognition unit 114. The detailed behavior matching unit 123 uniquely determines the task represented by the detailed behavior transition 50 from among the task candidates sent from the component behavior matching unit 122 and outputs the task recognition result 12.

[0065] The detailed behavior matching unit 123 matches only the tasks included in the task candidates sent from the element behavior matching unit 122 with the behavior transition table 30. Similar to the matching in the element behavior matching unit 122, a method for matching the detailed behavior transition 50 with the behavior transition table 30 may be, for example, time-series data clustering, which calculates the time-series distance relationship between the detailed behavior transition 50 and the transition pattern of the detailed behavior of each task in the behavior transition table 30 and narrows down the tasks with the closest distance relationship. In the example of FIG. 12 , of the task candidates "●● installation on the lower part of ▲▲" and "●● installation on the upper part of ▲▲" narrowed down by the element behavior matching unit 122, "●● installation on the upper part of ▲▲" differs from the detailed behavior transition 50 in terms of the left arm posture and right arm posture in transitions 2 and 3, i.e., there is a distance, and therefore the task candidate is narrowed down to "●● installation on the lower part of ▲▲" by time-series data clustering, etc. In this way, by utilizing detailed actions in the matching, it is possible to accurately distinguish between tasks with the same or very similar action transition patterns, such as tasks that differ only in installation location.

[0066] Although one embodiment of the present invention has been described above, this is merely an example for explaining the present invention, and the scope of the present invention is not limited to this embodiment. The present invention can be implemented in various other forms.

[0067] The above description can be summarized as follows: The following summary may include supplementary explanations and explanations of variations of the above description.

[0068] In the task recognition support system S of this embodiment, the arithmetic device (processor 3) further executes a process of acquiring data on the subject's action target from a sensor 6 for observing the action target, and a process of recognizing the object of the subject's action target based on the data on the action target, and when recognizing the time transition of the elemental actions, the time transition of the elemental actions may be recognized from a combination of the action and posture of each body part and the object.

[0069] This allows for cases where it is difficult to determine the work content based on the movements and postures of workers alone, and by taking into account the objects being worked on, such as screws, bolts, and specific parts, it becomes possible to accurately identify the work content. In turn, it becomes possible to accurately distinguish between tasks with similar transition patterns of elemental actions, enabling more efficient work recognition.

[0070] Furthermore, in the task recognition support system S of this embodiment, the arithmetic device (processor 3) may further execute a process of acquiring data on the position of the action target of the subject from a position observation sensor 6, and a process of recognizing the position of the action target of the subject based on the position data, and when recognizing the time transition of the elemental actions, the time transition of the elemental actions may be recognized from a combination of the action and posture of each body part and the position.

[0071] This allows for cases where it is difficult to determine the work content based on the movements and posture of workers alone, and makes it possible to accurately identify the work content by taking into account the position of the work object, such as at feet, waist height, or near the ceiling.Finally, it makes it possible to accurately distinguish between tasks with similar transition patterns of elemental actions, enabling more efficient work recognition.

[0072] Furthermore, in the task recognition support system S of this embodiment, the arithmetic device (processor 3) may further execute the following processes: acquiring data on the object of action of the subject from a sensor 6 for observing the object of action; recognizing the object of action of the subject based on the data on the object of action; acquiring data on the position of the object of action of the subject from a sensor 6 for position observation; and recognizing the position of the object of action of the subject based on the data on the position; and when recognizing the time transition of the elemental actions, the time transition of the elemental actions may be recognized from a combination of the action and posture of each body part, the object, and the position.

[0073] This allows for cases where it is difficult to determine the work content based on the movements and postures of workers, etc., and by taking into account the objects and positions, it becomes possible to identify the work content with greater accuracy. In turn, it allows for more accurate determination and more efficient work recognition, even between tasks with similar transition patterns of elemental actions.

[0074] Furthermore, in the task recognition support system S of this embodiment, when identifying the task performed by the subject, the arithmetic device (processor 3) may, if multiple tasks are identified as tasks performed by the subject as a result of comparing the time transition of the component tasks with the task transition table in the storage device, further compare the time transition of each of the component tasks, the object, and the position with the task transition table to identify the task performed by the subject from the task procedure.

[0075] According to this, in cases where it is possible to narrow down to one task based solely on the movements and postures of workers, etc., it is possible to improve processing efficiency by omitting processing that takes other elements (objects and positions) into consideration, and it is also possible to deal with cases where it is difficult to narrow down to one task based solely on the movements and postures of workers, etc., and by taking the above-mentioned objects and positions into consideration, it is possible to identify the task content with even greater accuracy. In turn, even between tasks with similar transition patterns of elemental actions, it is possible to distinguish with greater accuracy and recognize tasks more efficiently.

[0076] Furthermore, in the task recognition support system S of this embodiment, the arithmetic device (processor 3) may further execute a process of extracting, based on information in a task procedure manual 11 in which the task procedures are described, a correspondence between the identification information of each task and the actions and postures of each body part of the elemental actions that make up each task from explanatory information regarding the task content of each task in the task procedure manual 11, and generating the action transition table.

[0077] This allows the behavior transition table 30 to be efficiently generated based on the work procedure manual 11 and used for task recognition processing. Furthermore, tasks with similar transition patterns of component actions can be accurately distinguished, enabling more efficient task recognition.

[0078] Furthermore, in the task recognition support system S of this embodiment, when generating the behavior transition table, the arithmetic device (processor 3) may extract, from the explanatory information regarding the task content in the task procedure manual 11, the identification information of each task, the behavior and posture of each body part of the elemental behavior that constitutes each task, and the correspondence between the object and position of the action target, and generate the behavior transition table.

[0079] This makes it possible to efficiently generate a more detailed behavior transition table 30 based on the work procedure manual 11 and use it in the task recognition process. Furthermore, even tasks with similar transition patterns of elemental actions can be distinguished more accurately, enabling more efficient task recognition. [Explanation of symbols]

[0080] S Task Recognition Support System J Information processing device 1 Auxiliary storage device (storage device) P Program 2 Main memory (storage device) 3. Processor (arithmetic unit) 4 Output Devices 5. Communications equipment 6 sensors 10 Sensor Data 11 Work Procedures 12 Work recognition results 30 Action Transition Table 35 Sensor Data DB 40 Elemental Action Transition 50 Detailed behavior transition 110 Behavior recognition device 111 Data Acquisition Unit 112 Part behavior recognition unit 113 Action Target Recognition Unit 114 Action position recognition unit 115 Elemental Action Recognition Unit 120 Work recognition device 121 Action transition table creation section 122 Element Behavior Matching Unit 123 Detailed Behavior Matching Unit 125 Behavioral Matching Unit

Claims

1. a storage device that stores an action transition table that defines transition patterns of element actions required for each task that constitutes a work procedure; a computing device that executes a process of acquiring motion data on a subject of task recognition from a motion observation sensor, a process of recognizing the actions and postures of each body part of the subject in chronological order based on the motion data, a process of recognizing the time transition of elemental actions of each body part from the recognized actions and postures, and a process of identifying the task performed by the subject from the task procedure by comparing the time transition of the elemental actions with the action transition table in the storage device; A task recognition support system including:

2. The task recognition support system according to claim 1, The computing device further executing a process of acquiring data on a motion target of the subject from a motion target observation sensor, and a process of recognizing an object of the motion target of the subject based on the data on the motion target; when recognizing the time transition of the elemental actions, the time transition of the elemental actions is recognized from a combination of the action and posture of each body part and the object; A task recognition support system characterized by:

3. The task recognition support system according to claim 1, The computing device further executing a process of acquiring data on the position of a motion target on the subject from a position observation sensor, and a process of recognizing the position of the motion target on the subject based on the data on the position; when recognizing the time transition of the elemental actions, the time transition of the elemental actions is recognized from a combination of the action and posture of each body part and the position; A task recognition support system characterized by:

4. The task recognition support system according to claim 1, The computing device further executing a process of acquiring data on a motion target of the subject from a motion target observation sensor, a process of recognizing an object of the motion target of the subject based on the data on the motion target, a process of acquiring data on the position of the motion target of the subject from a position observation sensor, and a process of recognizing the position of the motion target of the subject based on the data on the position, when recognizing the time transition of the elemental actions, the time transition of the elemental actions is recognized from a combination of the action and posture of each body part, the object, and the position; A task recognition support system characterized by:

5. The task recognition support system according to claim 4, The computing device When identifying the work performed by the subject, if a plurality of works are identified as the work performed by the subject as a result of comparing the time transition of the elemental actions with the action transition table in the storage device, further comparing the time transition of each of the elemental actions, the object, and the position with the action transition table to identify the work performed by the subject from the work procedure. A task recognition support system characterized by:

6. The task recognition support system according to claim 1, The computing device and extracting, based on information in a work procedure manual in which the work procedures are described, a correspondence relationship between identification information of each work and the behavior and posture of each body part of the element behavior constituting each work from explanatory information on the work content of each work in the work procedure manual, and generating the behavior transition table. A task recognition support system characterized by:

7. The task recognition support system according to claim 6, The computing device When generating the behavior transition table, extracting, from the explanatory information on the work content in the work procedure manual, identification information of each work, the behavior and posture of each body part of the element behavior constituting each work, and the correspondence between the object and position of the action target, and generating the behavior transition table. A task recognition support system characterized by:

8. The information processing device a storage device for storing an action transition table that defines transition patterns of element actions required for each task constituting the task procedure; The system executes the following processes: acquiring motion data on a subject of task recognition from a motion observation sensor; recognizing the actions and postures of each body part of the subject in chronological order based on the motion data; recognizing the time transition of elemental actions of each body part from the recognized actions and postures; and comparing the time transition of elemental actions with the action transition table in the storage device to identify the task performed by the subject from the task procedure. A task recognition support method comprising:

Images