Task recognition system, task recognition method and program

Abstract

To provide a task recognition system capable of high-speed and advanced task recognition by reducing the processing time required for determination based on unnecessary task component recognition models while selecting a task component recognition model according to the task progress.SOLUTION: The task recognition system uses multiple task component recognition models in which information of movements, touch, hearing, position of a worker during the work, camera, equipment, and tools are input. The task recognition system includes: a product selection filtering section that selects task component recognition models based on production instructions; a progress filtering unit that selects a task component recognition model depending on the progress of the work; and a task recognition unit that recognizes the work out of the selected task component recognition models.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] The present disclosure relates to an activity recognition system, an activity recognition method, and a program. [Background technology]

[0002] Patent Document 1 discloses a work support device that links a work ability model created based on worker instruction information and the physical characteristics of the worker in a work process with information obtained by sensing the physical characteristics of the worker, and selects and associates a work ability model for each worker. The work support device includes a storage unit that stores work instruction information including instructions for a work procedure for making the worker execute a specified work process, and work ability model information including a plurality of models determined based on the physical characteristics of the worker in each work process. The work support device also includes a work sensing processing unit that senses the physical characteristics of the worker in the work performed based on the work instruction information and generates sensing information. The work support device also includes a work ability evaluation processing unit that selects one or more models corresponding to the sensing information from the work ability model information for each worker, and generates work ability evaluation information by associating the selected model with the sensing information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-051620 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a need to provide workers with work support information, such as work progress information, work load alarms, and work defect indications, in real time while they are working. In this case, it is necessary to perform complex work recognition at high speed, but there is a possibility that recognition using multiple models for advanced work recognition will be required, which may make it difficult to achieve both shortening the processing time and ensuring the recognition accuracy. Therefore, the object of the present disclosure is to provide a work recognition system that selects products based on production instructions, selects work component recognition models according to the progress of work, and can perform advanced work recognition at high speed by reducing the processing time required for judgment using unnecessary work component recognition models. [Means for solving the problem]

[0005] The task recognition system of the present disclosure comprises: A task recognition system using a plurality of task component recognition models to which information on a worker's motion, touch, hearing, position, camera, and equipment / tools during the task is input, a product selection filtering unit for selecting the work component recognition model based on a production instruction; a progress filtering unit that selects the task component recognition model according to a task progress; and an activity recognition unit that recognizes an activity from the selected activity component recognition model.

[0006] With the above configuration, it is possible to provide a work recognition system that selects products based on production instructions, selects work component recognition models according to the progress of the work, and reduces the processing time required for judgment using unnecessary work component recognition models, thereby enabling high-speed, advanced work recognition.

[0007] The task recognition system of the present disclosure comprises: The product selection filtering unit narrows down the work component recognition model to work for one product.

[0008] With the above configuration, it is possible to narrow down the work component recognition model from multiple products to one product.

[0009] The task recognition system of the present disclosure comprises: The progress filtering unit excludes the task component recognition model for a completed task.

[0010] With the above configuration, it is possible to exclude task component recognition models for completed tasks and narrow down the number of task component recognition models.

[0011] The task recognition system of the present disclosure comprises: After the product selection filtering unit selects the product, it is determined whether the operation can be recognized; If the task cannot be recognized, the product selection filtering unit is removed, the task is recognized, and the task recognition result is output, and the progress filtering unit selects the task. When the task is recognized, the task recognition result is outputted and is selected by the progress filtering unit.

[0012] With the above configuration, work can be recognized even when unusual work such as duplicate work or non-scheduled work is performed.

[0013] The task recognition system of the present disclosure comprises: The present invention is characterized by comprising a task recognition model library that stores a task component recognition model referenced by the product selection filtering unit.

[0014] With the above configuration, the task recognition model can be stored in advance.

[0015] The task recognition method of the present disclosure includes: A method for recognizing work using a plurality of work component recognition models to which information on a worker's motion, touch, hearing, position, camera, and equipment / tools during work is input, Selecting the work component recognition model based on a production instruction; Selecting the task component recognition model according to the task progress; The task recognition method recognizes the task from the selected task component recognition model.

[0016] With the above configuration, it is possible to provide a task recognition method that selects products based on production instructions, selects learning models according to the progress of work, and reduces the processing time required for judgment based on unnecessary learning models, thereby enabling high-speed, advanced task recognition.

[0017] The program of the present disclosure is A program to be executed by an information processing device that recognizes work using a plurality of work component recognition models to which information on a worker's motion, touch, hearing, position, camera, and equipment / tools during work is input, Selecting the work component recognition model based on a production instruction; Selecting the task component recognition model according to the task progress; The program causes an information processing device to recognize the task from the selected task component recognition model.

[0018] With the above configuration, it is possible to provide a program that causes an information processing device to perform product selection based on production instructions, select a work component recognition model according to the progress of the work, and perform high-speed, advanced work recognition by reducing the processing time required for judgment using unnecessary work component recognition models. Effect of the Invention

[0019] The present disclosure makes it possible to provide a work recognition system that selects products based on production instructions, selects work component recognition models according to the progress of the work, and reduces the processing time required for judgment using unnecessary work component recognition models, thereby enabling high-speed, advanced work recognition. [Brief description of the drawings]

[0020] [Figure 1] 1 is a block diagram showing a configuration of a task recognition system according to an embodiment. [Diagram 2] FIG. 2 is a correlation block diagram of components related to unit task recognition and advanced task recognition according to the embodiment. [Diagram 3] FIG. 2 is a diagram showing a relationship between a unit verb and an action verb according to an embodiment. [Figure 4]FIG. 2 is a diagram illustrating an example of a relationship between a unit task and a recognition model according to the embodiment. [Diagram 5] FIG. 5 is a continuation of the example of the relationship between the unit tasks and the recognition model in FIG. 4 according to the embodiment. [Figure 6] 2 is a block diagram showing a relationship between a task component recognition model and a unit task recognition model according to the embodiment; FIG. [Figure 7] 1 is a flowchart of unit operation recognition in the production of a product according to an embodiment. [Figure 8] 4 is a diagram illustrating a reference coordinate setting unit that sets the coordinates of a fixed reference and a transported object (target object) reference according to the embodiment. FIG. [Figure 9] 11 is a flowchart of a reference coordinate setting process in the production of a product according to an embodiment. [Figure 10] FIG. 13 is a diagram showing an example in which a task position is added to a unit task according to the embodiment; [Figure 11] 12 is a continuation of the example in which a work position is added to the unit work of FIG. 11 according to the embodiment. FIG. [Figure 12] 2 is a block diagram showing the relationship between a unit task recognition model, a task component recognition model for advanced task recognition, and an advanced task recognition model according to the embodiment; FIG. [Figure 13] 1 is a flowchart of advanced task recognition in product production according to an embodiment. [Figure 14] 1 is a schematic diagram (top) and a flowchart (bottom) showing product selection filtering and progress filtering in the production of products according to an embodiment. [Figure 15] This figure (top diagram) shows an example of coordination of data acquisition of sensors corresponding to work areas in an embodiment, a schematic diagram of the work areas (bottom left diagram), and a correspondence table between work areas and sensors (bottom right diagram). [Figure 16] 11 is a flowchart showing a process of prioritizing and switching a plurality of sensors or devices according to a work position in the diversity input control according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Embodiment Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiment. Furthermore, not all of the configurations described in the embodiment are necessarily essential as means for solving the problem. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and repeated explanations are omitted as necessary.

[0022] (Description of the task recognition system according to the embodiment) 1 is a block diagram showing a configuration of an activity recognition system according to an embodiment, and the activity recognition system according to the embodiment will be described with reference to FIG.

[0023] 1 , the task recognition system 100 includes an external device / equipment 101, a wearable device 102, and an task recognition platform 134. In addition, as upstream peripheral systems of the task recognition system 100, the task recognition system 100 includes a bill of materials (BOM) 135, a product data management (PDM), bill of process (BOP), and bill of equipment (BOE) 136, and a manufacturing execution system (MES) 141. In addition, the task recognition system 100 includes an application (APPL) 142 as a downstream peripheral system.

[0024] The external device / equipment 101 is an external element provided in the process for purposes other than those of this system, such as a camera, an indoor GPS (Global Positioning System), other equipment, jigs and tools, a transport system, etc.

[0025] The wearable device 102 is a variety of sensors worn by the worker, such as a first-person perspective camera, a microphone, a vibration sensor, a pressure sensor, an IMU (Inertial Measurement Unit), a strain sensor, a vital sensor, and the like.

[0026] The task recognition platform 134 is a part configured with an information processing device (not shown) that is the core of the task recognition system of the present disclosure. The information processing device at least includes a processor (e.g., a CPU (Central Processing Unit)) that executes a program to execute processing, and a memory that stores the program. The information processing device may be configured with a cloud server in which some or all of the functions are distributed. The task recognition platform 134 includes an input I / F (Interface) 103, a time series synchronization unit 104, task recognition data preparation 105, a learning / setting unit 114 by task recognition data preparation, a task recognition model library 115, a task recognition model library dynamic filtering 131, a task recognition multimodal AI 132, and a unit task task performance data output unit 133.

[0027] The input I / F 103 is an interface for connecting to the external device / equipment 101 and the wearable device 102 .

[0028] The time series synchronization unit 104 synchronizes the data of each of the external device / equipment 101 and the wearable device 102 in terms of time.

[0029] Task recognition data preparation 105 prepares data for a model for recognizing task components related to a worker, a task object, equipment, tools, etc. Task recognition data preparation 105 includes a body / finger skeleton estimation unit 106, a tactile / auditory sensor data unit 107, a body part position estimation unit 108, a worker identification unit 109, etc., with respect to the worker. Task recognition data preparation 105 includes a position estimation unit 110 and an appearance image unit 111 with respect to the task object. Task recognition data preparation 105 includes an equipment / jig / tool ​​signal / data unit 112 with respect to equipment / tools. Task recognition data preparation also includes a diversity input control unit 113.

[0030] The body and finger skeleton estimation unit 106 estimates the skeletons of the body and fingers using data from the sensors or devices. The body and finger skeleton estimation unit 106 also prioritizes and switches the sensors or devices used for skeleton estimation according to instructions from the diversity input control unit 113.

[0031] The tactile / auditory sensor data unit 107 acquires input data such as vibration, friction, reaction force, and sound from a sensor or device. The body part position estimation unit 108 estimates the position of each body part using data from the sensor or device. The body part position estimation unit 108 prioritizes and switches sensors or devices used for estimating the position of each body part according to instructions from the diversity input control unit 113. The worker identification unit 109 identifies the worker by ID (Identification) or the like.

[0032] The position estimation unit 110 estimates the position of the work target object. The appearance image unit 111 captures an image of the appearance of the work target object using the external device / equipment 101 and the wearable device 102 to obtain an appearance image.

[0033] An equipment / jig / tool ​​signal / data unit 112 acquires data on equipment and tools in use. A diversity input control unit 113 prioritizes and switches input sensors or devices according to the work position.

[0034] The learning / setting unit 114 by task recognition data preparation is a learning / setting unit for constructing a task component recognition model 143, which will be described later. The data prepared by the task recognition data preparation 105 is learned or set, and each part of the task component recognition model 143 is constructed.

[0035] The task recognition model library 115 is a reference unit of the task recognition platform 134. The task recognition model library 115 includes a task component recognition model 143, a task component recognition model-based learning / setting unit 123, a general-purpose task recognition module 124, an area-based reference coordinate control unit 127, a position definition unit 128, a task position-based learning / setting unit 129, and an advanced task recognition model 130.

[0036] The work component recognition model 143 is a storage unit that stores each recognition model. The work component recognition model 143 includes a posture and behavior recognition model 116 and a gesture and voice recognition model 117 for the worker. The work component recognition model 143 includes an object recognition model 118, a physical F / B (Feedback) recognition model 119, and a state recognition and measurement model 120 for the work object. The work component recognition model 143 includes an equipment and tool data recognition model 121 for the equipment and tools. The work component recognition model 143 includes a work position estimation model 122 for the work position.

[0037] The posture and action recognition model 116 is a model that recognizes posture and action by learning or setting from the body and finger skeleton estimation unit 106 and the tactile and auditory sensor data unit 107. The gesture and voice recognition model 117 is a model that recognizes gesture and voice by learning or setting from the body and finger skeleton estimation unit 106 and the tactile and auditory sensor data unit 107.

[0038] The object recognition model 118 is a model that recognizes an object by learning or setting from the body / finger skeleton estimation unit 106, the tactile / auditory sensor data unit 107, the body part position estimation unit 108, and the appearance image unit 111. The physical F / B recognition model 119 is a model that recognizes physical feedback by learning or setting from the tactile / auditory sensor data unit 107. The state recognition / measurement model 120 is a model that recognizes and measures the state of a work object by learning or setting from the appearance image unit 111.

[0039] The equipment and tool data recognition model 121 is a model that recognizes equipment and tool data by learning or setting from the equipment and tool signal and data unit 112. The work position estimation model 122 is a model that estimates the work position by acquiring information from the body part position estimation unit 108, the work object position estimation unit 110, and the equipment and tool signal and data unit 112. The work position estimation model 122 is set based on the process or product. The work position estimation model 122 also acquires information from the area-based reference coordinate control unit 127. Furthermore, the work position estimation model 122 provides data of the position estimation result to the work position learning / setting unit 129.

[0040] The task component recognition model learning / setting unit 123 is a learning / setting unit for constructing a unit task recognition model 125, which will be described later. The data prepared by the task component recognition model 143 is learned or set, and the unit task recognition model 125 is constructed.

[0041] The general-purpose task recognition module 124 is a unit task recognition section that inputs the motion, touch, hearing, position, camera, and equipment / tool ​​information of the worker when working on an object into the unit task model, and recognizes the unit task. In other words, the general-purpose task recognition module 124 recognizes what the worker did and how he / she did it. The general-purpose task recognition module 124 includes a unit task recognition model 125 and a unit task definition section 126.

[0042] The unit task recognition model 125 is a model for recognizing tasks that can be defined generically regardless of the product among various tasks. The unit task recognition model 125 is generated by combining product-independent components of the task component recognition model based on the task content in the unit task recognition model generation unit. For example, a generic task that is not product-dependent is a combination of verbs related to tasks such as tightening, fitting, and removal on a task object such as a part that does not include product-specific information and is common across products, and will be described later. The unit task recognition model 125 is a model that recognizes a unit task by learning or setting from the posture and behavior recognition model 116, the gesture and voice recognition model 117, the object recognition model 118, the physical F / B recognition model 119, the state recognition and measurement model 120, and the equipment and tool data recognition model 121.

[0043] The unit task definition section 126 defines a task for creating the unit task recognition model 125 .

[0044] The area-specific reference coordinate control unit 127 performs control to change the reference coordinates according to the area of ​​the process. The area-specific reference coordinate control unit 127 provides the work position estimation model 122 with information on the fixed reference and the product reference.

[0045] The position definition unit 128 defines the position from the product 3D, the equipment 3D, etc., as described later. The position definition unit 128 also adds position information to the unit operation definition unit 126 in order to define the position where the unit operation is performed. The position definition unit 128 provides learning or setting data to the learning / setting unit 129 based on the operation position.

[0046] The work position based learning / setting unit 129 learns or sets from the work position estimation model 122, the general-purpose work recognition module 124, the position definition unit 128, and work items 138 of product-specific process information 137 described later. The work position based learning / setting unit 129 learns or sets and constructs an advanced work recognition model 130.

[0047] The advanced task recognition model 130 is a model for recognizing where, what, and how a worker did it. The advanced task recognition model 130 recognizes tasks with high accuracy by adding information specific to the product, such as the task position on the product, to general-purpose unit task recognition.

[0048] The task recognition model library dynamic filtering 131 excludes models other than the relevant product from a process library that includes models of multiple vehicle models, based on production instructions from the MES 141. Furthermore, it excludes completed models based on the task progress from the start of the task to the completion of the task. The task recognition model library dynamic filtering 131 obtains information on who the worker is from the worker identification unit.

[0049] The task recognition multimodal AI (Artificial Intelligence) 132 is an artificial intelligence for recognizing who, when, where, what, and how a worker did something. The task recognition multimodal AI 132 acquires information from the task recognition data preparation 105 and the task recognition model library dynamic filtering 131.

[0050] The unit task work performance data output unit 133 outputs work performance data for each unit task. The unit task work performance data output unit 133 acquires information from the task recognition multimodal AI 132. The data output by the unit task work performance data output unit 133 is utilized in an application (APPL) 142 for improving safety and ergonomics, improving quality assurance, improving productivity, etc.

[0051] BOM 135 is a bill of materials that contains information about parts required to manufacture a product and is basic information for understanding the configuration in which the product is assembled.

[0052] The PDM·BOP·BOE 136 includes a work element master 139 , product-specific process information 137 , and product and process drawing information 140 .

[0053] The work element master 139 is master information that standardizes and generalizes the components of work across products, and includes, for example, work objects, work verbs, tools, work positions, etc. The product-specific process information 137 includes information on work items 138, which are combinations of work elements defined in the work element master 139, and detailed work procedures. The product / process drawing information 140 includes information on parts 3D, process layouts, equipment 3D, etc.

[0054] The work element master 139 provides information to the unit work definition unit 126. The product-specific process information 137 acquires information from the work element master 139, provides the information to the product / process drawing information 140, and provides learning or setting data to the work position learning / setting unit 129. In addition, the product-specific process information 137 provides information to the position definition unit 128 and the area-specific reference coordinate control unit 127 via the work element master 139 that is defined in a standard manner across products.

[0055] The MES 141 is a manufacturing execution system that monitors and manages the work of factory equipment and workers by linking with each part of the factory production line. The MES 141 obtains information from the BOP / BOE 136. The MES also has a production instruction database to provide production instruction information to the work recognition model library dynamic filtering 131. The production instructions are instructions to people and equipment such as parts to be used and procedures in addition to the type and specifications of individual products in accordance with the production plan.

[0056] (Description of General-Purpose Task Recognition Module According to the Embodiment) FIG. 2 is a correlation block diagram of components related to unit task recognition and advanced task recognition according to the embodiment. FIG. 3 is a diagram showing the relationship between task verbs and unit verbs according to the embodiment. FIG. 4 is a diagram showing an example of the relationship between unit tasks and recognition models according to the embodiment. FIG. 5 is a continuation of the example of the relationship between unit tasks and recognition models in FIG. 4 according to the embodiment. FIG. 6 is a block diagram showing the relationship between a task component recognition model and a unit task recognition model according to the embodiment. FIG. 7 is a flowchart of unit task recognition in product production according to the embodiment. Unit task recognition according to the embodiment will be described with reference to FIGS. 2 to 7.

[0057] As shown in Fig. 2, the generic task recognition module 124 recognizes task items broken down into task elements such as generic task verbs and task objects. Furthermore, the generic task recognition module 124 defines, learns, or sets correlations with the task component recognition model 143 using, as a master, unit tasks defined by unit verbs that break down the task verb portion into smaller units at a level suitable for task recognition. Therefore, the generic task recognition module 124 constitutes a generic task recognition module that is not dependent on individual products and processes. By sharing the generic task recognition module 124 between different products, maintenance and operation can be performed efficiently, including responding to changes in products and processes.

[0058] The work element master 139 includes information on a work object 202 and a work verb / tool ​​203 as general-purpose work elements. The work element master 139 also includes information on a work position 204 as a product-specific work element. The information on the work object 202, the work verb / tool ​​203, and the work position 204 is obtained from a work item 206 of a BOP 205.

[0059] A work item 206 defines a series of tasks for producing a product by a combination of work objects and work verbs. Work elements are the elements that make up the work item 206. The main work elements are work verbs, work objects, tools, work positions, etc. Of these, by defining the work verbs, work objects, and tools as general-purpose master items, data collection and analysis across products and factories can be performed. As the work position may differ depending on the product even for the same part, the work position classification code, etc. can be standardized, but the work position itself is treated as a work element specific to the product.

[0060] The general-purpose task recognition module 124 includes a unit task definition section (general purpose) 209 , a task component recognition model correlation definition section (other than task position) 211 , and a general-purpose unit task recognition model 125 .

[0061] Moreover, the unit work 208 is a further subdivision of the contents of general-purpose work elements such as work verbs, work objects, tools, etc. Specifically, the work verbs are broken down into elements suitable for work recognition, and work is divided into these units to form a unit work. The unit work 208 comprises a unit work definition section (general purpose) 209 and a unit work definition section (product-specific) 210.

[0062] The unit work definition section (general purpose) 209 includes information on steps, which are the work sequence, unit verbs defined in a hierarchical structure according to granularity, recognition categories, and recognition items. The unit work definition section (general purpose) 209 is associated with the work object 202 and the work verb / tool ​​203. The unit verb is a verb with a granularity that allows work recognition. The work verbs in the work item 206 can be arbitrarily defined with various granularities, but the work recognition disclosed herein aims to specifically and in detail recognize various works performed at the manufacturing site. For this reason, it is necessary to subdivide the work verbs with a coarse granularity to a granularity that allows work recognition. If the granularity difference between the work verbs and the unit verbs is large, subdivision may be performed in multiple hierarchies.

[0063] There are two recognition categories, motion and event, which are set as targets for recognizing unit work, and are defined according to the characteristics and recognition purpose of the work. This allows the general-purpose work recognition module 124 to recognize various works in detail. Motion recognizes work as a block performed continuously in a chronological order, thereby recognizing continuous movements such as walking. Events recognize specific aspects of work such as the start of tightening and completion of tightening, thereby recognizing accurate work segmentation timing.

[0064] The recognition item is used to define a specific recognition target such as a start and an end for a recognition section of a unit verb.

[0065] The work component recognition model correlation definition unit (other than work position) 211 assigns a work component recognition model (other than the work position recognition model, which is a work component specific to a product) suitable for recognition according to the work characteristics to the work element (work verb × work object) × unit work (unit verb, recognition category, item) and defines the correlation. The work component recognition model correlation definition unit (other than work position) 211 includes information on necessity, pattern, variation, and reliability.

[0066] The degree of necessity is defined for each task component recognition model 143, thereby enabling efficient allocation of unit tasks 208 and task component recognition models 143, and maintaining and controlling robustness in the event that output from some models is missing. The patterns and variations are the patterns and detailed variations to be recognized by the task component recognition model 143. The reliability is defined for each task component recognition model 143, enabling recognition accuracy evaluation to be performed within the system. The reliability is set arbitrarily or based on actual recognition result data. This evaluation is used to rank multiple candidates when multiple candidates appear when task recognition is performed by the general-purpose task recognition module 124, for example.

[0067] The general-purpose task recognition module 124 is shared and reused across products when there are common parts or similar parts between products.

[0068] As shown in Fig. 3, a work object n and a work verb n are obtained from a work item 206 of a BOP 205. The work verb n is divided into unit verb 1, unit verb 2, ... unit verb n. The unit verb 1 is further divided into unit verb 1.2, unit verb 1.3, and unit verb 1.4. Note that Fig. 3 shows an example of the relationship between the work verbs and the unit verbs, and a third or subsequent hierarchical level may be provided to further subdivide the unit verbs.

[0069] Specifically, as shown in the example in Figure 4, the take-out of unit verb 1 is decomposed into unit verb 1.1 (empty hand carry), unit verb 1.2 (grab), and unit verb 1.3 (carry).

[0070] As shown in FIG. 6, the work component recognition model correlation definition unit (other than work position) 211 is reflected in the unit work recognition model 125 through the learning / setting unit 123 by the work component recognition model learned or set in the work component recognition model 143. FIG. 5 is a continuation of the vertical column of FIG. 4, and for example, as shown in FIG. 5, the unit verb 1.2 "grab" is associated with the object recognition model 118 and the physical F / B recognition model 119. Also, the unit verb 1.2 "grab" is associated with the posture and action recognition model 116. Furthermore, the unit verb 1.2 "grab" is associated with the equipment and tool data recognition model 121. In this way, the work component recognition model correlation definition unit (other than work position) 211 associates the unit verb with the work component recognition model 143.

[0071] The method of unit task recognition in the production of a product is shown in FIG. 7. The information processing device first receives a production instruction from the MES 141 (701). The information processing device obtains the product production sequence by receiving the production instruction. Next, the information processing device obtains product selection filtering from the task recognition model library 115 (702). Product selection filtering will be described later. The information processing device obtains a task recognition model required for the product from the library by the product selection filtering, configures a set of models required for performing task recognition for the product, and the process N starts (705). Then, task item N of the process N starts (706). Next, the information processing device inputs data from devices and equipment (704) as needed during the task execution process and performs time series synchronization and processing (703), and then performs task component recognition (707). The general-purpose task recognition unit of the task recognition multimodal AI (132) continuously obtains task component recognition results (708) and passes them to the unit task recognition (general purpose) (709). In unit task recognition (general purpose), a general-purpose unit task is recognized 709 by matching with the unit task recognition model 125, and the recognition result is output 710. Note that, when multiple unit tasks are recognized during unit task recognition, multiple candidates ranked in order of reliability are output as described above.

[0072] The result output is passed to the advanced task recognition model 130. Moreover, the processes of 703 to 710 are continuously performed until the task of step N is completed.

[0073] This provides a general-purpose task recognition module that can recognize unit tasks independently of the product.

[0074] (Description of area-specific reference coordinate control unit according to the embodiment) Fig. 8 is a diagram for explaining a reference coordinate setting unit for setting the coordinates of a fixed reference and a transported object (target object) reference according to the embodiment. Fig. 9 is a flowchart for setting reference coordinates in the production of a product according to the embodiment. The area-specific reference coordinate control unit 127 according to the embodiment will be explained with reference to Figs. 8 and 9.

[0075] As shown in Fig. 8, in a line-type production site, work objects are produced by flowing through a line in a factory. For example, there are cases where the position of the work object does not move relative to the floor during part processing and assembly work on a fixed workbench, and cases where the work object is transported to a transport system, such as when assembling parts to a vehicle body transported by a conveyor. For each case, two reference coordinates for the work position estimation model are set: a fixed reference and a transported object (object) reference. By dynamically switching between these two reference coordinates according to the worker's work area, the work position coordinates required for work recognition can be accurately grasped.

[0076] As shown in the upper diagram of Fig. 8, assuming that objects 800, 801, 802, and 803 are moving on a conveyance system, the area-specific reference coordinate control unit 127 in Fig. 9 defines the area where work is performed on the conveyed object 800, etc. as a conveyed object reference application area, and the other areas as fixed reference application areas. Regarding switching of the reference coordinates, the above-mentioned areas are divided in advance on the line layout, and the reference coordinates to be applied to each area are defined, and the reference coordinates defined sequentially according to the position of the worker are applied.

[0077] As shown in the middle diagram of Fig. 8, the fixed reference is based on coordinates fixed to the floor surface of a line such as a building pillar or a workbench. The fixed reference is set in an area where the work object does not move relative to the floor surface while the worker is working. On the other hand, as shown in the bottom diagram of Fig. 8, the transported object reference is based on the coordinates of the transported object when the object 800 or the like is flowing on the transport system. When the work object moves relative to the floor surface, the transported object reference is set.

[0078] In cases where the transported object reference is applied, the position information of the work object on the transport system is obtained in real time by connecting to the transport system, and the work position is estimated based on the transported object (more precisely, based on the transported work object).

[0079] The setting of the fixed reference and the object reference is performed by the area-specific reference coordinate control unit 127, which is a reference setting unit, as shown in FIG. 9. First, the information processing device inputs data from the external device / equipment 101 and the wearable device 102 and performs time series synchronization / processing 703. The information processing device inputs the data to the work position estimation model 122. Then, the information processing device recognizes the worker's presence area 901. The area-specific reference coordinate control unit 127 executes reference coordinate control 903 when the worker's presence area acquired from the worker presence area recognition 901 moves to a different area based on the area-specific reference coordinate setting 902 set in advance with reference to the process layout / equipment 3D. Next, a reference coordinate switching instruction is output to the work position estimation model 122 904. In this way, the area-specific reference coordinate is controlled according to the area where the worker is present.

[0080] (Description of Advanced Task Recognition According to the Embodiment) The advanced task recognition according to the embodiment will be described with reference to Fig. 2 and Figs. 10 to 13. Fig. 10 is a diagram showing an example in which a task position is added to a unit task according to the embodiment. Fig. 11 is a diagram continuing from the example in which a task position is added to the unit task of Fig. 11 according to the embodiment. Fig. 12 is a block diagram showing the relationship between a unit task recognition model, a task component recognition model for advanced task recognition, and an advanced task recognition model according to the embodiment. Fig. 13 is a flowchart of advanced task recognition in product production according to the embodiment.

[0081] 2, the advanced task recognition 213 performs high-definition and high-precision task recognition by combining task position information specific to a product or process with the output obtained by the generic task recognition module 124. In some cases, the advanced task recognition 213 may further supplementarily add an individual recognition model. Furthermore, by operating the advanced task recognition 213 in combination with the generic task recognition module 124, it is possible to efficiently configure an advanced task recognition model that corresponds to differences in products and processes while maximizing versatility.

[0082] As shown in FIG. 2, the advanced task recognition 213 includes a unit task definition section (product-specific) 210, a task component recognition model correlation definition section (task position) 212, and the advanced task recognition model 130.

[0083] The unit operation definition part (product-specific) 210 includes the reference coordinates, target part, position type, and range specification method, which are the operation position details. The unit operation definition part (product-specific) 210 is included in the unit operation 208, and acquires the operation position code set as a standard common to all products from the product-specific operation position 204 of the operation element master 139. The unit operation definition part (product-specific) 210 also acquires the product's 3D coordinate information and process layout information as detailed operation positions from the PDM·BOP 207.

[0084] The reference coordinate system is selected from either the fixed reference system or the conveyed object reference system defined in FIG.

[0085] The target part indicates a body part of the worker for which the working position is to be acquired, such as the head, arms, or legs.

[0086] The position type is the method of specifying the position. For example, the position type is two-dimensional plane and three-dimensional space.

[0087] The range specification method is a method for specifying the range of a work position. For example, the range may be specified by two diagonal points of a solid object or a plane figure, or by a route.

[0088] The work component recognition model correlation definition unit (work position) 212 assigns a work component recognition model suitable for recognition to the work position (a work position recognition model that is a work component specific to the product) and defines the correlation. The work component recognition model correlation definition unit (work position) 212 includes information on the necessity, reference coordinates, target part, position type, and reliability.

[0089] The degree of necessity is defined for each work component recognition model 143, so that the allocation of the work position and the work component recognition model 143 can be efficiently performed, and robustness can be maintained and controlled in the case where the output from some models is missing.

[0090] The reference coordinates, target portion, and position type use information acquired in the unit operation definition section (product-specific) 210.

[0091] The reliability is defined for each task component recognition model 143, and allows the recognition accuracy to be evaluated within the system. The reliability is set arbitrarily or based on the actual recognition result data. This evaluation is used to rank multiple candidates when multiple candidates are generated during task recognition in the advanced task recognition 213, for example.

[0092] The advanced task recognition model 130 combines a general-purpose task recognition module 124 and a task position estimation model 122 for task component recognition (with other models added as necessary), enabling detailed and highly accurate task recognition according to the product.

[0093] As shown in FIG. 12, the work component recognition model correlation definition unit (work position) 212 associates the unit task recognition model 125 of the general-purpose task recognition module 124, the task position estimation model 122 of the work component recognition model 143, and other models 1201, and learning or setting is performed in the task position learning / setting unit 129.

[0094] As shown in Fig. 13, advanced task recognition is performed. Fig. 13 illustrates advanced task recognition in addition to the content explained in Fig. 7. Therefore, the content explained in Fig. 7 will be omitted.

[0095] The information processing device outputs the unit task recognition (general purpose) result 710, and the advanced task recognition unit of the task recognition multimodal AI 132 acquires the unit task recognition result (general purpose) 1301. In parallel, the area-specific reference coordinate control unit 127 acquires the worker's position information from the task component recognition 707, and transmits the reference coordinate information to the task position estimation model 122. The task position estimation model 122 estimates task position coordinates by acquiring the worker's position information from the task component recognition 707 and acquiring the reference coordinate information from the area-specific reference coordinate control unit 127, and outputs the result to the advanced task recognition unit of the task recognition multimodal AI 132. The advanced task recognition unit acquires the task position coordinate estimation result 1306, and performs a matching evaluation between the unit task and the actual task position using the advanced task recognition model 130, narrows down the unit task candidates from the corresponding candidates 1307, and outputs a candidate ranking 1308.

[0096] Next, the advanced task recognition unit of the task recognition multimodal AI 132 recognizes (estimates) 1302 the most accurate unit task from the unit task recognition candidates and candidate rankings acquired from the unit task recognition (general) result output 710 and the unit task recognition (product-specific) result output 1308. In parallel, the advanced task recognition unit performs task element recognition 1303 by monitoring the individual unit task recognition data in time series for task elements that are a collection of multiple unit tasks (task object = part X, task verb = tightening, tool = tool Y, task position code = Z, task status: in progress, completed, etc.). Next, the advanced task recognition unit performs task item recognition 1304 in conjunction with the output of task element recognition results for task items that are combinations of task elements (e.g. task item = tightening part X, task status: in progress, completed, etc.). The advanced task recognition unit then outputs 1305 the above-mentioned recognition results (unit task recognition 1302, task element recognition 1303, task item recognition 1304) as needed. The recognition results are sent to a database, data analysis, utilization applications, etc. Furthermore, the processing of the advanced task recognition section is carried out continuously until the task of step N is completed.

[0097] When the task element recognition 1303 fails to recognize an element, the process returns to the unit task recognition 1302. The task element recognition 1303 recognizes an element based on multiple results of the unit task recognition 1302.

[0098] In this way, the advanced task recognition system includes an advanced task recognition unit that recognizes tasks at an advanced level using as input unit task recognition (general) candidates that are the output results of the general-purpose task recognition unit of the task recognition multimodal AI 132, and unit task recognition (product-specific) candidates related to product-specific task positions that have as input the estimation results of the task position estimation model. In this way, a task recognition system and a task recognition method are provided.

[0099] (Description of Task Recognition Model Filtering According to the Embodiment) 14 is a schematic diagram (upper diagram) and a flowchart (lower diagram) showing product selection filtering and progress filtering in the production of products according to the embodiment. Task recognition model filtering according to the embodiment will be described with reference to FIG.

[0100] As shown in the upper diagram of Figure 14, for a company that mass-produces a variety of products, the model library for all products will be huge, for example, 15 products x 5 factories. Therefore, when operating these huge task recognition models on a mixed production line, problems may arise in terms of processing speed and the capacity of the information processing device. Therefore, when moving from the production preparation stage to the mass production stage, the corresponding models are narrowed down by factory, so that the factory library becomes, for example, 3 products x 1 factory in Figure 14. By further narrowing down the corresponding models by process, the process library becomes, for example, 3 products x 1 process in Figure 14. Here, if the breakdown of the three products is product A, product B, and product C, in mass production of the product, for example, when a production instruction for product B is received, it is possible to narrow down to 1 product x 1 process by narrowing down to the model corresponding to product B. In this way, in a process library that contains models of multiple vehicle models, excluding models other than the corresponding product according to the production instruction is called product selection filtering.

[0101] Furthermore, excluding completed models according to the work progress from the start of the work to the completion of the work is referred to as progress filtering.

[0102] As shown in the lower diagram of FIG. 14, the information processing device receives a production instruction from the MES 141 and acquires a product production sequence 701. Next, the information processing device executes product selection filtering to select an execution model set for process N from the work recognition model library 115 702. Work for process N is started 705. Then, work recognition for work items 1, 2, ..., N is performed 1401. In parallel, the information processing device performs work recognition 1402. If the work can be recognized (Yes in step 1402), the information processing device outputs the work recognition result. If the work cannot be recognized (No in step 1402), the information processing device cancels, i.e., removes, the product selection filtering 1404, and performs work recognition again 1405. The reason for canceling the product selection filtering is that it is considered that an unusual work such as a duplicated work or an unscheduled work has been performed. The cancellation range is preset arbitrarily. (For example, the relevant process x the relevant product, the relevant process x all products, the relevant factory x product x the relevant product, etc.) If the rework can be recognized, the information processing device outputs the work recognition result 1403. Next, the information processing device selects 1406 by excluding models for the completed work from the execution model set using progress filtering. This updates the execution model. The processes 1402 to 1406 are performed continuously until the work of process N is completed.

[0103] In this way, the work recognition system includes a product selection filtering unit that selects a work recognition model based on production instructions, a progress filtering unit that selects a work recognition model according to work progress, and a work recognition unit that recognizes work from the selected work recognition model. The work recognition system also narrows down the work recognition models to work for one product by the product selection filtering unit. The work recognition system also excludes work recognition models for completed work by the progress filtering unit. The work recognition system further determines whether the work can be recognized, and if the work cannot be recognized, removes the product selection filtering, recognizes the work again, and outputs the work recognition result.

[0104] In this way, a task recognition system can be provided that can select learning models according to the progress of the task, shorten the processing time required for judgment based on unnecessary learning models, and perform high-speed, advanced task recognition.

[0105] (Description of Diversity Input Control Unit According to the Embodiment) Fig. 15 is a diagram (upper diagram) showing an example of cooperation of data acquisition of sensors corresponding to a working area according to an embodiment, a schematic diagram of the working area (lower left diagram), and a correspondence table between the working area and the sensor (lower right diagram). Fig. 16 is a flowchart showing a process of prioritizing and switching a plurality of sensors or devices according to a working position in the diversity input control according to an embodiment. The diversity input control unit according to an embodiment will be described with reference to Figs. 15 and 16.

[0106] When work performance data is acquired using multiple sensors or devices such as cameras, GPS, IMUs, etc., these may become obstacles depending on the positional relationship with the process or product, etc., and some devices may not be usable (camera: blind spots, etc.) or may not function properly (GPS: radio wave reflection, obstruction, etc.). Therefore, in a system equipped with multiple sensors or devices, the diversity input control unit 113 ensures robustness and accuracy of data acquisition by automatically selecting the most suitable device according to the work position.

[0107] As shown in the lower left diagram of Figure 15, the surrounding area of ​​the object is divided into an object-away area, which is an area (Zone A) away from the object and not affected by obstacles such as equipment, for example the work area, an object-approaching area, which is an area in contact with the object approaching the object (Zone B), and an object-inside area, which is an area within the object (Zone C).

[0108] Taking the acquisition of the work position as an example, as shown in the lower right diagram of Fig. 15, in Zone A, the work position can be detected with high accuracy by indoor GPS. Also, the position can be estimated by IMU-based position estimation. However, since indoor GPS has higher accuracy, indoor GPS is prioritized in Zone A, and the IMU is used as a backup. When capturing an image of the target, Zone A cannot be used because it is outside the imaging range.

[0109] Similarly, in Zone B, "location estimation from images" or "location estimation by image skeleton estimation or image object recognition, etc." can be obtained from images of captured objects, so image skeleton estimation-based location estimation is the most accurate (however, other devices may be more accurate depending on the camera depression angle and distance from the camera). Indoor GPS loses accuracy in areas with many radio wave obstructions and reflectors, so in Zone B, the IMU is used as a second backup and the indoor GPS as a third backup.

[0110] In Zone C, position estimation by the IMU takes top priority. If the object blocks radio waves inside the object, indoor GPS cannot be used or its accuracy will be extremely poor. Also, "position estimation by image" that captures the object or "position estimation by image skeleton estimation or image object recognition, etc." can be used partially in areas that are not blind spots, so they are used as backups.

[0111] When a worker performs work along a work route as shown in the upper diagram of Fig. 15, indoor GPS is used for work position 1, and image skeleton estimation-based position estimation using the position data is used for work positions 2 and 3. Similarly, IMU-based position estimation using the position data is used for work positions 4 and 5, and image skeleton estimation-based position estimation using the position data is used for work position 6.

[0112] As shown in Fig. 16, first, the information processing device receives a production instruction from the MES 141 701. Next, the information processing device acquires 1602 a diversity input control setting from a diversity input control setting unit 1601. In parallel, work for process N is started 705. At the start, if the work start position for each process is determined as a predetermined position, the position is reset 1603. The information processing device acquires information on the position reset. Next, the information processing device continuously acquires 1604 work position estimation results for work items 1, 2, ... n using the work position estimation model 122. During work for process N, the information processing device instructs 1605 the selection of an input device as needed depending on the work position estimation result.

[0113] In this way, the diversity input control unit includes a work position information acquisition unit that acquires the position of the worker, and a control unit that switches and prioritizes the input of multiple sensors or devices according to the work area divided in advance. The divided areas are an object-separated area away from the object, an object-approaching area that contacts the object, and an object-inside area. The object-separated area has its position information acquired by preferentially using, for example, indoor GPS, the object-approaching area has its position information acquired by preferentially using, for example, image skeleton estimation, and the object-inside area has its position information acquired by preferentially using, for example, IMU. The diversity input control unit further includes a receiving unit that receives a production instruction, and an input control setting acquisition unit that acquires input control settings of multiple sensors or devices according to the production instruction.

[0114] In this way, a location information acquisition system and a location information acquisition method can be provided that are capable of ensuring robustness and accuracy of data acquisition by providing multiple sensors or devices and automatically selecting the optimal sensor or device depending on the work location.

[0115] A part or all of the processing in the above-mentioned information processing device can be realized as a computer program. Such a program can be stored and supplied to a computer using various types of non-transitory computer-readable media. The non-transitory computer-readable media includes various types of tangible recording media. Examples of the non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of temporary computer-readable media. Examples of the temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable media can supply the program to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0116] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0117] 100 Task recognition system, 101 External device / equipment, 102 Wearable device, 103 Input I / F, 104 Time series synchronization unit, 105 Data preparation for task recognition, 106 Body / finger skeleton estimation unit, 107 Tactile / auditory sensor data unit, 108 Body part position estimation unit, 109 Worker identification unit, 110 Position estimation unit, 111 Appearance image unit, 112 Equipment / tool ​​signal / data unit, 113 Diversity input control unit, 114 Learning / setting unit by preparing data for task recognition, 115 Task recognition model library, 116 Posture / behavior recognition model, 117 Gesture / voice recognition model, 118 Object recognition model, 119 Physical F / B recognition model, 120 State recognition / measurement model, 121 Equipment / tool ​​data recognition model, 122 Task position estimation model, 123 Learning / setting unit using work component recognition model, 124 General-purpose work recognition module, 125 Unit work recognition model, 126 Unit work definition unit, 127 Area-specific reference coordinate control unit, 128 Position definition unit, 129 Learning / setting unit using work position, 130 Advanced work recognition model, 131 Work recognition model library dynamic filtering, 132 Work recognition multimodal AI, 133 Unit work-specific work performance data output unit, 134 Work recognition platform, 135 BOM, 136 PDM·BOP·BOE, 137 Product-specific process information, 138 Work item, 139 Work element master, 140 Product and process drawing information, 141 MES, 142 Application, 143 Work component recognition model, 202 Work object, 203 Work verb and tool, 204 Work position, 205 BOP, 206 Work item, 207 PDM·BOP, 208 Unit work 209 unit work definition section (general purpose), 210 unit work definition section (product specific), 211 work component recognition model correlation definition section (other than work position), 212 work component recognition model correlation definition section (work position), 213 advanced work recognition, 704 device, 800 object, 801 object, 802 object, 803 object, 1201 other model, 1601 diversity input control setting section

Claims

1. A task recognition system using a plurality of task component recognition models to which information on a worker's motion, touch, hearing, position, camera, and equipment / tools during the task is input, a product selection filtering unit for selecting the work component recognition model based on a production instruction; a progress filtering unit that selects the task component recognition model according to a task progress; an activity recognition unit that recognizes an activity from the selected activity component recognition model.

2. The task recognition system according to claim 1 , wherein the product selection filtering unit narrows down the task component recognition model to tasks of one product.

3. The task recognition system according to claim 1 , wherein the progress filtering unit excludes the task component recognition model for a completed task.

4. After the product selection filtering unit selects the product, it is determined whether the operation can be recognized; If the task cannot be recognized, the product selection filtering unit is removed, the task is recognized, and the task recognition result is output, and the task is selected by the progress filtering unit. The task recognition system according to claim 1 , wherein, if the task is recognized, a task recognition result is output and selected by the progress filtering unit.

5. The task recognition system according to claim 1 , further comprising: a task recognition model library that stores a task component recognition model referenced by the product selection filtering unit.

6. A method for recognizing work using a plurality of work component recognition models to which information on a worker's motion, touch, hearing, position, camera, and equipment / tools during work is input, comprising: Selecting the work component recognition model based on a production instruction; Selecting the task component recognition model according to the task progress; and recognizing the task from the selected task component recognition model.

7. A program to be executed by an information processing device that recognizes work using a plurality of work component recognition models to which information on a worker's motion, touch, hearing, position, camera, and equipment / tools during work is input, Selecting the work component recognition model based on a production instruction; Selecting the task component recognition model according to the task progress; A program that causes an information processing device to recognize the task from the selected task component recognition model.

Images