Systems, methods, and programs
The work recognition system addresses the lack of versatility in existing systems by generating a general-purpose model and adjusting coordinates, enabling efficient and accurate recognition of unit tasks and work positions.
Patent Information
- Application Number
- JP2026207582
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2026-09-16
- Publication Date
- 2026-11-19
AI Technical Summary
Existing work recognition systems lack versatility and require modification of recognition logic and learning models for new processes and products, leading to inefficiencies in adapting to new tasks.
A work recognition system that utilizes a storage unit for multiple work component recognition models, generates a general-purpose unit work recognition model by combining product-independent components, and includes a reference setting unit to adjust coordinates based on object movement, enabling advanced work recognition and position estimation.
The system achieves universal recognition of unit tasks and accurate work position estimation, facilitating efficient and sophisticated task recognition across various products and processes.
Smart Images

Figure 2026183136000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a work recognition system, a work recognition method, and a program.
Background Art
[0002] Patent Document 1 describes a work recognition system that recognizes worker's body information and an object using a sensor and associates them with a work. The work recognition system of Patent Document 1 includes a sensor data acquisition unit that acquires sensor data, and a body part information acquisition unit that detects a body part of a worker based on the sensor data and acquires body part information related to the body part of the worker. The work recognition system further includes an object information acquisition unit that detects an object based on the sensor data and acquires object information related to the object, and an association unit that associates the object with the body part of the worker who performed the work using the object, based on the body part information and the object information. Further, the work recognition system includes a recognition result analysis unit that recognizes the work performed by the worker based on association information related to the association result associated by the association unit.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of Invention
Problems to be Solved by the Invention
[0004] There is a demand for improved accuracy in technologies for recognizing the work of workers. When dedicated recognition logic and learning models are introduced for work elements as in Patent Document 1, the versatility is poor. Therefore, in the case of new processes and new products, etc., it is necessary to modify the recognition logic and learning models, which takes time for preparation. Accordingly, an object of the present disclosure is to provide a work recognition system that recognizes universally defined unit works and leads to advanced work recognition.
Means for Solving the Problems
[0005] The work recognition system disclosed herein is A storage unit that stores multiple work component recognition models that take as input information about the worker's actions, touch, hearing, position, camera, and equipment / tools during work on an object, and decompose the work into work components for determination, A unit work recognition model generation unit generates a general-purpose unit work recognition model by combining product-independent components from the aforementioned work component recognition model based on the work content, The work recognition system comprises a unit work recognition unit that recognizes a unit work by inputting information about the worker's actions, touch, hearing, position, camera, and equipment / tools during work on an object into the aforementioned unit work recognition model.
[0006] With the above configuration, we can provide a work recognition system that recognizes generally defined unit tasks and leads to advanced work recognition.
[0007] The work recognition system disclosed herein is If the object does not move relative to the floor surface during the worker's work, the reference coordinates are set to a fixed reference. The system is characterized by including a reference setting unit that sets the reference coordinates relative to the object if the object moves relative to the floor surface during the worker's work.
[0008] The above configuration makes it possible to accurately estimate the worker's position.
[0009] The work recognition system disclosed herein is A work position estimation unit takes the output result of the unit work recognition unit and the reference coordinates of the work area as input and estimates the work position, The system is characterized by comprising an advanced work recognition unit that takes the output result of the unit work recognition unit and the estimation result of the work position estimation unit as inputs and recognizes work in an advanced manner.
[0010] With the above configuration, it is possible to recognize a general-purpose unit task and estimate the work location, thereby enabling highly sophisticated task recognition.
[0011] The method of recognizing work in this disclosure is: The system takes as input information about the worker's actions, touch, hearing, position, camera, and equipment / tools during work on an object, and stores multiple work component recognition models that decompose and determine the work into work components. Based on the work content, a general-purpose unit work recognition model is generated by combining product-independent components from the aforementioned work component recognition model. This is a work recognition method that recognizes a unit of work by inputting information about the worker's actions, touch, hearing, position, camera, and equipment / tools during work on an object into a unit work recognition model.
[0012] The above configuration provides a work recognition method that recognizes generally defined unit tasks and leads to advanced work recognition.
[0013] The program disclosed herein is The system takes as input information about the worker's actions, touch, hearing, position, camera, and equipment / tools during work on an object, and stores multiple work recognition models that decompose and determine the work into work components. Based on the work content, a general-purpose unit work recognition model is generated by combining product-independent components from the aforementioned work component recognition model. This program inputs information about the worker's actions, touch, hearing, position, camera, and equipment / tools during work on an object into the aforementioned unit work recognition model, and causes the information processing device to perform the unit work recognition.
[0014] With the above configuration, the information processing device can be made to recognize a generally defined unit task and use that recognition to perform a more advanced task. [Effects of the Invention]
[0015] This disclosure provides a work recognition system that recognizes generally defined unit tasks and connects them to the recognition of more advanced tasks. [Brief explanation of the drawing]
[0016] [Figure 1] FIG. 2 is a block diagram showing a configuration of a work recognition system according to an embodiment. [Figure 2] FIG. 3 is a correlation block diagram of constituent elements related to unit work recognition and advanced work recognition according to the embodiment. [Figure 3] FIG. 4 is a diagram showing a relationship between unit verbs and work verbs according to the embodiment. [Figure 4] FIG. 5 is a diagram showing an example of a relationship between unit work and a recognition model according to the embodiment. [Figure 5] FIG. 6 is a diagram continuing from the example of the relationship between unit work and the recognition model in FIG. 4 according to the embodiment. [Figure 6] FIG. 7 is a block diagram showing a relationship between a work component recognition model and a unit work recognition model according to the embodiment. [Figure 7] FIG. 8 is a flowchart of unit work recognition in product production according to the embodiment. [Figure 8] FIG. 9 is a diagram for explaining a reference coordinate setting unit that sets coordinates of a fixed reference and a conveyed object (target object) reference according to the embodiment. [Figure 9] FIG. 10 is a flowchart of reference coordinate setting in product production according to the embodiment. [Figure 10] FIG. 11 is a diagram showing an example in which a work position is added to unit work according to the embodiment. [Figure 11] FIG. 12 is a diagram continuing from the example in which the work position is added to the unit work of FIG. 11 according to the embodiment. [Figure 12] FIG. 13 is a block diagram showing a relationship between a unit work recognition model, a work component recognition model for advanced work recognition, and an advanced work recognition model according to the embodiment. [Figure 13] FIG. 14 is a flowchart of advanced work recognition in product production according to the embodiment. [Figure 14] FIG. 15 is a schematic diagram (upper diagram) and a flowchart (lower diagram) showing product selection filtering and progress filtering in product production according to the embodiment. [Figure 15]The diagram (top) shows an example of data acquisition coordination between sensors corresponding to the work area according to the embodiment, a schematic diagram of the work area (bottom left), and a correspondence table between the work area and sensors (bottom right). [Figure 16] This flowchart shows the process of prioritizing and switching between multiple sensors or devices according to the work position in the diversity input control according to the embodiment. [Modes for carrying out the invention]
[0017] Embodiment Embodiments of the present invention will be described below with reference to the drawings. However, the invention claimed is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential for solving the problem. For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations have been omitted where necessary.
[0018] (Description of the work recognition system according to the embodiment) Figure 1 is a block diagram showing the configuration of the work recognition system according to the embodiment. The work recognition system according to the embodiment will be described with reference to Figure 1.
[0019] As shown in Figure 1, the work recognition system 100 comprises an external device / equipment 101, a wearable device 102, and a work recognition platform 134. Furthermore, the upstream peripheral systems of the work recognition system 100 include a BOM (Bill of Materials) 135, a PDM (Product Data Management), BOP (Bill of Process), and BOE (Bill of Equipment) 136, and an MES (Manufacturing Execution System) 141. Additionally, the downstream peripheral system of the work recognition system 100 includes an APPL (Application) 142.
[0020] External devices and equipment 101 are external elements provided to the process for purposes other than those of this system. Examples of external devices and equipment 101 include cameras, indoor GPS (Global Positioning System), other equipment, jigs and fixtures, conveying systems, etc.
[0021] Wearable devices 102 are various sensors worn by workers. Examples of wearable devices 102 include first-person view cameras, microphones, vibration sensors, pressure sensors, IMUs (Inertial Measurement Units), strain sensors, vital signs sensors, etc.
[0022] The work recognition platform 134 is a part consisting of an information processing device (not shown) that forms the core of the work recognition system of this disclosure. The information processing device comprises at least a processor (e.g., a CPU (Central Processing Unit)) that executes a program to perform processing and memory for storing the program. The information processing device may consist of a cloud server that distributes some or all of its functions. The work recognition platform 134 comprises an input I / F (Interface) 103, a time-series synchronization unit 104, a work recognition data preparation unit 105, a learning / setting unit 114 based on the work recognition data preparation unit 114, a work recognition model library 115, a work recognition model library dynamic filtering unit 131, a work recognition multimodal AI 132, and a unit work performance data output unit 133.
[0023] Input I / F 103 is an interface for connecting to external devices / equipment 101 and wearable device 102.
[0024] The time-series synchronization unit 104 synchronizes the data of each external device / equipment 101 and wearable device 102 in time.
[0025] The work recognition data preparation unit 105 prepares data for a model to recognize work components related to the worker, work object, equipment, tools, etc. Regarding the worker, the work recognition data preparation unit 105 includes a body / finger skeleton estimation unit 106, a tactile / auditory sensor data unit 107, a body part position estimation unit 108, and a worker identification unit 109, etc. Regarding the work object, the work recognition data preparation unit 105 includes a position estimation unit 110 and an appearance image unit 111. Regarding equipment and tools, the work recognition data preparation unit 105 includes an equipment / jig and tool signal / data unit 112. In addition, the work recognition data preparation unit includes a diversity input control unit 113.
[0026] The body and finger skeleton estimation unit 106 uses data from sensors or devices to estimate the skeleton of the body and fingers. The body and finger skeleton estimation unit 106 also prioritizes and switches the sensors or devices used for skeleton estimation based on instructions from the diversity input control unit 113.
[0027] The tactile and auditory sensor data unit 107 acquires input data such as vibration, friction, reaction force, and sound from sensors or devices. The body part position estimation unit 108 estimates the position of each body part using the data from sensors or devices. The body part position estimation unit 108 prioritizes and switches the sensors or devices used for estimating the position of each body part based on instructions from the diversity input control unit 113. The worker identification unit 109 identifies the worker using an ID (Identification), etc.
[0028] The position estimation unit 110 estimates the position of the work object. The appearance image unit 111 captures an image of the appearance of the work object using an external device / equipment 101 and a wearable device 102 to acquire an appearance image.
[0029] The equipment / jig and tool signal / data unit 112 acquires data on equipment and tools being used during operation. The diversity input control unit 113 prioritizes and switches the sensors or devices to be input according to the work position.
[0030] The learning / setting unit 114, which prepares data for work recognition, is a learning / setting unit for constructing the work component recognition model 143, which will be explained later. The data prepared in the work recognition data preparation 105 is learned or set to construct each part of the work component recognition model 143.
[0031] The work recognition model library 115 is the reference unit of the work recognition platform 134. The work recognition model library 115 comprises a work component recognition model 143, a learning / setting unit 123 based on the work component recognition model, a general-purpose work recognition module 124, an area-specific reference coordinate control unit 127, a position definition unit 128, a learning / setting unit 129 based on the work position, and an advanced work recognition model 130.
[0032] The work component recognition model 143 is a memory unit that stores each recognition model. The work component recognition model 143 includes a posture / action recognition model 116 and a gesture / voice recognition model 117 with respect to the worker. The work component recognition model 143 includes an object recognition model 118, a physical F / B (Feedback) recognition model 119 and a state recognition / measurement model 120 with respect to the work object. The work component recognition model 143 includes an equipment / tool data recognition model 121 with respect to equipment / tools. The work component recognition model 143 includes a work position estimation model 122 with respect to the work position.
[0033] The posture / action recognition model 116 is a model that recognizes posture and actions by learning or setting from the body / finger skeleton estimation unit 106 and the tactile / auditory sensor data unit 107. The gesture / speech recognition model 117 is a model that recognizes gestures and speech by learning or setting from the body / finger skeleton estimation unit 106 and the tactile / auditory sensor data unit 107.
[0034] The object recognition model 118 is a model that recognizes objects by learning or setting from the body / finger skeleton estimation unit 106, the tactile / auditory sensor data unit 107, the body part position estimation unit 108, and the appearance image unit 111. The physical feedback recognition model 119 is a model that recognizes physical feedback by learning or setting from the tactile / auditory sensor data unit 107. The state recognition / measurement model 120 is a model that recognizes and measures the state of the work object by learning or setting from the appearance image unit 111.
[0035] The equipment / tool data recognition model 121 is a model that recognizes equipment and tool data by learning from or setting from the equipment / jig and fixture signal / data unit 112. The work position estimation model 122 is a model that estimates the work position by acquiring information from the body part position estimation unit 108, the work object position estimation unit 110, and the equipment / jig and fixture signal / data unit 112. The work position estimation model 122 is set by process or product standard. The work position estimation model 122 also acquires information from the area-specific reference coordinate control unit 127. Furthermore, the work position estimation model 122 provides data of the position estimation result to the work position learning / setting unit 129.
[0036] The learning / setting unit 123, which uses a work component recognition model, is a learning / setting unit for constructing the unit work recognition model 125, which will be explained later. The data prepared by the work component recognition model 143 is learned or set to construct the unit work recognition model 125.
[0037] The general-purpose work recognition module 124 is a unit work recognition unit that recognizes unit work by inputting information about the worker's actions, touch, hearing, position, camera, and equipment / tools during work on an object into a unit work model. In other words, the general-purpose work recognition module 124 recognizes what the worker did and how they did it. The general-purpose work recognition module 124 comprises a unit work recognition model 125 and a unit work definition unit 126.
[0038] The unit task recognition model 125 is a model for recognizing tasks that can be defined universally regardless of the product, among a variety of tasks. The unit task recognition model 125 is generated in the unit task recognition model generation unit by combining product-independent components of the task component recognition model based on the task content. For example, a general task that is not product-specific is a combination of verbs related to tasks such as tightening, fitting, and removing from work objects such as parts that are common across products and do not contain product-specific information, which will be explained later. The unit task recognition model 125 is a model that recognizes unit tasks by learning from or setting the posture / action recognition model 116, the gesture / voice recognition model 117, the object recognition model 118, the physical F / B recognition model 119, the state recognition / measurement model 120, and the equipment / tool data recognition model 121.
[0039] The unit task definition unit 126 defines the tasks for creating the unit task recognition model 125.
[0040] The area-specific reference coordinate control unit 127 controls the change of reference coordinates according to the area of the process. The area-specific reference coordinate control unit 127 provides the work position estimation model 122 with information on fixed references and product references.
[0041] The position definition unit 128 defines positions from product 3D, equipment 3D, etc., as will be explained later. The position definition unit 128 also adds position information to the unit work definition unit 126 to define the position where a unit work is performed. The position definition unit 128 provides learning or setting data to the work position learning / setting unit 129.
[0042] The work position learning / setting unit 129 learns or sets from the work position estimation model 122, the general-purpose work recognition module 124, the position definition unit 128, and the work items 138 of the product-specific process information 137, which will be described later. The work position learning / setting unit 129 learns or sets and constructs the advanced work recognition model 130.
[0043] The advanced work recognition model 130 is a model for recognizing where, what, and how a worker performed an action. The advanced work recognition model 130 recognizes tasks with high accuracy by adding product-specific information, such as the work position on the product, to general-purpose unit work recognition.
[0044] The work recognition model library dynamic filtering 131 excludes models other than the relevant product in a process library containing models for multiple vehicle types, based on production instructions from the MES 141. Furthermore, it excludes completed models based on the progress of the work from start to finish. The work recognition model library dynamic filtering 131 obtains information on who the worker is from the worker identification unit.
[0045] The work recognition multimodal AI (Artificial Intelligence) 132 is an artificial intelligence designed to recognize who, when, where, what, and how a worker performed an action. The work recognition multimodal AI 132 obtains information from the work recognition data preparation 105 and the work recognition model library dynamic filtering 131.
[0046] The unit-by-unit work performance data output unit 133 outputs work performance data for each unit of work. The unit-by-unit work performance data output unit 133 acquires information from the work recognition multimodal AI 132. The data output by the unit-by-unit work performance data output unit 133 is used in applications (APPL) 142 for safety and ergonomics improvement, quality assurance improvement, productivity improvement, etc.
[0047] BOM135 is a bill of materials that contains essential information for manufacturing a product, including the components needed and how the product is assembled.
[0048] PDM·BOP·BOE136 comprises a work element master 139, product-specific process information 137, and product / process drawing information 140.
[0049] The work element master 139 is master information that standardizes and generalizes the components of a work across products, and includes, for example, the work object, work verb, tool, work location, etc. The product-specific process information 137 includes work items 138, which are combinations of work elements defined in the work element master 139, and detailed work procedure information. The product / process drawing information 140 includes information such as 3D parts, process layout, and 3D equipment.
[0050] The work element master 139 provides information to the unit work definition unit 126. The product-specific process information 137 acquires information from the work element master 139, provides information to the product / process drawing information 140, and provides learning or setting data to the work position learning / setting unit 129. In addition, the product-specific process information 137 is transmitted across products via the work element master 139, which is defined as a standard, and the general-purpose work recognition module 124 provides information to the position definition unit 128 and the area-specific reference coordinate control unit 127.
[0051] MES141 is a manufacturing execution system that monitors and manages factory equipment and worker operations by linking with each part of the factory's production line. MES141 acquires information from BOP / BOE136. In addition, MES has a production instruction database to provide production instruction information to the work recognition model library dynamic filtering 131. Production instructions are instructions to people and equipment, including the type and specifications of individual products, as well as the parts to be used and procedures, in accordance with the production plan.
[0052] (Description of the general-purpose work recognition module according to the embodiment) Figure 2 is a correlation block diagram of the components related to unit work recognition and advanced work recognition according to the embodiment. Figure 3 is a diagram showing the relationship between work verbs and unit verbs according to the embodiment. Figure 4 is a diagram showing an example of the relationship between unit work and recognition model according to the embodiment. Figure 5 is a continuation of the example of the relationship between unit work and recognition model in Figure 4 according to the embodiment. Figure 6 is a block diagram showing the relationship between the work component recognition model and the unit work recognition model according to the embodiment. Figure 7 is a flowchart of unit work recognition in product production according to the embodiment. Unit work recognition according to the embodiment will be explained with reference to Figures 2 to 7.
[0053] As shown in Figure 2, the general-purpose work recognition module 124 recognizes work items that have been broken down into general-purpose work verbs and work elements such as work objects. Furthermore, the general-purpose work recognition module 124 defines, learns, or sets correlations with the work component recognition model 143 using unit work, which is defined by unit verbs that have been broken down into unit verbs at a level suitable for work recognition, as a master. Therefore, the general-purpose work recognition module 124 constitutes a general-purpose work recognition module that is independent of individual products and processes. By sharing the general-purpose work recognition module 124 across different products, maintenance and operation can be carried out efficiently, including responding to changes in products and processes.
[0054] The work element master 139 includes information on the work object 202 and the work verb / tool 203 as general-purpose work elements. The work element master 139 also includes information on the work location 204 as a product-specific work element. This information on the work object 202, the work verb / tool 203, and the work location 204 is obtained from work item 206 of the BOP 205.
[0055] Work item 206 defines a series of tasks for producing a product, using combinations of work objects and work verbs. Work elements are the components that make up work item 206. The main work elements are work verbs, work objects, tools, and work locations. Of these, work verbs, work objects, and tools are defined as general-purpose master items, enabling data aggregation and analysis across products and factories. Since the work location may differ depending on the product even for the same part, while classification codes for work locations can be standardized, the work location itself is treated as a product-specific work element.
[0056] The general-purpose work recognition module 124 comprises a unit work definition unit (general purpose) 209, a work component recognition model correlation definition unit (other than work position) 211, and a general-purpose unit work recognition model 125.
[0057] Furthermore, the unit work 208 is a further subdivision of the content of general work elements such as work verbs, work objects, and tools. Specifically, work verbs are broken down into units suitable for work recognition, and the work is divided into these units to form a unit work. The unit work 208 comprises a unit work definition unit (general purpose) 209 and a unit work definition unit (product specific) 210.
[0058] The unit work definition unit (general purpose) 209 includes information on steps, which are the work sequence, unit verbs defined in a hierarchical structure according to granularity, recognition categories, and recognition items. The unit work definition unit (general purpose) 209 is associated with the work object 202 and the work verbs / tools 203. Unit verbs are verbs with a granularity sufficient for work recognition. Work verbs in work items 206 can be arbitrarily defined at various granularities, but the work recognition in this disclosure aims to recognize various tasks performed on the manufacturing site in a concrete and detailed manner. Therefore, coarse-grained work verbs need to be subdivided to a granularity sufficient for work recognition. If the difference in granularity between work verbs and unit verbs is large, the subdivision may be carried out in multiple hierarchical levels.
[0059] The recognition categories consist of two types, motion and event, defined according to the characteristics of the work and the purpose of recognition, for recognizing unit tasks. Therefore, the general-purpose work recognition module 124 can recognize a wide variety of tasks with precision. Motion recognizes tasks as a series of actions performed sequentially over time. Therefore, it recognizes continuous movements such as walking. Events recognize specific sections of a task, such as the start and completion of tightening. Therefore, it can recognize precise work segmentation timing.
[0060] Recognition items are used to define specific recognition targets, such as start and end, for the recognition categories of unit verbs.
[0061] The work component recognition model correlation definition unit (excluding work position) 211 assigns a work component recognition model (excluding the work position recognition model, which is a product-specific work component) suitable for recognition according to the work characteristics to work elements (work verb × work object) × unit work (unit verb, recognition category, item) and defines the correlation. The work component recognition model correlation definition unit (excluding work position) 211 includes information on necessity, pattern, variation, and reliability.
[0062] The degree of necessity is defined for each work component recognition model 143, enabling efficient allocation of unit tasks 208 to work component recognition models 143, while maintaining and controlling robustness in case of missing outputs from some models. Patterns and variations are the patterns and detailed variations that are recognized by the work component recognition model 143. The confidence level is defined for each work component recognition model 143, allowing recognition accuracy evaluation to be performed within the system. The confidence level is set arbitrarily or based on actual recognition result data. This evaluation is used, for example, to rank multiple candidates when multiple candidates are generated during work recognition with the general-purpose work recognition module 124.
[0063] The general-purpose work recognition module 124 is shared and reused across products if there are common or similar parts between them.
[0064] As shown in Figure 3, the work object n and work verb n are obtained from work item 206 of BOP205. Work verb n is divided into unit verb 1, unit verb 2, ... unit verb n. Unit verb 1 is further divided into unit verb 1.2, unit verb 1.3, and unit verb 1.4. Note that Figure 3 shows the relationship between work verbs and unit verbs as an example, and a third level or higher may be added to further subdivide unit verbs.
[0065] Specifically, as shown in the example in Figure 4, the extraction of unit verb 1 is broken down into empty transport of unit verb 1.1, grasping of unit verb 1.2, and transport of unit verb 1.3.
[0066] As shown in Figure 6, the work component recognition model correlation definition unit (excluding work position) 211 is reflected in the unit work recognition model 125 through the learning / setting unit 123 by the work component recognition model learned or set in the work component recognition model 143. Figure 5 is a continuation of the vertical columns in Figure 4, but for example, as shown in Figure 5, the unit verb 1.2 "grasp" is associated with the object recognition model 118 and the physical F / B recognition model 119. In addition, the unit verb 1.2 "grasp" is associated with the posture / action recognition model 116. Furthermore, the unit verb 1.2 "grasp" is associated with the equipment / tool data recognition model 121. In this way, the work component recognition model correlation definition unit (excluding work position) 211 associates unit verbs with the work component recognition model 143.
[0067] The method for recognizing unit tasks in product production is shown in Figure 7. First, the information processing device receives a production instruction from the MES 141 701. The information processing device obtains the product production sequence upon receiving the production instruction. Next, the information processing device obtains product selection filtering from the task recognition model library 115 702. Product selection filtering will be explained later. The information processing device obtains the task recognition model required for the product from the library through product selection filtering, constructs a set of models necessary for executing task recognition for the product, and process N begins 705. Then, task item N of process N begins 706. Next, the information processing device performs data input and time-series synchronization and processing from devices and equipment 704 as needed during the execution of the task 703, and then performs task component recognition 707. The general-purpose task recognition unit of the task recognition multimodal AI 132 continuously acquires the task component recognition results 708 and passes them on to unit task recognition (general-purpose) 709. The unit work recognition (general purpose) recognizes a general-purpose unit work by matching it with the unit work recognition model 125, 709 and outputting the recognition result 710. If multiple unit works are recognized during unit work recognition, multiple candidates ranked in order of confidence are output as described above.
[0068] The result output is passed to the advanced work recognition model 130. Furthermore, processes 703-710 are performed continuously until the completion of process N.
[0069] This provides a general-purpose work recognition module that can recognize unit tasks independently of the product.
[0070] (Description of the area-specific reference coordinate control unit according to the embodiment) Figure 8 is a diagram illustrating the reference coordinate setting unit for setting the coordinates of a fixed reference and a transported object (target object) reference according to the embodiment. Figure 9 is a flowchart of the reference coordinate setting in product production according to the embodiment. The area-specific reference coordinate control unit 127 according to the embodiment will be explained with reference to Figures 8 and 9.
[0071] As shown in Figure 8, in a line-type production site, the workpiece moves along a line within the factory during production. For example, during work such as parts processing and assembly on a fixed workbench, the position of the workpiece does not move relative to the floor. In contrast, there are cases where the workpiece is transported by a transport system, such as when assembling parts onto the body of a vehicle transported by a conveyor. For each case, two types of reference coordinates are established for the work position estimation model: a fixed reference and a transported object (workpiece) reference. By dynamically switching these two reference coordinates according to the worker's work area, the work position coordinates necessary for work recognition can be accurately grasped.
[0072] As shown in the upper diagram of Figure 8, the area-specific reference coordinate control unit 127, assuming that objects 800, 801, 802, and 803 are flowing on the transport system, designates the area where work is performed on the transported objects 800, etc., as the transported object reference application area, and the other areas as the fixed reference application area. Regarding the switching of reference coordinates, the aforementioned areas are divided in advance on the line layout, and the reference coordinates to be applied to each area are defined, so that the defined reference coordinates are applied sequentially according to the worker's position.
[0073] As shown in the middle diagram of Figure 8, a fixed reference is a coordinate system based on fixed coordinates relative to the floor surface of building columns, workbenches, etc. Fixed references are set in areas where the work object does not move relative to the floor surface during the worker's work. On the other hand, as shown in the lower diagram of Figure 8, a transported object reference is a system based on the coordinates of the transported object when the object 800, etc., is moving along a transport system. If the work object moves relative to the floor surface, the transported object reference is set.
[0074] In cases where the transported object standard is applied, the position information of the work object on the transport system is acquired in real time by connecting it to the transport system, and the work position is estimated based on the transported object standard (more precisely, the transported work object standard).
[0075] The setting of fixed references and object references is performed by the area-specific reference coordinate control unit 127, which is the reference setting unit, as shown in Figure 9. First, the information processing device 703 inputs data and performs time-series synchronization and processing from external devices / equipment 101 and wearable device 102. The information processing device inputs the data into the work position estimation model 122. The information processing device then recognizes the worker's location area 901. Based on the area-specific reference coordinate setting 902, which was set in advance by referring to the process layout / equipment 3D, the area-specific reference coordinate control unit 127 executes reference coordinate control 903 when the worker's location area, acquired from the worker location area recognition 901, moves to a different area. Next, 904 outputs a reference coordinate switching instruction to the work position estimation model 122. In this way, the area-specific reference coordinates are controlled according to the area where the worker is located.
[0076] (Explanation of advanced work recognition according to the embodiment) The advanced work recognition according to the embodiment will be explained with reference to Figures 2 and 10 to 13. Figure 10 is a diagram showing an example in which a work position is added to a unit work according to the embodiment. Figure 11 is a continuation of the example in which a work position is added to the unit work in Figure 11 according to the embodiment. Figure 12 is a block diagram showing the relationship between the unit work recognition model, the work component recognition model for advanced work recognition, and the advanced work recognition model according to the embodiment. Figure 13 is a flowchart of advanced work recognition in the production of a product according to the embodiment.
[0077] As shown in Figure 2, the advanced work recognition 213 performs high-definition and high-precision work recognition by combining the output obtained from the general-purpose work recognition module 124 with work position information specific to the product and process. In some cases, the advanced work recognition 213 may further supplement with individual recognition models. Furthermore, by operating the advanced work recognition 213 in combination with the general-purpose work recognition module 124, it is possible to efficiently configure an advanced work recognition model that responds to differences in products and processes while maximizing versatility.
[0078] As shown in Figure 2, the advanced work recognition 213 comprises a unit work definition unit (product-specific) 210, a work component recognition model correlation definition unit (work position) 212, and an advanced work recognition model 130.
[0079] The unit work definition unit (product-specific) 210 includes work position details such as reference coordinates, target part, position type, and range specification method. The unit work definition unit (product-specific) 210 is included in the unit work 208 and obtains a work position code that is set as a standard common to all products from the product-specific work position 204 of the work element master 139. In addition, the unit work definition unit (product-specific) 210 obtains 3D coordinate information and process layout information of the product as detailed work position from the PDM / BOP 207.
[0080] The reference coordinates can be selected from either the fixed reference or the transported object reference as defined in Figure 8.
[0081] The target body part refers to the part of the worker's body from which the work position is to be acquired. Examples of target body parts include the head, arms, and legs.
[0082] The location type is the method of specifying a location. Examples of location types include a two-dimensional plane and a three-dimensional space.
[0083] The range specification method is a method for specifying the range of the work area. For example, the range can be specified by specifying the position range using two diagonal points of a three-dimensional object or a two-dimensional figure, or by specifying the position range along a path.
[0084] The work component recognition model correlation definition unit (work position) 212 assigns a work component recognition model (a work position recognition model that is a work component specific to the product) suitable for recognition to the work position and defines the correlation. The work component recognition model correlation definition unit (work position) 212 includes information on necessity, reference coordinates, target part, position type, and confidence level.
[0085] The degree of necessity is defined for each work component recognition model 143, enabling efficient assignment of work locations to work component recognition models 143, while maintaining and controlling robustness in the event of missing outputs from some models.
[0086] The reference coordinates, target area, and position type are determined using information obtained from the unit work definition unit (product-specific) 210.
[0087] A confidence level is defined for each work component recognition model 143, allowing for recognition accuracy evaluation within the system. The confidence level is set arbitrarily or based on actual recognition result data. This evaluation is used, for example, to rank multiple candidates when multiple candidates are generated during work recognition using advanced work recognition 213.
[0088] The advanced work recognition model 130 enables detailed and highly accurate work recognition tailored to the product by combining the general-purpose work recognition module 124 with the work component recognition work position estimation model 122 (additional models may be added as needed).
[0089] As shown in Figure 12, the work component recognition model correlation definition unit (work position) 212 associates the unit work recognition model 125 of the general-purpose work recognition module 124, the work position estimation model 122 of the work component recognition model 143, and other models 1201, and the work position learning / setting unit 129 performs learning or setting.
[0090] As shown in Figure 13, advanced task recognition is performed. Figure 13 includes the advanced task recognition in addition to the content explained in Figure 7. Therefore, the content explained in Figure 7 is omitted.
[0091] The information processing device outputs the unit work recognition (general purpose) result 710, and the advanced work recognition unit of the work recognition multimodal AI 132 acquires the unit work recognition result (general purpose) 1301. In parallel, the area-specific reference coordinate control unit 127 acquires the worker's location information from the work component recognition 707 and transmits the reference coordinate information to the work position estimation model 122. The work position estimation model 122 acquires the worker's location information from the work component recognition 707 and also acquires the reference coordinate information from the area-specific reference coordinate control unit 127 to estimate the work position coordinates, and outputs the result to the advanced work recognition unit of the work recognition multimodal AI 132. The advanced work recognition unit acquires the work position coordinate estimation result 1306 and uses the advanced work recognition model 130 to perform a matching evaluation between the unit work and the actual work position, narrows down the candidate unit work candidates 1307, and outputs a candidate ranking 1308.
[0092] Next, the advanced work recognition unit of the work recognition multimodal AI 132 recognizes (estimates) the unit work with the highest accuracy from the unit work recognition candidates and candidate ranking obtained from the result output 710 of unit work recognition (general purpose) and the result output 1308 of unit work recognition (product specific), 1302. In parallel, the advanced work recognition unit performs work element recognition for work elements, which are a collection of multiple unit works, by monitoring the individual unit work recognition data in a time series, 1303 (work object = part X, work verb = tightening, tool = tool Y, work position code = Z, work status: in progress, completed, etc.). Next, the advanced work recognition unit performs work item recognition for work items, which are combinations of work elements, in conjunction with the output of the work element recognition results, 1304. Then, the advanced work recognition unit outputs the aforementioned recognition results (unit work recognition 1302, work element recognition 1303, work item recognition 1304) as needed, 1305. The recognition results are sent to databases, data analysis systems, and application tools. Furthermore, the advanced work recognition unit processes continuously until the completion of process N.
[0093] If the work element recognition 1303 fails to recognize an element, it returns to the unit work recognition 1302. The work element recognition 1303 is recognized based on multiple results from the unit work recognition 1302.
[0094] Thus, the advanced work recognition system comprises an advanced work recognition unit that recognizes work at a high level, taking as input a unit work recognition (general purpose) candidate, which is the output result of the general-purpose work recognition unit of the work recognition multimodal AI132, and a unit work recognition (product-specific) candidate related to the work position, which is the estimation result of the work position estimation model. In this way, a work recognition system and a work recognition method are provided.
[0095] (Description of the work recognition model filtering according to the embodiment) Figure 14 shows a schematic diagram (top) and a flowchart (bottom) illustrating product selection filtering and progress filtering in the production of a product according to the embodiment. The work recognition model filtering according to the embodiment will be explained with reference to Figure 14.
[0096] As shown in the upper diagram of Figure 14, for companies that mass-produce a variety of products, the model library for all products is enormous, for example, 15 products x 5 factories. Therefore, when operating these vast number of work recognition models on a mixed-model production line, problems may arise in terms of processing speed and the capabilities of the information processing equipment. To address this, when moving from the production preparation stage to the mass production stage, the relevant models are narrowed down by factory, so the factory library becomes, for example, 3 products x 1 factory in Figure 14. Furthermore, by narrowing down the relevant models by process, the process library becomes, for example, 3 products x 1 process in Figure 14. If the three products are Product A, Product B, and Product C, then in the mass production of a product, for example, if a production instruction for Product B is received, the model can be narrowed down to the model corresponding to Product B, thus narrowing it down to 1 product x 1 process. In this way, in a process library that includes models for multiple vehicle types, excluding models other than the relevant product according to the production instruction is called product selection filtering.
[0097] Furthermore, progress filtering is used to exclude completed models according to the work progress from the start to the completion of the work.
[0098] As shown in the lower part of Figure 14, the information processing device receives production instructions from the MES 141 and acquires the product production sequence 701. Next, the information processing device performs product selection filtering 702 to select the execution model set for process N from the work recognition model library 115. Work for process N is started 705. Subsequently, work recognition is performed for work items 1, 2...N 1401. The information processing device performs work recognition in parallel 1402. If the work is recognized (Yes in step 1402), the information processing device outputs the work recognition result. If the work is not recognized (No in step 1402), the information processing device cancels, i.e., removes, the product selection filtering 1404 and performs work recognition again 1405. The reason for canceling the product selection filtering is that it is considered that an unusual task, such as a duplicate task or an unspecified task, has been performed. The cancellation range can be arbitrarily set in advance. (Examples: process in question × product in question, process in question × all products, factory in question × product × product in question, etc.) If a rework can be recognized, the information processing device outputs the work recognition result 1403. Next, the information processing device uses progress filtering to exclude and select models for completed work from the execution model set 1406. This updates the execution model. Processes 1402 to 1406 are performed continuously until the work in process N is completed.
[0099] Thus, the work recognition system comprises a product selection filtering unit that selects work recognition models based on production instructions, a progress filtering unit that selects work recognition models according to the progress of the work, and a work recognition unit that recognizes work from the selected work recognition models. Furthermore, the work recognition system narrows down the work recognition models to the work of one product using the product selection filtering unit. The work recognition system also excludes work recognition models for completed work using the progress filtering unit. In addition, the work recognition system determines whether or not the work can be recognized, and if the work cannot be recognized, it removes the product selection filtering and tries to recognize the work again and outputs the work recognition result.
[0100] In this way, a work recognition system can be provided that selects learning models according to the progress of the work, reduces the processing time required for judgments made by unnecessary learning models, and enables high-speed, advanced work recognition.
[0101] (Description of the diversity input control unit according to the embodiment) Figure 15 shows an example of data acquisition coordination between sensors corresponding to the work area according to the embodiment (top figure), a schematic diagram of the work area (bottom left figure), and a correspondence table between the work area and sensors (bottom right figure). Figure 16 is a flowchart showing the process of prioritizing and switching multiple sensors or devices according to the work position in the diversity input control according to the embodiment. The diversity input control unit according to the embodiment will be described with reference to Figures 15 and 16.
[0102] When acquiring work performance data using multiple sensors or devices such as cameras, GPS, and IMUs, depending on their position relative to the process or product, these may become obstacles, making some devices unusable (camera: blind spots, etc.) or causing them to malfunction (GPS: radio wave reflection / shielding, etc.). Therefore, the diversity input control unit 113 ensures robustness and accuracy of data acquisition by automatically selecting the optimal device according to the work position in a system equipped with multiple sensors or devices.
[0103] As shown in the lower left diagram of Figure 15, the area surrounding the object is divided into, for example, an object separation area (Zone A), which is the area away from the object and unaffected by obstacles such as equipment; an object approach area (Zone B), which is the area that is close to the object and in contact with it; and an object interior area (Zone C), which is the area inside the object.
[0104] Taking the acquisition of work position as an example, as shown in the lower right of Figure 15, in Zone A, the work position can be accurately detected by indoor GPS. Position estimation is also possible using IMU-based position estimation. However, since indoor GPS is more accurate, indoor GPS is prioritized in Zone A, and the IMU is used as a backup. When imaging an object, Zone A is outside the imaging range and therefore cannot be used.
[0105] Similarly, in Zone B, since position estimation based on image skeletal estimation or image object recognition can be obtained from images of the target object, position estimation based on image skeletal estimation is the most advantageous in terms of accuracy (however, depending on the camera's depression angle and distance from the camera, other devices may be more accurate). Indoor GPS accuracy decreases in areas with many radio wave obstructions and reflectors, so in Zone B, the IMU is used as a second backup and indoor GPS as a third backup.
[0106] In Zone C, IMU-based position estimation takes precedence. Within an object, if the object blocks radio waves, indoor GPS cannot be used, or its accuracy deteriorates drastically. Additionally, "position estimation by image" or "position estimation by image skeletal estimation or image object recognition," which involves imaging the object, can be used in areas that are not blind spots, and are therefore used as a backup.
[0107] When a worker performs work along a work path as shown in the upper diagram of Figure 15, at work location 1, indoor GPS is used for position estimation, and at work locations 2 and 3, position estimation is performed based on image skeletal estimation, inheriting the position data. Similarly, at work locations 4 and 5, IMU-based position estimation is performed, inheriting the position data, and at work location 6, position estimation is performed based on image skeletal estimation, inheriting the position data.
[0108] As shown in Figure 16, first the information processing device receives a production instruction from the MES 141 701. Next, the information processing device acquires diversity input control settings from the diversity input control setting unit 1601 1602. Simultaneously, the work of process N is started 705. At the start, if the work start position is predetermined for each process, the position is reset 1603. The information processing device acquires information about the position reset. Next, the information processing device continuously acquires work position estimation results for work items 1, 2...n using the work position estimation model 122 1604. While the work of process N is in progress, the information processing device promptly instructs the selection of an input device according to the work position estimation results 1605.
[0109] Thus, the diversity input control unit comprises a work position information acquisition unit that acquires the worker's position, and a control unit that switches and prioritizes inputs from multiple sensors or devices according to pre-divided work areas. The divided areas are an object separation area, which is far from the object, an object proximity area, which is in contact with the object, and an object interior area. Furthermore, in the object separation area, for example, location information is acquired using an indoor GPS preferentially; in the object proximity area, for example, location information is acquired using image skeletal estimation preferentially; and in the object interior area, for example, location information is acquired using an IMU preferentially. In addition, the diversity input control unit comprises a receiving unit that receives production instructions, and an input control setting acquisition unit that acquires input control settings for multiple sensors or devices according to the production instructions.
[0110] In this way, a location information acquisition system and method can be provided that ensure robustness and accuracy of data acquisition by providing multiple sensors or devices and automatically selecting the optimal sensor or device according to the work location.
[0111] Some or all of the processing in the information processing device described above can be implemented as a computer program. Such programs can be stored and supplied to a computer using various types of non-temporary computer-readable media. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.
[0112] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. [Explanation of symbols]
[0113] 100 Work Recognition System, 101 External Devices / Equipment, 102 Wearable Devices, 103 Input I / F, 104 Time Series Synchronization Unit, 105 Work Recognition Data Preparation, 106 Body / Finger Skeleton Estimation Unit, 107 Tactile / Auditory Sensor Data Unit, 108 Body Part Position Estimation Unit, 109 Worker Identification Unit, 110 Position Estimation Unit, 111 Appearance Image Unit, 112 Equipment / Jig / Tool Signal / Data Unit, 113 Diversity Input Control Unit, 114 Learning / Setting Unit by Work Recognition Data Preparation, 115 Work Recognition Model Library, 116 Posture / Action Recognition Model, 117 Gesture / Voice Recognition Model, 118 Object Recognition Model, 119 Physical F / B Recognition Model, 120 State Recognition / Measurement Model, 121 Equipment / Tool Data Recognition Model, 122 Work Position Estimation Model, 123 124 Learning / Setting Unit by Work Component Recognition Model, 125 General-Purpose Work Recognition Module, 126 Unit Work Definition Unit, 127 Area-Specific Reference Coordinate Control Unit, 128 Position Definition Unit, 129 Learning / Setting Unit by Work Position, 130 Advanced Work Recognition Model, 131 Work Recognition Model Library Dynamic Filtering, 132 Work Recognition Multimodal AI, 133 Work Performance Data Output Unit for Each Unit Work, 134 Work Recognition Platform, 135 BOM, 136 PDM / BOP / BOE, 137 Process Information by Product, 138 Work Items, 139 Work Element Master, 140 Product / Process Drawing Information, 141 MES, 142 Application, 143 Work Component Recognition Model, 202 Work Object, 203 Work Verbs / Tools, 204 Work Position, 205 BOP, 206 Work Items, 207 PDM / BOP, 208 Unit Work 209 Unit work definition unit (general purpose), 210 Unit work definition unit (product specific), 211 Work component recognition model correlation definition unit (other than work position), 212 Work component recognition model correlation definition unit (work position), 213 Advanced work recognition, 704 Device, 800 Object, 801 Object, 802 Object, 803 Object, 1201 Other models, 1601 Diversity input control setting unit
Claims
[Claim 1] A storage unit that stores multiple work component recognition models that take as input information about the worker's actions, touch, hearing, position, camera, and equipment / tools during work on an object, and decompose the work into work components for evaluation, A unit work recognition model generation unit generates a general-purpose unit work recognition model by combining product-independent components from the aforementioned work component recognition model based on the work content, A work recognition system comprising a unit work recognition unit that recognizes a unit work by inputting information on the worker's actions, touch, hearing, position, camera, and equipment / tools during work on an object into the unit work recognition model.
Citation Information
Patent Citations
Work recognition device and work recognition method
WO2018087844A1
