Job recognition system, job recognition method, and program

Abstract

Provided is a work recognition system provided with: a storage unit that stores a plurality of work component recognition models for decomposing a work into work components and determining the work components, using as input information about the motion, the tactile sensation, the hearing, the position, the camera, and the equipment / tool when a worker performs a work on an object; a unit job recognition model generation unit that generates a common unit job recognition model by combining components that do not depend on a product in the job component recognition model on the basis of the job content; and a unit work recognition unit that recognizes the unit work by inputting, to the unit work model, information on the operation, the tactile sense, the hearing sense, the position, the camera, and the equipment / tool when the worker performs work on the object.

Description

Technical Field

[0001] The present invention relates to a job identification system, a job identification method and a program. Background Art

[0002] WO2018 / 087844A1 describes a job identification system that uses sensors to identify the operator's physical information and objects and associates them with the job.

[0003] The operation recognition system of WO2018 / 087844A1 includes: a sensor data acquisition unit that acquires sensor data; and a body part information acquisition unit that detects the body part of the operator based on the sensor data and acquires body part information related to the body part of the operator. In addition, the operation recognition system includes: an object information acquisition unit that detects the object based on the sensor data and acquires object information related to the object; and an association unit that associates the object with the body part of the operator who performed the operation using the object based on the body part information and the object information. In addition, the operation recognition system includes a recognition result analysis unit that recognizes the operation performed by the operator based on the association information related to the association result associated by the association unit. Summary of the invention

[0004] It is desirable to improve the accuracy of the technology for identifying the operation of an operator. When a dedicated recognition logic and learning model are introduced for the elements of the operation as in Patent Document 1, there is a lack of versatility. Therefore, in the case of new processes and new products, the recognition logic and learning model have to be changed, so preparation takes time. Therefore, the purpose of the present disclosure is to provide a work recognition system that recognizes a unit operation defined in general, which helps advanced work recognition.

[0005] The operation identification system disclosed in the present invention comprises: A storage unit storing a plurality of work component recognition models, wherein the plurality of work component recognition models take as input information of a worker's action, touch, hearing, position, camera, and equipment / tool ​​when working on an object, decompose the work into work components, and make judgments; a unit work identification model generating unit for generating a general unit work identification model by combining components that are not dependent on products in the work component identification model based on the work content; and The unit operation recognition unit inputs information on the action, touch, hearing, position, camera, and equipment / tool ​​of the operator when operating the object into the unit operation recognition model to recognize the unit operation.

[0006] According to the above configuration, it is possible to provide a work recognition system that recognizes commonly defined unit work and facilitates advanced work recognition.

[0007] The task identification system disclosed in the present invention is characterized in that: Equipped with a benchmark setting unit, When the object does not move relative to the ground during the operation of the operator, the reference setting unit sets the reference coordinates as a fixed reference. The reference setting unit sets the reference coordinates as an object reference when the object moves relative to the ground during the work of the operator.

[0008] According to the above configuration, the working position of the worker can be accurately estimated.

[0009] The task identification system disclosed in the present invention is characterized by comprising: a work position estimating unit that estimates a work position by taking the output result of the unit work identifying unit and the reference coordinates of the work area as input; and A high-level work recognition unit receives the output result of the unit work recognition unit and the estimation result of the work position estimation unit as input, and recognizes the work at a high level.

[0010] According to the above configuration, by recognizing a unit work defined in general and estimating the work position, it is possible to recognize the work at a high level.

[0011] In the operation identification method disclosed in the present invention, storing a plurality of work component recognition models, wherein the plurality of work component recognition models take as input the information of the action, touch, hearing, position, camera, and equipment / tool ​​when the operator performs work on the object, decompose the work into work components and make judgments, Based on the job content, the components in the job component identification model that are not dependent on the product are combined to generate a general unit job identification model, The unit operation recognition model is input with information on the operator's actions, touch, hearing, position, camera, and equipment / tools when performing operations on an object, and the unit operation is recognized.

[0012] According to the above configuration, it is possible to provide a work recognition method that recognizes a unit work that is generally defined and facilitates advanced work recognition.

[0013] The program of the present disclosure causes the information processing device to execute: storing a plurality of work component recognition models, wherein the plurality of work component recognition models take as input the information of the action, touch, hearing, position, camera, and equipment / tool ​​when the operator performs work on the object, decompose the work into work components and make judgments, Based on the job content, the components in the job component identification model that are not dependent on the product are combined to generate a general unit job identification model, The unit operation recognition model is input with information on the actions, touch, hearing, position, camera, and equipment / tools of the operator when performing operations on the object, so as to recognize the unit operation.

[0014] According to the above configuration, it is possible to cause the information processing apparatus to recognize a unit job that is commonly defined, thereby facilitating high-level job recognition.

[0015] According to the present disclosure, it is possible to provide a work recognition system that recognizes a unit work that is commonly defined and facilitates advanced work recognition. The above and other objects, features and advantages of the present disclosure will be more fully understood from the detailed description given below and the accompanying drawings which are given by way of example only and thus should not be regarded as limiting the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a block diagram showing the configuration of a task recognition system according to an embodiment. Figure 2 This is a block diagram showing components related to unit task identification and high-level task identification according to an embodiment of the present invention. Figure 3 It is a diagram showing the relationship between unit verbs and work verbs according to the embodiment. Figure 4 It is a diagram showing an example of the relationship between the unit task and the recognition model according to the embodiment. Figure 5 This is a diagram showing the embodiments involved. Figure 4 Continuation diagram of an example of the relationship between unit operations and the recognition model. Figure 6 It is a block diagram showing the relationship between the work component identification model and the unit work identification model involved in the embodiment. Figure 7 This is a flowchart of unit operation identification in the production of a product according to an embodiment. Figure 8 It is a diagram for explaining a reference coordinate setting unit for setting coordinates based on a fixed reference and a conveyance object (object) reference according to the embodiment. Fig. 9 This is a flowchart of setting reference coordinates in production of a product according to an embodiment. Fig.10 This is a diagram showing an example of adding a work position to a unit work according to the embodiment. Fig.11 This is a diagram showing the embodiments involved. Fig.10 A continuation diagram of an example of adding a job position to a unit job. Fig.12It is a block diagram showing the relationship between the unit task identification model involved in the embodiment, the task component identification model for high-level task identification, and the high-level task identification model. Fig.13 It is a flowchart of high-level task identification in the production of products involved in the implementation mode. Fig.14 The following are a schematic diagram (upper diagram) and a flow chart (lower diagram) showing product selection filtering and progress filtering in the production of products involved in the embodiment. Fig.15 A diagram showing an example of collaboration in data acquisition of sensors corresponding to a work area according to an embodiment (upper diagram), a schematic diagram of the work area (lower left diagram), and a correspondence table between the work area and sensors (lower right diagram). Fig.16 This is a flowchart showing a process of assigning and switching priorities to a plurality of sensors or devices according to work positions in the diversity input control according to the embodiment. DETAILED DESCRIPTION

[0017] Implementation Below, the embodiments of the present invention are described with reference to the accompanying drawings. However, the invention involved in the claims is not limited to the following embodiments. In addition, as a means for solving the technical problem, all the structures described in the embodiments are not necessarily required. In order to clearly explain, the following description and drawings are appropriately omitted and simplified. In each of the drawings, the same elements are given the same figure numbers, and repeated descriptions are omitted as needed.

[0018] (Description of the Work Recognition System According to the Embodiment) Figure 1 2 is a block diagram showing the configuration of a work recognition system according to an embodiment. Figure 1 , the job identification system involved in the implementation method is explained.

[0019] like Figure 1As shown, the operation identification system 100 includes: an external device / equipment 101, a wearable device 102, and an operation identification platform 134. In addition, as a peripheral system on the upstream side of the operation identification system 100, it includes BOM (Bill Of Materials) 135, PDM (Product Data Management) / BOP (Bill Of Process) / BOE (Bill Of Equipment) 136, and MES (Manufacturing Execution System) 141. In addition, as a peripheral system on the downstream side of the operation identification system 100, it includes APPL (Application) 142.

[0020] The external device / equipment 101 is an external element provided in the process for purposes other than the system itself, and is, for example, a camera, an indoor GPS (Global Positioning System), other equipment, jigs and tools, a conveyor system, and the like.

[0021] The wearable device 102 is various sensors worn by the operator, such as a first-person viewpoint camera, a microphone, a vibration sensor, a pressure sensor, an IMU (Inertial Measurement Unit), a strain sensor, a life sensor, and the like.

[0022] The job identification platform 134 is a part composed of an information processing device (not shown) that is the core of the job identification system of the present disclosure. The information processing device has at least a processor (for example, a CPU (Central Processing Unit)) that executes a program in order to perform processing and a memory that stores the program. The information processing device can also be composed of a cloud server that distributes part or all of the functions. The job identification platform 134 includes: an input I / F (Interface) 103, a time series synchronization unit 104, a job identification data preparation 105, a learning / setting unit based on the job identification data preparation 114, a job identification model library 115, a job identification model library dynamic filtering 131, a job identification multimodal AI 132, and a job performance data output unit 133 for each unit job.

[0023] The input I / F 103 is an interface for connecting to the external device / equipment 101 and the wearable device 102 .

[0024] The time series synchronization unit 104 synchronizes the data of each external device / equipment 101 and the wearable device 102 in terms of time.

[0025] The data preparation 105 for operation identification prepares data for models for identifying operation components related to operators, operation objects, equipment / tools, etc. The data preparation 105 for operation identification includes: a body / finger skeleton estimation unit 106, a tactile / auditory sensor data unit 107, a body part position estimation unit 108, an operator identification unit 109, etc. Regarding operators, the data preparation 105 for operation identification includes: a position estimation unit 110 and an appearance image unit 111. The data preparation 105 for operation identification includes: an equipment / jig signal / data unit 112 regarding equipment / tools. In addition, the data preparation for operation identification includes a diversity input control unit 113.

[0026] The body / finger skeleton estimation unit 106 estimates the skeleton of the body and fingers using data from sensors or devices. In addition, the body / finger skeleton estimation unit 106 assigns and switches the priority of sensors or devices used for skeleton estimation according to instructions from the diversity input control unit 113.

[0027] The tactile / auditory sensor data unit 107 obtains input data such as vibration, friction, reaction force, sound, etc. from sensors or devices. The body part position estimation unit 108 estimates the position of each body part using the data from the sensors or devices. The body part position estimation unit 108 performs priority allocation and switching of sensors or devices used for position estimation of each body part according to instructions from the diversity input control unit 113. The operator identification unit 109 identifies the operator using ID (Identification) or the like.

[0028] The position estimation unit 110 estimates the position of the work object. The appearance image unit 111 acquires an appearance image by capturing the appearance of the work object using the external device / equipment 101 and the wearable device 102.

[0029] The equipment / jig signal / data unit 112 acquires data of equipment and tools during operation. The diversity input control unit 113 allocates and switches the priority of sensors or devices to be input according to the operation position.

[0030] The learning / setting unit 114 based on the work identification data preparation is a learning / setting unit for constructing the work component identification model 143 described later. The data prepared in the work identification data preparation 105 is learned or set to construct each part of the work component identification model 143.

[0031] The work identification model library 115 is a reference part of the work identification platform 134. The work identification model library 115 includes: a work component identification model 143, a learning / setting unit 123 based on the work component identification model, a general work identification module 124, a regional reference coordinate control unit 127, a position definition unit 128, a work position-based learning / setting unit 129, and an advanced work identification model 130.

[0032] The work component recognition model 143 is a storage unit for storing each recognition model. The work component recognition model 143 includes: a posture / action recognition model 116 and a gesture / sound recognition model 117 for the worker. The work component recognition model 143 includes: an object recognition model 118, a physical F / B (Feedback) recognition model 119, and a state recognition / measurement model 120 for the work object. The work component recognition model 143 includes: a device / tool ​​data recognition model 121 for the equipment / tool. The work component recognition model 143 includes: a work position estimation model 122 for the work position.

[0033] The posture / action recognition model 116 is a model for recognizing postures and actions by learning or setting from the body / finger skeleton estimation unit 106 and the tactile / auditory sensor data unit 107. The gesture / sound recognition model 117 is a model for recognizing gestures and sounds by learning or setting from the body / finger skeleton estimation unit 106 and the tactile / auditory sensor data unit 107.

[0034] The object recognition model 118 is a model for recognizing an object by learning or setting from the body / finger skeleton estimation unit 106, the tactile / auditory sensor data unit 107, the body part position estimation unit 108, and the appearance image unit 111. The physical F / B recognition model 119 is a model for recognizing physical feedback by learning or setting from the tactile / auditory sensor data unit 107. The state recognition / measurement model 120 is a model for recognizing and measuring the state of the work object by learning or setting from the appearance image unit 111.

[0035] The equipment / tool ​​data recognition model 121 is a model for recognizing equipment and tool data by learning or setting from the equipment / jig signal / data unit 112. The work position estimation model 122 is a model for estimating the work position by obtaining information from the body part position estimation unit 108, the work object position estimation unit 110, and the equipment / jig signal / data unit 112. The work position estimation model 122 is set in the process or product benchmark. In addition, the work position estimation model 122 obtains information from the regional benchmark coordinate control unit 127. In addition, the work position estimation model 122 provides data of the position estimation result to the learning / setting unit 129 based on the work position.

[0036] The learning / setting unit 123 based on the work component identification model is a learning / setting unit for constructing the unit work identification model 125 described later. The data prepared in the work component identification model 143 is learned or set, and the unit work identification model 125 is constructed.

[0037] The general operation recognition module 124 is a unit operation recognition unit that inputs information about the action, touch, hearing, position, camera, and equipment / tool ​​of the operator when operating the object into the unit operation model to recognize the unit operation. In other words, the general operation recognition module 124 recognizes how and what the operator has done. The general operation recognition module 124 includes a unit operation recognition model 125 and a unit operation definition unit 126.

[0038] The unit operation recognition model 125 is a model for identifying operations that can be universally defined without depending on the product among various operations. The unit operation recognition model 125 is generated by the unit operation recognition model generation unit by combining the components that are independent of the product in the operation component recognition model based on the operation content. For example, the so-called universal operation that is independent of the product is a combination of verbs related to the operation, such as tightening, embedding, and removing, for operation objects such as parts that do not contain product-specific information and are common across products, which will be explained in the following section. The unit operation recognition model 125 is a model for identifying unit operations by learning or setting from the posture / action recognition model 116, the gesture / sound recognition model 117, the object recognition model 118, the physical F / B recognition model 119, the state recognition / measurement model 120, and the equipment / tool ​​data recognition model 121.

[0039] The unit task definition unit 126 defines tasks for creating the unit task recognition model 125 .

[0040] The area-by-area reference coordinate control unit 127 performs control to change the reference coordinates for each process area. The area-by-area reference coordinate control unit 127 provides the work position estimation model 122 with information on the fixed reference and the product reference.

[0041] The position definition unit 128 defines a position based on the product 3D, the equipment 3D, etc. as described later. In addition, the position definition unit 128 adds position information to the unit operation definition unit 126 in order to define the position where the unit operation is performed. The position definition unit 128 provides learning or setting data to the learning / setting unit 129 based on the operation position.

[0042] The work position based learning / setting unit 129 learns or sets from the work position estimation model 122, the general work recognition module 124, the position definition unit 128, and the work item 138 of the product-by-product process information 137 described later. The work position based learning / setting unit 129 learns or sets to construct the high-level work recognition model 130.

[0043] The high-level task recognition model 130 is a model for recognizing where, how, and what a worker has done. The high-level task recognition model 130 recognizes tasks with high accuracy by adding information such as the task location on a product, which is product-specific information, to general unit task recognition.

[0044] The operation identification model library dynamic filter 131 excludes models other than the corresponding products in the process library containing models of various vehicle models according to the production instructions from MES 141. In addition, the models of the completed part are excluded according to the operation progress from the start of the operation to the completion of the operation. The operation identification model library dynamic filter 131 obtains information such as who is the operator from the operator identification unit.

[0045] The work recognition multimodal AI (Artificial Intelligence) 132 is an artificial intelligence for recognizing who, when, where, how, and what a worker did. The work recognition multimodal AI 132 obtains information from the work recognition data preparation 105 and the work recognition model library dynamic filtering 131.

[0046] The unit operation performance data output unit 133 outputs the unit operation performance data for each unit operation. The unit operation performance data output unit 133 obtains information from the operation recognition multimodal AI 132. In addition, the data output by the unit operation performance data output unit 133 is used in an application program (APPL) 142 for improving safety / ergonomics, improving quality assurance, improving productivity, etc.

[0047] The BOM 135 is basic information for understanding component information required to manufacture a product according to a component composition table and the composition in which the product is assembled.

[0048] PDM / BOP / BOE 136 includes main operation elements 139 , product-by-product process information 137 , and product / process drawing information 140 .

[0049] The main operation element 139 is the main information that standardizes / generalizes the components of the operation across products, and includes, for example, the operation object, operation verb, tool, operation position, etc. The product-specific process information 137 includes the operation item 138 and detailed operation instructions, and the operation item 138 is a combination of the operation elements defined by the main operation element 139. The product / process drawing information 140 includes information such as component 3D, process layout, and equipment 3D.

[0050] The main operation element 139 provides information to the unit operation definition unit 126. The product-by-product process information 137 obtains information from the main operation element 139, provides information to the product / process drawing information 140, and provides learning or setting data to the learning / setting unit 129 based on the operation position. In addition, the product-by-product process information 137 is provided via the main operation element 139 defined by the cross-product standard, and the universal operation recognition module 124 provides information to the position definition unit 128 and the area-by-area reference coordinate control unit 127.

[0051] MES 141 is a manufacturing execution system, which is a system that monitors / manages the equipment and workers' work in the factory by linking with each part of the factory's production line. MES 141 obtains information from BOP / BOE 136. In addition, MES has a production instruction database to provide production instruction information to the work recognition model library dynamic filter 131. Production instructions are instructions to people or equipment according to the production plan, the type and specifications of each product, and the components and procedures to be used.

[0052] (Description of the General Operation Identification Module According to the Embodiment) Figure 2 This is a block diagram showing components related to unit task identification and high-level task identification according to an embodiment of the present invention. Figure 3 This is a diagram showing the relationship between work verbs and unit verbs according to the embodiment. Figure 4 It is a diagram showing an example of the relationship between the unit task and the recognition model according to the embodiment. Figure 5 This is a diagram showing the embodiments involved. Figure 4 Continuation diagram of an example of the relationship between unit operations and the recognition model. Figure 6 It is a block diagram showing the relationship between the work component identification model and the unit work identification model involved in the embodiment. Figure 7 1 is a flowchart of unit operation identification in the production of a product according to an embodiment of the present invention. Figures 2 to 7 , the unit operation identification involved in the implementation method is explained.

[0053] like Figure 2As shown, the general operation identification module 124 uses the general operation verbs and operation objects decomposed into the operation items as the operation elements as the identification objects. In addition, the general operation identification module 124 defines, learns or sets the correlation with the operation component identification model 143 by using the unit operation defined by the unit verb as the master operation, wherein the unit verb is obtained by finely decomposing the operation verb part to a level suitable for operation identification. Therefore, the general operation identification module 124 constitutes a general operation identification module that is independent of individual products and processes. The general operation identification module 124 is shared between different products, thereby efficiently performing maintenance and operation including responding to changes in products / processes.

[0054] The main operation element 139 includes information on the operation object 202 and the operation verb / tool ​​203 as common operation elements. In addition, the main operation element 139 includes information on the operation position 204 as a product-specific operation element. The information on the operation object 202, the operation verb / tool ​​203, and the operation position 204 is obtained from the operation item 206 of the BOP 205.

[0055] The operation item 206 is a series of operations used to produce a product defined by a combination of an operation object and an operation verb. Operation elements are elements that constitute the operation item 206. The main operation elements are operation verbs, operation objects, tools, operation positions, etc. Among them, by defining operation verbs, operation objects, and tools as common main items, data statistics and analysis across products and factories can be performed. Regarding the operation position, even for the same part, the operation position may be different for different products, so although the classification code of the operation position can be standardized, the operation position itself is treated as an operation element inherent to the product.

[0056] The general-purpose task identification module 124 includes a unit task definition section (general-purpose) 209 , a task component identification model related definition section (excluding task positions) 211 , and a general-purpose unit task identification model 125 .

[0057] In addition, the unit operation 208 is formed by further subdividing the contents of common operation elements such as operation verbs, operation objects, and tools. Specifically, the operation verbs are decomposed into elements suitable for operation identification, and the operations obtained by dividing the operations into these units are set as unit operations. The unit operation 208 includes a unit operation definition unit (common) 209 and a unit operation definition unit (product-specific) 210.

[0058] The unit operation definition section (general) 209 includes: steps as operation sequence, unit verbs defined in a hierarchical structure according to granularity, identification classification, and information of identification items. The unit operation definition section (general) 209 is associated with the operation object 202 and the operation verb / tool ​​203. The unit verb is a verb of a granularity that enables operation identification. Although the operation verbs in the operation item 206 can be arbitrarily defined at various granularities, the purpose of the operation identification disclosed in the present invention is to specifically and in detail identify various operations performed at the manufacturing site. Therefore, for operation verbs with coarse granularity, it is necessary to subdivide them to a granularity that enables operation identification. In the case where the difference in granularity between the operation verb and the unit verb is large, the subdivision can also be performed in multiple levels.

[0059] The recognition classification is to set motion and event as two objects when identifying unit operations, and define them according to the characteristics of the operation and the purpose of recognition. Therefore, the general operation recognition module 124 can carefully recognize various operations. Motion recognition is a block of operations that are performed continuously in time sequence. Therefore, continuous actions such as walking are recognized. Events recognize specific sections of operations such as the start of tightening and the completion of tightening. Therefore, the precise operation segmentation time is recognized.

[0060] The recognition item is used when defining specific recognition targets such as start and end for the recognition classification of a unit verb.

[0061] The work component identification model related definition part (excluding work position) 211 defines the work element (work verb × work object) × unit work (unit verb / identification category / item) by assigning a work component identification model suitable for identification according to the work characteristics (excluding the work position identification model as a work component inherent in the product). The work component identification model related definition part (excluding work position) 211 includes information on necessity, mode, variation, and reliability.

[0062] By defining the degree of necessity for each job component identification model 143, the allocation of unit jobs 208 and job component identification models 143 can be performed efficiently, and the robustness when the output from a part of the model is missing can be maintained and controlled. Patterns and variants are patterns and detailed variants of objects to be recognized by the job component identification model 143. Reliability is defined for each job component identification model 143 so that recognition accuracy can be evaluated within the system. Reliability is set arbitrarily or based on actual recognition result data. This evaluation is used, for example, to sort multiple candidates when multiple candidates appear during job recognition by the general job recognition module 124.

[0063] When there are common components or similar components between products, the common operation identification module 124 is shared and reused across products.

[0064] like Figure 3 As shown, a work object n and a work verb n are obtained from the work item 206 of the BOP 205. The work verb n is divided into unit verb 1, unit verb 2, ... unit verb n. In addition, the unit verb 1 is divided into unit verb 1.2, unit verb 1.3, and unit verb 1.4. In addition, Figure 3 The relationship between work verbs and unit verbs is shown as an example. In order to further subdivide the unit verbs, levels after the third level may be provided.

[0065] Specifically, if Figure 4 As shown in the example, the unit verb 1 of taking out is decomposed into the unit verb 1.1 of carrying with bare hands, the unit verb 1.2 of grasping, and the unit verb 1.3 of carrying.

[0066] like Figure 6 As shown, the work component identification model related definition part (excluding the work position) 211 is reflected in the unit work identification model 125 through the learning / setting part 123 based on the work component identification model that has been learned or set in the work component identification model 143. Figure 5 yes Figure 4 The continuation of the column, for example, Figure 5 As shown, the grasping of the unit verb 1.2 is associated with the object recognition model 118 and the physical F / B recognition model 119. In addition, the grasping of the unit verb 1.2 is associated with the posture / action recognition model 116. In addition, the grasping of the unit verb 1.2 is associated with the equipment / tool ​​data recognition model 121. In this way, the work component recognition model related definition part (excluding the work position) 211 associates the unit verb with the work component recognition model 143.

[0067] The method of unit operation identification in product production is shown in Figure 7. The information processing device first receives a production instruction from MES141 (701). The information processing device obtains the product production sequence by receiving the production instruction. Next, the information processing device obtains the product selection filter from the job identification model library 115 (702). The product selection filter will be described later. The information processing device obtains the job identification model required for the product from the library through the product selection filter, and constitutes a group of models required for the execution of the job identification of the product, and process N becomes the job start (705). Then, job item N of process N starts (706). Next, during the execution of the job, the information processing device inputs data and synchronizes / processes time series (703) from the device / equipment (704) at any time, and performs job component identification on this basis (707). In the general job identification unit of the job identification multimodal AI132, the job component identification results are continuously obtained (708) and handed over to the unit job identification (general) (709). Unit task recognition (general purpose) identifies general unit tasks (709) by comparing the unit task recognition model 125, and outputs the recognition result (710). In addition, when multiple unit tasks are recognized during unit task recognition, multiple candidates are outputted in the order of reliability as described above.

[0068] The result output is handed over to the high-level work identification model 130. In addition, the processing of 703 to 710 is continuously performed until the work of step N is completed.

[0069] Thus, a universal job identification module capable of identifying unit jobs regardless of products is provided.

[0070] (Description of the Region-by-Region Reference Coordinate Control Unit According to the Embodiment) Figure 8 It is a diagram for explaining a reference coordinate setting unit for setting coordinates based on a fixed reference and a conveyance object (object) reference according to the embodiment. Fig. 9 This is a flowchart of setting reference coordinates in the production of a product according to an embodiment. Figure 8 and Fig. 9 , the regional reference coordinate control unit 127 involved in the implementation manner is explained.

[0071] like Figure 8As shown, in a production line type production site, the work object flows in the production line within the factory and is produced. There are the following situations, namely: for example, in the process of performing parts processing / assembly operations using a fixed workbench, the position of the work object does not move relative to the ground, and the work of assembling parts onto the body of a vehicle transported by a conveyor, etc., and the work object is transported to the conveying system. For each situation, two reference coordinates of the work position estimation model, namely a fixed reference and a conveyed object (object) reference, are pre-set. By dynamically switching these two reference coordinates according to the operator's work area, the work position coordinates required for work identification can be accurately grasped.

[0072] Fig. 9 The regional reference coordinate control unit 127 is as follows Figure 8 As shown in the figure above, when objects 800, 801, 802, and 803 flow on the conveying system, the area where the work is performed on the conveying object 800, etc. is set as the conveying object reference application area, and the other areas are set as fixed reference application areas. Regarding the switching of reference coordinates, the aforementioned areas are pre-divided on the production line layout, and the reference coordinates applied to each area are pre-defined, so that the reference coordinates defined in sequence are applied according to the position of the operator.

[0073] like Figure 8 As shown in the middle figure of the figure, the so-called fixed reference is the coordinates of the building column, workbench, etc. that are fixed relative to the ground of the production line. The fixed reference is set in the area where the work object does not move relative to the ground during the work of the operator. On the other hand, Figure 8 As shown in the figure below, the so-called conveyed object reference is the coordinates of the conveyed object when the object 800 flows on the conveying system. When the work object moves relative to the ground, it is set as the conveyed object reference.

[0074] In addition, when the transport object basis is applied, the position information of the work object on the transport system is acquired in real time by connecting to the transport system, and the work position is estimated based on the transport object basis (more precisely, based on the transported work object basis).

[0075] like Fig. 9As shown, the setting of the fixed reference and the object reference is performed by the regional reference coordinate control unit 127 which is a reference setting unit. First, the information processing device inputs data and synchronizes / processes time series from the external device / equipment 101 and the wearable device 102 (703). The information processing device inputs the data into the work position estimation model 122. Then, the information processing device identifies the operator registration area (901). The regional reference coordinate control unit 127 performs reference coordinate control (903) when the operator's registration area obtained from the operator registration area identification (901) moves to a different area based on the regional reference coordinate setting (902) set in advance with reference to the process layout / equipment 3D. Then, the reference coordinate switching instruction (904) is output to the work position estimation model 122. In this way, the regional reference coordinates are controlled according to the area registered by the operator.

[0076] (Description of Advanced Task Identification According to Embodiments) Reference Figure 2 as well as Figures 10 to 13 , the advanced job identification involved in the implementation method is explained. Fig.10 This is a diagram showing an example of adding a work position to a unit work according to the embodiment. Fig.11 This is a diagram showing the embodiments involved. Fig.10 A continuation diagram of an example of adding a job position to a unit job. Fig.12 It is a block diagram showing the relationship between the unit task identification model involved in the embodiment, the task component identification model for high-level task identification, and the high-level task identification model. Fig.13 It is a flowchart of high-level task identification in the production of products involved in the implementation mode.

[0077] like Figure 2 As shown, the advanced operation recognition 213 performs high-precision and high-accuracy operation recognition on the operation position information inherent in the output combination product / process obtained by the general operation recognition module 124. In some cases, the advanced operation recognition 213 can also supplement the individual recognition model. In addition, the advanced operation recognition 213 can maximize the versatility by combining with the general operation recognition module 124, and at the same time realize the efficient construction of the advanced operation recognition model corresponding to the differences in products and processes.

[0078] like Figure 2 As shown, the high-level job identification 213 includes a unit job definition section (product-specific) 210 , a job component identification model related definition section (job location) 212 , and a high-level job identification model 130 .

[0079] The unit operation definition part (product-specific) 210 includes reference coordinates, target parts, position types, and range designation methods as operation position details. The unit operation definition part (product-specific) 210 is included in the unit operation 208, and obtains the operation position code that is set in common and standard for all products from the product-specific operation position 204 of the main operation element 139. In addition, the unit operation definition part (product-specific) 210 obtains the 3D coordinate information and process layout information of the product from the PDM / BOP 207 as the detailed operation position.

[0080] The reference coordinates are selected in Figure 8 Any of the fixed reference and transport reference defined in.

[0081] The target part refers to the body part of the worker for which the work position is to be acquired. Examples of the target part include the head, arms, legs, and the like.

[0082] The position type is a method of specifying a position, and examples of the position type include two-dimensional plane and three-dimensional space.

[0083] The range designation method is a method for designating the range of a work position. For example, the range designation method specifies the position range using two diagonal points of a three-dimensional object or a two-dimensional figure, or specifies the position range using a path.

[0084] The work component identification model related definition part (work position) 212 assigns a work component identification model suitable for identification to the work position (a work position identification model of a work component inherent to the product) and defines it. The work component identification model related definition part (work position) 212 includes information on necessity, reference coordinates, target part, position type, and reliability.

[0085] By defining the degree of necessity for each work component identification model 143, it is possible to efficiently assign work positions and work component identification models 143, and to maintain and control the robustness when output from a part of the model is missing.

[0086] The reference coordinates, target part, and position type use information acquired by the unit operation definition unit (product-specific) 210 .

[0087] Reliability is defined for each work component recognition model 143 so that recognition accuracy can be evaluated within the system. Reliability is set arbitrarily or based on actual recognition result data. This evaluation is used, for example, to sort multiple candidates when multiple candidates appear during work recognition by high-level work recognition 213.

[0088] The high-level work identification model 130 is capable of performing detailed and highly accurate work identification corresponding to the product by combining the general work identification module 124 and the work position estimation model 122 for work component identification (other models are added as needed).

[0089] like Fig.12 As shown, in the job component identification model related definition section (job position) 212, the unit job identification model 125 of the general job identification module 124, the job position inference model 122 of the job component identification model 143 and other models 1201 are associated and learned or set by the learning / setting section 129 based on the job position.

[0090] like Fig.13 As shown, high-level job identification is performed. Fig.13 Apart from Figure 7 In addition to the contents described in , advanced job identification is also recorded. Figure 7 The contents described in are omitted.

[0091] After the information processing device outputs the unit operation recognition (general) result (710), the advanced operation recognition unit of the operation recognition multimodal AI 132 obtains the unit operation recognition result (general) (1301). In parallel, the regional reference coordinate control unit 127 obtains the position information of the operator from the operation component recognition (707) and sends the reference coordinate information to the operation position estimation model 122. The operation position estimation model 122 obtains the position information of the operator from the operation component recognition (707) and obtains the reference coordinate information from the regional reference coordinate control unit 127, thereby estimating the operation position coordinates and outputs the result to the advanced operation recognition unit of the operation recognition multimodal AI 132. The advanced operation recognition unit obtains the operation position coordinate estimation result (1306), and uses the advanced operation recognition model 130 to perform matching evaluation between the unit operation and the actual operation position, selects the corresponding candidate unit operation candidates (1307), and outputs the candidate ranking (1308).

[0092] Next, the advanced operation recognition unit of the operation recognition multimodal AI 132 recognizes (infers) the unit operation with the highest accuracy (1302) based on the unit operation recognition candidates and candidate rankings obtained from the result output (710) of the unit operation recognition (general) and the result output (1308) of the unit operation recognition (product-specific). In parallel, the advanced operation recognition unit performs operation element recognition (1303) on the operation element as a collection of multiple unit operations by monitoring each unit operation recognition data in a time series (operation object = component X, operation verb = tightening, tool = tool Y, operation position code = Z, operation status: in progress, completed, etc.). Next, the advanced operation recognition unit performs operation item recognition (e.g., operation item = tightening component X, operation status: in progress, completed, etc.) (1304) on the operation item as a combination of operation elements in conjunction with the output of the operation element recognition result. Then, the high-level operation recognition unit outputs the above-mentioned recognition results (unit operation recognition (1302), operation element recognition (1303), operation item recognition (1304)) at any time (1305). The recognition results are sent to a database, data analysis, application program, etc. In addition, the processing of the high-level operation recognition unit is continuously performed until the operation of process N is completed.

[0093] If the operation element recognition (1303) fails to recognize the operation element, it returns to the unit operation recognition (1302). The operation element recognition (1303) performs recognition based on the multiple results of the unit operation recognition (1302).

[0094] Thus, a high-level work recognition unit is provided, which uses as input the unit work recognition (general) candidate outputted from the general work recognition unit of the work recognition multimodal AI 132 and the unit work recognition (product-specific) candidate associated with the work position specific to the product using the estimation result of the work position estimation model as input, and recognizes the work in a high-level manner. Thus, a work recognition system and a work recognition method are provided.

[0095] (Description of the Filtering of the Job Identification Model According to the Embodiment) Fig.14 The following is a schematic diagram (upper figure) and a flow chart (lower figure) showing product selection filtering and progress filtering in the production of products involved in the embodiment. Fig.14 , the job identification model filtering involved in the implementation method is explained.

[0096] like Fig.14As shown in the figure above, for companies that mass-produce various products, the model library of all products is huge, such as 15 products x 5 factories. Therefore, when these huge job recognition models are used on mixed production lines, problems may arise in terms of processing speed and the capacity of information processing devices. Therefore, by selecting the corresponding model by factory when transferring from the production preparation stage to the mass production stage, the factory library, for example, Fig.14 In addition, by selecting the corresponding model by process, the process library, for example, Fig.14 Here, if the details of the three products are set as product A, product B, and product C, in mass production of the products, for example, when a production instruction for product B is received, by filtering to the model corresponding to product B, it is possible to filter to 1 product × 1 process. In this way, in a process library containing models of multiple vehicle models, the process of excluding models other than the corresponding product according to the production instruction is used as a product selection filter.

[0097] Furthermore, a process of excluding models of the completed part according to the progress of the work from the start of the work to the completion of the work is referred to as progress filtering.

[0098] like Fig.14 As shown in the following figure, the information processing device receives a production instruction from MES141 and obtains a product production sequence (701). Next, the information processing device obtains a product selection filter (702) for selecting the execution model group of process N from the job identification model library 115. The job of process N starts (705). Next, the job identification of job items 1, 2...N is performed (1401). In parallel, the information processing device performs job identification (1402). In the case of successful job identification (in the case of "Yes" in step 1402), the information processing device outputs the job identification result. In the case of failure to identify the job (in the case of "No" in step 1402), the information processing device cancels, i.e. removes the product selection filter (1404), and performs job identification again (1405). The product selection filter is canceled because it is considered that repeated jobs or unspecified jobs that do not usually occur are performed. In addition, the cancellation range is set arbitrarily in advance. When it is possible to perform another operation identification (e.g., this process × this product, this process × all products, this factory × product × this product, etc.), the information processing device outputs the operation identification result (1403). Next, the information processing device excludes the model of the operation completed part from the execution model group through progress filtering and selects it (1406). In this way, the execution model is updated. The processing of 1402 to 1406 is continuously performed until the operation of process N is completed.

[0099] In this way, the job identification system includes: a product selection filter unit that selects a job identification model according to a production instruction, a progress filter unit that selects a job identification model according to a job progress, and a job identification unit that identifies a job according to the selected job identification model. In addition, the job identification system filters the job identification model into jobs of one product through the product selection filter unit. In addition, the job identification system excludes job identification models of completed parts of the job through the progress filter unit. In addition, the job identification system determines whether the job can be identified. If the job cannot be identified, the product selection filter is removed and the job is identified again, and the job identification result is output.

[0100] In this way, it is possible to provide a work recognition system that selects a learning model according to the progress of the work, shortens the processing time required for determination based on unnecessary learning models, and performs high-level work recognition at high speed.

[0101] (Description of Diversity Input Control Unit According to Embodiment) Fig.15 A diagram showing an example of collaboration in data acquisition of sensors corresponding to a work area according to an embodiment (upper diagram), a schematic diagram of the work area (lower left diagram), and a correspondence table between the work area and sensors (lower right diagram). Fig.16 1 is a flowchart showing a process of assigning and switching a plurality of sensors or devices according to a work position in a diversity input control according to an embodiment. Fig.15 and Fig.16 , the diversity input control unit involved in the implementation method is explained.

[0102] When using multiple sensors or devices such as cameras, GPS, and IMU to obtain work performance data, they may become obstacles depending on the positional relationship between the process and the product, and may become unusable (camera: blind spot, etc.) or not function properly (GPS: radio wave reflection / shielding, etc.) depending on the device. Therefore, the diversity input control unit 113 automatically selects the best device according to the work position in a system equipped with multiple sensors or devices, thereby ensuring the robustness and accuracy of data acquisition.

[0103] like Fig.15 As shown in the lower left figure, the surrounding area of ​​the object is divided into, for example: an area that separates the working area from the object and is not affected by obstacles such as equipment (area A), namely the object separation area, an area close to the object and in contact with the object (area B), namely the object close area, and an area inside the object (area C), namely the area inside the object.

[0104] Here, taking the acquisition of the working position as an example, Fig.15As shown in the lower right figure of , in area A, the working position can be detected with high accuracy by indoor GPS. In addition, position estimation can also be performed by position estimation based on IMU. However, since indoor GPS has higher accuracy, indoor GPS is prioritized in area A and IMU is used as a backup. When shooting an object, area A cannot be used because it is outside the shooting range.

[0105] Similarly, in area B, since "position estimation based on image" or "position estimation based on image skeleton estimation or image object recognition, etc." can be obtained from the image of the captured object, position estimation based on image skeleton estimation is the most advantageous in terms of accuracy (however, depending on the depression angle of the camera or the distance from the camera, other devices may have higher accuracy). Since the accuracy of indoor GPS decreases in areas with many obstructions / reflectors of radio waves, IMU is used as the second backup in area B and indoor GPS is used as the third backup.

[0106] In area C, position estimation by IMU is given the highest priority. Indoor GPS cannot be used or the accuracy is extremely degraded when the object blocks the radio waves. In addition, "position estimation based on images" or "position estimation based on image skeleton estimation or image object recognition" of the photographed object can be partially used in areas that are not blind spots, so it is used as a backup.

[0107] In the operating staff Fig.15 When working along the working path shown in the figure above, indoor GPS is used at working position 1, and position estimation based on image skeleton estimation that inherits the position data is performed at working positions 2 and 3. Similarly, position estimation based on IMU that inherits the position data is performed at working positions 4 and 5, and position estimation based on image skeleton estimation that inherits the position data is performed at working position 6.

[0108] like Fig.16 As shown, first, the information processing device receives a production instruction from MES141 (701). Next, the information processing device obtains the diversity input control setting from the diversity input control setting unit 1601 (1602). In parallel, the operation of process N starts (705). At the beginning, when the operation start position is set to a specified position for each process, the position is reset (1603). The information processing device obtains the information of the position reset. Next, the information processing device uses the operation position estimation model 122 to continuously obtain the operation position estimation results of operation items 1, 2...n (1604). During the operation of process N, the information processing device instructs the selection of input devices at any time according to the operation position estimation results (1605).

[0109] Thus, the diversity input control unit includes: a work position information acquisition unit that acquires the position of the operator; and a control unit that switches / prioritizes the multiple sensors or devices to be input according to the pre-divided work area. In addition, the divided areas are: an object separation area away from the object, an object proximity area in contact with the object, and an area inside the object. In addition, the object separation area, for example, gives priority to using indoor GPS to obtain position information, the object proximity area, for example, gives priority to using image skeleton estimation to obtain position information, and the area inside the object, for example, gives priority to using IMU to obtain position information. The diversity input control unit also includes: a receiving unit that receives production instructions; an input control setting acquisition unit that acquires input control settings of multiple sensors or devices according to the production instructions.

[0110] In this way, a position information acquisition system and a position information acquisition method can be provided, which are equipped with multiple sensors or devices and automatically select the best sensor or device according to the working position, thereby ensuring the robustness and accuracy of data acquisition.

[0111] Part or all of the processing in the above-mentioned information processing device can be implemented as a computer program. Such a program can be stored and provided to a computer using various types of non-temporary computer-readable media. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, hard disk drives) and magneto-optical recording media (e.g., magneto-optical disks), CD-Read Only Memory (ROM), CD-R, CD-R / W, semiconductor memories (e.g., mask ROM, Programmable ROM (PROM, Programmable Read-Only Memory)), Erasable PROM (EPROM, Erasable Read-Only Memory), Flash ROM, Random Access Memory (RAM, Random Access Memory)). In addition, the program can also be provided to the computer via various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can provide programs to a computer via wired communication paths such as wires and optical fibers, or wireless communication paths.

[0112] In addition, the present invention is not limited to the above-mentioned embodiment, and can be appropriately modified within the scope not departing from the gist. From the disclosure thus described, it is obvious that the embodiments of the present disclosure may be varied in a number of ways. Such variations should not be regarded as departing from the spirit and scope of the present disclosure, and all such modifications obvious to those skilled in the art are intended to be included within the scope of the claims.

Claims

1. A task identification system comprising: A storage unit storing a plurality of work component recognition models, wherein the plurality of work component recognition models take as input information of a worker's action, touch, hearing, position, camera, and equipment / tool ​​when working on an object, decompose the work into work components, and make judgments; a unit operation identification model generating unit for generating a general unit operation identification model by combining components that are not dependent on products in the operation component identification model based on the operation content; as well as The unit operation recognition unit inputs information on the action, touch, hearing, position, camera, and equipment / tool ​​of the operator when operating the object into the unit operation recognition model to recognize the unit operation.

2. The operation identification system according to claim 1, wherein: Equipped with a benchmark setting unit, When the object does not move relative to the ground during the operation of the operator, the reference setting unit sets the reference coordinates as a fixed reference. The reference setting unit sets the reference coordinates as an object reference when the object moves relative to the ground during the work of the operator.

3. The operation identification system according to claim 1 or 2, wherein: have: a work position estimating unit that estimates a work position by taking the output result of the unit work identifying unit and the reference coordinates of the work area as input; and A high-level work recognition unit receives the output result of the unit work recognition unit and the estimation result of the work position estimation unit as input, and recognizes the work at a high level.

4. A method for identifying a job, wherein: storing a plurality of work component recognition models, wherein the plurality of work component recognition models take as input the information of the action, touch, hearing, position, camera, and equipment / tool ​​of the operator when performing work on the object, decompose the work into work components and make judgments, Based on the job content, the components in the job component identification model that are not dependent on the product are combined to generate a general unit job identification model, The unit operation recognition model is input with information on the operator's actions, touch, hearing, position, camera, and equipment / tools when performing operations on an object, and the unit operation is recognized.

5. A program product that causes an information processing device to execute: storing a plurality of work component recognition models, wherein the plurality of work component recognition models take as input the information of the action, touch, hearing, position, camera, and equipment / tool ​​of the operator when performing work on the object, decompose the work into work components and make judgments, Based on the job content, the components in the job component identification model that are not dependent on the product are combined to generate a general unit job identification model, The unit operation recognition model is input with information on the actions, touch, hearing, position, camera, and equipment / tools of the operator when performing operations on the object, so as to recognize the unit operation.

Images