Work level conversion device, work level conversion method, and work level conversion program

Abstract

To provide an operation level conversion device that can convert a plurality of lower level operations into upper level operations.SOLUTION: An operation level conversion device comprises: a generation unit 13 that generates series data of target movement information 19b in which a target movement detected from time series information on an operator's operations and incidental information incidental to the target operations are associated with each other; and a conversion output unit 14 that, based on a learned model 19d that converts lower level operation series data specified by the combination of the target movement and the incidental information into upper level operation series data composed of a plurality of lower level operations and the series data of the target movement information 19b, converts the series data of the target movement information 19b into the upper level operation series data and outputs the resultant data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an activity level conversion device, an activity level conversion method, and an activity level conversion program. [Background technology]

[0002] At production sites such as factories, the actions of workers are analyzed to recognize the work performed by the workers and to evaluate the content of the work. For example, the following Patent Document 1 discloses a work recognition device that associates an object with a body part of the worker who performed the work using the object, recognizes the work performed by the worker, and evaluates the content of the work based on the recognition result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6444573 Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, the work performed at a production site can be divided into work levels with different granularity. For example, the work of packing a box can be divided into the work of fixing a box to a jig and the work of storing a product in a box. In this case, the work of packing a box is a higher level work, and the work of fixing a box to a jig and the work of storing a product in a box are lower level work. The work of fixing a box to a jig, which has been divided into this lower level work, can be further divided into the work of gripping the box, the work of transporting the box, and the work of adjusting the box. In this case, the work of fixing the box to the jig is a higher level work, and the work of gripping the box, the work of transporting the box, and the work of adjusting the box are lower level work.

[0005] When evaluating the content of work, the content of the evaluation may be set for each level of work. Therefore, if different levels of work can be recognized when recognizing work, more effective evaluation can be performed.

[0006] However, in Patent Document 1, it is only possible to recognize tasks at the same level, and it is not possible to convert the recognized tasks into tasks at a different level.

[0007] Here, Japanese Patent Application No. 2019-200826 by the applicant of the present application describes a method of converting a higher-level task into multiple lower-level tasks by dividing a task based on a target motion into tasks on a component-by-component basis. However, there is no description of a method of converting multiple lower-level tasks into a higher-level task.

[0008] Therefore, the present invention provides an activity level conversion device, an activity level conversion method, and an activity level conversion program capable of converting a plurality of lower-level activities into higher-level activities. [Means for solving the problem]

[0009] An activity level conversion device according to one aspect of the present disclosure includes a generation unit that generates series data of target activity information that associates a target activity detected from time series information related to a worker's activity with accompanying information accompanying the target activity, a converter that converts lower-level activity series data specified by a combination of the target activity and the accompanying information into higher-level activity series data composed of a plurality of lower-level activities, and a conversion output unit that converts the series data of the target activity information based on the series data of the target activity information and outputs the higher-level activity series data.

[0010] According to this aspect, series data of target action information is generated based on the target action and accompanying information detected from time series information regarding the worker's actions, and the series data of the target action information is converted into higher-level work series data based on a converter that converts lower-level work series data identified by a combination of the target action and accompanying information into higher-level work series data and the series data of the target action information, and can be output.

[0011] In the above aspect, the converter may be configured with a trained model that uses task information indicating a correspondence between lower-level task series data and higher-level task series data as teacher data, and is trained to convert the sequence data of the target action information into higher-level task series data and output it.

[0012] This makes it possible to obtain higher-level task sequence data by inputting sequence data of target action information into the trained model.

[0013] In the above aspect, the conversion output section may further output, for each target motion, start timing information indicating whether or not a higher level task has been started by the target motion.

[0014] This makes it possible to identify the timing at which a higher level operation has started.

[0015] In the above aspect, the supplementary information may include information that can be recognized from the time-series information and information that is annotated to the time-series information.

[0016] This makes it possible to improve the accuracy of identifying the target action.

[0017] An activity level conversion method according to another aspect of the present disclosure is a method executed by a processor, and includes generating series data of target activity information that associates a target activity detected from time series information related to a worker's activity with accompanying information accompanying the target activity, a converter that converts lower-level activity series data identified by a combination of the target activity and the accompanying information into higher-level activity series data composed of a plurality of lower-level activities, and converting the series data of the target activity information into higher-level activity series data based on the series data of the target activity information and outputting the higher-level activity series data.

[0018] According to this aspect, series data of target action information is generated based on the target action and accompanying information detected from time series information regarding the worker's actions, and the series data of the target action information is converted into higher-level work series data based on a converter that converts lower-level work series data identified by a combination of the target action and accompanying information into higher-level work series data and the series data of the target action information, and can be output.

[0019] An activity level conversion program according to another aspect of the present disclosure causes a computer to function as a generation unit that generates series data of target activity information that associates a target activity detected from time series information related to a worker's activity with accompanying information accompanying the target activity, a converter that converts lower-level task series data identified by a combination of the target activity and the accompanying information into higher-level task series data composed of a plurality of lower-level tasks, and a conversion output unit that converts the series data of the target activity information into higher-level task series data based on the series data of the target activity information and outputs the converted data.

[0020] According to this aspect, series data of target action information is generated based on the target action and accompanying information detected from time series information regarding the worker's actions, and the series data of the target action information is converted into higher-level work series data based on a converter that converts lower-level work series data identified by a combination of the target action and accompanying information into higher-level work series data and the series data of the target action information, and can be output. Effect of the Invention

[0021] According to the present invention, it is possible to provide an activity level conversion device, an activity level conversion method, and an activity level conversion program capable of converting a plurality of lower level activities into higher level activities. [Brief description of the drawings]

[0022] [Figure 1] 1 is a diagram illustrating an example of an overview of a system including an operation level conversion device according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating an example of a functional configuration of a system including an operation level conversion device. [Diagram 3] 10 is a diagram illustrating an example of target action information stored in the activity level conversion device. [Figure 4] 10 is a diagram illustrating an example of task information stored in the task level conversion device. [Diagram 5] FIG. 2 is a diagram illustrating a hardware configuration of an operation level conversion device. [Figure 6] 11 is a flowchart illustrating an example of an operation level conversion process executed by the operation level conversion device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, referred to as "the present embodiment") will be described with reference to the drawings. In each drawing, the same reference numerals denote the same or similar configurations.

[0024] §1 Examples of application First, an example of a scene to which the present invention is applied will be described with reference to Fig. 1. In a system 100 including an activity level conversion device 10 according to this embodiment, the image sensors 20a, 20b, and 20c capture the actions of a worker A performed in a certain activity area R, and the activity level conversion device 10, which acquires the captured video, outputs a higher level activity based on a target action corresponding to a lower level activity detected from the video. The higher and lower activity levels will be described below.

[0025] When work levels with different granularity are set, the work set at the coarser (higher) work level becomes the higher-level work, and the work set at the finer (lower) work level becomes the lower-level work.

[0026] For example, if a "packing operation" is composed of an "operation of fixing a box to a jig" and an "operation of storing a product in a box," then the "packing operation" will be a higher-level operation, and the "operation of fixing a box to a jig" and the "operation of storing a product in a box" will be lower-level operations.

[0027] Furthermore, if the lower-level task of "fixing the box to the jig" is composed of "holding the box," "transporting the box," and "adjusting the box," then "fixing the box to the jig" will be the higher-level task, and "holding the box," "transporting the box," and "adjusting the box" will be the lower-level tasks.

[0028] Here, grasping is defined as the action of moving a hand to a work object and grabbing it, for example, the action of going to grab a part or tool. Transport is defined as the action of moving a work object to a target location, for example, the action of carrying a part or tool toward a product being assembled. Adjustment is defined as the action of moving a task to a target state, for example, the action of assembling parts.

[0029] The present invention involves training a model to learn the correspondence between lower-level work series data and higher-level work series data to generate a trained model for work level conversion, inputting work series data detected from a video into the trained model, and outputting the converted higher-level work series data from the trained model.

[0030] This makes it possible to convert task sequence data consisting of a plurality of tasks detected from a video into higher-level task sequence data.

[0031] That is, the task level conversion device 10 according to this embodiment can convert a plurality of tasks at a lower level into tasks at a higher level.

[0032] §2 Configuration Example [Function configuration] Next, an example of the functional configuration of a system 100 including an activity level conversion device 10 according to the present embodiment will be described with reference to FIG. 2. The system 100 includes three image sensors 20a, 20b, and 20c, and an activity level conversion device 10. In the following, the three image sensors 20a, 20b, and 20c will be referred to as an image sensor 20 when there is no need to distinguish them from one another. The activity level conversion device 10 has, as its functional configuration, an acquisition unit 11, a detection unit 12, a generation unit 13, a conversion output unit 14, and a storage unit 19, for example. The storage unit 19 stores, for example, a video 19a, target action information 19b, activity information 19c, and a trained model 19d.

[0033] The functional components of the system 100 including the task level conversion device 10 will be described in detail below.

[0034] <Image sensor> The image sensor 20 is, for example, a general-purpose camera, and captures a video including a scene in which the worker A is performing an action in the working area R. The image sensor 20 has, for example, a detection unit as a functional configuration. The detection unit detects the action of the worker A, and outputs a video showing the action as time-series information.

[0035] Each of the image sensors 20a, 20b, 20c is disposed so as to capture the entire working area R and the entire body of the worker A. In this case, for example, each of the image sensors 20a, 20b, 20c may be disposed so as to capture the entire working area R and the entire body of the worker A, or each of the image sensors 20a, 20b, 20c may be disposed so as to capture a part of the working area R and the worker A and to cover the entire working area R and the entire body of the worker A by combining the respective videos. Also, each of the image sensors 20a, 20b, 20c may capture the working area R and the worker A at different magnifications. It is not necessary to provide three image sensors 20, and it is sufficient to provide at least one or more image sensors.

[0036] <Acquisition part> The acquisition unit 11 acquires time-series information (video in this embodiment) related to the actions performed by the worker A from the image sensor 20. The time-series information acquired by the acquisition unit 11 is transmitted to the storage unit 19 and stored as video 19a. The acquisition unit 11 also acquires the video 19a stored in the storage unit 19.

[0037] <Detection section> The detection unit 12 recognizes the position on the image where the hand of the worker A is present from the video 19a, and detects the target motion based on the motion at the recognized position. For example, grasping, carrying, and adjustment can be set as the target motion. In addition, for example, a motion of grabbing a box or part can be set as a motion corresponding to grasping, for example, a motion of carrying a box or part can be set as a motion corresponding to carrying, and for example, a motion of fixing or assembling a box or part can be set as a motion corresponding to adjustment. The target motion and the motion corresponding to the target motion can be set appropriately according to the work content, and it is preferable that the correspondence between them is stored in advance in the storage unit 19.

[0038] Specifically, the detection unit 12 recognizes the position on the video 19a where the hand of the worker A is present, and when it determines that the worker A is making a motion to grasp a part, it detects grasping as the target motion.

[0039] <Generation part> The generating unit 13 generates object action information 19b in which the object action detected by the detecting unit 12 is associated with the accompanying information accompanying the object action. The object action information 19b will be described with reference to FIG.

[0040] The target action information 19b is sequence data having, for example, a NO item, a target action item, and an additional information item as data items. The NO item stores the order in which the target action occurred. Instead of the order in which the target action occurred, the time at which the target action occurred or the elapsed time from the start of the work may be stored. The target action item stores the content of the target action. The additional information item stores additional information that is additional to the target action. The additional information may include, for example, information that can be recognized from the video 19a and information that is annotated to the video 19a. Specifically, this includes objects such as parts and boxes, the contact / separation state between the parts and boxes and the hands, the worker, the work process, and information that is known at the time the video was shot.

[0041] For example, in the target motion information with a NO of "1" in FIG. 3, "grasp" is stored as the target motion, and "box" is stored as the additional information. This target motion information allows specification that the worker is performing the task of gripping a box at the time of NO "1". Furthermore, in the target motion information with a NO of "4", "grasp" is stored as the target motion, and "part 1" is stored as the additional information. This target motion information allows specification that the worker is performing the task of gripping part 1 at the time of NO "4". Furthermore, in the target motion information with a NO of "6", "transport" is stored as the target motion, and "part 1" is stored as the additional information. This target motion information allows specification that the worker is performing the task of gripping a box at the time of NO "6".

[0042] <Conversion output section> Returning to the description of Fig. 2, the conversion output unit 14 inputs the sequence data of the target action information 19b generated by the generation unit 13 to the trained model 19d, and causes the trained model 19d to output the converted higher-level task sequence data.

[0043] The trained model 19d is a trained model for task level conversion, and is generated by, for example, having a model learn the correspondence between lower-level task series data and higher-level task series data. As the trained model, for example, a deep learning model used in the field of natural language processing (NLP), such as Transformer, can be used.

[0044] In this embodiment, for example, task information 19c indicating the correspondence between lower-level task series data identified by a combination of target actions and accompanying information and higher-level task series data is used as training data to generate a trained model 19d.

[0045] The task information 19c will be described with reference to Fig. 4. In Fig. 4, "Level 1", "Level 2", and "Level 3" are set as task levels, for example. "Level 1" is the lowest level, "Level 3" is the highest level, and "Level 2" is the middle level.

[0046] At "Level 1," lower level task sequence data is set in which the contents of a target action, which is the smallest unit of work, and the associated information associated with the target action are arranged in chronological order.

[0047] In "Level 2", higher level work sequence data corresponding to each of the time series data constituting the lower level work sequence data is set. In the "Level 2" work sequence data, start timing information indicating the timing when each work in "Level 2" started is set. For example, "1" is set to data when work started, and "0" is set to other data.

[0048] For example, the three data in the top row of level 2 are common to the task of "fixing the box to the jig," and the start timing information of the data in the top row is set to "1." Therefore, by identifying the task whose start timing information is set to "1," it is possible to identify the timing at which the task of "fixing the box to the jig" started.

[0049] The five data in the lower row of level 2 alternate between the tasks of "putting part 1 in a box" and "putting part 2 in a box," and task sequence data like this is set when, for example, the task of "putting part 1 in a box" is performed with the right hand and the task of "putting part 2 in a box" is performed in parallel with the left hand. In this type of task, "1" is set to the start timing information corresponding to the timing when the task of "putting part 1 in a box" is started with the right hand, and the timing when the task of "putting part 2 in a box" is started with the left hand.

[0050] It should be noted that when each work item includes "level 1" additional information, such as "level 2," the setting of the additional information can be omitted.

[0051] In "Level 3", the highest level of task sequence data corresponding to each of the time series data constituting the task sequence data of "Level 1" and "Level 2" is set. When one task is set as the highest level, such as in "Level 3", the settings of the incidental information and start timing information can be omitted.

[0052] Here, when outputting the converted higher-level task sequence data, the conversion output unit 14 may further output start timing information for each task (target motion).

[0053] [Hardware configuration] Next, an example of the hardware configuration of the task level conversion device 10 according to this embodiment will be described with reference to Fig. 5. The task level conversion device 10 has a CPU (Central Processing Unit) 10a equivalent to a calculation device, a RAM (Random Access Memory) 10b and a ROM (Read only Memory) 10c equivalent to a storage unit 19, a communication device 10d, an input device 10e, and a display device 10f. These components are connected via a bus so that they can transmit and receive data to each other. Note that, although a case where the task level conversion device 10 is configured by one computer will be described in this embodiment, the task level conversion device 10 may be realized using multiple computers.

[0054] The CPU 10a executes programs stored in the RAM 10b or the ROM 10c and functions as a control unit that performs calculations and processing of data. The CPU 10a receives various input data from the input device 10e or the communication device 10d, and displays the results of calculations performed on the input data on the display device 10f or stores the results in the RAM 10b or the ROM 10c.

[0055] The RAM 10b is made up of, for example, a semiconductor memory device, and stores rewritable data.The ROM 10c is made up of, for example, a semiconductor memory device, and stores readable but non-rewritable data.

[0056] The communication device 10d is an interface that connects the activity level conversion device 10 to an external device. The communication device 10d is connected to the image sensor 20 via a communication network such as a LAN (Local Area Network) or the Internet, for example, and receives moving images from the image sensor 20.

[0057] The input device 10e is an interface that accepts data input from a user, and can include, for example, a keyboard, a mouse, and a touch panel.

[0058] The display device 10f is an interface that visually displays the results of calculations performed by the CPU 10a, and can be configured, for example, by an LCD (Liquid Crystal Display).

[0059] The program may be provided by being stored in a computer-readable storage medium such as RAM 10b or ROM 10c, or may be provided via a communication network connected by communication device 10d. In the activity level conversion device 10, the CPU 10a executes the program to perform the operations of the acquisition unit 11, detection unit 12, generation unit 13, and conversion output unit 14 shown in FIG. 2. Note that these physical configurations are merely examples, and do not necessarily have to be independent configurations. For example, the activity level conversion device 10 may be provided with an LSI (Large-Scale Integration) that integrates the CPU 10a with the RAM 10b and ROM 10c.

[0060] §3 Example of operation FIG. 6 is a flowchart showing an example of the task level conversion process executed by the task level conversion device 10 according to this embodiment.

[0061] First, the acquisition unit 11 of the activity level conversion device 10 acquires a video, which is time-series information on the activity of the worker A, from the image sensor 20 (step S101). The acquired video is stored in the storage unit 19 as a video 19a.

[0062] Next, the detection unit 12 of the activity level conversion device 10 recognizes the position on the image where the hand of the worker A is present, based on the video 19a, and detects the target activity (step S102).

[0063] Next, the generating unit 13 of the activity level converting device 10 generates the target action information 19b based on the target action detected in the above step S102 and the accompanying information accompanying the target action (step S103). The generated target action information 19b is stored in the storage unit 19.

[0064] Next, the conversion output unit 14 of the activity level conversion device 10 inputs the sequence data of the target action information 19b generated in the above step S103 to the trained model 19d, and outputs the converted higher-level activity sequence data from the trained model 19d (step S104). Then, the activity level conversion process is terminated.

[0065] As described above, according to the work level conversion device 10 of this embodiment, sequence data of target action information 19b is generated based on the target action and incidental information detected from the video 19a related to the action of worker A, and the sequence data of the target action information 19b is converted to higher-level work sequence data based on the trained model 19d that converts lower-level work sequence data specified by a combination of the target action and the incidental information into higher-level work sequence data and the sequence data of the target action information 19b, and can be output.

[0066] Therefore, the task level conversion device 10 according to this embodiment can convert a plurality of tasks at a lower level into tasks at a higher level.

[0067] § 4 Variations It should be noted that the present invention is not limited to the above-described embodiment, and can be embodied in various other forms without departing from the spirit and scope of the present invention. Therefore, the above-described embodiment is merely illustrative in all respects and should not be interpreted as being restrictive.

[0068] For example, in the above-described embodiment, a trained model (converter) is used when converting lower-level work sequence data into higher-level work sequence data, but this is not limited to this. For example, a work information table (converter) as shown in FIG. 4 may be used to convert lower-level work sequence data into higher-level work sequence data.

[0069] In the above embodiment, the time-series information is a moving image, but the time-series information is not limited to a moving image. Specifically, the time-series information may be information on coordinate values ​​indicating the motion of the worker A measured by a motion capture device provided instead of the image sensor 20, or information indicating the motion of the worker A measured by having the worker A wear an acceleration sensor or a gyro sensor instead of the image sensor 20. The time-series information may be information indicating a change in pressure value measured by a pressure sensor arranged in the working area R instead of the image sensor 20, or information indicating a change in an event detected by a photoelectric sensor arranged in the working area R instead of the image sensor 20. Furthermore, the time-series information is not limited to using each of the above information individually, and may be a combination of two or more of the above information.

[0070] In addition, the embodiment of the present invention may be described as in the following appendices. However, the embodiment of the present invention is not limited to the form described in the following appendices. In addition, the embodiment of the present invention may be a form in which the descriptions between the appendices are replaced or combined.

[0071] [Appendix 1] a generation unit (13) that generates sequence data of target action information (19b) in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action; a converter for converting lower-level task sequence data specified by a combination of the target motion and the incidental information into higher-level task sequence data composed of a plurality of the lower-level tasks, and a conversion output unit (14) for converting the sequence data of the target motion information (19b) into the higher-level task sequence data based on the sequence data of the target motion information (19b) and outputting the converted sequence data; An operating level conversion device (10) comprising:

[0072] [Appendix 2] The converter is configured by a trained model (19d) that uses task information (19c) indicating a correspondence relationship between the lower level task series data and the higher level task series data as teacher data, and trains the trained model (19d) to convert the sequence data of the target action information (19b) into the higher level task series data and output the converted data. 2. The working level conversion device (10) of claim 1.

[0073] [Appendix 3] The conversion output unit (14) further outputs, for each of the target movements, start timing information indicating whether or not a higher-level task has been started by the target movement. 3. The working level conversion device (10) of claim 2.

[0074] [Appendix 4] The supplementary information includes information that can be recognized from the time-series information and information that is annotated to the time-series information. 3. The working level conversion device (10) of claim 2.

[0075] [Appendix 5] 1. A processor-implemented method, comprising: generating sequence data of target action information (19b) in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action; a converter for converting lower-level task sequence data specified by a combination of the target motion and the incidental information into higher-level task sequence data composed of a plurality of the lower-level tasks, and converting the sequence data of the target motion information (19b) into the higher-level task sequence data based on the sequence data of the target motion information (19b) and outputting the converted sequence data; A working level conversion method including:

[0076] [Appendix 6] Computer, a generation unit (13) that generates sequence data of target action information (19b) in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action; a converter for converting lower-level task sequence data specified by a combination of the target motion and the incidental information into higher-level task sequence data composed of a plurality of the lower-level tasks, and a conversion output unit (14) for converting the sequence data of the target motion information (19b) into the higher-level task sequence data based on the sequence data of the target motion information (19b) and outputting the converted sequence data; A working level conversion program that acts as a. [Explanation of symbols]

[0077] 10...task level conversion device, 10a...CPU, 10b...RAM, 10c...ROM, 10d...communication device, 10e...input device, 10f...display device, 11...acquisition unit, 12...detection unit, 13...generation unit, 14...conversion output unit, 19...storage unit, 19a...video, 19b...target action information, 19c...task information, 19d...trained model, 20...image sensor, 100...system, A...worker, R...work area

Claims

1. a generation unit that generates sequence data of target action information in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action; a trained model that uses task information that associates lower-level task series data specified by a combination of the target task and the incidental information with higher-level task series data composed of a plurality of the lower-level tasks as teacher data, inputs sequence data of the target task information, and trains the trained model to convert the input sequence data of the target task information into the higher-level task series data and output the converted sequence data; Equipped with The trained model outputs the higher-level work sequence data, with start timing information indicating whether or not a higher-level work included in the higher-level work sequence data is started, set for each target action. Working level conversion device.

2. The supplementary information includes information that can be recognized from the time-series information and information that is annotated to the time-series information.

2. The working level conversion device according to claim 1.

3. 1. A processor-implemented method, comprising: generating sequence data of target action information in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action; using task information that associates lower-level task series data identified by a combination of the target task and the incidental information with higher-level task series data composed of a plurality of tasks at the lower level as teacher data, inputting the sequence data of the target task information, and converting the input sequence data of the target task information into the higher-level task series data and outputting the converted sequence data, thereby generating a trained model; Including, The trained model outputs the higher-level work sequence data, with start timing information indicating whether or not a higher-level work included in the higher-level work sequence data is started, set for each target action. How to convert work levels.

4. Computer, a generation unit that generates sequence data of target action information in which a target action detected from time-series information related to a task of a worker is associated with attendant information associated with the target action; a trained model that uses task information that associates lower-level task series data specified by a combination of the target task and the incidental information with higher-level task series data composed of a plurality of the lower-level tasks as teacher data, inputs the sequence data of the target task information, and trains the trained model to convert the input sequence data of the target task information into the higher-level task series data and output the converted sequence data; Function as a The trained model outputs the higher-level work sequence data, with start timing information indicating whether or not a higher-level work included in the higher-level work sequence data is started, set for each target action. Work level conversion program.

Images