WORK ESTIMATION DEVICE, CONTROL METHOD FOR WORK ESTIMATION DEVICE, INFORMATION PROCESSING PROGRAM, AND RECORDING MEDIUM

Abstract

To allow for estimating a content of a task being done on a target object by an operator from a captured image with sufficient accuracy.SOLUTION: An information processing device (10) provided herein is configured to verify a content of prediction made using a task prediction model (171), a trained model, by a content of prediction made without using the task prediction model (171).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a task estimation device that estimates the content of a task being performed by a worker in a workplace. [Background technology]

[0002] Conventionally, various attempts have been known to use imaging data capturing images of the implementation status of work processes at production sites such as factories to improve the work processes. For example, Patent Document 1 listed below discloses a safety management system that uses the analysis results of image data to determine the behavior of workers. Research is also being conducted on HOI detection, which detects actions performed by humans with respect to objects captured in the captured images as an interaction between the human and the object (HOI). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2018-173957 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is a problem in that the accuracy of work content prediction based on HOI detection alone is not sufficient.

[0005] An object of one aspect of the present invention is to provide an activity estimation device or the like that can estimate, with sufficient accuracy from a captured image, the content of an activity that a worker is performing on a work object. [Means for solving the problem]

[0006] In order to solve the above problems, an work estimation device according to one aspect of the present invention is a work estimation device that estimates content of work being performed by a worker in a workplace, and includes: a prediction unit that predicts content of work being performed by the worker on a work object imaged in the workplace image from the workplace image or the feature information using a work prediction model, which is a trained model that receives as input a workplace image captured of the workplace or feature information generated from the workplace image, and outputs content of work being performed by the worker imaged in the workplace image on a work object imaged in the workplace image; a determination unit that determines whether the prediction by the prediction unit is correct or not, using at least one of the work content predicted without using the work prediction model and a determination result of whether the worker is present in the workplace; and an estimation unit that, when the determination unit determines that the prediction by the prediction unit is correct, estimates the work content predicted by the prediction unit as the work content being performed by the worker in the workplace.

[0007] According to the above configuration, the work estimation device determines whether the work content predicted using the work prediction model is correct using at least one of the work content predicted without using the work prediction model and the determination result of the presence or absence of the worker in the workplace. Then, when it is determined that the prediction using the work prediction model is correct, the work estimation device estimates that the work content predicted using the work prediction model is the work content being performed by the worker in the workplace.

[0008] In other words, the work estimation device verifies the accuracy of the work content predicted using the work prediction model using at least one of the work content predicted without using the work prediction model and the determination result of whether or not the worker is present in the workplace.

[0009] Therefore, the task estimation device has the effect of being able to estimate with high accuracy the content of the task being performed by the worker in the workplace from the workplace image.

[0010] The work estimation device according to one embodiment of the present invention may further include a work detection unit that predicts content of the work being performed by the worker in the workplace based on detection information from a sensor that detects a tool used for the work, the work object, and at least one state of the workplace, and the determination unit may determine that the prediction by the prediction unit is correct when the work content predicted by the prediction unit matches the work content predicted by the work detection unit.

[0011] According to the above configuration, the work estimation device determines whether the work content predicted from the detection information of the sensor matches the work content predicted using the work prediction model. If the work estimation device determines that the two match, it estimates that the work content predicted using the work prediction model is the work content being performed by the worker in the workplace.

[0012] That is, the activity estimation device verifies whether the content of the activity predicted using the activity prediction model is correct or not, based on the content of the activity predicted from the detection information of the sensor.

[0013] Therefore, the task estimation device has the effect of being able to estimate with high accuracy the content of the task being performed by the worker in the workplace from the workplace image.

[0014] The work estimation device according to one aspect of the present invention may further include a presence / absence prediction unit that predicts whether the worker is present in the workplace from the overall image, or the workplace image, or the feature information generated from the overall image or the workplace image, using a presence / absence model that is a trained model that receives as input (A) an overall image of a plurality of workplaces including the workplace, or (B) the workplace image, or (C) feature information generated from the overall image or the workplace image, and outputs whether the worker is present in the workplace, and when the presence / absence prediction unit predicts that the worker is present in the workplace, the determination unit may determine that the prediction by the prediction unit is correct.

[0015] According to the above configuration, the work estimation device predicts whether the worker is present in the workplace using the presence / absence model. Then, when the work estimation device predicts that the worker is present in the workplace, it estimates that the work content predicted using the work prediction model is the work content being performed by the worker in the workplace.

[0016] In other words, the activity estimation device verifies the accuracy of the content of the activity predicted using the activity prediction model based on the presence or absence of the worker in the workplace predicted using the presence or absence model.

[0017] Therefore, the task estimation device has the effect of being able to estimate with high accuracy the content of the task being performed by the worker in the workplace from the workplace image.

[0018] The work estimation device according to one aspect of the present invention may further include a work detection unit that predicts content of the work being performed by the worker in the workplace from detection information from a sensor that detects at least one state of a tool used in the work, the work object, and the workplace, and a presence / absence prediction unit that predicts whether the worker is present in the workplace from the overall image, the workplace image, or feature information generated from the overall image or the workplace image using a presence / absence model that is a trained model that receives as input (A) an overall image of a plurality of workplaces including the workplace, or (B) the workplace image, or (C) feature information generated from the overall image or the workplace image, and outputs whether the worker is present in the workplace, and the determination unit may determine that the prediction by the prediction unit is correct when the presence / absence prediction unit predicts that the worker is present in the workplace and the content of the work predicted by the prediction unit matches the content of the work predicted by the work detection unit.

[0019] According to the above configuration, the work estimation device predicts the content of the work predicted from the detection information of the sensor and predicts the presence or absence of the worker in the workplace. Then, when the work estimation device predicts that the worker will be present in the workplace and the content predicted from the detection information of the sensor matches the content predicted using the work prediction model, the work estimation device determines that the content predicted using the work prediction model is correct.

[0020] In other words, the work estimation device verifies the accuracy of the work content predicted using the work prediction model based on the work content predicted from the detection information of the sensor and the presence or absence of the worker in the workplace predicted using the presence / absence model.

[0021] Therefore, the task estimation device has the effect of being able to estimate with high accuracy the content of the task being performed by the worker in the workplace from the workplace image.

[0022] In a work estimation device according to one aspect of the present invention, the work prediction model may be constructed by machine learning using a learning dataset in which the content of the work being performed by the worker in the workplace, predicted from detection information of a sensor that detects a tool used in the work, the work object, and at least one state of the workplace, is assigned as a label to an image of the workplace or feature information generated from the image of the workplace.

[0023] According to the above configuration, the work estimation device constructs the work prediction model by machine learning using a learning dataset to which “the content of the work being performed by the worker in the workplace” predicted from the detection information of the sensor is labeled.

[0024] It is known that supervised learning requires a large amount of training data (learning dataset). The labeling process required to generate such training data and the labels (metadata) attached to the training data are also called "annotations."

[0025] For example, even if the teacher data learned when constructing the work prediction model does not require the setting of a bounding box for detecting the worker, the following five annotations are required for the work place image. That is, it is necessary to set the X and Y coordinates of the lower left point and the upper right point of the four points that indicate the range of the bounding box for detecting the work object for the work place image, that is, four annotations are required. Furthermore, it is necessary to assign information (label) to the work place image that indicates "the content of the work that the worker captured in the work place image is performing on the work object captured in the work place image," that is, one annotation is required.

[0026] Furthermore, if we were to have a human determine "the content of the work that the worker captured in the workplace image is performing on the work object captured in the workplace image" for each of the large number of workplace images, the cost required to do so would be extremely high.

[0027] Therefore, when generating teacher data to be learned when constructing the task prediction model, the "task content" predicted from the detection information of the sensor, rather than the "task content" determined by a human from the workplace image, is assigned as a label indicating the "task content." In other words, in the teacher data to be learned when constructing the task prediction model, the "task content" predicted from the detection information of the sensor, rather than the "task content" determined by a human from the workplace image, is assigned as a label indicating the "task content."

[0028] Therefore, the cost required to construct the task prediction model (especially the annotation cost required to generate the learning dataset) can be significantly reduced compared to the case where "content determined by a human from the workplace image" is used as a label indicating the "task content." For example, one of the above-mentioned five annotations required for the workplace image is changed from "content determined by a human" to "content predicted by a device such as the task estimation device from the detection information of the sensor," thereby reducing the annotation cost by 20%.

[0029] Therefore, with the above configuration, the work estimation device has the effect of being able to estimate with high accuracy the content of the work being performed by the worker in the workplace from the workplace image, using the work prediction model in which annotation costs are significantly reduced.

[0030] In order to solve the above problems, a control method according to one aspect of the present invention is a control method for a work estimation device that estimates content of work being performed by a worker in a workplace, the control method including: a prediction step of predicting content of work being performed by the worker on a work object imaged in the workplace from the workplace image or the feature information using a work prediction model, which is a trained model that receives as input a workplace image captured of the workplace or feature information generated from the workplace image and outputs content of work being performed by the worker imaged in the workplace image on a work object imaged in the workplace image; a determination step of determining whether the prediction in the prediction step is correct or not using at least one of the work content predicted without using the work prediction model and a determination result of whether the worker is present in the workplace; and an estimation step of estimating, when it is determined in the determination step that the prediction in the prediction step is correct, the work content predicted in the prediction step as the work content being performed by the worker in the workplace.

[0031] According to the above configuration, the control method determines whether the work content predicted using the work prediction model is correct or not using at least one of the work content predicted without using the work prediction model and the determination result of the presence or absence of the worker in the workplace. Then, when it is determined that the prediction using the work prediction model is correct, the control method estimates that the work content predicted using the work prediction model is the work content being performed by the worker in the workplace.

[0032] In other words, the control method verifies the accuracy of the work content predicted using the work prediction model using at least one of the work content predicted without using the work prediction model and the determination result of whether or not the worker is present in the workplace.

[0033] Therefore, the control method has the effect of being able to estimate with high accuracy the content of the work being performed by the worker in the workplace from the workplace image. Effect of the Invention

[0034] Advantageous Effects of Invention According to one aspect of the present invention, it is possible to estimate with sufficient accuracy the content of the work being performed by a worker on a work object from a captured image. [Brief description of the drawings]

[0035] [Figure 1] 1 is a block diagram showing a configuration of a main part of an information processing device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an overall outline of a control system including the information processing device of FIG. 1. [Diagram 3] 10 is a flow diagram illustrating an overall overview of processing executed by the information processing device in FIG. 1 when verifying the first prediction using only the work prediction of the second prediction. [Figure 4] 10 is a flow diagram illustrating an overall overview of processing executed by the information processing device in FIG. 1 when the first prediction is verified using only the presence / absence prediction of the second prediction. [Diagram 5] 10 is a flow diagram illustrating an overall outline of processing executed by the information processing device of FIG. 1 when a first prediction is verified using both presence / absence prediction and work prediction. [Figure 6] FIG. 13 is a diagram illustrating an example of annotations required for generating teacher data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] [Embodiment 1] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. Note that the same or corresponding parts in the drawings are given the same reference characters and their description will not be repeated. In the present embodiment, an information processing device 10 will be described as a typical example of "an operation estimation device that estimates the content of an operation Op being performed by a worker Hu at a work site Ar."

[0037] In the following description, "n", "m", "p", "q", "x", and "y" each represent an integer greater than or equal to 1, with "p" and "q" being different integers and "x" and "y" being different integers.

[0038] § 1. Examples of Application To facilitate understanding of an information processing device 10 (task estimation device) according to an embodiment of the present invention, first, an environment to which the information processing device 10 is applied will be described. For example, the information processing device 10 uses an analysis result of a work place image Ic obtained by capturing an image of a work place Ar to generate information (support information) that contributes to the efficiency of a work Op performed by a worker Hu on a work object Ob. First, a worker Hu, a work object Ob, a work Op, a work place Ar, and a factory Fa including one or more work places Ar will be described.

[0039] (About the factory and work process) In a factory Fa, which is a production site, for example, various products are produced through one or more work processes Pr(1), Pr(2), Pr(3), ... Pr(n). Each of the multiple work processes Pr(1), Pr(2), Pr(3), ... Pr(n) carried out in the factory Fa is, for example, a "painting" process, a "main work assembly" process, a "installation of the main work into the body" process, and an "inspection" process.

[0040] When it is necessary to distinguish between multiple factories Fa, they will be distinguished by adding suffixes such as "(A)", "(B)", "(C)", ..., and "(Z)" to the reference numbers. For example, they will be distinguished by writing "Factory Fa(A)", "Factory Fa(B)", "Factory Fa(X)", ..., and "Factory Fa(Z)". When there is no particular need to distinguish between each of the multiple factories Fa, they will simply be referred to as "Factory Fa".

[0041] In addition, when it is necessary to distinguish between multiple work processes Pr, they are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., and "(n)" to the reference numerals. For example, they are distinguished by notation such as "work process Pr(1)", "work process Pr(2)", "work process Pr(3)", ..., and "work process Pr(n)". When there is no particular need to distinguish between multiple work processes Pr, they are simply referred to as "work process Pr".

[0042] (About the work) In a work process Pr, a worker Hu performs one or more operations Op(1), Op(2), Op(3), ... Op(n). For example, in a work process Pr(x), a worker Hu performs operations Op(x1), Op(x2), and Op(x3). As an example, when a work process Pr(m) is a "painting" process, the operation Op(x1) may be "holding" the work object Ob, the operation Op(x2) may be "painting" the work object Ob, and the operation Op(x3) may be "transporting" the work object Ob.

[0043] Regarding an operation Op, when it is necessary to distinguish between multiple operations Ops, they will be distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the symbol, and when there is no particular need to distinguish between them, they will be simply referred to as "operation Op."

[0044] (About the workplace) The factory Fa includes one or more work areas Ar(1), Ar(2), Ar(3), ... Ar(n). The work area Ar is also called a "cell", that is, the factory Fa includes one or more cells. Each of the one or more work areas Ar(1), Ar(2), Ar(3), ... Ar(n) corresponds to one or more operation processes Pr(1), Pr(2), Pr(3), ... Pr(n). In other words, the work area Ar(m) is an area where the worker Hu performs the operation Op(m) included in the operation process Pr(m).

[0045] Regarding workplaces Ar, when it is necessary to distinguish between multiple workplaces Ar, they will be distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they will be simply referred to as "workplaces Ar".

[0046] (About the work object) In the workplace Ar, one or more work objects Ob are placed, which are the targets of work Op performed by a worker Hu. For example, in the workplace Ar(x), work objects Ob(x1), Ob(x2), Ob(x3), ..., Ob(xn) are placed.

[0047] The work target Ob is an entity other than the worker Hu, for example, a so-called "workpiece," or, for example, a tool used by the worker Hu when performing the work Op.

[0048] Regarding work objects Ob, when it is necessary to distinguish between multiple work objects Ob, they will be distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numeral, and when there is no particular need to distinguish between them, they will be simply referred to as "work objects Ob".

[0049] (About the workers) In the factory Fa, there are one or more workers Hu who perform the work Op included in the work process Pr, for example, workers Hu(1), Hu(2), Hu(3), ..., Hu(n). The workers Hu may be identified, for example, by a worker ID attached to the top of the hat worn by the worker Hu. Specifically, the workers Hu(1) and Hu(2) in the factory Fa may be identified by the worker ID(1) attached to the hat worn by the worker Hu(1) and the worker ID(2) attached to the hat worn by the worker Hu(2), respectively. In addition, a certain worker Hu(1) may be regarded as a work object Ob for another worker Hu(2).

[0050] Regarding workers Hu, when it is necessary to distinguish between multiple workers Hu, they will be distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they will be simply referred to as "worker Hu."

[0051] (About the equipment and operation) The factory Fa includes one or more devices 40(1), 40(2), 40(3), ... 40(n). The devices 40 include an input device 41 such as a sensor, and an output device 42 such as an actuator.

[0052] The output device 42 is a device whose output is controlled by the PLC 20. When performing an operation Op, the worker Hu may use the output of the output device 42. Specifically, the worker Hu may perform the operation Op by using an action Ac executed by the output device 42. That is, when performing the operation Op, the worker Hu may use the output device 42 as a tool. In other words, the output device 42 may be a tool used by the worker Hu when performing the operation Op. For example, when the worker Hu performs an operation Op(x) included in a work process Pr(x), the output device 42(x) may be used. Specifically, the worker Hu may perform the operation Op(x) by using an action Ac(x) executed by the output device 42(x).

[0053] The input device 41 is a sensor (detection device) that detects at least one state of the tools (e.g., the output device 42 and a tool 90 other than the output device 42 described later) used in the work Op, the work object Ob, and the work area Ar. For example, the input device 41 detects the power consumption of the tools, the gripping of the tools, work object Ob, etc. by the worker Hu, the pressing of a button provided on the tools, work area Ar, work object Ob, etc. by the worker Hu, the opening and closing of an opening / closing mechanism provided on the tools, work area Ar, work object Ob, etc.

[0054] Specifically, the input device 41 may be, for example, a current detection sensor, and may detect a current flowing through an "instrument 90(x): electric screwdriver" used by the worker Hu when performing an "operation Op(x): screw tightening operation." The input device 41 may also be, for example, a grip sensor, and may detect that the "instrument 90(x): electric screwdriver" is gripped by the worker Hu. The input device 41 may also be, for example, a start / end button, and may detect pressing of the button by the worker Hu. The input device 41 may also be, for example, a photoelectric sensor, and may detect opening and closing of an opening / closing mechanism provided in the tool, work area Ar, work object Ob, etc.

[0055] The input device 41 periodically outputs information indicating the detected state (hereinafter also referred to as "detection information Id") to the PLC 20 via the control network 50. In particular, the input device 41 periodically outputs to the PLC 20 the detection information Id including "the time at which at least one state of the tool used in the work Op, the work target Ob, and the work area Ar was detected (detection time Td)."

[0056] Regarding the devices 40, when it is necessary to distinguish between the multiple devices 40, they are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they are simply referred to as "devices 40". Similarly, regarding the input devices 41, when it is necessary to distinguish between the multiple input devices 41, they are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they are simply referred to as "input devices 41". Furthermore, regarding the output devices 42, when it is necessary to distinguish between the multiple output devices 42, they are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they are simply referred to as "output devices 42".

[0057] Furthermore, with regard to operations Ac, when it is necessary to distinguish between multiple operations Ac, they will be distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the symbol, and when there is no particular need to distinguish between them, they will be simply referred to as "operations Ac."

[0058] In addition, the detection information Id indicating "the state of at least one of the tool (e.g., the output device 42(x) or the tool 90(x)) used in the work Op(x), the work object Ob(x), and the work area Ar(x)" may be specifically referred to as "detection information Id(x)."

[0059] (About tools) The factory Fa may include, in addition to the output device 42, a tool 90 (not shown) other than the output device 42. When performing an operation Op, the worker Hu may use the tool 90. Specifically, the worker Hu may perform the operation Op by using an action Ac performed by the tool 90. For example, when the worker Hu performs an operation Op(x) included in a work process Pr(x), the tool 90(x) may be used. Specifically, the worker Hu may perform the operation Op(x) by using an action Ac(x) performed by the tool 90(x). As an example, the worker Hu may perform "operation Op(x): screw tightening operation" by using "action Ac(x): rotation operation" performed by "tool 90(x): electric screwdriver".

[0060] As described above, the state of the appliance 90 (for example, a state in which the appliance is being used by the worker Hu) is detected by the input device 41, and the input device 41 periodically outputs the detected "status of the appliance 90" to the PLC 20 via the control network 50. That is, when the input device 41 detects the "status of the appliance 90", it periodically outputs detection information Id including the detected "status of the appliance 90" and the time (detection time Td) at which the "status of the appliance 90" was detected to the PLC 20.

[0061] (About the workshop images) The work place image Ic is a captured image of the work place Ar, and each of the multiple work place images Ic is a captured image of each of the multiple work places Ar. For example, the work place image Ic is a captured image of a worker Hu performing a task Op in the work place Ar from substantially the front, and the work place image Ic is a captured image generated for each work place Ar.

[0062] The multiple work place images Ic may be distinguished from each other by the work place Ar captured in each of the work place images Ic. For example, the multiple work place images Ic may be distinguished from each other by associating each of the multiple work place images Ic with the work place Ar captured in each of the work place images Ic.

[0063] That is, the work place image Ic captured in the work place Ar(1) may be referred to as work place image Ic(1), and the work place image Ic captured in the work place Ar(2) may be referred to as work place image Ic(2). Similarly, the work place image Ic captured in the work place Ar(n) may be referred to as work place image Ic(n).

[0064] When there is no need to distinguish the multiple work place images Ic from one another based on the work place Ar captured in each of the work place images Ic, they are simply referred to as "work place images Ic."

[0065] (About the whole image) While the work area image Ic is an image of each work area Ar, the overall image Ia is an image in which multiple work areas Ar are captured, for example, an image of the entire factory Fa as viewed down from the ceiling of the factory Fa.

[0066] First, an overview of a control system 1 including an information processing device 10 will be described below with reference to FIG.

[0067] (Overview of the system including the information processing device) 2 is a diagram showing an overall overview of the control system 1 and the like including the information processing device 10. In the environment described above, the information processing device 10 estimates with high accuracy, from a workplace image Ic obtained by capturing an image of the workplace Ar, the content of an operation Op being performed by a worker Hu captured in the workplace image Ic.

[0068] The cell camera 30 is installed in a workplace Ar (i.e., a cell) and generates a workplace image Ic. For example, a cell camera 30 is installed for each workplace Ar, and each of the multiple cell cameras 30 is installed for each workplace Ar at a position where it can capture an image of a worker Hu performing a task Op in the workplace Ar from approximately the front. The cell camera 30 outputs the generated workplace image Ic to the information processing device 10, and in particular outputs the workplace image Ic to the information processing device 10 together with the image capturing time Tp, which is the time when the workplace Ar is captured.

[0069] When distinguishing between the cell cameras 30 installed in each work area Ar in the factory Fa, each of the cell cameras 30 may be distinguished by associating each of the cell cameras 30 with the work area Ar in which the cell camera 30 is installed. For example, the cell camera 30 installed in the work area Ar(1) may be described as cell camera 30(1), and the cell camera 30 installed in the work area Ar(2) may be described as cell camera 30(2). Similarly, the cell camera 30 installed in the work area Ar(n) may be described as cell camera 30(n).

[0070] The ceiling camera 80 is installed, for example, on the ceiling of the factory Fa, and generates an overall image Ia looking down diagonally from above on multiple work areas Ar, for example, an overall image Ia capturing the entire factory Fa is generated by a single ceiling camera 80. However, it is not essential that a single ceiling camera 80 is installed on the ceiling of the factory Fa, and each of multiple ceiling cameras 80 installed on the ceiling of the factory Fa may generate an overall image Ia capturing multiple work areas Ar in the factory Fa. In other words, it is sufficient for the ceiling camera 80 to be able to generate an overall image Ia capturing multiple work areas Ar.

[0071] In the following description, an example will be described in which one ceiling camera 80, which is a wide-area imaging camera, is installed on the ceiling of a factory Fa, and an overall image Ia is generated by capturing an image of the entire factory Fa from the ceiling camera 80. The ceiling camera 80 outputs the generated overall image Ia to the information processing device 10, and in particular outputs the overall image Ia to the information processing device 10 together with the overall time Ta, which is the time when the entire factory Fa was captured (i.e., the time when the overall image Ia was generated).

[0072] (Prediction of work content through image analysis of workplace images) The information processing device 10 is included in the control system 1 and is communicatively connected to the cell camera 30 via, for example, a USB (Universal Serial Bus) cable. The information processing device 10 acquires a workplace image Ic of a workplace Ar from the cell camera 30, performs image analysis on the acquired workplace image Ic, and predicts the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob. The information processing device 10 predicts "the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob" from the workplace image Ic using a HOI (Human Object Interaction) detection algorithm. In particular, the information processing device 10 acquires from the cell camera 30 the workplace image Ic and the imaging time Tp, which is the time when the workplace image Ic was generated, and performs the following predictions from the acquired workplace image Ic and the imaging time Tp. That is, the information processing device 10 predicts "the content of the work Op being performed by the worker Hu on the work object Ob in the work place Ar at the imaging time Tp" The "content of the work Op being performed by the worker Hu on the work object Ob in the work place Ar at the imaging time Tp" predicted from the work place image Ic, particularly the "content of the work Op being performed by the worker Hu on the work object Ob in the work place Ar at the imaging time Tp" is also referred to as a first prediction Pf.

[0073] For example, the information processing device 10 performs image analysis on a workplace image Ic(x) obtained by capturing an image of the workplace Ar(x) to predict the content of an operation Op(x) being performed by a worker Hu(x) on a work object Ob(x) in the workplace Ar(x). In particular, the information processing device 10 predicts, from the workplace image Ic(x), "the content of an operation Op(x) being performed by a worker Hu(x) on a work object Ob(x) in the workplace Ar(x) at the capturing time Tp when the workplace image Ic(x) is captured." The "content of an operation Op(x) being performed by a worker Hu(x) on a work object Ob(x) in the workplace Ar(x) at the capturing time Tp when the workplace image Ic(x) is captured," predicted from the workplace image Ic(x), may be particularly referred to as a "first prediction Pf(x)."

[0074] (Prediction of worker presence by image analysis of the entire image) The information processing device 10 is included in the control system 1 and is communicatively connected to the ceiling camera 80 via, for example, a USB (Universal Serial Bus) cable. The information processing device 10 acquires an overall image Ia captured from the ceiling camera 80 overlooking the entire factory Fa, and performs image analysis on the acquired overall image Ia to predict whether the worker Hu is in the work area Ar. The information processing device 10 predicts, from the overall image Ia, "whether the worker Hu is in the work area Ar captured in the overall image Ia" using a movement line detection algorithm (object detection algorithm) such as YOLO. In particular, the information processing device 10 acquires, from the ceiling camera 80, the overall image Ia and the overall time Ta, which is the time when the overall image Ia was generated, and executes the following prediction from the acquired overall image Ia and overall time Ta. That is, the information processing device 10 predicts "whether the worker Hu is in the work area Ar (particularly, the work area Ar captured in the work area image Ic) at the overall time Ta".

[0075] Therefore, the information processing device 10 can predict "whether or not a worker Hu is present in the workplace Ar (particularly, the workplace Ar captured in the workplace image Ic) at the total time Ta corresponding to the imaging time Tp".

[0076] The prediction of "whether or not the worker Hu is in the workplace Ar" from the overall image Ia, and in particular the prediction of "whether or not the worker Hu is in the workplace Ar at the overall time Ta corresponding to the imaging time Tp", is also referred to as presence / absence prediction Pa.

[0077] For example, the information processing device 10 predicts "whether a worker Hu(x) is in the workplace Ar(x) captured in the workplace image Ic(x) generated by the cell camera 30(x)" by image analysis of the overall image Ia generated by the ceiling camera 80. In particular, the information processing device 10 predicts "whether a worker Hu(x) is in the workplace Ar(x) at the overall time Ta corresponding to the 'imaging time Tp at which the workplace image Ic(x) is captured'" from the overall image Ia. The prediction of "whether a worker Hu(x) is in the workplace Ar(x) at the overall time Ta corresponding to the 'imaging time Tp at which the workplace image Ic(x) is captured'" predicted from the overall image Ia may be particularly referred to as "presence prediction Pa(x)".

[0078] (Prediction of work content using detection information) Furthermore, the information processing device 10 acquires, from a PLC (Programmable Logic Controller) 20, process information Ip generated from an operation result La indicating the content and result of the operation Ac executed by an apparatus 40 whose operation Ac is controlled by the PLC 20. The process information Ip includes, for example, at least one of "the time when the apparatus 40 executed the operation Ac" and "the time when the execution of the operation Ac by the apparatus 40 was detected by the input device 41 or the like" as a detection time Td.

[0079] The information processing device 10 grasps the state of the output device 42 (e.g., an operation state including the content and result of the operation Ac) from the process information Ip acquired from the PLC 20, and predicts the content of the work Op being performed by the worker Hu using the output device 42 from the grasped state. In particular, the information processing device 10 grasps the "state of the output device 42 at 'detection time Td corresponding to the imaging time Tp'" from the process information Ip including the detection time Td. Then, the information processing device 10 predicts the "content of the work Op being performed by the worker Hu using the output device 42 at 'detection time Td corresponding to the imaging time Tp'" from the grasped "state of the output device 42 at 'detection time Td corresponding to the imaging time Tp'".

[0080] However, the PLC 20 may grasp the state of the output device 42 from the output of the equipment 40, etc., and predict the content of the work Op being performed by the worker Hu using the output device 42 from the grasped state. Then, the PLC 20 may output process information Ip including the predicted content of the work Op to the information processing device 10. In other words, it may be the PLC 20 or the information processing device 10 that predicts the content of the work Op being performed by the worker Hu using the output device 42 from the state of the output device 42. It is sufficient for the information processing device 10 to acquire or generate "the content of the work Op being performed by the worker Hu using the output device 42" predicted from the state of the output device 42.

[0081] Here, the process information Ip is information indicating the state of the output device 42 and the like, which is generated from "the output of the equipment 40 controlled by the PLC 20 and the control contents of the output device 42 by the PLC 20" and the like. Therefore, the PLC 20 can also be regarded as a detection device that detects the state of the output device 42 from "the output of the output device 42 obtained as feedback from the output device 42 and the control contents of the output device 42" and the like. In addition, the process information Ip can also be regarded as a type of "detection information Id" which is information indicating "at least one state of the tools (e.g., the output device 42 and the tool 90) used in the work Op, the work target Ob, and the work area Ar."

[0082] Furthermore, the information processing device 10 acquires detection information Id from the PLC 20, which indicates "at least one state of the tools (e.g., the output device 42 and the implement 90) used in the work Op, the work object Ob, and the work area Ar" detected by the input device 41.

[0083] The information processing device 10 may obtain the detection information Id indicating the state detected by the input device 41 directly from the input device 41 without going through the PLC 20.

[0084] The detection information Id includes a detection time Td, which is the time when the input device 41 detects "at least one state of the tool used in the work Op, the work target Ob, and the work area Ar."

[0085] The information processing device 10 grasps the state of at least one of the tools used in the work Op, the work object Ob, and the work area Ar based on the detection information Id acquired from the PLC 20, and predicts the content of the work Op being performed by the worker Hu from the grasped states.

[0086] However, it may be the PLC 20 that grasps the state of at least one of the tool, the work object Ob, and the work area Ar using the detection information Id acquired from the input device 41 and predicts the content of the work Op being performed by the worker Hu from the grasped state. Then, the PLC 20 may output the process information Ip including the predicted content of the work Op to the information processing device 10. In other words, it may be the PLC 20 or the information processing device 10 that predicts the content of the work Op being performed by the worker Hu from the state of at least one of the tool, the work object Ob, and the work area Ar. It is sufficient for the information processing device 10 to acquire or generate "the content of the work Op being performed by the worker Hu" predicted from the state of at least one of the tool, the work object Ob, and the work area Ar.

[0087] That is, the information processing device 10 acquires from the PLC 20 the detection information Id (including the process information Ip) indicating "at least one state of the tool used in the work Op, the work object Ob, and the work area Ar." The information processing device 10 predicts "the content of the work Op being performed by the worker Hu" from the acquired detection information Id, and in particular, predicts "the content of the work Op being performed by the worker Hu at the 'detection time Td corresponding to the imaging time Tp'." However, the information processing device 10 may acquire from the PLC 20 the "content of the work Op being performed by the worker Hu" predicted from the detection information Id by the PLC 20 together with the detection information Id or instead of the detection information Id. For example, the information processing device 10 may acquire from the PLC 20 the "content of the work Op being performed by the worker Hu at the 'detection time Td corresponding to the imaging time Tp'" together with the detection information Id or instead of the detection information Id.

[0088] The "content of the work Op being performed by the worker Hu" predicted from the detection information Id (including the process information Ip), and in particular the "content of the work Op being performed by the worker Hu at the 'detection time Td corresponding to the imaging time Tp'", is also referred to as the work prediction Po.

[0089] For example, the information processing device 10 predicts the content of the work Op(x) being performed by a worker Hu(x) in a work site Ar(x) from the following detection information Id(x) (including process information Ip(x)). That is, the information processing device 10 predicts the content of the work Op(x) from the detection information Id(x) indicating "the tools provided in the work site Ar(x), the work object Ob(x) placed in the work site Ar(x), and at least one state of the work site Ar(x)." The "tools provided in the work site Ar(x) (for example, at least one of the output device 42(x) and the appliance 90(x))" are the tools used by the worker Hu(x) when performing the work Op(x). In particular, the information processing device 10 predicts "the content of the work Op(x) being performed by the worker Hu(x) on the work object Ob(x) in the workplace Ar(x) at 'the detection time Td corresponding to the imaging time Tp'" from the process information Ip(x). The "content of the work Op(x) being performed by the worker Hu(x) at 'the detection time Td corresponding to the imaging time Tp'" predicted from the detection information Id(x) (including the process information Ip(x)) may be referred to as "work prediction Po(x)."

[0090] (Verifying predicted work content and outputting estimated results) The information processing device 10 verifies the "contents of the work Op being performed by the worker Hu on the work object Ob" predicted from the workplace image Ic, using at least one of the following two predictions. That is, the information processing device 10 verifies the contents predicted from the workplace image Ic, using at least one of the "presence or absence of the worker Hu" predicted from the overall image Ia, and the "contents of the work Op being performed by the worker Hu" predicted from the detection information Id. That is, the information processing device 10 verifies the accuracy of the first prediction Pf, using at least one of the presence or absence prediction Pa and the work prediction Po, and in particular, verifies the accuracy of the first prediction Pf(x), using at least one of the presence or absence prediction Pa(x) and the work prediction Po(x).

[0091] The first prediction Pf(x) is a prediction of "the content of the work Op(x) being performed by a worker Hu(x) on a work object Ob(x) in a workplace Ar(x)" predicted from the workplace image Ic(x) by the information processing device 10. The "first prediction Pf(x)" is, for example, "the content of the work Op(x) being performed by a worker Hu(x) on a work object Ob(x) in a workplace Ar(x) at an imaging time Tp when the workplace image Ic(x) is captured" predicted from the workplace image Ic(x).

[0092] The presence / absence prediction Pa(x) is a prediction of "whether or not a worker Hu is present in the workplace Ar(x)" predicted from the overall image Ia by the information processing device 10. The presence / absence prediction Pa(x) is, for example, a prediction of "whether a worker Hu(x) is in the workplace Ar(x) at the overall time Ta corresponding to 'the imaging time Tp at which the workplace image Ic(x) is captured'" predicted from the overall image Ia.

[0093] The work prediction Po(x) is a prediction of "the content of work Op(x) being performed by a worker Hu(x) at a workplace Ar(x)" predicted from the detection information Id(x) by the information processing device 10. The work prediction Po(x) is, for example, "the content of work Op(x) being performed by a worker Hu(x) at a workplace Ar(x) at 'the detection time Td corresponding to the imaging time Tp'" predicted from the detection information Id(x).

[0094] For example, the information processing device 10 (particularly, the determination unit 150 described later) determines that "the first prediction Pf(x) is correct" when the presence / absence prediction Pa(x) predicts that "the worker Hu is present in the work area Ar(x)". More specifically, the information processing device 10 determines that "the first prediction Pf(x) is correct" when the presence / absence prediction Pa(x) predicts that "the worker Hu(x) is in the work area Ar(x) at 'the total time Ta corresponding to the imaging time Tp'". Then, the information processing device 10 (particularly, the estimation unit 160 described later) outputs the first prediction Pf(x) determined to be correct by the determination unit 150 to the outside as an estimation result.

[0095] For example, when the work Op(x) in the work prediction Po(x) and the work Op(x) in the first prediction Pf(x) match, the information processing device 10 (particularly, the determination unit 150 described later) determines that "the first prediction Pf(x) is correct." More specifically, the information processing device 10 checks whether or not "the content of the work Op(x) at the 'detection time Td corresponding to the imaging time Tp'" in the presence / absence prediction Pa(x) matches with "the content of the work Op(x) at the imaging time Tp" in the first prediction Pf(x). Then, when the information processing device 10 confirms that the two match, it determines that "the first prediction Pf(x) is correct." Then, the information processing device 10 (particularly, the estimation unit 160 described later) outputs the first prediction Pf(x) determined to be correct by the determination unit 150 to the outside as an estimation result.

[0096] For example, when the presence / absence prediction Pa(x) predicts that "the worker Hu is present in the work place Ar(x)", the information processing device 10 (particularly, the determination unit 150 described later) checks whether the work prediction Po(x) and the first prediction Pf(x) match. More specifically, when the presence / absence prediction Pa(x) predicts that "the worker Hu(x) is in the work place Ar(x) at 'total time Ta corresponding to the imaging time Tp'", the information processing device 10 checks whether the work prediction Po(x) and the first prediction Pf(x) match. When the information processing device 10 checks whether the "content of the work Op(x) at the detection time Td corresponding to the imaging time Tp" in the presence / absence prediction Pa(x) matches with the "content of the work Op(x) at the imaging time Tp" in the first prediction Pf(x), the information processing device 10 determines that the first prediction Pf(x) is correct. Then, the information processing device 10 (particularly the estimation unit 160 described later) outputs the first prediction Pf(x) determined to be correct by the determination unit 150 to the outside as an estimation result.

[0097] (Master-slave control system) In the control system 1, the devices 40 used to perform the work process Pr in the factory Fa are controlled by the PLC 20 as a line controller. That is, the control system 1 is constructed as a master-slave control system in which the PLC 20 is the master and each of the devices 40 is a slave, and each of the devices 40 is communicatively connected to the PLC 20 via a network (control network 50). The PLC 20 is called a "master" in the sense that it manages data transmission via the control network 50. The terms "master" and "slave" are defined with a focus on the control function of data transmission on the control network 50, and there are no particular limitations on the type of information transmitted and received between the devices.

[0098] The PLC 20 is a control device (controller) that controls the entire control system 1, and is connected to each of the multiple devices 40 so as to be able to communicate with them. The PLC 20 acquires information from each of the multiple devices 40 as an input device 41 (measurement device) as input data. The PLC 20 executes arithmetic processing using the acquired input data according to a user program previously installed. The PLC 20 executes the arithmetic processing to determine control content for the control system 1, and for example, determines control content for each of the multiple devices 40 as an output device 42 such as an actuator. Then, the PLC 20 outputs control data corresponding to the determined control content to each of the multiple devices 40 (particularly, the output device 42). The PLC 20 repeatedly acquires input data from each of the multiple devices 40 (specifically, the input device 41) and acquires control data for each of the multiple devices 40 (specifically, the output device 42) at a predetermined period (control period). For example, a display unit and an operation unit (not shown) may be connected to the PLC 20. The display unit is composed of a liquid crystal panel capable of displaying images, and the operation unit is typically composed of a touch panel, a keyboard, a mouse, and the like.

[0099] The device 40 is a slave in the control system 1, which is a master-slave control system in which the PLC 20 is the master. The device 40 is an input device 41 that repeatedly transmits input data to the PLC 20 at every predetermined control period, or an output device 42 that repeatedly receives control data from the PLC 20 at every predetermined control period and operates according to the received control data.

[0100] The equipment 40 (particularly, the input device 41) is, for example, a sensor that transmits detection results or the like as input data to the PLC 20. For example, the input device 41 may be a photoelectric sensor, a barcode reader that transmits read results, or an inspection machine (tester) that transmits inspection results.

[0101] The device 40 (specifically, the output device 42) may be a robot or the like serving as an output device that performs screw tightening, picking, and the like, or may be various types of actuators.

[0102] Furthermore, the device 40 may be a PT (Programmable Terminal) to which a plurality of input devices 41 are connected.

[0103] The control network 50 transmits various data received by the PLC 20 or transmitted by the PLC 20, and typically may use various types of industrial Ethernet (registered trademark), and may also be called a field network. Known examples of industrial Ethernet (registered trademark) include EtherCAT (registered trademark), Profinet IRT, MECHATROLINK (registered trademark)-III, Powerlink, SERCOS (registered trademark)-III, and CIP Motion, and any of these may be used. Furthermore, a field network other than the industrial Ethernet (registered trademark) may also be used. For example, if motion control is not performed, DeviceNet, CompoNet / IP (registered trademark), etc. may be used.

[0104] In this embodiment, a control system 1 will be described in which data is transmitted and received between a PLC 20 (master) and a device 40 (slave) by sequentially transferring data frames on a control network 50. That is, data is repeatedly transmitted and received between the PLC 20 and the device 40 for each control period by sequentially transferring data frames on the control network 50 at a predetermined control period. Data may be transmitted and received between a plurality of devices 40, i.e., a plurality of slaves, by sequentially transferring data frames on the control network 50.

[0105] The information processing device 10 is included in the control system 1, but the information processing device 10 does not have to be a slave in the control system 1.

[0106] In the control system 1 which is a master-slave control system in which the device 40 is a slave, the PLC 20 which is the master receives the operation result La and the detection information Id from the device 40 which is a slave, for example, repeatedly at every predetermined control period.

[0107] (Regarding process information) In the control system 1, which is a master-slave control system in which the device 40 is a slave, the PLC 20, which is the master, receives an operation result La from the device 40, which is a slave, for example, repeatedly at a predetermined control period. The operation result La is information indicating "the content and result of the operation Ac performed by the device 40." That is, the device 40 transmits the operation result La, which indicates the content and result of the operation Ac actually performed when performing the work process Pr, to the PLC 20 at a predetermined period. For example, the device 40(x) transmits the operation result La(x), which indicates the content and result of the operation Ac(x) performed when performing the work process Pr(x), to the PLC 20 at a control period.

[0108] The PLC 20 acquires, for example, a measurement result that is a result of a measurement operation performed by the device 40 as the input device 41 (measurement device) as the operation result La of the device 40. Furthermore, if the device 40 is an inspection machine, the PLC 20 acquires a result of an inspection operation performed by the device 40, for example, an inspection result such as "whether or not the inspection standard was met," as the operation result La of the device 40. Furthermore, the PLC 20 acquires, for example, a result of an output operation performed by the device 40 as the output device 42 as the operation result La of the device 40. If the device 40 is a robot that performs screw tightening, picking, etc., the PLC 20 acquires, for example, the operation result La of the number of screw tightenings, the picking result (picking success or picking error), etc. as the operation result La of the device 40.

[0109] The PLC 20 repeatedly receives, at a predetermined cycle, from the device 40, operation results La indicating the contents and results of operations Ac actually performed by the device 40 when carrying out the work process Pr, and transmits, i.e., transfers, the received operation results La to the information processing device 10 as process information Ip. In addition, the PLC 20 transmits information generated using the operation results La repeatedly received at a predetermined cycle from the device 40 to the information processing device 10 as the process information Ip.

[0110] Furthermore, the PLC 20 may transmit the operation result La repeatedly received from the device 40 at a predetermined cycle as process information Ip to the outside of the control system 1. For example, the PLC 20 may transmit the operation result La repeatedly received from the device 40 at a predetermined cycle as process information Ip to an in-house LAN (Local Area Network) shown in FIG. 2 connected to an MES (Manufacturing Execution System) or the like.

[0111] The information processing device 10 identifies the operation start time Tms, operation completion time Tme, and operation period Da of the operation Ac performed by the device 40 when carrying out the work process Pr, from the process information Ip (operation result La) acquired from the PLC 20. Then, the information processing device 10 executes various judgments on the operation Ac using the identified operation start time Tms, operation completion time Tme, operation period Da, and a predetermined operation criterion Sa corresponding to the operation Ac.

[0112] However, the above-mentioned judgment on the operation result La (particularly, the operation Ac) included in the process information Ip may be performed by PLC 20, and PLC 20 may transmit the result of the above-mentioned judgment to the information processing device 10 by including it in the process information Ip or instead of the process information Ip.

[0113] In the above description, the operation start time Tms is the time when the device 40 used in the work process Pr starts to execute the operation Ac when performing the work process Pr, and the operation completion time Tme is the time when the device 40 completes the execution of the operation Ac. The operation period Da is the period from the operation start time Tms to the operation completion time Tme.

[0114] As described above, the process information Ip can also be considered as information indicating "the state of the tools (particularly, the output device 42) used in the work Op." The process information Ip can also be considered as information indicating, for example, "the state of the tools (particularly, the output device 42) used in the work Op at 'the detection time Td corresponding to the imaging time Tp.'" Therefore, the process information Ip can be understood as a type of detection information Id indicating "at least one state of the tools (e.g., the output device 42 and the tool 90) used in the work Op, the work object Ob, and the work area Ar."

[0115] (Regarding detection information) In the control system 1, the PLC 20 receives detection information Id from the slave device 40 (particularly, the input device 41) repeatedly, for example, at a predetermined control period. The detection information Id is information indicating "at least one state of the tool used in the work Op (e.g., the output device 42 and a tool 90 other than the output device 42 described later), the work target Ob, and the work area Ar" detected by the input device 41. That is, the device 40 (particularly, the input device 41) transmits "at least one state of the tool used in the work Op by the worker Hu, the work target Ob, and the work area Ar" to the PLC 20 repeatedly at a predetermined period. The detection information Id includes information indicating the time (i.e., the detection time Td) at which the input device 41 detected "at least one state of the tool used in the work Op, the work target Ob, and the work area Ar."

[0116] The PLC 20 acquires, for example, information indicating the magnitude of the current flowing through the appliance 90(x) (such as the power consumption of the appliance 90(x)) from the equipment 40(y) serving as the input device 41(y) as detection information Id(x) about the appliance 90(x).

[0117] The PLC 20 acquires, for example, from the equipment 40(y) serving as the input device 41(y), information indicating whether the tool 90(x) is held by the worker Hu, as detection information Id(x) about the tool 90(x).

[0118] The PLC 20 acquires, for example, information indicating the opening and closing of an opening / closing mechanism provided in the work object Ob(x) from the equipment 40(y) serving as the input device 41(y), as detection information Id(x) about the work object Ob(x).

[0119] The PLC 20 acquires, for example, information indicating whether a button on the work object Ob(x) has been pressed from an apparatus 40(y) serving as an input device 41(y), as detection information Id(x) about the work object Ob(x).

[0120] The PLC 20 acquires, for example, information indicating the opening and closing of an opening / closing mechanism provided in the workplace Ar(x) from the equipment 40(y) serving as the input device 41(y) as detection information Id(x) about the workplace Ar(x).

[0121] The PLC 20 acquires, for example, information indicating whether a button provided in the workplace Ar(x) has been pressed from equipment 40(y) serving as an input device 41(y), as detection information Id(x) for the workplace Ar(x).

[0122] The PLC 20 repeatedly receives the detection information Id from the input device 41 at a predetermined cycle, and transmits, that is, transfers, the received detection information Id to the information processing device 10.

[0123] Furthermore, the PLC 20 may transmit the detection information Id repeatedly received from the input device 41 at a predetermined cycle to the outside of the control system 1, similar to the process information Ip. For example, the PLC 20 may transmit the detection information Id repeatedly received from the input device 41 at a predetermined cycle to an in-house LAN (Local Area Network) shown in FIG. 2 connected to an MES (Manufacturing Execution System) or the like.

[0124] The information processing device 10 identifies "at least one state of the tool used by the worker Hu in the work Op, the work object Ob, and the work area Ar" and the detection time Td at which the input device 41 detected the state, from the detection information Id acquired from the PLC 20. Then, the information processing device 10 predicts the content of the work Op being performed by the worker Hu at the detection time Td, from the identified detection time Td and "at least one state of the tool used by the worker Hu in the work Op, the work object Ob, and the work area Ar." In other words, the information processing device 10 predicts the content of the work Op being performed by the worker Hu at the detection time Td, from the detection information Id acquired from the PLC 20.

[0125] However, the process of predicting the content of the work Op being performed by the worker Hu at the detection time Td from the detection information Id may be executed by the PLC 20. In this case, the PLC 20 may transmit the prediction result to the information processing device 10 by including it in the detection information Id or instead of the detection information Id.

[0126] (For systems and devices other than master-slave control systems) 2 shows an in-house LAN system, other network systems, etc., in addition to the control system 1 as a master-slave control system. The in-house LAN is connected to a process information DB (Database) also called MES, etc. The process information DB stores information indicating "standard operations that should be performed by the equipment 40 used to perform the work process Pr" as an operation standard Sa.

[0127] 2, an event management device 60 that monitors and manages various events that occur in the factory Fa is connected to the process information DB as the MES via an in-house LAN. However, it is not essential that the event management device 60 is connected to the process information DB via the in-house LAN, and the event management device 60 does not have to be provided.

[0128] Furthermore, the process information DB is connected to the PLC 20 via an internal LAN. Although not shown, the process information DB may be connected to the information processing device 10. Furthermore, in addition to the MES, an ERP (Enterprise Resources Planning), a WMS (Warehouse Management System), etc. (not shown) may be connected to the internal LAN.

[0129] In FIG. 2, the process information DB is connected to a video storage server or the like via an "other network" different from the control network 50 and the in-house LAN. The information processing device 10 is connected to the video storage server or the like via the other network, and the work site image Ic, the overall image Ia, the contents of the work Op estimated by the information processing device 10, support information, and the like transmitted from the information processing device 10 are stored in the video storage server or the like. In addition, an external device 70 realized by a PC (Personal Computer) or the like is connected to the video storage server or the like. The external device 70 displays, for example, the work site image Ic, the overall image Ia, the contents of the work Op estimated by the information processing device 10, support information, and the like, and performs visualization of the process information Ip, and the like. That is, the external device 70 displays a list of information required for improving the work process Pr, and displays information indicating the work process Pr that is a bottleneck, the date and time of an error that occurred in the work process Pr, and the like in association with the corresponding work site image Ic.

[0130] As described above, the cell camera 30 captures the workplace Ar to generate a workplace image Ic, and transmits the generated workplace image Ic to the information processing device 10 via a communication cable, such as a USB (Universal Serial Bus) cable.

[0131] Moreover, the ceiling camera 80 captures an overhead image of the entire factory Fa to generate an overall image Ia, and transmits the generated overall image Ia to the information processing device 10 via a communication cable, such as a USB cable.

[0132] The information processing device 10 may be, for example, a data extraction device realized by a PC or the like, which combines the process information Ip acquired from the PLC 20 and the work place image Ic acquired from the cell camera 30 to enable efficient use of both. The information processing device 10 visualizes the work process Pr by combining the process information Ip including the "operation result La showing the contents and results of the actual operations Ac of the multiple devices 40 in the factory Fa" with the "workplace image Ic" and the "contents of the work Op estimated from the work place image Ic". The information processing device 10 visualizes the work process Pr by combining the process information Ip acquired from the PLC 20, the work place image Ic acquired from the cell camera 30, and the contents of the work Op estimated from the work place image Ic, and also efficiently and precisely analyzes the work process Pr.

[0133] For example, the information processing device 10, in addition to visualizing the process information Ip, extracts a work process Pr that is a bottleneck, and combines a work site image Ic that captures the implementation status of the extracted work process Pr with the process information Ip of the extracted work process Pr. Therefore, for example, the user can easily identify the cause of the bottleneck and the cause of the defect for the work process Pr that is a bottleneck, and can efficiently perform improvement work for the work process Pr. In addition, the information processing device 10 can also be used for traceability when a defect occurs.

[0134] Furthermore, the information processing device 10 can improve the accuracy of both the analysis of the workplace image Ic and the analysis of the process information Ip by combining the workplace image Ic and each of the process information Ip of the multiple devices 40. For example, the information processing device 10 uses the process information Ip (operation result La) acquired from the PLC 20 to identify the operation start time Tms at which the device 40 starts the operation Ac and the operation completion time Tme at which the operation Ac is completed. The information processing device 10 can associate the operation Ac of the device 40 and the work Op of the worker Hu in the operation period Da with high accuracy by performing an analysis on the workplace image Ic captured during the operation period Da, which is the period from the operation start time Tms to the operation completion time Tme.

[0135] Furthermore, the information processing device 10 acquires a behavior standard Sa indicating "standard behavior that should be performed by the equipment 40 used to perform the work process Pr" from the process information DB. Then, the information processing device 10 executes a judgment on "actual behavior Ac executed by the equipment 40 used to perform the work process Pr" using the acquired behavior standard Sa. The information processing device 10 may execute an analysis on a workplace image Ic capturing an execution status of the work process Pr corresponding to the behavior Ac determined to be different from the standard behavior indicated by the behavior standard Sa.

[0136] (Estimation of work content) The information processing device 10 analyzes the workplace image Ic to predict the content of the work Op performed by the worker Hu on the work object Ob. In particular, the information processing device 10 predicts "the content of the work Op(x) performed by the worker Hu(x) on the work object Ob(x) in the workplace Ar(x) at the imaging time Tp when the workplace image Ic(x) is captured" from the workplace image Ic(x) capturing the workplace Ar(x). The work Op performed by the worker Hu in the workplace Ar is generally a work Op performed on the work object Ob. Therefore, the work Op can be regarded as an interaction between the worker Hu and the work object Ob for a combination (pair) of the worker Hu and the work object Ob.

[0137] Here, conventionally, a method for detecting a human object interaction (HOI), which is an interaction between a human captured in the workplace image Ic and an object captured in the workplace image Ic, from the workplace image Ic has been studied. For example, a HOI detection technique is known that localizes both a human and an object and identifies the interaction between the two.

[0138] The information processing device 10 can use an HOI detection algorithm to estimate the content of the operation Op that the worker Hu is performing on the work object Ob. That is, the information processing device 10 can predict the content of the operation Op that the worker Hu is performing on the work object Ob for a "pair of the worker Hu and the work object Ob" captured in the workplace image Ic by using the HOI detection algorithm that detects the HOI.

[0139] However, it is difficult to say that the accuracy of prediction using the HOI detection algorithm is sufficiently high. In other words, it is difficult to say that the accuracy of prediction of the HOI detection algorithm, which predicts the content of the work Op that the worker Hu captured in the workplace image Ic is performing on the work object Ob captured in the workplace image Ic from the workplace image Ic, is sufficiently high. In particular, it is known that when the following phenomena occur in the workplace image Ic, the accuracy of HOI detection for that workplace image Ic generally tends to decrease.

[0140] In other words, it is known that the accuracy of HOI detection for a work area image Ic in which an obstacle or the like in the foreground hides all or part of at least one of the worker Hu and the work object Ob, i.e., an "occlusion," generally tends to decrease.

[0141] In addition, it is known that when the size (scale) of at least one of the worker Hu and the work object Ob captured in the work place image Ic is (extremely) small or large, the accuracy of HOI detection for that work place image Ic generally tends to decrease.

[0142] It is known that the accuracy of HOI detection for a work area image Ic that has a "truncation" in which all or part of at least one of the worker Hu and the work object Ob is not captured because it goes outside the imaging range, tends to generally decrease.

[0143] It is known that when at least one of the worker Hu and the work object Ob is entirely or partially transparent, the accuracy of HOI detection for the work place image Ic generally tends to decrease.

[0144] Furthermore, it is known that the accuracy of HOI detection for a workplace image Ic in which a work object Ob is captured at an angle, size, etc. that is not normally captured tends to generally decrease. For example, when the angle (rotation), size, etc. of the work object Ob captured in the workplace image Ic are of a rare type, it is known that the accuracy of HOI detection for that workplace image Ic generally tends to decrease.

[0145] It is also known that if the lighting at the time of capturing an image is too bright or too dark, the accuracy of HOI detection for that work place image Ic generally tends to decrease.

[0146] In the following description, an example will be described in which the information processing device 10 detects the worker Hu and the work object Ob from the workplace image Ic using a bounding box for HOI detection (i.e., prediction of the contents of the work Op). However, it is not essential for the information processing device 10 to use a bounding box to detect the worker Hu and the work object Ob. The information processing device 10 may use "Mask R-CNN" to execute a classification task (segmentation) of the worker Hu and the work object Ob for the workplace image Ic. In addition, the information processing device 10 may detect the worker Hu and the work object Ob from the workplace image Ic using an Attention mechanism used in deep learning simultaneously with "Mask R-CNN". The information processing device 10 may improve the prediction accuracy of HOI detection by using "Mask R-CNN", an Attention mechanism, etc.

[0147] (Verification of HOI detection results) As described above, the accuracy of prediction using the HOI detection algorithm is not sufficiently high. Therefore, the information processing device 10 verifies the content (i.e., the first prediction Pf(x)) of the HOI (i.e., the operation Op) predicted using the HOI detection algorithm using at least one of the presence / absence prediction Pa(x) and the operation prediction Po(x).

[0148] The information processing device 10 improves the robustness and estimation accuracy of the estimation of the "contents of the work Op" by using the HOI detection algorithm in combination with "a flow line detection algorithm such as YOLO" and "detection information Id from the input device 41, etc." In other words, the information processing device 10 uses at least one of the presence / absence prediction Pa by the flow line detection algorithm and the work prediction Po predicted from the detection information Id as a noise filter for the first prediction Pf (HOI detection result) by the HOI detection algorithm.

[0149] The information processing device 10 is an operation estimation device that estimates "the contents of the operation Op" by using at least one of a "traffic line detection algorithm" and "detection information Id from the input device 41 or the like" in addition to the HOI detection algorithm.

[0150] (Reduction of annotation costs) The task prediction model 171 as a trained model used in the HOI detection algorithm is constructed by supervised learning on a set of teacher data Dt1 (learning data set DS1) in which the following labels are attached to the workplace image Ic or its feature information. That is, the task prediction model 171 is constructed by machine learning using a set of teacher data Dt1 in which "the content of the task Op that the worker Hu captured in the workplace image Ic is performing on the task object Ob captured in the workplace image Ic" is attached as a label. The "label" attached to the workplace image Ic or its feature information in the teacher data Dt1 is also referred to as "annotation" or "metadata".

[0151] In this embodiment, the "content of the work Op being performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic" attached as a label is a work prediction Po predicted from the detection information Id. That is, the work prediction model 171 is constructed by supervised learning on a set of teacher data Dt1 in which "the 'content of the work Op being performed by the worker Hu' predicted from the detection information Id is attached as a label to the work place image Ic or its feature information".

[0152] In general, it is known that it takes a lot of time and effort to generate a large amount of training data Dt1 required to build a trained model, and that the so-called annotation cost is extremely high. For example, when a human is to determine "the content of the work Op that a worker Hu captured in the workplace image Ic is performing on a work object Ob captured in the workplace image Ic" for each of a large number of workplace images Ic, the cost required for this is extremely high.

[0153] Therefore, in this embodiment, the "contents of the work Op being performed by the worker Hu" predicted from the detection information Id is used as a "label" to be attached to the work site image Ic or its feature information. This makes it possible to reduce the cost (annotation cost) required for generating a large amount of teacher data Dt1, that is, the cost required for constructing the work prediction model 171. Therefore, the information processing device 10 can realize highly accurate and robust estimation of the "contents of the work Op" using the work prediction model 171 constructed at low cost.

[0154] (Summary of the information processing device according to this embodiment) The contents described above with reference to Fig. 2 can be summarized as follows. That is, the information processing device 10 is an operation estimation device that estimates the contents of an operation Op being performed by a worker Hu at a workplace Ar, and includes a first prediction unit 120 (prediction unit), a determination unit 150, and an estimation unit 160.

[0155] The first prediction unit 120 predicts "the content of the work Op being performed by the worker Hu on the work object Ob" from "the work place image Ic or the feature information generated from the work place image Ic" using the work prediction model 171. The work prediction model 171 is a trained model that receives as input "the work place image Ic obtained by capturing an image of the work place Ar" or "the feature information generated from the work place image Ic" and outputs the following information. In other words, the work prediction model 171 is a trained model that outputs "the content of the work Op being performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic".

[0156] The determination unit 150 determines whether the prediction by the first prediction unit 120 is correct or not, using at least one of the "contents of the work Op predicted without using the work prediction model 171" and the "determination result of whether or not the worker Hu is present in the work site Ar." In other words, the determination unit 150 determines whether the first prediction Pf is correct or not, using at least one of the work prediction Po and the presence / absence prediction Pa.

[0157] When the determination unit 150 determines that the prediction by the first prediction unit 120 is correct, the estimation unit 160 estimates that the content of the work Op predicted by the first prediction unit 120 is "the content of the work Op being performed by the worker Hu at the workplace Ar." In other words, when the estimation unit 160 verifies that the first prediction Pf is correct using at least one of the work prediction Po and the presence / absence prediction Pa, it outputs the first prediction Pf as an estimation result for "the content of the work Op being performed by the worker Hu at the workplace Ar."

[0158] According to the above configuration, the information processing device 10 judges whether the content of the work Op predicted using the work prediction model 171 (i.e., the first prediction Pf) is correct or not by using at least one of the following two predictions: That is, the information processing device 10 judges whether the first prediction Pf is correct or not by using at least one of the content of the work Op predicted without using the work prediction model 171 (work prediction Po) and the judgment result of the presence or absence of the worker Hu in the workplace Ar (presence prediction Pa).

[0159] Then, when the information processing device 10 determines that the prediction using the work prediction model 171 is correct, it estimates that the content of the work Op predicted using the work prediction model 171 is "the content of the work Op being performed by the worker Hu at the workplace Ar."

[0160] In other words, the information processing device 10 verifies the accuracy of the content of the work Op predicted using the work prediction model 171 using at least one of the content of the work Op predicted without using the work prediction model 171 and the determination result of whether or not a worker Hu is present in the workplace Ar.

[0161] Therefore, the information processing device 10 has an effect of being able to estimate, with high accuracy, "the content of the work Op being performed by the worker Hu in the workplace Ar" from the workplace image Ic.

[0162] The information processing device 10 further includes a work prediction unit 142 (work detection unit) that predicts "the contents of the work Op being performed by the worker Hu in the work site Ar" from the detection information Id. The detection information Id is information indicating "at least one state of the tool used in the work Op, the work target Ob, and the work site Ar" detected by the input device 41 (sensor).

[0163] When the content of the work Op predicted by the first prediction unit 120 (i.e., the first prediction Pf) matches the content of the work Op predicted by the work prediction unit 142 (i.e., the work prediction Po), the judgment unit 150 judges that the prediction of the first prediction unit 120 is correct.

[0164] According to the above configuration, the information processing device 10 determines whether the content of the work Op predicted from the detection information Id of the input device 41 matches the content of the work Op predicted using the work prediction model 171. Then, when the information processing device 10 determines that the two match, it estimates that the content of the work Op predicted using the work prediction model 171 is "the content of the work Op being performed by the worker Hu at the workplace Ar."

[0165] That is, the information processing device 10 verifies whether the content of the work Op predicted using the work prediction model 171 is correct or not, based on the content of the work Op predicted from the detection information Id of the input device 41.

[0166] Therefore, the information processing device 10 has an effect of being able to estimate, with high accuracy, "the content of the work Op being performed by the worker Hu in the workplace Ar" from the workplace image Ic.

[0167] The information processing device 10 further includes a presence / absence prediction unit 141, and when the presence / absence prediction unit 141 predicts that "the worker Hu is present in the workplace Ar", the determination unit 150 determines that the prediction of the first prediction unit 120 is correct. The presence / absence prediction unit 141 predicts "whether the worker Hu is present in the workplace Ar" from the overall image Ia, or the workplace image Ic, or feature information generated from the overall image Ia or the workplace image Ic, using the presence / absence model 172. The presence / absence model 172 is a trained model, and its input is (A) the overall image Ia capturing a plurality of workplaces Ar including the workplace Ar, or (B) the workplace image Ic, or (C) feature information generated from the overall image Ia or the workplace image Ic. The output of the presence / absence model 172 is information indicating "whether the worker Hu is present in the workplace Ar".

[0168] According to the above configuration, the information processing device 10 predicts "whether or not a worker Hu is present in the workplace Ar" using the presence / absence model 172. Then, when the information processing device 10 predicts that "a worker Hu is present in the workplace Ar", it estimates that the content of the work Op predicted using the work prediction model 171 is "the content of the work Op being performed by the worker Hu in the workplace Ar".

[0169] That is, the information processing device 10 verifies whether the content of the work Op predicted using the work prediction model 171 is correct or not based on the “presence or absence of the worker Hu in the work area Ar” predicted using the presence or absence model 172.

[0170] Therefore, the information processing device 10 has an effect of being able to estimate, with high accuracy, "the content of the work Op being performed by the worker Hu in the workplace Ar" from the workplace image Ic.

[0171] The information processing device 10 may include both the work prediction unit 142 and the presence prediction unit 141. When the presence prediction unit 141 predicts that the worker Hu is present in the work area Ar and the following condition is satisfied, the determination unit 150 determines that the prediction of the first prediction unit 120 is correct. That is, when the content of the work Op predicted by the first prediction unit 120 and the content of the work Op predicted by the work prediction unit 142 match, the determination unit 150 determines that the prediction of the first prediction unit 120 is correct.

[0172] According to the above configuration, the information processing device 10 predicts the content of the work Op predicted from the detection information Id of the input device 41, and predicts the presence or absence of a worker Hu in the work site Ar. Then, when the information processing device 10 predicts that a worker Hu is present in the work site Ar and the following conditions are satisfied, it determines that the prediction of the first prediction unit 120 is correct. That is, when the content predicted from the detection information Id of the input device 41 matches the content predicted using the work prediction model 171, the information processing device 10 determines that the content predicted using the work prediction model 171 is correct.

[0173] In other words, the information processing device 10 verifies the accuracy of the content of the work Op predicted using the work prediction model 171 based on the content of the work Op predicted from the detection information Id of the input device 41 and the presence or absence of the worker Hu in the workplace Ar predicted using the presence / absence model 172.

[0174] Therefore, the information processing device 10 has an effect of being able to estimate, with high accuracy, "the content of the work Op being performed by the worker Hu in the workplace Ar" from the workplace image Ic.

[0175] §2. Example configuration The information processing device 10, the outline of which has been explained above, will now be explained in detail with reference to FIG.

[0176] Fig. 1 is a block diagram showing a main configuration of an information processing device 10. The information processing device 10 illustrated in Fig. 1 includes, as functional blocks, for example, a first acquisition unit 110, a first prediction unit 120, a second acquisition unit 130, a second prediction unit 140, a determination unit 150, an estimation unit 160, and a storage unit 170.

[0177] The information processing device 10 may include the following units in addition to the above-mentioned functional blocks. That is, the information processing device 10 may include a management unit that controls the storage (saving) of the work area image Ic in a predetermined storage area (storage area), a measurement unit that measures the time (stay time) that the worker Hu stays in the work area Ar, an analysis unit that associates the process information Ip with the stay time, and the like. To ensure conciseness of the description, the configuration of the information processing device 10 that is not directly related to this embodiment is omitted from the description and block diagram. However, the information processing device 10 may include these omitted configurations in accordance with the actual situation of the implementation.

[0178] The above-mentioned functional blocks of the information processing device 10 can be realized, for example, by the arithmetic unit reading out a program stored in a storage device (storage unit 170) realized by a read only memory (ROM), a non-volatile random access memory (NVRAM), etc., into a random access memory (RAM) (not shown) or the like, and executing the program. Examples of devices that can be used as the arithmetic unit include a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these.

[0179] First, the first acquisition unit 110, the first prediction unit 120, the second acquisition unit 130, the second prediction unit 140, the determination unit 150, and the estimation unit 160 will be described in detail below.

[0180] (Function blocks other than the memory unit) The first acquisition unit 110 acquires, from the cell camera 30 , a workplace image Ic captured by the cell camera 30 of the workplace Ar, and outputs the acquired workplace image Ic to the first prediction unit 120 .

[0181] The first prediction unit 120 performs image analysis on the workplace image Ic captured by the cell camera 30, and predicts "the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic." The first prediction unit 120 performs, for example, model-based image analysis on the workplace image Ic, and predicts "the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic."

[0182] The first prediction unit 120, which executes model-based image analysis, is realized as, for example, a neural network, and detects an operation Op as an HOI by executing an HOI detection algorithm. Specifically, the first prediction unit 120 refers to the storage unit 170 to acquire an operation prediction model 171. The operation prediction model 171 is a trained model that receives the work place image Ic as an input and outputs "the contents of the operation Op that the worker Hu captured in the work place image Ic is performing on the work object Ob captured in the work place image Ic". The first prediction unit 120 predicts "the contents of the operation Op that the worker Hu captured in the work place image Ic is performing on the work object Ob captured in the work place image Ic" from the work place image Ic, using the operation prediction model 171 acquired by referring to the storage unit 170. Hereinafter, "neural network" may be abbreviated as "NN".

[0183] For example, the first prediction unit 120 generates the following first prediction Pf(x) from the work place image Ic(x) using the work prediction model 171. That is, the first prediction unit 120 generates the first prediction Pf(x), which is a prediction of "the content of the work Op(x) being performed by the worker Hu, who is imaged in the work place image Ic(x), on the work object Ob(x), which is imaged in the work place image Ic(x)."

[0184] In particular, the first prediction unit 120 performs the following prediction from the workplace image Ic(x) using the imaging time Tp, which is the time when the cell camera 30 captures the workplace Ar(x) and generates the workplace image Ic(x). That is, the first prediction unit 120 predicts, from the workplace image Ic(x), "the content of the work Op(x) being performed by the worker Hu(x) on the work object Ob(x) in the workplace Ar(x) at the imaging time Tp."

[0185] The first prediction unit 120 outputs the content of the work Op predicted from the work place image Ic using the work prediction model 171 as a first prediction Pf to the determination unit 150. For example, the first prediction unit 120 outputs the first prediction Pf(x) generated from the work place image Ic(x) to the determination unit 150.

[0186] The second acquisition unit 130 acquires various information for generating the second prediction Ps from the outside, and outputs the acquired various information to the second prediction unit 140. The second prediction Ps is a prediction for verifying (the correctness of) the first prediction Pf, and includes at least one of a presence / absence prediction Pa and an operation prediction Po. The second acquisition unit 130 illustrated in FIG. 1 includes an entire image acquisition unit 131 and a detection information acquisition unit 132.

[0187] The overall image acquisition unit 131 acquires an overall image Ia from the ceiling camera 80, which is captured by the ceiling camera 80 overlooking the entire factory Fa, and outputs the acquired overall image Ia to the second prediction unit 140, in particular to the presence / absence prediction unit 141.

[0188] The detection information acquisition unit 132 acquires detection information Id (including process information Ip) indicating "at least one state of the tool used in the work Op, the work object Ob, and the work area Ar" from the PLC 20. As described above, the detection information Id is information indicating "at least one state of the tool used in the work Op, the work object Ob, and the work area Ar" detected by the input device 41 as a detection device (measurement device), for example. The detection information acquisition unit 132 outputs the acquired detection information Id to the first prediction unit 120.

[0189] The second prediction unit 140 generates a second prediction Ps including at least one of the presence / absence prediction Pa and the work prediction Po, and outputs the generated second prediction Ps to the determination unit 150. The second prediction unit 140 includes a presence / absence prediction unit 141 and a work prediction unit 142.

[0190] The presence / absence prediction unit 141 performs image analysis on the overall image Ia acquired from the second acquisition unit 130 (particularly the overall image acquisition unit 131) to predict "whether a worker Hu is in the workplace Ar captured in the workplace image Ic", that is, to generate a presence / absence prediction Pa. The presence / absence prediction unit 141 performs, for example, model-based image analysis on the overall image Ia to predict "whether a worker Hu is in the workplace Ar captured in the workplace image Ic".

[0191] The presence / absence prediction unit 141, which executes model-based image analysis, is realized as, for example, a NN, and predicts the presence / absence of a worker Hu in the workplace Ar by executing a movement line detection algorithm (object detection algorithm) such as YOLO. Specifically, the presence / absence prediction unit 141 refers to the storage unit 170 to acquire a presence / absence model 172. The presence / absence model 172 is a trained model that receives the entire image Ia as an input and outputs "whether a worker Hu is present in the workplace Ar (more precisely, each of the multiple workplaces Ar in the factory Fa)." The presence / absence prediction unit 141 uses the presence / absence model 172 acquired by referring to the storage unit 170 to predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic" from the entire image Ia. For example, the presence prediction unit 141 uses the presence model 172 from the overall image Ia to generate a presence prediction Pa(x), which is a prediction as to whether a worker Hu is present in the workplace Ar(x) captured in the workplace image Ic(x).

[0192] In particular, the presence / absence prediction unit 141 performs the following prediction from the overall image Ia using the overall time Ta, which is the time when the ceiling camera 80 captures multiple workplaces Ar including 'workplace Ar(x) captured in the workplace image Ic(x)' to generate the overall image Ia.' In other words, the presence / absence prediction unit 141 predicts from the overall image Ia whether or not the worker Hu(x) is in the workplace Ar(x) at the overall time Ta corresponding to 'capture time Tp when the workplace image Ic(x) is captured.'

[0193] The presence / absence prediction unit 141 may use the presence / absence model 172 to predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic," particularly "whether a worker Hu is present in the workplace Ar at the capture time Tp," from the entire image Ia for a predetermined period (e.g., 5 seconds). The prediction of the presence / absence of the worker Hu by the presence / absence prediction unit 141 using the presence / absence model 172 may result in failure, such as erroneous detection or non-detection. Therefore, if the presence / absence prediction unit 141 predicts that "the worker Hu is present" for only 1 second out of 5 seconds, but predicts that "the worker Hu is not present" for the remaining 4 seconds, the prediction that "the worker Hu is present" may be regarded as a erroneous detection (i.e., an erroneous prediction). Even if it is predicted that "the worker Hu is present" for only 1 second, if it is predicted that "the worker Hu is not present" for the remaining 4 seconds, the presence / absence prediction unit 141 may predict that "the worker Hu is not present" from the entire image Ia for those 5 seconds. In other words, the presence / absence prediction unit 141 may generate a presence / absence prediction Pa that "the worker Hu was not present in 'the workplace Ar captured in the workplace image Ic' during those 5 seconds" from the entire image Ia for those 5 seconds.

[0194] To summarize, the presence / absence prediction unit 141 may generate the presence / absence prediction Pa that "the worker Hu was present" for the entirety of a predetermined period only when it predicts that "the worker Hu was present" for a period of a predetermined percentage (e.g., 50%) or more of the predetermined period from the entire image Ia of the predetermined period. For example, the presence / absence prediction unit 141 may generate the presence / absence prediction Pa that "the worker Hu was present during those 5 seconds" only when the period in which it was predicted that "the worker Hu was present" is 2.5 seconds or more for the entire image Ia of 5 seconds.

[0195] The movement line detection algorithm used by the presence / absence prediction unit 141 to predict the presence / absence of the worker Hu is, for example, YOLO, but it is not essential that the movement line detection algorithm used by the presence / absence prediction unit 141 is YOLO, and a movement line detection algorithm other than YOLO may be used. The presence / absence prediction unit 141 only needs to be able to predict "whether the worker Hu is in the workplace Ar captured in the workplace image Ic" from the overall image Ia using the movement line detection algorithm.

[0196] The presence / absence prediction unit 141 outputs to the judgment unit 150 a presence / absence prediction Pa (e.g., a presence / absence prediction Pa(x)), which is a prediction as to whether a worker Hu is present in the workplace Ar captured in the workplace image Ic, predicted from the overall image Ia using the presence / absence model 172.

[0197] The work prediction unit 142 predicts "the contents of the work Op being performed by the worker Hu" by using the detection information Id (including the process information Ip) acquired from the second acquisition unit 130 (particularly, the detection information acquisition unit 132), that is, generates a work prediction Po. As described above, the detection information Id is information indicating "the state of at least one of the tools (e.g., the output device 42 and the tool 90) used in the work Op, the work target Ob, and the work site Ar". For example, the work prediction unit 142 generates the following work prediction Po(x) from the detection information Id(x). That is, the work prediction unit 142 generates a work prediction Po(x) that is a prediction of "the contents of the work Op(x) being performed by the worker Hu(x) in the work site Ar(x)" from the detection information Id(x). As mentioned above, the detection information Id(x) is information that indicates "the status of at least one of the tool (e.g., the output device 42(x) or the tool 90(x)) used in the work Op(x), the work object Ob(x), and the work area Ar(x)."

[0198] The work prediction unit 142 predicts, for example, from "the magnitude of the current flowing through a tool such as an appliance 90(x)," that "an action Ac(x) was performed, and an action Op(x) corresponding to the action Ac(x) was performed by a worker Hu at a workplace Ar(x)."

[0199] The work prediction unit 142 predicts, for example, from the "power consumption of the tool" that "an action Ac(x) was performed, and an action Op(x) corresponding to the action Ac(x) was performed by a worker Hu at a work site Ar(x)."

[0200] For example, the work prediction unit 142 predicts that "a work Op(x) was performed by a worker Hu in a work area Ar(x)" based on "a tool or a work object Ob(x) was gripped."

[0201] The work prediction unit 142 predicts that "a tool, or a work object Ob(x), or an opening / closing mechanism equipped in the work area Ar(x) was opened or closed," and therefore "a work Op(x) was performed by a worker Hu in the work area Ar(x)."

[0202] The work prediction unit 142 predicts that "a tool, a work object Ob(x), or a button on the work area Ar(x) was pressed," and therefore "a work Op(x) was performed by a worker Hu at the work area Ar(x)."

[0203] As described above, the work prediction unit 142 predicts "the content of the work Op being performed by the worker Hu" from the detection information Id, for example, predicts "the content of the work Op(x) being performed by the worker Hu(x) at the workplace Ar(x)" from the detection information Id(x).

[0204] In particular, the work prediction unit 142 performs the following prediction using the detection time Td, which is the time when "at least one state of the tool used in the work Op(x), the work object Ob(x), and the work area Ar(x)" included in the detection information Id(x) was detected. That is, the work prediction unit 142 predicts "the content of the work Op(x) being performed by the worker Hu(x) at the detection time Td corresponding to the 'imaging time Tp when the work area image Ic(x) was captured'" from the detection information Id(x).

[0205] The work prediction unit 142 outputs "the content of the work Op being performed by the worker Hu" predicted from the detection information Id as a work prediction Po to the determination unit 150. For example, the work prediction unit 142 outputs to the determination unit 150 a work prediction Po(x), which is "the content of the work Op(x) being performed by the worker Hu(x) in the workplace Ar(x)" predicted from the detection information Id(x).

[0206] The determination unit 150 verifies the first prediction Pf (whether it is correct or not) by using the second prediction Ps, that is, verifies the first prediction Pf (whether it is correct or not) by using at least one of the presence / absence prediction Pa and the work prediction Po. Specifically, the determination unit 150 verifies the first prediction Pf(x) (whether it is correct or not) by using at least one of the presence / absence prediction Pa(x) and the work prediction Po(x).

[0207] As described above, the first prediction Pf(x) is, for example, the "contents of the work Op(x) that the worker Hu(x) is performing on the work object Ob(x) in the work area Ar(x) at the imaging time Tp" predicted from the work area image Ic(x). The presence / absence prediction Pa(x) is, for example, the prediction of "whether the worker Hu(x) is in the work area Ar(x) at the total time Ta corresponding to the 'imaging time Tp at which the work area image Ic(x) is captured'" predicted from the overall image Ia. The work prediction Po(x) is, for example, the "contents of the work Op(x) that the worker Hu(x) is performing at the detection time Td corresponding to the 'imaging time Tp at which the work area image Ic(x) is captured'" predicted from the detection information Id(x).

[0208] For example, if the presence / absence prediction Pa(x) predicts that "the worker Hu is present in the workplace Ar(x)", the determination unit 150 determines that "the first prediction Pf(x) is correct". In particular, if the presence / absence prediction Pa(x) predicts that "the worker Hu is present in the workplace Ar(x) at the imaging time Tp, which is the time when the cell camera 30 generates the workplace image Ic(x)", the determination unit 150 determines that "the first prediction Pf(x) is correct".

[0209] For example, when the work Op(x) in the work prediction Po(x) and the work Op(x) in the first prediction Pf(x) match, the determination unit 150 determines that "the first prediction Pf(x) is correct." In particular, when "the work Op(x) at the imaging time Tp(x) in the work prediction Po(x)" and "the work Op(x) at the imaging time Tp(x) in the first prediction Pf(x)" match, the determination unit 150 determines that "the first prediction Pf(x) is correct."

[0210] For example, the determination unit 150 determines that "the worker Hu is present in the work site Ar(x)" and the work Op(x) in the work prediction Po(x) and the work Op(x) in the first prediction Pf(x) match, and determines that "the first prediction Pf(x) is correct". In particular, the determination unit 150 determines that "the first prediction Pf(x) is correct" when both of the following two conditions are satisfied. That is, the first condition is that the work prediction Po(x) predicts that "the worker Hu is present in the work site Ar(x) at the image capture time Tp, which is the time when the cell camera 30 generates the work site image Ic(x)". The second condition is that "the work Op(x) at the image capture time Tp(x) in the work prediction Po(x)" and "the work Op(x) at the image capture time Tp(x) in the first prediction Pf(x)" match.

[0211] The determination unit 150 notifies the estimation unit 160 of the first prediction Pf together with the determination result (verification result). The determination unit 150 may further notify the estimation unit 160 of the second prediction Ps (at least one of the presence / absence prediction Pa and the work prediction Po).

[0212] When the judgment unit 150 judges that "the first prediction Pf(x) is correct," the estimation unit 160 estimates that the first prediction Pf(x) is "the content of the work Op being performed by the worker Hu at the workplace Ar," and outputs the first prediction Pf(x) to the outside as an estimation result.

[0213] When the determining unit 150 determines that "the first prediction Pf(x) is incorrect", the estimation unit 160 outputs to the outside the estimation result that "there is no worker Hu in the workplace Ar(x)" or that "there is no content of the work Op(x) that could be estimated with the desired accuracy". The estimation unit 160 may output the estimation result described above together with the workplace image Ic to the video storage server of FIG. 2 or the like.

[0214] (Regarding the memory unit) The storage unit 170 is a storage device that stores various data used by the information processing device 10. The storage unit 170 may non-temporarily store (1) a control program, (2) an OS program, (3) application programs for executing various functions possessed by the information processing device 10, and (4) various data read when executing the application programs, which are executed by the information processing device 10. The above data (1) to (4) are stored in a non-volatile storage device such as a read only memory (ROM), a flash memory, an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM (registered trademark)), or a hard disk drive (HDD). The information processing device 10 may include a temporary storage unit (not shown). The temporary storage unit is a so-called working memory that temporarily stores data used for calculations and calculation results, etc., during the course of various processes executed by the information processing device 10, and is composed of a volatile storage device such as a random access memory (RAM). Which data is stored in which storage device is appropriately determined based on the purpose of use, convenience, cost, physical constraints, etc. of the information processing device 10. The storage unit 170 further stores an operation prediction model 171 and a presence / absence model 172 .

[0215] The work prediction model 171 is a model for the first prediction unit 120 to predict "the content of the work Op being performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic" from the work place image Ic. The work prediction model 171 is a trained model that receives, for example, the work place image Ic as input and outputs information indicating "the content of the work Op being performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic".

[0216] The presence / absence model 172 is a model for the presence / absence prediction unit 141 to predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic" from the entire image Ia. The presence / absence model 172 is a trained model that receives the entire image Ia as an input and outputs "whether a worker Hu is present in the workplace Ar (more precisely, each of the multiple workplaces Ar in the factory Fa)."

[0217] §3. Example of operation 3, 4, and 5 are flow diagrams each illustrating an overview of the processing executed by the information processing device 10. The information processing device 10 can estimate "the content of the work Op being performed by the worker Hu" with high accuracy by verifying (the correctness of) the first prediction Pf using the second prediction Ps.

[0218] (Example of processing when verification is performed using only the work forecast Po) 3 is a flow diagram illustrating an overview of a process executed by the information processing device 10 when verifying (the correctness of) the first prediction Pf using only the work prediction Po of the second prediction Ps. As illustrated in FIG. 3, the information processing device 10 (particularly, the determination unit 150) may verify (the correctness of) the first prediction Pf using only the work prediction Po.

[0219] As shown in FIG. 3, the judgment unit 150 first checks whether or not the “contents of the work Op being performed by the worker Hu” are predicted in the work prediction Po, that is, whether or not “it is predicted from the detection information Id that some work Op is being performed” (S110).

[0220] When it is confirmed that "it is predicted from the detection information Id that some work Op is being performed" (YES in S110), the determination unit 150 further confirms whether or not "the content of the work Op being performed by the worker Hu" is predicted in the first prediction Pf. In other words, the determination unit 150 confirms "whether or not it is predicted that some work Op is being performed using the work prediction model 171" (S120).

[0221] When it is confirmed that "the work prediction model 171 has been used to predict that some work Op is being performed" (YES in S120), the determination unit 150 executes the following determination. That is, the determination unit 150 determines whether "the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171" (S130). That is, the determination unit 150 determines whether "the work Op in the work prediction Po" matches "the work Op(x) in the first prediction Pf".

[0222] When the determination unit 150 determines that "the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171" (YES in S130), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the content of the work Op predicted by the first acquisition unit 110 using the work prediction model 171 from the work place image Ic as the estimation result (S140), that is, outputs the first prediction Pf(x) as the estimation result.

[0223] When it is confirmed that "it was not predicted from the detection information Id that any work Op was being performed" (NO in S110), that is, when the work prediction unit 142 was unable to predict the content of the work Op from the detection information Id, the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy" (S150).

[0224] When it is confirmed that "using the work prediction model 171, it was not predicted that any work Op was being performed" (NO in S120), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy" (S150). In other words, if the first prediction unit 120 cannot predict the content of the work Op from the work place image Ic, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy".

[0225] When it is determined that "the content of the work Op predicted from the detection information Id does not match the content of the work Op predicted using the work prediction model 171" (NO in S130), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy" (S150).

[0226] The information processing device 10 verifies the first prediction Pf (whether it is correct or not) using the operation prediction Po, that is, uses the HOI detection algorithm in combination with "detection information Id from the input device 41 or the like." The information processing device 10 includes a first prediction unit 120 that executes the HOI detection algorithm and an operation prediction unit 142 that generates the operation prediction Po from the detection information Id, thereby canceling erroneous detection (prediction error) by the first prediction unit 120. Therefore, the information processing device 10 can cancel erroneous detection (prediction error) of the HOI detection algorithm and achieve highly accurate estimation.

[0227] The work prediction unit 142 predicts "the contents of the work Op being performed by the worker Hu" from the detection information Id from the input device 41, etc., that is, generates a work prediction Po. The prediction accuracy of "the contents of the work Op being performed by the worker Hu" by the work prediction unit 142 using the detection information Id is higher than the prediction accuracy of "the contents of the work Op being performed by the worker Hu" by the first prediction unit 120 that executes the HOI detection algorithm. In other words, the prediction accuracy of the work prediction Po is higher than the prediction accuracy of the first prediction Pf.

[0228] Therefore, the information processing device 10 can estimate "the contents of the work Op being performed by the worker Hu" with high accuracy by using the first prediction Pf and the work prediction Po. The types of "the work Op being performed by the worker Hu" that the first prediction unit 120 can predict by executing the HOI detection algorithm may be the same as the types of "the work Op being performed by the worker Hu" that the work prediction unit 142 can predict from the detection information Id, or the former may be greater than the latter.

[0229] In general, HOI detection by an HOI detection algorithm (i.e., prediction of "contents of work Op") fails if at least one of the worker Hu and the work object Ob cannot be detected (object detection) from the work place image Ic. Therefore, failure of HOI detection, that is, "the work prediction model 171 was used to predict that some work Op was being performed" may be regarded as "failure to detect an object (e.g., worker Hu) (i.e., absence)."

[0230] (Example of processing when verifying using only presence / absence prediction Pa) 4 is a flow diagram illustrating an overview of a process executed by the information processing device 10 when verifying (the correctness of) the first prediction Pf using only the presence / absence prediction Pa of the second prediction Ps. As illustrated in FIG. 4, the information processing device 10 (particularly, the determination unit 150) may verify (the correctness of) the first prediction Pf using only the presence / absence prediction Pa.

[0231] As shown in FIG. 4, the judgment unit 150 first checks whether or not the presence prediction Pa predicts that "the worker Hu is present in the workplace Ar", that is, whether or not "the presence of the worker Hu is predicted to be present in the workplace Ar using the presence model 172" (S210).

[0232] As described above, the presence / absence prediction unit 141 may predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic during a predetermined period (e.g., 5 seconds)" from the entire image Ia. The presence / absence prediction unit 141 may predict "the worker Hu was present in the workplace Ar captured in the workplace image Ic during the predetermined period" only when it predicts that "the worker Hu is present" for a period of at least a predetermined percentage (e.g., 50%) of the predetermined period.

[0233] When it is confirmed that "the presence of the worker Hu in the work area Ar is predicted using the presence model 172" (YES in S210), the determination unit 150 further confirms whether or not "the content of the work Op being performed by the worker Hu" is predicted in the first prediction Pf. In other words, the determination unit 150 confirms "whether or not it is predicted that some work Op is being performed using the work prediction model 171" (S220).

[0234] When the determination unit 150 confirms that "using the work prediction model 171, it has been predicted that some work Op is being performed" (YES in S220), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the content of the work Op predicted by the first acquisition unit 110 from the work place image Ic using the work prediction model 171 as the estimation result (S230), that is, outputs the first prediction Pf(x) as the estimation result.

[0235] When it is confirmed that "using the presence / absence model 172, it is predicted that the worker Hu is not present in the workplace Ar" (NO in S210), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the worker Hu is not present in the workplace Ar" (S250). In other words, when the presence / absence prediction unit 141 predicts from the entire image Ia that "the worker Hu is not present in the workplace Ar captured in the workplace image Ic", the estimation unit 160 outputs the estimation result that "the worker Hu is not present in the workplace Ar".

[0236] When it is confirmed that "using the work prediction model 171, it was not predicted that any work Op was being performed" (NO in S220), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy" (S240). In other words, if the first prediction unit 120 cannot predict the content of the work Op from the work place image Ic, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy".

[0237] The information processing device 10 verifies the first prediction Pf (whether it is correct or not) using the presence / absence prediction Pa, that is, uses the HOI detection algorithm in combination with a "movement line detection algorithm such as YOLO". That is, the information processing device 10 includes a first prediction unit 120 that executes the HOI detection algorithm and a presence / absence prediction unit 141 that executes the movement line detection algorithm, thereby being able to cancel erroneous detection (prediction errors) by the first prediction unit 120. The movement line detection algorithm can only predict (detect) the presence / absence of the worker Hu, but the prediction accuracy of the movement line detection algorithm is higher than that of the HOI detection algorithm. Therefore, the information processing device 10 can estimate "the contents of the work Op being performed by the worker Hu" with high accuracy by using the first prediction Pf and the presence / absence prediction Pa.

[0238] (Example of processing when verification is performed using both presence prediction Pa and work prediction Po) Fig. 5 is a flow diagram illustrating an overview of a process executed by the information processing device 10 when verifying (the correctness of) the first prediction Pf using both the presence / absence prediction Pa and the work prediction Po. As illustrated in Fig. 5, the information processing device 10 (particularly, the determination unit 150) may verify (the correctness of) the first prediction Pf using both the presence / absence prediction Pa and the work prediction Po.

[0239] As shown in FIG. 5, the judgment unit 150 first checks whether or not the presence prediction Pa predicts that "the worker Hu is present in the workplace Ar", that is, whether or not "the presence of the worker Hu is predicted to be present in the workplace Ar using the presence model 172" (S310).

[0240] As described above, the presence / absence prediction unit 141 may predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic during a predetermined period" from the entire image Ia for that predetermined period. The presence / absence prediction unit 141 may predict "the worker Hu was present in the workplace Ar captured in the workplace image Ic during that predetermined period" only when it predicts that "the worker Hu is present" for at least a predetermined percentage of the predetermined period.

[0241] When it is confirmed that "the presence of the worker Hu in the work area Ar is predicted using the presence model 172" (YES in S310), the determination unit 150 next checks whether or not "the contents of the work Op being performed by the worker Hu" are predicted in the work prediction Po. In other words, the determination unit 150 checks "whether or not it is predicted from the detection information Id that some work Op is being performed" (S320).

[0242] When it is confirmed that "it is predicted from the detection information Id that some work Op is being performed" (YES in S320), the determination unit 150 checks whether or not "the content of the work Op being performed by the worker Hu" is predicted in the first prediction Pf. In other words, the determination unit 150 checks "whether or not it is predicted that some work Op is being performed using the work prediction model 171" (S330).

[0243] When it is confirmed that "the work prediction model 171 has been used to predict that some work Op is being performed" (YES in S330), the determination unit 150 executes the following determination. That is, the determination unit 150 determines whether "the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171" (S340). That is, the determination unit 150 determines whether "the work Op in the work prediction Po" matches "the work Op(x) in the first prediction Pf".

[0244] When the determination unit 150 determines that "the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171" (YES in S340), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the content of the work Op predicted by the first acquisition unit 110 from the work place image Ic using the work prediction model 171 as the estimation result (S350), that is, outputs the first prediction Pf(x) as the estimation result.

[0245] When it is confirmed that "using the presence / absence model 172, it is predicted that the worker Hu is not present in the workplace Ar" (NO in S310), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the worker Hu is not present in the workplace Ar" (S360). In other words, when the presence / absence prediction unit 141 predicts from the entire image Ia that "the worker Hu is not present in the workplace Ar captured in the workplace image Ic", the estimation unit 160 outputs the estimation result that "the worker Hu is not present in the workplace Ar".

[0246] When it is confirmed that "it was not predicted from the detection information Id that any work Op was being performed" (NO in S320), that is, when the work prediction unit 142 was unable to predict the content of the work Op from the detection information Id, the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy" (S370).

[0247] When it is confirmed that "using the work prediction model 171, it was not predicted that any work Op was being performed" (NO in S330), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy" (S370). In other words, if the first prediction unit 120 cannot predict the content of the work Op from the work place image Ic, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy".

[0248] When it is determined that "the content of the work Op predicted from the detection information Id does not match the content of the work Op predicted using the work prediction model 171" (NO in S340), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy" (S370).

[0249] As described above, events that can be predicted from the detection information Id (including the process information Ip) are different from events that can be predicted by a movement line detection algorithm. Specifically, the work prediction Po generated by the work prediction unit 142 from the detection information Id is a prediction of "the contents of the work Op being performed by the worker Hu." In contrast, the presence / absence prediction Pa generated by the presence / absence prediction unit 141 using the movement line detection algorithm is a prediction of "whether the worker Hu is in the work area Ar captured in the work area image Ic."

[0250] Therefore, the information processing device 10 (particularly, the determination unit 150) can use at least one of the work prediction Po and the presence / absence prediction Pa to verify the first prediction Pf (whether it is correct or not). Specifically, the determination unit 150 may verify the first prediction Pf (whether it is correct or not) using only the work prediction Po as shown in Fig. 3, or may verify the first prediction Pf (whether it is correct or not) using only the presence / absence prediction Pa as shown in Fig. 4.

[0251] Furthermore, the determination unit 150 may verify (the correctness of) the first prediction Pf using both the presence / absence prediction Pa and the work prediction Po. In particular, as shown in Fig. 5, the determination unit 150 may first filter the presence / absence of the worker Hu using the presence / absence prediction Pa, and when predicting that the worker Hu is present, may further verify (the correctness of) the first prediction Pf using the work prediction Po.

[0252] The processing executed by the information processing device 10 described above with reference to Figures 3, 4, and 5 can be summarized as follows. That is, the control method executed by the information processing device 10 is a control method of an operation estimation device that estimates "the content of the operation Op being performed by the worker Hu at the workplace Ar." The control method executed by the information processing device 10 includes prediction steps (S120, S220, S330), determination steps (S130, S210, S310, S340), and estimation steps (S140, S230, S350).

[0253] The prediction step predicts "the content of the work Op being performed by the worker Hu on the work object Ob" from "the work place image Ic or the feature information generated from the work place image Ic" using the work prediction model 171. The work prediction model 171 is a trained model that receives as input "the work place image Ic obtained by capturing an image of the work place Ar" or "the feature information generated from the work place image Ic" and outputs the following information. That is, the work prediction model 171 is a trained model that outputs "the content of the work Op being performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic".

[0254] The determination step determines whether the prediction in the prediction step is correct or not using at least one of the content of the work Op predicted without using the work prediction model 171 and the determination result of whether the worker Hu is present in the work site Ar. That is, the determination step determines whether the first prediction Pf is correct or not using at least one of the work prediction Po and the presence / absence prediction Pa.

[0255] In the estimation step, if it is determined in the determination step that "the prediction in the prediction step is correct," the content of the work Op predicted in the prediction step is estimated to be "the content of the work Op being performed by the worker Hu at the workplace Ar." In other words, in the estimation step, if it is verified that the first prediction Pf is correct using at least one of the work prediction Po and the presence / absence prediction Pa, the first prediction Pf is output as an estimation result for "the content of the work Op being performed by the worker Hu at the workplace Ar."

[0256] According to the above configuration, the control method determines whether the content of the work Op predicted using the work prediction model 171 is correct or not using at least one of the content of the work Op predicted without using the work prediction model 171 and the determination result of the presence or absence of the worker Hu at the workplace Ar. Then, when it is determined that the prediction using the work prediction model 171 is correct, the control method estimates that the content of the work Op predicted using the work prediction model 171 is "the content of the work Op being performed by the worker Hu at the workplace Ar".

[0257] In other words, the control method verifies the accuracy of the content of the work Op predicted using the work prediction model 171 using at least one of the content of the work Op predicted without using the work prediction model 171 and the determination result of whether or not a worker Hu is present in the workplace Ar.

[0258] Therefore, the control method has an effect of being able to estimate with high accuracy "the content of the work Op being performed by the worker Hu in the workplace Ar" from the workplace image Ic.

[0259] (Example of the first prediction section and the presence / absence prediction section implemented as a neural network (NN)) As described above, at least one of the first prediction unit 120 and the presence / absence prediction unit 141 may be realized as a NN. In particular, at least one of the first prediction unit 120 and the presence / absence prediction unit 141 may be realized as a CNN (Convolutional Neural Network) or a DNN (Deep Neural Network).

[0260] For example, the first prediction unit 120 (work prediction model 171) outputs a first prediction Pf indicating "the content of the work Op being performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic" from the work place image Ic. Also, for example, the presence prediction unit 141 (presence model 172) outputs a presence prediction Pa indicating "whether the worker Hu is in the work place Ar captured in the work place image Ic" from the entire image Ia.

[0261] Next, the learning process for constructing the task prediction model 171 and the presence / absence model 172, which are trained models, will be described below.

[0262] (Learning process) (About the work forecast model) The first prediction unit 120 realized as a NN constructs an activity prediction model 171, which is a trained model, by, for example, supervised learning on a training data set DS1, which is a collection of training data Dt1. The training data Dt1 is data in which information indicating at least "the content of an activity Op being performed by a worker Hu captured in the workshop image Ic on a task object Ob captured in the workshop image Ic" is attached as a label to the workshop image Ic. The "training data" is also called "training data." In addition, the process in which "the first prediction unit 120 constructs the activity prediction model 171 by supervised learning on the training data set DS1" is also called "learning process SS1."

[0263] In the learning process SS1, the first prediction unit 120 realized as a NN constructs, as a learned model, an operation prediction model 171 that outputs the following information in response to an input of a workplace image Ic, by supervised learning on the learning dataset DS1: That is, the operation prediction model 171 is a learned model that receives an input of the workplace image Ic, and outputs "the content of an operation Op being performed by a worker Hu captured in the workplace image Ic on a work object Ob captured in the workplace image Ic."

[0264] That is, in the teacher data Dt1, the workplace image Ic is labeled with information indicating "the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic." The information attached as a label to the workplace image Ic in the teacher data Dt1 is similar to the information output in response to the input of the workplace image Ic by the work prediction model 171 constructed by machine learning on the learning dataset DS1, which is a collection of the teacher data Dt1.

[0265] The first prediction unit 120 realized as a NN stores in the storage unit 170 an activity prediction model 171 constructed by the learning process SS1.

[0266] (About preparing training data) FIG. 6 is a diagram for explaining an example of annotations required for generating the teacher data Dt1. In the teacher data Dt1 illustrated in FIG. 6, the work place image Ic is given the following labels (annotations). That is, the work place image Ic is given a label indicating the "range of the bounding box for detecting the worker Hu" surrounded by a dashed line. In addition, the work place image Ic is given a label indicating the "range of the bounding box for detecting the work object Ob" surrounded by a dashed line. Furthermore, the work place image Ic is given a straight line with arrows on both ends as a label indicating the "contents of the work Op performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic". The straight line with arrows on both ends connects the approximate center of the "range of the bounding box for detecting the worker Hu" and the approximate center of the "range of the bounding box for detecting the work object Ob".

[0267] In order to prepare the teacher data Dt1 as illustrated in FIG. 6, even if it is not necessary to set a bounding box for detecting the worker Hu, the following five annotations are required for the workplace image Ic. That is, it is necessary to set the X and Y coordinates of the lower left and upper right points of the four points that indicate the range of the bounding box for detecting the work object Ob for the workplace image Ic, that is, four annotations are required. Furthermore, one annotation is required for the workplace image Ic that indicates "the content of the work Op that the worker Hu captured in the workplace image Ic is performing on the work object Ob captured in the workplace image Ic."

[0268] Here, for supervised learning, a large amount of teacher data Dt1 is generally required, and in order to prepare a large amount of teacher data Dt1, it is necessary to perform the above-mentioned five annotations for each of the large amount of workplace images Ic. Therefore, when a human is to determine "the content of the work Op that the worker Hu captured in the workplace image Ic is performing on the work object Ob captured in the workplace image Ic" for each of the large amount of workplace images Ic, the cost required for this is extremely large.

[0269] Therefore, in this embodiment, when preparing the teacher data Dt1, the "contents of the work Op" predicted from the detection information Id, rather than the "contents of the work Op" determined by a human from the workplace image Ic, is attached to the workplace image Ic as a label indicating the "contents of the work Op". In other words, in the teacher data Dt1, the "contents of the work Op" predicted by the work prediction unit 142 from the detection information Id, rather than the "contents of the work Op" determined by a human from the workplace image Ic, is attached as a label indicating the "contents of the work Op".

[0270] Therefore, the cost required to construct the work prediction model 171 can be significantly reduced compared to when "content determined by a human from the work place image Ic" is used as a label indicating "content of the work Op". In particular, the annotation cost required to generate the learning dataset DS1 can be significantly reduced compared to when "content determined by a human from the work place image Ic" is used as a label indicating "content of the work Op". For example, one of the above-mentioned five annotations required for the work place image Ic is changed from "content determined by a human" to "content predicted by the work prediction unit 142 from the detection information Id", so that the annotation cost can be reduced by 20%.

[0271] For example, the teacher data Dt1(x) is generated by labeling the work site image Ic(x) generated by the cell camera 30 capturing an image of the work site Ar(x) at a certain image capturing time Tp with the following "contents of work Op(x)". That is, the teacher data Dt1(x) is generated by labeling the work site image Ic(x) with the "contents of work Op(x) performed by the worker Hu(x) in the work site Ar(x) at the image capturing time Tp" predicted by the work prediction unit 142 from the detection information Id(x). The detection information Id(x) is information indicating "the tool (e.g., the output device 42(x) or the tool 90(x)) used in the work Op(x), the work object Ob(x), and at least one of the states of the work site Ar(x)" at the image capturing time Tp.

[0272] Specifically, the work prediction unit 142 predicts, for example, that "at the imaging time Tp, the worker Hu performed work Op(x): screw tightening work in the work site Ar(x)" from "the magnitude of the current flowing through the tool 90(x): electric screwdriver at the imaging time Tp." This predicted content (i.e., "content of work Op(x): screw tightening work") is added to the work site image Ic(x) to generate the teacher data Dt1(x).

[0273] As described above, in the information processing device 10, the work prediction model 171 is a trained model constructed by machine learning using the training dataset DS1. In the training dataset DS1, the "workplace image Ic or feature information generated from the workplace image Ic" is labeled with the following information. That is, the "contents of the work Op performed by the worker Hu in the workplace Ar" predicted from the detection information Id of the input device 41 that detects "the tool used in the work Op, the work object Ob, and at least one state of the workplace Ar" is labeled.

[0274] According to the above-described configuration, the information processing device 10 constructs an operation prediction model 171 by machine learning using a learning dataset labeled with “the content of the operation Op being performed by the worker Hu at the workplace Ar” predicted from the detection information Id of the input device 41.

[0275] It is known that supervised learning requires a large amount of training data (learning dataset). The labeling process required to generate such training data and the labels (metadata) attached to the training data are also called "annotations."

[0276] For example, even if the teacher data Dt1 learned when constructing the work prediction model 171 does not require the setting of a bounding box for detecting the worker Hu, the following five annotations are required for the work place image Ic. That is, it is necessary to set the X coordinates and Y coordinates of the lower left point and the upper right point of the four points indicating the range of the bounding box for detecting the work object Ob for the work place image Ic, that is, four annotations are required. Furthermore, it is necessary to assign information (label) indicating "the content of the work Op that the worker Hu captured in the work place image Ic is performing on the work object Ob captured in the work place image Ic" to the work place image Ic, that is, one annotation is required.

[0277] Furthermore, if we were to have a human determine the content of the work Op that the worker Hu captured in the workplace image Ic is performing on the work object Ob captured in the workplace image Ic for each of the large number of workplace images Ic, the cost required to do so would be extremely high.

[0278] Therefore, when generating the teacher data Dt1, the "contents of the work Op" predicted from the detection information Id of the input device 41, rather than the "contents of the work Op" determined by a human from the workplace image Ic, is attached as a label indicating the "contents of the work Op". In other words, in the teacher data Dt1, the "contents of the work Op" predicted from the detection information Id of the input device 41, rather than the "contents of the work Op" determined by a human from the workplace image Ic, is attached as a label indicating the "contents of the work Op".

[0279] Therefore, the cost required to construct the work prediction model 171 can be significantly reduced compared to when "contents determined by humans from the work place image Ic" is used as a label indicating "contents of the work Op". In particular, the annotation cost required to generate the learning dataset DS1 can be significantly reduced compared to when "contents determined by humans from the work place image Ic" is used as a label indicating "contents of the work Op". For example, one of the above-mentioned five annotations required for the work place image Ic is changed from "contents determined by humans" to "contents predicted by a device such as the information processing device 10 from the detection information Id", so that the annotation cost can be reduced by 20%.

[0280] Therefore, according to the above-mentioned configuration, the information processing device 10 has the effect of being able to estimate with high accuracy “the content of the work Op being performed by the worker Hu” from the workplace image Ic by using the work prediction model 171 in which annotation costs are significantly reduced.

[0281] (About the existence model) The presence / absence prediction unit 141 realized as a NN constructs the presence / absence model 172, which is a trained model, by, for example, supervised learning on a data set DS2 which is a collection of training data Dt2. The training data Dt2 is data in which information indicating "whether a worker Hu is present in a workplace Ar (more precisely, in each of the multiple workplaces Ar in the factory Fa)" is attached as a label to the entire image Ia. The process in which "the presence / absence prediction unit 141 constructs the presence / absence model 172 by supervised learning on the data set DS2" is also referred to as "learning process SS2".

[0282] In learning process SS2, presence / absence prediction unit 141 realized as a NN constructs, through supervised learning on dataset DS2, a presence / absence model 172 that outputs the following information in response to an input of entire image Ia, as a learned model: That is, presence / absence model 172 is a learned model that receives input of entire image Ia and outputs "whether worker Hu is present in workplace Ar (more precisely, each of the multiple workplaces Ar in factory Fa)."

[0283] That is, in the teacher data Dt2, information indicating "whether a worker Hu is present in the workplace Ar (more precisely, in each of the multiple workplaces Ar in the factory Fa)" is attached as a label to the whole image Ia. The information attached as a label to the whole image Ia in the teacher data Dt2 is similar to the information output by the presence / absence model 172 constructed by machine learning for the data set DS2, which is a collection of the teacher data Dt2, in response to the input of the whole image Ia.

[0284] The presence / absence prediction unit 141 realized as a NN stores in the storage unit 170 a presence / absence model 172 constructed by the learning process SS2.

[0285] However, it is not essential that the presence / absence prediction unit 141 constructs the presence / absence model 172 by the learning process SS2. The presence / absence prediction unit 141 may use a general learned model used in a traffic line detection algorithm as the presence / absence model 172 to predict "whether a worker Hu is in a workplace Ar (more precisely, in each of the multiple workplaces Ar in the factory Fa)" from the overall image Ia.

[0286] § 4. Variations (About machine learning) So far, an example has been described in which the task prediction model 171 and the presence / absence model 172 are constructed as learned data by machine learning on supervised data. However, the machine learning for constructing the task prediction model 171 and the presence / absence model 172 is not limited to machine learning on supervised data. The machine learning for constructing the task prediction model 171 and the presence / absence model 172 is not limited to supervised learning, and may be other machine learning such as unsupervised learning or reinforcement learning.

[0287] (About the whole image) So far, an example has been described in which the presence / absence prediction unit 141 predicts "whether the worker Hu is in the workplace Ar captured in the workplace image Ic" from the entire image Ia. However, the presence / absence prediction unit 141 may predict "whether the worker Hu is in the workplace Ar captured in the workplace image Ic" from the workplace image Ic. That is, it is not essential for the information processing device 10 to acquire the entire image Ia separately from the workplace image Ic. The information processing device 10 (particularly, the determination unit 150) can verify the correctness of the first prediction Pf generated from the workplace image Ic by the presence / absence prediction Pa generated from at least one of the entire image Ia and the workplace image Ic. For example, if the presence / absence prediction Pa(x) generated from at least one of the entire image Ia and the workplace image Ic(x) is a prediction that "the worker Hu is in the workplace Ar(x)", the determination unit 150 determines that "the first prediction Pf(x) is correct".

[0288] (About work forecast) So far, an example has been described in which the information processing device 10 is the entity that predicts "the content of the work Op being performed by the worker Hu" from the detection information Id. That is, an example has been described in which the work prediction unit 142 grasps "at least one state of the tool used in the work Op, the work target Ob, and the work area Ar" from the detection information Id, and generates a work prediction Po from the grasped state.

[0289] However, the entity that grasps "at least one state of the tool used in the work Op, the work object Ob, and the work site Ar" from the detection information Id and predicts "the content of the work Op being performed by the worker Hu" from the grasped state does not have to be the information processing device 10. It is sufficient for the information processing device 10 to acquire or generate "the content of the work Op being performed by the worker Hu" predicted from at least one state of the tool, the work object Ob, and the work site Ar.

[0290] For example, the PLC 20 may generate the work prediction Po from the detection information Id. In this case, the information processing device 10 (particularly, the detection information acquisition unit 132) acquires the result of the prediction performed by the PLC 20, that is, the work prediction Po, from the PLC 20. Then, the determination unit 150 verifies the first prediction Pf (whether it is correct or not) using the work prediction Po generated by the PLC 20.

[0291] In other words, when the information processing device 10 (particularly, the judgment unit 150) uses the work prediction Po to verify the first prediction Pf (whether it is correct or not), the entity that generates the work prediction Po from the detection information Id may be the information processing device 10 or may not be the information processing device 10.

[0292] (About the second prediction) Although an example in which the information processing device 10 includes both the presence / absence prediction unit 141 and the work prediction unit 142 has been described so far, it is not essential that the information processing device 10 includes both the presence / absence prediction unit 141 and the work prediction unit 142. The determination unit 150 only needs to be able to verify the first prediction Pf (whether it is correct or not) using the second prediction Ps, that is, it only needs to be able to verify the first prediction Pf (whether it is correct or not) using at least one of the presence / absence prediction Pa and the work prediction Po. Therefore, when the presence / absence prediction Pa is not used to verify the first prediction Pf (whether it is correct or not), the information processing device 10 does not need to include the presence / absence prediction unit 141. Furthermore, when the work prediction Po is not used to verify the first prediction Pf (whether it is correct or not), the information processing device 10 does not need to include the work prediction unit 142.

[0293] [Software implementation example] The functional blocks of the information processing device 10 (specifically, the first acquisition unit 110, the first prediction unit 120, the second acquisition unit 130, the second prediction unit 140, the determination unit 150, and the estimation unit 160) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like. Moreover, these functional blocks may be realized by software using a CPU, a GPU, a DSP, or the like.

[0294] In the latter case, the information processing device 10 includes a CPU, GPU, DSP, etc., which executes instructions of a program, which is software that realizes each function, a ROM or storage device (these are referred to as "recording medium") in which the program and various data are recorded so as to be readable by a computer (or CPU), and a RAM or the like in which the program is expanded. The object of the present invention is achieved by having the computer (or CPU, GPU, DSP, etc.) read and execute the program from the recording medium. As the recording medium, a "non-transient tangible medium", for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, etc., can be used. The program may also be supplied to the computer via any transmission medium (such as a communication network or a broadcast wave) capable of transmitting the program. The present invention may also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

[0295] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0296] 10 Information processing device (task estimation device) 41 Input devices (sensors) 42 Output devices (tools) 90 Equipment (tools) 120 First Prediction Department (Prediction Department) 141 Prediction Department 142 Work prediction unit (work detection unit) 150 Judgment section 160 Estimation part 171 Work Prediction Model 172 Existence Model Ar Workshop DS1 training dataset Hu worker Ia Overall image Ic Workshop Images Id Detection Information Ob Work Object Op Work S120 (Prediction step) S130 (Decision step) S140 (Estimation step) S210 (Decision step) S220 (Prediction step) S230 (Estimation step) S310 (Decision step) S330 (Prediction step) S340 (Decision step) S350 (Estimated Step)

Claims

1. A task estimation device that estimates the content of a task performed by a worker in a workplace, a prediction unit that predicts the content of the work that the worker is performing on the work object captured in the workplace from the workplace image or the feature information using a work prediction model that is a trained model that receives an input of a workplace image captured in the workplace or feature information generated from the workplace image and outputs the content of the work that the worker, whose image is captured in the workplace image, is performing on the work object captured in the workplace image; a determination unit that determines whether the prediction by the prediction unit is correct or not, using at least one of the work content predicted without using the work prediction model and a determination result of whether the worker is present in the work area; and an estimation unit that, when the determination unit determines that the prediction by the prediction unit is correct, estimates the content of the work predicted by the prediction unit as the content of the work being performed by the worker in the workplace; and In addition to providing a presence / absence prediction unit that predicts whether the worker is present in the workplace from the overall image, or the workplace image, or the feature information generated from the overall image or the workplace image, using a presence / absence model that is a trained model that inputs (A) an overall image of a plurality of workplaces including the workplace, or (B) an image of the workplace, or (C) feature information generated from the overall image or the workplace image, and outputs whether the worker is present in the workplace, The determination unit determines that the prediction by the presence prediction unit is correct when the presence prediction unit predicts that the worker is present in the work area.

2. a work detection unit that predicts the content of the work being performed by the worker in the work area based on detection information from a sensor that detects at least one state of a tool used in the work, the work object, and the work area; Further equipped with The determination unit determines that the prediction by the prediction unit is correct when the presence prediction unit predicts that the worker is present in the work area and the content of the work predicted by the prediction unit coincides with the content of the work predicted by the work detection unit. The task estimation device according to claim 1 .

3. The work prediction model is constructed by machine learning using a learning dataset in which the content of the work being performed by the worker in the workplace, predicted from detection information of a sensor that detects at least one state of the tool used in the work, the work object, and the workplace, is labeled with the workplace image or feature information generated from the workplace image. The task estimation device according to claim 1 or 2.

4. A control method for an activity estimation device that estimates the content of an activity performed by a worker in a workplace, comprising the steps of: a prediction step of predicting the content of the work being performed by the worker on the work object, based on the workplace image or the feature information, using a work prediction model which is a trained model that receives an input of a workplace image of the workplace or feature information generated from the workplace image and outputs the content of the work being performed by the worker captured in the workplace image on the work object, based on the workplace image or the feature information; a determination step of determining whether the prediction in the prediction step is correct or not, using at least one of the work content predicted without using the work prediction model and a determination result of whether the worker is present in the work area; an estimation step of estimating, when it is determined in the determination step that the prediction in the prediction step is correct, the content of the work predicted in the prediction step as the content of the work being performed by the worker in the workplace; Including, (A) an overall image of a plurality of workplaces including the workplace is input, or (B) the workplace image is input, or (C) feature information generated from the overall image or the workplace image is input, and a presence / absence model is used as a trained model that outputs whether the worker is present in the workplace from the overall image, or the workplace image, or the feature information generated from the overall image or the workplace image, The control method, wherein the judging step judges that the prediction made in the predicting step is correct when the presence of the worker in the work area is predicted by the presence predicting step.

5. An information processing program for causing a computer to function as the task estimation device according to claim 1 , the information processing program causing a computer to function as each of the units.

6. 6. A computer-readable recording medium on which the information processing program according to claim 5 is recorded.

Images