Processing system, processing method, program, and storage medium

Abstract

To provide a processing system, a processing method, a program, and a storage medium, which can estimate a task with higher accuracy.SOLUTION: A processing system generates first graph data on the basis of a worker's posture which is estimated from a first image of the worker. The first graph data includes a plurality of first nodes corresponding respectively to the plurality of joints of the worker, and a plurality of first edges corresponding respectively to the plurality of skeletal parts of the worker. The processing system inputs the first graph data to a neural network including a graph neural network (GNN). The processing system estimates a task being performed by the worker using a result output from the neural network.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] FIELD An embodiment of the present invention relates to a processing system, a processing method, a program, and a storage medium. [Background technology]

[0002] There are systems that automatically estimate the tasks being performed. There is a demand for technology that can estimate the tasks more accurately for these systems. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-91249 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a processing system, a processing method, a program, and a storage medium that are capable of estimating an operation with higher accuracy. [Means for solving the problem]

[0005] A processing system according to an embodiment generates first graph data based on a posture of a worker estimated from a first image showing the worker. The first graph data includes a plurality of first nodes corresponding to a plurality of joints of the worker, respectively, and a plurality of first edges corresponding to a plurality of bones of the worker, respectively. The processing system inputs the first graph data to a neural network including a graph neural network (GNN). The processing system estimates an operation being performed by the worker using a result output from the neural network. [Brief description of the drawings]

[0006] [Figure 1]FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. [Diagram 2] Fig. 2(a) is a schematic diagram showing a worker and an object, and Fig. 2(b) is an example of an image captured by an imaging device. [Diagram 3] FIG. 3 shows the result of pose estimation being performed on the image shown in FIG. 2(b). [Figure 4] FIG. 4 is a schematic diagram illustrating the structure of the first graph data. [Diagram 5] FIG. 5 is a schematic diagram illustrating the structure of a neural network. [Figure 6] Fig. 6(a) is a diagram for explaining a specific structure of the first graph data, and Fig. 6(b) is a diagram illustrating an example of a feature vector. [Figure 7] 7(a) to 7(c) are diagrams showing specific examples of feature vectors. [Figure 8] FIG. 8 is a flowchart illustrating an example of a processing method according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram illustrating an output result by the processing system according to the embodiment. [Figure 10] FIG. 10 is a schematic diagram illustrating another structure of graph data. [Figure 11] 11(a) and 11(b) are schematic diagrams illustrating other structures of graph data. [Figure 12] FIG. 12 is a schematic diagram illustrating another structure of a neural network. [Figure 13] FIG. 13 is a schematic diagram illustrating a specific structure of an LSTM network. [Figure 14] FIG. 14 is a table showing the accuracy of the task estimation results. [Figure 15] FIG. 15 is a table illustrating the operations performed. [Figure 16] FIG. 16 is a schematic diagram illustrating the structure of the first graph data. [Figure 17] FIG. 17 is a schematic diagram illustrating the structure of the first graph data. [Figure 18] FIG. 18 is a schematic diagram illustrating another structure of a neural network. [Figure 19] FIG. 19 is a schematic diagram illustrating another structure of a neural network. [Figure 20] FIG. 20 is a flowchart showing another example of the processing method according to the embodiment. [Figure 21] FIG. 21 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as the actual ones. Even when the same part is shown, the dimensions and ratios of each part may be shown differently depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. The processing system according to the embodiment is used to estimate an operation being performed by a worker from an image. As shown in Fig. 1, the processing system 1 includes an imaging device 10, a processing device 20, a storage device 30, an input device 40, and an output device 50.

[0009] The imaging device 10 captures an image of a worker performing a task and acquires the image. The processing device 20 processes the acquired image and estimates the task being performed. The storage device 30 stores images or videos as well as data required for processing by the processing device 20. The input device 40 is used by a user to input data to the processing device 20. The processing device 20 causes the output device 50 to output the data obtained by processing so that the user can recognize it.

[0010] The processing by the processing system 1 will be described below with reference to a specific example.

[0011] Fig. 2(a) is a schematic diagram showing a worker and an object, Fig. 2(b) is an example of an image captured by an imaging device, and Fig. 3 is a diagram showing an example of a posture estimation result. For example, as shown in Fig. 2(a), an item A1 is placed on a carrier C. A worker W performs a predetermined task on the item A1. The item A1 is a semi-finished product, a unit used in a finished product, etc. The imaging device 10 captures an image of the worker W and the item A1 and acquires an image. Fig. 2(b) shows an image IMG acquired by the imaging device 10.

[0012] Preferably, the imaging device 10 is attached to a wall or ceiling, and captures images of the worker W and the item A1 from above. This makes it easier to capture the worker W and the item A1. The imaging device 10 may capture images directly below or at an angle to the vertical direction. The imaging device 10 repeatedly captures images. Alternatively, the imaging device 10 may capture a video. In this case, still images are repeatedly extracted from the video. The imaging device 10 stores the images or videos in the storage device 30.

[0013] The processing device 20 accesses the storage device 30 and acquires the image captured by the imaging device 10. The processing device 20 estimates the posture of the worker W from the image in which the worker W appears. For example, the processing device 20 inputs the image into a posture estimation model prepared in advance. The posture estimation model is trained in advance to estimate the posture of a person appearing in an image in response to the input of the image. The processing device 20 acquires an estimation result by the posture estimation model. For example, the posture estimation model includes a neural network. It is preferable that the posture estimation model includes a convolutional neural network (CNN). As the posture estimation model, OpenPose, DarkPose, CenterNet, or the like can be used.

[0014] FIG. 3 shows a result of pose estimation performed on the image shown in FIG. 2(b). As shown in FIG. 3, the positions of multiple joints 100 of the worker W are estimated by pose estimation. The processing device 20 generates first graph data based on the estimated pose. The first graph data has a graph-type data structure. The first graph data includes multiple nodes and multiple edges. Nodes that are related to each other are connected by edges.

[0015] FIG. 4 is a schematic diagram illustrating the structure of the first graph data. As shown in FIG. 4, the first graph data GD1 includes a plurality of first nodes n1 and a plurality of first edges e1. The plurality of first nodes n1 correspond to a plurality of joints of the worker, respectively. The plurality of first edges e1 correspond to a plurality of skeletons of the worker, respectively. In the example shown in FIG. 4, the positions of the ankles, knees, waist, chest, wrists, elbows, and head are estimated from the image, and a plurality of first nodes n1 corresponding thereto are set. In addition, a plurality of first edges e1 are set corresponding to the skeletons connecting those joints. Therefore, no edge is set between joints that are not actually connected. For example, the wrist node and the elbow node are connected by an edge, but the wrist node and the waist node are not connected.

[0016] The first graph data GD1 may include more first nodes and first edges than the illustrated example. For example, first nodes corresponding to the neck, shoulders, fingers, etc., and first edges corresponding to these first nodes may be further set. By increasing the number of nodes, the accuracy of the task estimation can be further improved. The number of nodes to be set can be appropriately changed according to the processing capacity of the processing device 20.

[0017] The processing device 20 inputs the first graph data to a neural network and obtains an output result from the neural network. The neural network includes a graph neural network (GNN) so as to process the graph data.

[0018] FIG. 5 is a schematic diagram illustrating the structure of a neural network. For example, a neural network 200 shown in Fig. 5 is used for task estimation. The neural network 200 includes an input layer 210, a GNN 220, and a fully connected layer 230. The GNN 220 includes a plurality of convolution layers 220a and a pooling layer 220b. The input layer 210 receives the first graph data GD1 as input. The size of the input layer 210 is set according to the number of nodes in the first graph data GD1 and the number of edges in the first graph data GD1.

[0019] Fig. 6(a) is a diagram for explaining a specific structure of the first graph data, Fig. 6(b) is a diagram illustrating an example of a feature vector, and Figs. 7(a) to 7(c) are diagrams illustrating specific examples of the feature vector. As shown in Fig. 6(a), the first graph data includes a set vector V and an adjacency matrix A. The set vector V includes values ​​of each of a plurality of nodes. Each node is represented by a feature vector v. As shown in Fig. 6(b), the feature vector v includes two values, an X coordinate of a joint and a Y coordinate of the joint.

[0020] As an example, Fig. 7(a) to Fig. 7(c) respectively show a feature vector v1 corresponding to a left wrist node, a feature vector v2 corresponding to a left elbow node, and a feature vector v3 corresponding to a chest node. The feature vector v1 includes the X coordinate of the left wrist and the Y coordinate of the left wrist. The feature vector v2 includes the X coordinate of the left elbow and the Y coordinate of the left elbow. The feature vector v3 includes the X coordinate of the chest and the Y coordinate of the chest. The feature vectors v1 to v3 are examples of the first node n1.

[0021] In the adjacency matrix A shown in Fig. 6(a), the relevance between nodes is represented by "0" or "1". That is, the existence of each edge is represented by a value of "0" or "1". "0" indicates that the edge does not exist. "1" indicates that the edge exists.

[0022] In the GNN 220, each convolutional layer 220a convolves each feature vector v according to the following formula (1). In formula (1), v i is the initial feature vector of the i-th node. conv(t) is the updated feature vector of the i-th node, obtained by t times of convolution. A(i) is the set of the feature vector of the i-th node in the set vector V and the feature vectors of the nodes defined as adjacent to the i-th node by the adjacency matrix A. Here, "adjacent" refers to being connected by an edge. W(t) is the weight updated t times. The output result by the convolution layer 220a is further reduced by the pooling layer 220b. The output result by the pooling layer 220b is converted to one-dimensional data by the fully connected layer 230. The output result by the fully connected layer 230 corresponds to the estimated result of the work.

number

[0023] (Processing method) FIG. 8 is a flowchart illustrating an example of a processing method according to the embodiment. As shown in FIG. 8, the imaging device 10 captures an image of a worker and an object, and acquires a video (step S10). The processing device 20 extracts an image from the video (step S20). The processing device 20 estimates the posture of the worker from the image (step S30). The processing device 20 generates first graph data based on the estimated posture of the worker (step S40). The processing device 20 inputs the first graph data into a neural network, and acquires an output result from the neural network (step S50). The processing device 20 estimates the work being performed by the worker using the output result from the neural network (step S60).

[0024] (study) The neural network used to estimate the task is trained in advance. A plurality of training data are used for training. Each training data includes input data and teaching data (labels). The input data has a graph structure and is generated using an image showing an actual task. The input data may be prepared using a synthetic image that mimics an actual task. The above-mentioned graph data generation method can be applied to generate the input data. The teaching data is a label that indicates the task corresponding to the input data. Each intermediate layer of the neural network is trained so that teaching data is output in response to the input of input data.

[0025] (Example output) FIG. 9 is a schematic diagram illustrating an output result by the processing system according to the embodiment. In Fig. 9, the horizontal direction indicates time. The vertical direction indicates the estimation results of the item state and the work location based on the image at each time. The top row indicates the work estimated based on the item state and the work location.

[0026] While the work is being performed, images showing the state of the work are repeatedly acquired. The processing device 20 repeatedly estimates the work based on each image. This allows estimation of what work the worker was performing at each time. For example, the processing system 1 estimates the work in real time while the work is being performed.

[0027] Advantages of the embodiment will be described. Various methods have been used to attempt to estimate the work being performed. Work generally involves the repetition of the same movements or small changes in movements. For this reason, estimating work is often more difficult than estimating physical actions such as running, jumping, and bending. In order to accurately estimate work, there is a method in which multiple sensors are attached to the body of a worker. In this method, the work is estimated by determining the worker's small movements from the data of each sensor. In this case, many expensive sensors are required, which requires high costs. In addition, it is time-consuming to attach sensors to the worker, and the sensors may get in the way of the work.

[0028] Regarding this issue, the processing system 1 according to the embodiment uses the posture of the worker to estimate the work being performed. The posture of the worker changes depending on the work being performed. Therefore, by using the posture information, it becomes possible to estimate the work.

[0029] Furthermore, in an embodiment of the present invention, in order to further improve the accuracy of estimation, graph data is generated using posture information. The graph data includes a plurality of nodes and a plurality of edges. The edges indicate that the nodes are related to each other. For example, the connections between the joints of the body are represented by edges. The processing system 1 inputs the graph data into the GNN to obtain an estimation result of the task.

[0030] Some joints in the body are connected to each other by a skeleton. The movements of these joints are related to each other. On the other hand, the movement of joints that are not connected by a skeleton is less related to each other. For example, in the body, there are combinations of joints with high movement correlation, such as a combination of a wrist and an elbow, or a combination of an ankle and a knee. On the other hand, there are combinations of joints with low movement correlation, such as a combination of an ankle and a wrist. By using graph data, the relationship between nodes can be expressed. In other words, a weight can be assigned to each node by an edge. When data used for estimating an operation does not have a graph structure, the relationship between data is learned and calculated indiscriminately. By using graph data, the relationship between nodes is taken into account, and a more accurate estimation result can be obtained.

[0031] In addition, the posture of the worker can be estimated from the image. The work can be estimated without attaching a sensor to the worker. Therefore, the work can be estimated without disturbing the work performed by the worker. The cost required for work estimation can also be reduced.

[0032] According to an embodiment of the present invention, it is possible to estimate an operation more simply and accurately.

[0033] 10, 11(a) and 11(b) are schematic diagrams illustrating other structures of graph data. Changes in the node values ​​over time may be used for the estimation. For example, as shown in FIG. 10, first graph data GD1 is generated from an image (first image) obtained at time t1. The first graph data GD1 includes a plurality of first nodes n1 and a plurality of first edges e1. Second graph data GD2 is generated from an image (second image) obtained at time t2 after time t1. The second graph data GD2 includes a plurality of first nodes n1 and a plurality of first edges e1, similar to the first graph data GD1. The processing device 20 connects the plurality of first nodes n1 of the first graph data GD1 and the plurality of first nodes n1 of the second graph data GD2 by a plurality of edges e4. Note that in FIG. 10, only some of the edges e4 are shown, and other edges e4 are omitted.

[0034] 11(a), images may be acquired at three or more times t1 to t3, and the first graph data GD1 to third graph data GD3 may be generated from the three images. For example, between the first graph data GD1 and the second graph data GD2 based on images captured at adjacent times, corresponding first nodes n1 are connected by an edge e4. Between the second graph data GD2 and the third graph data GD3 based on images captured at adjacent times, corresponding first nodes n1 are connected by an edge e4.

[0035] One graph data may be selected from the multiple graph data, and the nodes of the selected graph data may be connected to the nodes of the other graph data. For example, the third graph data GD3 may be selected from the first graph data GD1 to the third graph data GD3. As shown in FIG. 11(b), the first node n1 of the third graph data GD3 may be connected to the first node n1 of the first graph data GD1 and the first node n1 of the second graph data GD2. In this case, the first node n1 of the first graph data GD1 and the first node n1 of the second graph data GD2 are not connected by an edge.

[0036] By representing the changes in each node over time in a graph structure, the accuracy of task estimation can be improved.

[0037] In the example shown in FIG. 11(a) or FIG. 11(b), all of the temporally consecutive images do not have to be used to generate the graph data. For example, when 10 images are acquired between times t1 and t10, 10 pieces of graph data may be generated from the 10 images, and the nodes of the graph data may be connected to each other. A portion of the 10 images may be selected. In this case, graph data is generated from each of the selected images, and the nodes of the graph data are connected to each other. In either case, a plurality of graph data items that are connected to each other are input to the neural network 200.

[0038] FIG. 12 is a schematic diagram illustrating another structure of a neural network. In order to use the temporal change of the node for estimation, the neural network may include a Long Short Term Memory (LSTM) network. The neural network 200a shown in FIG. 12 further includes an LSTM network 225, in comparison with the neural network 200 shown in FIG. 5. The data (values) output from the GNN 220 are input to the LSTM network 225. The data output from the LSTM network 225 is input to the fully connected layer 230, and the fully connected layer 230 outputs an estimation result of the task.

[0039] (LSTM network) FIG. 13 is a schematic diagram illustrating a specific structure of an LSTM network. As shown in Figure 13, each neuron N in the LSTM network 300 includes a forget gate 310, an input gate 320, and an output gate 330. t represents the input value to neuron N at time t. C t represents the state of neuron N at time t. t represents the output value of the forget gate 310 at time t. trepresents the output value of the input gate at time t. t represents the output value of the output gate at time t. t represents the output value of neuron N at time t. t , i t , C t ,o t , and h t are respectively expressed by the following Formulas 2 to 6.

number

number

number

number

number

[0040] (Example) The inventors of the present application have verified the accuracy of work estimation for the above-mentioned embodiment. First, the imaging device 10 captures the state of work. The frame rate of the imaging device 10 is 25 fps. That is, the imaging device 10 captures 25 images per second. The processing device 20 captures images captured at each time from time t to time t+99 in order to estimate the work being performed at time t. The processing device 20 estimates the posture of the worker from each of the 100 captured images. The processing device 20 generates graph data from the estimated posture for each image. As shown in FIG. 11(a), the processing device 20 connects graph data based on images captured at adjacent times by edges. The processing device 20 inputs the obtained graph data to the neural network 200 shown in FIG. 5. The processing device 20 estimates the work being performed at time t based on the results output from the neural network 200.

[0041] In addition to the above estimation, the processing device 20 generates 100 pieces of graph data using the postures estimated from the 100 images. The processing device 20 sequentially inputs the 100 pieces of graph data to a neural network 200a shown in Fig. 12. The processing device 20 estimates the task being performed at time t based on the results output from the neural network 200a.

[0042] In the method using the neural network 200 shown in Fig. 5 and the graph data shown in Fig. 11(a), or in the method using the neural network 200a shown in Fig. 12 and the graph data shown in Fig. 4, multiple images captured at different times are used. These methods have in common the fact that they can use the time change of the node for estimation.

[0043] Fig. 14 is a table showing the accuracy of the estimation result of the work. Fig. 15 is a table showing an example of the work to be performed. Nine workers W1 to W9 shown in Fig. 14 each perform a task. Each worker sequentially performs tasks "1" to "16" shown in Fig. 15. Using images of the worker performing each task, the task was estimated by each of the first method and the second method.

[0044] The first method is an estimation method using the neural network 200 shown in FIG. 5 and the graph data shown in FIG. 11(a). The second method is an estimation method using the neural network 200a shown in FIG. 12 and multiple graph data shown in FIG. 4. The same images were used for estimating the tasks by the first method and the second method. In addition, the number of convolution layers 220a in the neural networks 200 and 200a was set to two. The accuracy (correct answer rate) of the estimation for the 16 tasks was investigated for each worker and each method. The results are shown in FIG. 14.

[0045] The experimental results showed that both methods have in common the ability to use temporal changes in nodes for estimation, but there are large differences in the accuracy of the estimation. Specifically, the second method improved the estimation accuracy of all tasks by more than 10% compared to the first method. For some tasks, the second method improved the estimation accuracy of the tasks by more than 25% compared to the first method.

[0046] From these results, the following can be inferred. GNNs are effective at learning spatial associations, such as skeletal connections. Furthermore, for associations between temporal data, recurrent neural networks such as LSTM networks are more effective than GNNs at learning. In other words, it is effective to use a GNN to learn associations between spatial nodes, and an LSTM network to learn associations between temporal nodes. It can be inferred that by dividing the roles between a GNN and an LSTM network, it is possible to improve the accuracy of estimation.

[0047] In addition, in the processing system 1, auxiliary sensors may be attached to the body of the worker. For example, in addition to the posture of the worker, the state of the item, and the work location, the acceleration and angular velocity of a part of the body may be used to estimate the work. Even in this case, the number of sensors required can be reduced compared to when the work is estimated from the sensors alone.

[0048] In addition to the posture, other information may be estimated from the image. For example, the processing device 20 estimates one or more selected from the state of the article and the location of the work on the article from the image. The processing device 20 generates first graph data based on the posture and one or more selected from the state and the location of the work. The processing device 20 inputs the first graph data to a neural network and estimates the work being performed using the output result.

[0049] The state (appearance) of the item changes as the work progresses. The part of the item where the worker is working also changes depending on the work being performed. Therefore, by using this information, the accuracy of work estimation can be further improved.

[0050] A state estimation model is used to estimate the state of an item. The processing device 20 inputs an image into the state estimation model. The processing device 20 acquires an estimation result by the state estimation model. The state estimation model is trained in advance so as to estimate the state of an item depicted in an image in response to the input of the image. The state estimation model is trained using an image depicting an item and a label indicating the state of the item. For example, the state estimation model includes a neural network. It is preferable that the state estimation model includes a convolutional neural network (CNN).

[0051] Template matching may be used to estimate the state of the item. The processing device 20 compares the image with a plurality of template images prepared in advance. Each template image is associated with a state of the item. The processing device 20 calculates the similarity between the image and each template image. The processing device 20 extracts the template image with the highest similarity. The processing device 20 estimates the state associated with the extracted template image as the state of the item depicted in the image.

[0052] A work location estimation model is used to estimate the location of work performed on an item. The processing device 20 inputs an image into the work location estimation model. The processing device 20 acquires an estimation result by the work location estimation model. The work location estimation model is trained in advance to estimate the location of work performed by a worker in response to the input of an image. The work location estimation model is trained using an image showing an item and a label indicating the work location in the image. For example, the work location estimation model includes a neural network. It is preferable that the work location estimation model includes a CNN.

[0053] 16 and 17 are schematic diagrams illustrating the structure of the first graph data. The processing device 20 generates the first graph data based on one or more selected from the state and the working location in addition to the posture. For example, the first graph data GD1a shown in Fig. 16 includes the first data D1, the second data D2, and the third data D3. The first data D1, the second data D2, and the third data D3 are separated from each other and are not connected by an edge.

[0054] The first data D1 is generated based on the posture estimation result. The first data D1 includes a plurality of first nodes n1 and a plurality of first edges e1. The plurality of first nodes n1 correspond to a plurality of joints of the worker, respectively. The plurality of first edges e1 correspond to a plurality of bones of the worker, respectively.

[0055] The second data D2 is generated based on the state estimation result. The second data D2 includes a plurality of second nodes n2 and a plurality of second edges e2. The plurality of second nodes n2 correspond to a plurality of states that the item can take. Each of the plurality of second edges e2 corresponds to a transition of the state of the item. In the example shown in FIG. 16, eight second nodes n2 are set because the item can take eight states. Nine second edges e2 are set corresponding to the possibility of transition of those states. For example, a transition from the first state to the second state, a transition from the second state to the first state, and a transition from the first state to the third state may occur. The second edges e2 are set corresponding to these transitions.

[0056] The third data D3 is generated based on the estimation result of the work location. The third data D3 includes a plurality of third nodes n3 and a plurality of third edges e3. The plurality of third nodes n3 correspond to a plurality of locations where the item can be worked on. Each of the plurality of third edges e3 indicates the relationship between the locations where the work can be performed. For example, in actual work, the locations where transitions can occur are connected by edges. In the example shown in FIG. 16, the north node and the east node are connected by an edge. This indicates that the location where the work can be performed can transition from the north location to the east location, or from the east location to the north location. Similarly, edges are set between the east node and the south node, between the south node and the west node, and between the west node and the north node. This indicates that the location where the work can be performed can transition between the respective nodes. In addition, no edges are set between the north location and the south location, and between the east location and the west location. This indicates that there are no transitions of work locations between the North location and the South location, and between the East location and the West location.

[0057] The first graph data GD1a shown in Fig. 16 is input to the neural network 200 shown in Fig. 5. The processing device 20 uses the results output from the neural network 200 to estimate the task being performed.

[0058] As in the first graph data GD1b shown in Fig. 17, the first node n1 and the second node n2 may be connected by an edge e, and the first node n1 and the third node n3 may be connected by another edge e. Each second node n2 is connected to the first node n1 with which it has a relationship. Each third node n3 is connected to the first node n1 with which it has a relationship.

[0059] As an example, when the relevance between the position of the worker's waist and a certain state of the item is higher than the relevance of other combinations, the first node n1 of the waist and the second node n2 of that state of the item are connected by an edge e. When the relevance between the position of the worker's left elbow and another state of the item is higher than the relevance of other combinations, the first node n1 of the left elbow and the second node n2 of that other state of the item are connected by an edge e. When a certain part of the item is mainly worked on by the worker's right hand, the third node n3 corresponding to that part is connected to the first node n1 corresponding to the right hand by an edge e. When another part of the item is mainly worked on by the worker's left hand, the third node n3 corresponding to that other part is connected to the first node n1 corresponding to the left hand by an edge e.

[0060] 18 and 19 are schematic diagrams illustrating other structures of neural networks. As shown in FIG. 18, the neural network 200b may include input layers 211 to 213, GNNs 221 to 223, and a fully connected layer 230. The input layer 211 receives first data D1 related to the posture. The GNN 221 includes a convolutional layer 221a and a pooling layer 221b, and processes the first data D1. The input layer 212 receives second data D2 related to the state of the item. The GNN 222 includes a convolutional layer 222a and a pooling layer 222b, and processes the second data D2. The input layer 213 receives third data D3 related to the work location on the item. The GNN 223 includes a convolutional layer 223a and a pooling layer 223b, and processes the third data D3. The convolutional layers 221a to 223a convolute each feature vector of each node, similar to the convolutional layer 220a. The pooling layers 221b to 223b reduce the convoluted feature vectors in the same manner as the pooling layer 220b. The output results of the GNNs 221 to 223 are combined in the fully connected layer 230. The fully connected layer 230 outputs an estimation result of the task.

[0061] The second node n2 corresponding to the state of the item and the third node n3 corresponding to the work location on the item are represented by feature vectors, similar to the example shown in Fig. 6(a). The second edge e2 and the third edge e3 are represented by values ​​of "0" or "1".

[0062] The posture, state, and working location may be estimated from a plurality of images, and graph data may be generated for each. In this case, the plurality of graph data are connected by edges as shown in FIG. 10, FIG. 11(a), or FIG. 11(b), and input to the neural network 200 shown in FIG. 5. The plurality of graph data may be input sequentially to the neural network 200a including the LSTM network 225.

[0063] 19, compared to the neural network 200b, LSTM networks 225a to 225c may be further provided. The data output from the GNNs 221 to 223 are input to the LSTM networks 225a to 225c, respectively. The data output from the LSTM networks 225a to 225c are input to the fully connected layer 230, and the fully connected layer 230 outputs an estimation result of the task.

[0064] (Processing method) FIG. 20 is a flowchart showing another example of the processing method according to the embodiment. First, steps S10 to S30 are executed in the same manner as in the flowchart shown in FIG. 8. The processing device 20 estimates the state of the item from the image (step S31). The processing device 20 estimates the work location on the item from the image (step S32). The processing device 20 generates first graph data based on the posture of the worker, the state of the item, and the work location on the item (step S40). The processing device 20 inputs the first graph data into a neural network and obtains an output result from the neural network (step S50). The processing device 20 estimates the work being performed by the worker using the output result from the neural network (step S60).

[0065] By using information on the state of the item and the location of work on the item in addition to the posture, the accuracy of work estimation can be improved.

[0066] In particular, when handling (manufacturing) an object, complex and detailed motions appear. Furthermore, similar motions may appear even in different tasks. When a large made-to-order product (indented product) is manufactured, thousands to tens of thousands of parts may be assembled. This results in a very large number of tasks. When estimating tasks based only on posture, there is a possibility that accuracy will decrease. By using one or more selected from the state and the task location in addition to the posture, it is possible to estimate the tasks with higher accuracy.

[0067] In the above example, three pieces of information are used to estimate the work: the posture of the worker, the state of the item, and the work location on the item. The embodiment of the present invention is not limited to this example, and one selected from the state and the work location, and the posture of the worker may be used to estimate the work. For example, if the state (appearance) of the item changes as the work progresses, the work can be accurately estimated even without information on the work location. If the work location changes as the work progresses, the work can be accurately estimated even without information on the state of the item.

[0068] FIG. 21 is a schematic diagram showing a hardware configuration. The processing device 20 includes, for example, the hardware configuration shown in Fig. 21. A computer 90 shown in Fig. 21 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0069] The ROM 92 stores a program for controlling the operation of the computer. The ROM 92 stores programs necessary for the computer to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0070] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0071] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs.

[0072] The input interface (I / F) 95 connects the computer 90 and the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0073] The output interface (I / F) 96 connects the computer 90 and the output device 96a. The output I / F 96 is, for example, a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI (registered trademark)). The CPU 91 transmits data to the output device 96a via the output I / F 96 and can cause the output device 96a to display an image.

[0074] A communication interface (I / F) 97 connects the computer 90 to a server 97a external to the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97. The camera 99 photographs an item and stores the image in the server 97a.

[0075] The storage device 94 includes one or more selected from a hard disk drive (HDD) and a solid state drive (SSD). The input device 95a includes one or more selected from a mouse, a keyboard, a microphone (audio input), and a touch pad. The output device 96a includes one or more selected from a monitor, a projector, a speaker, and a printer. A device having the functions of both the input device 95a and the output device 96a, such as a touch panel, may be used.

[0076] The storage device 94 can be used as the storage device 30. The camera 99 can be used as the imaging device 10.

[0077] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0078] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0079] Implementations of the invention may include the following features. (Feature 1) generating first graph data including a plurality of first nodes corresponding to a plurality of joints of the worker, respectively, and a plurality of first edges corresponding to a plurality of bones of the worker, respectively, based on a posture of the worker estimated from a first image in which the worker is captured; A processing system that inputs the first graph data into a neural network, including a graph neural network (GNN), and uses results output from the neural network to infer an operation being performed by the worker. (Feature 2) generating second graph data including a plurality of first nodes and a plurality of first edges based on a posture of the worker estimated from a second image acquired after the first image; The processing system according to feature 1, further comprising: inputting the second graph data into the neural network after inputting the first graph data; and estimating the task using the result output from the neural network. (Feature 3) The processing system of feature 2, wherein the neural network includes the GNN and a Long Short Term Memory (LSTM) network to which output from the GNN is input. (Feature 4) generating second graph data including a plurality of nodes and a plurality of edges based on a posture of the worker estimated from a second image acquired after the first image; The processing system according to feature 1, further comprising: inputting graph data in which the plurality of nodes in the first graph data and the plurality of nodes in the second graph data are connected by a plurality of edges to the neural network; and estimating the operation using the result output from the neural network. (Feature 5) generating the first graph data based on a state of the article estimated from the first image in which the article is captured in addition to the posture; The processing system according to any one of features 1 to 4, wherein the first graph data includes a plurality of second nodes each corresponding to a plurality of first nodes, a plurality of first edges, and a plurality of states that the item can take. (Feature 6) generating the first graph data based on a work location on the object estimated from the first image showing the object in addition to the posture; The processing system according to any one of features 1 to 4, wherein the first graph data includes the plurality of first nodes, the plurality of first edges, and a plurality of third nodes respectively corresponding to a plurality of locations of the article. (Feature 7) The processing system according to any one of features 1 to 6, wherein the plurality of nodes included in the first graph data indicate coordinates of each of the plurality of joints in the first image. (Feature 8) The processing system according to any one of features 1 to 7, further comprising a display device that displays a time when the first image was acquired and an estimation result of the task at the time. (Feature 9) The processing device includes: generating first graph data including a plurality of first nodes corresponding to a plurality of joints of the worker, respectively, and a plurality of first edges corresponding to a plurality of bones of the worker, respectively, based on a posture of the worker estimated from a first image in which the worker is captured; inputting the first graph data into a neural network including a graph neural network (GNN), and estimating the task being performed by the worker using a result output from the neural network; Processing methods. (Feature 10) A program for causing the processing device to execute the processing method according to feature 9. (Feature 11) A storage medium storing a program for causing the processing device to execute the processing method according to feature 9.

[0080] According to the embodiment described above, a processing system, a processing method, a program, and a storage medium are provided that are capable of estimating an operation more simply and accurately.

[0081] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0082] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0083] 1: Processing system, 10: Imaging device, 20: Processing device, 30: Storage device, 40: Input device, 50: Output device, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 95a: Input device, 96: Output interface, 96a: Output device, 97: Communication interface, 97a: Server, 98: System bus, 99: Camera, 100: Joint, 200, 200a, 200b: Neural network, 210-213: Input layer, 220-223: GNN, 220a-223a: Convolution layer, 220b-223b: Pooling layer, 225: LSTM network, 230: Fully connected layer, 300: LSTM network, 310: Forget gate, 320: input gate, 330: output gate, D1: first data, D2: second data, D3: third data, GD1, GD1a, GD1b: first graph data, GD2: second graph data, GD3: third graph data, N: neuron, V: set vector, W: worker, e: edge, e1: first edge, e2: second edge, e3: third edge, e4: edge, n1: first node, n2: second node, n3: third node, v: feature vector

Claims

1. generating first graph data including a plurality of first nodes corresponding to a plurality of joints of the worker, respectively, and a plurality of first edges corresponding to a plurality of bones of the worker, respectively, based on a posture of the worker estimated from a first image in which the worker is captured; A processing system that inputs the first graph data into a neural network, including a graph neural network (GNN), and uses results output from the neural network to infer an operation being performed by the worker.

2. generating second graph data including a plurality of first nodes and a plurality of first edges based on a posture of the worker estimated from a second image acquired after the first image; The processing system of claim 1 , further comprising: inputting the second graph data into the neural network after inputting the first graph data; and estimating the task using the result output from the neural network.

3. The processing system of claim 2 , wherein the neural network includes the GNN and a Long Short Term Memory (LSTM) network to which an output from the GNN is input.

4. generating second graph data including a plurality of nodes and a plurality of edges based on a posture of the worker estimated from a second image acquired after the first image; 2. The processing system according to claim 1, wherein graph data in which the plurality of nodes in the first graph data and the plurality of nodes in the second graph data are connected by a plurality of edges is input to the neural network, and the operation is estimated using the result output from the neural network.

5. generating the first graph data based on a state of the article estimated from the first image in which the article is captured in addition to the posture; The processing system according to any one of claims 1 to 4, wherein the first graph data includes a plurality of second nodes each corresponding to a plurality of first nodes, a plurality of first edges, and a plurality of states that the item can take.

6. generating the first graph data based on a work location on the object estimated from the first image showing the object in addition to the posture; The processing system according to any one of claims 1 to 4, wherein the first graph data includes the plurality of first nodes, the plurality of first edges, and a plurality of third nodes each corresponding to a plurality of locations of the item.

7. The processing system according to claim 1 , wherein the plurality of nodes included in the first graph data indicate coordinates of each of the plurality of joints in the first image.

8. 5. The processing system according to claim 1, further comprising a display device that displays a time when the first image was acquired and an estimation result of the work at the time.

9. The processing device includes: generating first graph data including a plurality of first nodes corresponding to a plurality of joints of the worker, respectively, and a plurality of first edges corresponding to a plurality of bones of the worker, respectively, based on a posture of the worker estimated from a first image in which the worker is captured; inputting the first graph data into a neural network including a graph neural network (GNN), and estimating the task being performed by the worker using a result output from the neural network; Processing methods.

10. A program for causing the processing device to execute the processing method according to claim 9.

11. A storage medium storing a program for causing the processing device to execute the processing method according to claim 9.

Images