Processing system, processing method, program, and storage medium

Abstract

To provide a processing system, a processing method, a program, and a storage medium, which can estimate a task with higher accuracy.SOLUTION: A processing system estimates a worker's posture from a first image of the worker and an article. The processing system estimates at least one selected from a state of the article and a work location of the worker on the article on the basis of the first image. The processing system generates first graph data including a plurality of nodes and a plurality of edges on the basis of the posture and the at least one selected the state and the work location. The processing system inputs the first graph data to a neural network including a graph neural network (GNN). The processing system estimates a task being performed by the worker using a result output from the neural network.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] FIELD An embodiment of the present invention relates to a processing system, a processing method, a program, and a storage medium. [Background technology]

[0002] There are systems that automatically estimate the tasks being performed. There is a demand for technology that can estimate the tasks more accurately for these systems. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-91249 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a processing system, a processing method, a program, and a storage medium that are capable of estimating an operation with higher accuracy. [Means for solving the problem]

[0005] A processing system according to an embodiment estimates a posture of a worker from a first image showing the worker and an object. The processing system estimates one or more selected from a state of the object and a location of work performed on the object by the worker from the first image. The processing system generates first graph data including a plurality of nodes and a plurality of edges based on the one or more selected from the state and the location of work, and the posture. The processing system inputs the first graph data to a neural network including a graph neural network (GNN). The processing system estimates the work being performed by the worker using a result output from the neural network. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. [Diagram 2] Fig. 2(a) is a schematic diagram showing a worker and an object, and Fig. 2(b) is an example of an image captured by an imaging device. [Diagram 3] FIG. 3 is a flowchart illustrating an example of a processing method according to the embodiment. [Figure 4] 4(a) to 4(c) are diagrams for explaining the processing by the processing system according to the embodiment. [Diagram 5] 5(a) to 5(c) are diagrams for explaining the processing by the processing system according to the embodiment. [Figure 6] 6(a) to 6(c) are diagrams for explaining the processing by the processing system according to the embodiment. [Figure 7] 7(a) to 7(d) are diagrams for explaining the processing by the processing system according to the embodiment. [Figure 8] 8(a) and 8(b) are diagrams for explaining the processing by the processing system according to the embodiment. [Figure 9] 9(a) to 9(c) are diagrams for explaining the processing by the processing system according to the embodiment. [Figure 10] 10(a) to 10(d) are diagrams for explaining the processing by the processing system according to the embodiment. [Figure 11] FIG. 11 is a flowchart showing a method for estimating an item position. [Figure 12] FIG. 12 is a schematic diagram for explaining a result of position estimation when performing tracking processing. [Figure 13] FIG. 13 is a diagram for explaining the processing by the processing system according to the embodiment. [Figure 14] 14(a) to 14(d) are diagrams for explaining the processing by the processing system according to the embodiment. [Figure 15]15(a) to 15(d) are diagrams for explaining the processing by the processing system according to the embodiment. [Figure 16] FIG. 16 is a flowchart showing an outline of the tracking process. [Figure 17] FIG. 17 is a flowchart showing the update process in the tracking process. [Figure 18] 18(a) to 18(c) are images for explaining the processing by the processing system according to the embodiment. [Figure 19] 19(a) to 19(c) are images for explaining the processing performed by the processing system according to the embodiment. [Figure 20] FIG. 20 is a schematic diagram for explaining a method for estimating a work location. [Figure 21] FIG. 21 is a flowchart showing a method for estimating a work location. [Figure 22] FIG. 22 is a schematic diagram illustrating the structure of the first graph data. [Figure 23] FIG. 23 is a schematic diagram illustrating the structure of the first graph data. [Figure 24] FIG. 24 is a schematic diagram illustrating the structure of the first graph data. [Diagram 25] FIG. 25 is a schematic diagram illustrating the structure of a neural network. [Figure 26] Fig. 26(a) is a diagram for explaining a specific structure of the first graph data, and Fig. 26(b) is a diagram illustrating an example of a feature vector. [Figure 27] 27(a) to 27(f) are diagrams showing specific examples of feature vectors. [Figure 28] FIG. 28 is a schematic diagram illustrating another structure of a neural network. [Figure 29] FIG. 29 is a schematic diagram illustrating another structure of graph data. [Diagram 30] 30(a) and 30(b) are schematic diagrams illustrating other structures of graph data. [Diagram 31]FIG. 31 is a schematic diagram illustrating another structure of a neural network. [Diagram 32] FIG. 32 is a schematic diagram illustrating another structure of a neural network. [Diagram 33] FIG. 33 is a schematic diagram illustrating a specific structure of an LSTM network. [Diagram 34] FIG. 34 is a schematic diagram illustrating an output result by the processing system according to the embodiment. [Diagram 35] FIG. 35 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as the actual ones. Even when the same part is shown, the dimensions and ratios of each part may be shown differently depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. The processing system according to the embodiment is used to estimate an operation performed by a worker from an image. As shown in Fig. 1, the processing system 1 includes an imaging device 10, a processing device 20, a storage device 30, an input device 40, and an output device 50.

[0009] Fig. 2(a) is a schematic diagram showing a worker and an object, and Fig. 2(b) is an example of an image captured by an imaging device. For example, as shown in Fig. 2(a), an item A1 is placed on a carrier C. A worker W performs a predetermined task on the item A1. The item A1 is a semi-finished product, a unit used in a finished product, etc. The imaging device 10 captures an image of the worker W and the item A1 and acquires an image. Fig. 2(b) shows an image IMG acquired by the imaging device 10.

[0010] Preferably, the imaging device 10 is attached to a wall or ceiling, and captures images of the worker W and the item A1 from above. This makes it easier to capture the worker W and the item A1. The imaging device 10 may capture images directly below or at an angle to the vertical direction. The imaging device 10 repeatedly captures images. Alternatively, the imaging device 10 may capture a video. In this case, still images are repeatedly extracted from the video. The imaging device 10 stores the images or videos in the storage device 30.

[0011] The processing device 20 accesses the storage device 30 and acquires the image captured by the imaging device 10. The processing device 20 estimates the posture of the worker W, the position of the item A1, the orientation of the item A1, and the state of the item A1 from the image. Furthermore, the processing device 20 estimates the location of the work performed on the item A1 by the worker W based on the posture, position, and orientation. The processing device 20 estimates the work being performed by the worker W using the posture, the state of the item, and the location of the work performed on the item.

[0012] The storage device 30 stores images or videos as well as data necessary for processing by the processing device 20. The input device 40 is used by a user to input data to the processing device 20. The processing device 20 causes the output device 50 to output data obtained by processing so that the user can recognize the data.

[0013] (Processing method) FIG. 3 is a flowchart illustrating an example of a processing method according to the embodiment. With reference to FIG. 3, an overview of the operation of the processing system according to the embodiment will be described. The imaging device 10 captures an image of a worker and an item, and acquires a video (step S10). The processing device 20 extracts an image from the video (step S20). The processing device 20 estimates the posture of the worker from the image (step S30). The processing device 20 estimates the position and orientation of the item from the image (step S40). The processing device 20 estimates the state of the item from the image (step S50). The processing device 20 estimates the work location on the item based on the posture of the worker, the position of the item, and the orientation of the item (step S60). The processing device 20 generates first graph data based on the posture of the worker, the state of the item, and the work location on the item (step S70). The processing device 20 inputs the first graph data to a neural network, and acquires an output result from the neural network (step S80). The neural network includes a graph neural network (GNN). The processing device 20 uses the output result from the neural network to estimate the task being performed by the worker (step S90).

[0014] Hereinafter, each process executed by the processing device 20 will be specifically described.

[0015] (Posture Estimation) The processing device 20 estimates the posture of the worker W from an image in which the worker W appears. For example, the processing device 20 inputs an image into a posture estimation model prepared in advance. The posture estimation model is trained in advance to estimate the posture of a person appearing in an image in response to the input of the image. The processing device 20 acquires an estimation result by the posture estimation model. For example, the posture estimation model includes a neural network. It is preferable that the posture estimation model includes a convolutional neural network (CNN). As the posture estimation model, OpenPose, DarkPose, CenterNet, or the like can be used.

[0016] (Position and orientation estimation) The processing device 20 extracts two images captured at different times from a plurality of images. The processing device 20 estimates motion information from the two images. The motion information is information indicating the motion of an object between one image and another image. For example, dense optical flow is calculated as the motion information. Any method for calculating dense optical flow may be used, and recurrent all-pairs field transforms (RAFT) or total variation (TV)-L1, etc., may be used.

[0017] Figures 4(a) to 4(c), 5(a) to 5(c), 6(a) to 6(c), 7(a) to 7(d), 8(a), 8(b), 9(a) to 9(c), and 10(a) to 10(d) are diagrams for explaining processing by the processing system of the embodiment. FIG. 4(a) shows image It1 captured at time t1. FIG. 4(b) shows image It2 captured at time t2. Time t2 is after time t1. FIG. 4(c) shows movement information from image It1 to image It2, calculated by the processing device 20. In normal work, mainly workers and items related to the work move. When workers and items related to a different work are not captured in the image, the movement information indicates the area in the image where the workers and items are captured. Here, the part of the image including the workers and items indicated by the movement information is called a "partial area."

[0018] The movement information used to estimate the position of an item may include the movement of other tools in addition to the movement of the worker and the movement of the item. The tools are tools, jigs, etc. However, the shape of the tools and the appearance of these tools in the movement information are sufficiently different from the shape of the worker, the shape of the item, and the appearance of the worker and the item in the movement information. For this reason, as described below, by using the "certainty" regarding the shape or position of the item, the influence of the movement of the tools, the movement of the jigs, etc. on the estimation of the position of the item can be sufficiently reduced.

[0019] The result of the above-mentioned pose estimation indicates the area in the image where the worker is shown. Here, the area in which the worker is shown as a result of the pose estimation is referred to as the "worker area". The processing device 20 estimates the worker area in the image from the result of the pose estimation. The processing device 20 uses the worker area as a mask to remove the worker area from the movement information. This allows only the movement information of the item to be obtained. The movement information of the item indicates the area in the image where the item is shown. Here, the area in which the item is shown as indicated by the movement information of the item is referred to as the "item area". The item area is estimated from the movement information of the item.

[0020] FIG. 5(a) shows the result of pose estimation for image It1 shown in FIG. 4(a). The positions of multiple joints 100 are estimated by pose estimation. From the result of pose estimation, a worker area 101 is identified as shown in FIG. 5(a). FIG. 5(b) shows movement information 102 from time t1 to time t2. By removing a part of FIG. 5(b) using the worker area 101 as a mask, the movement information of the item shown in FIG. 5(c) is obtained. The movement information of the item indicates an item area 103 in which an item is captured in image It1.

[0021] The processing device 20 copies the motion information shown in FIG. 5(c). The processing device 20 calculates the correlation coefficient with the copied motion information while shifting the position of the motion information up, down, left, and right. This results in a two-dimensional correlation coefficient map. The processing device 20 estimates the coordinates at which the maximum correlation coefficient is obtained from the obtained correlation coefficient map as the center of the object area. When calculating the correlation coefficient, preprocessing may be applied to the copied motion information. For example, the copied motion information may be inverted up and down, left and right, or up and down and left and right to obtain the center coordinates of an object that translates up and down, an object that translates left and right, or an object that rotates. The preprocessing applied to the copied motion information is not limited to these.

[0022] The processing device 20 scans the article area in N directions at equal intervals from the center of the article area to estimate the article's contour points. For example, the point in the correlation coefficient map where the value first drops in the scanning direction is adopted as the contour point. In this way, N contour points are obtained. As an example, N is set to 36.

[0023] The processing device 20 extracts n contour points from N contour points. The value n is smaller than the value N. For example, the processing device 20 extracts n contour points by a greedy method. In the greedy method, the angle between a contour point of interest and an adjacent contour point is calculated. The processing device 20 calculates the angle between each contour point and an adjacent contour point. The processing device 20 extracts n contour points in ascending order of angle. For example, when the shape of an article is equal to or can be approximated to an m-polygon when viewed from above, the value n is set to a value m. Note that when an article is circular, the angles between adjacent contour points are substantially equal. In this case, the value n may be equal to the value N. That is, the process of extracting n contour points may be omitted.

[0024] FIG. 6(a) shows the result of estimating the center 103a and setting N contour points 103b for the object region 103 shown in FIG. 5(c). In this example, N is set to 30. Therefore, 30 contour points 103b are set. In the example shown in FIG. 5(c) and FIG. 6(a), the shape of the object A1 is rectangular. At the corners of the rectangle, the angle between adjacent contour points is small. As shown in FIG. 6(b), the processing device 20 extracts four contour points 103b corresponding to the corners of the rectangle from the 30 contour points. The contour of the object A1 is estimated by connecting the four contour points 103b.

[0025] The processing device 20 uses the n contour points to search for a polygon that is most likely to represent the shape of the object. Specifically, the processing device 20 depicts a preset shape of the object using one of the n sides as a reference. Based on the estimated contour, the processing device 20 calculates the likelihood of the position of the depicted shape. The processing device 20 depicts the shape using each side as a reference and calculates the likelihood for each. The processing device 20 adopts the position of the shape with the highest likelihood as the position of the shape of the object A1 depicted in the image.

[0026] As shown in Fig. 6(c), a rectangle 104 based on the contour point 103b assumed by Fig. 6(b) has four sides 104a to 104d. Fig. 7(a) shows the result of drawing a preset rectangle based on the side 104a. Similarly, Figs. 7(b) to 7(d) show the result of drawing a preset rectangle based on the sides 104b to 104d.

[0027] The processing device 20 calculates the likelihood between the rectangle 104 shown in FIG. 6(c) and the rectangles 105a to 105d shown in FIG. 7(a) to FIG. 7(d) as the likelihood of the rectangles 105a to 105d. The average value inside the rectangle of the obtained correlation coefficient map is used as the likelihood. As an example, for the rectangles 105a to 105d shown in FIG. 7(a) to FIG. 7(d), "0.9", "0.4", "0.2", and "0.8" are calculated as the likelihood, respectively. The processing device 20 adopts the rectangle 105a for which the maximum likelihood is obtained. Alternatively, in the calculation of the likelihood, the image may be cut out based on each of the rectangles 105a to 105d. The processing device 20 may input each image into a model for state classification described later, and obtain the confidence level for the classification result as the likelihood.

[0028] The processing device 20 adopts the position of the shape for which the maximum likelihood is obtained as the position of the shape of the article at the time when one of the two images is captured. The processing device 20 calculates coordinates indicating the position of the article from the adopted shape. For example, the processing device 20 sets the center coordinates of the adopted shape as the article position. Alternatively, the processing device 20 may calculate the article position from the adopted shape according to preset conditions. The processing device 20 outputs the coordinates as an estimation result of the article position.

[0029] In addition, it is preferable that the image capturing times of the two images used to estimate the motion information are far enough apart that the movement of the worker or the object is sufficiently visible. As an example, the image capturing device 10 captures a video at 25 fps. Therefore, when images captured at adjacent times are extracted, the difference between their image capturing times is 1 / 25 seconds. The movement of the worker or the object is unlikely to appear within a 1 / 25 second interval. The influence of image noise, etc. becomes large, and erroneous motion information is likely to be generated. For example, it is preferable that the difference between the image capturing times of the two images used to estimate the motion information is longer than 1 / 20 seconds and shorter than 1 / 2 seconds.

[0030] Furthermore, the sampling rate of the video captured by the image capture device 10 may be dynamically changed. For example, if the worker or object is moving quickly, the sampling rate is increased. This change in speed can be determined based on the magnitude of the immediately preceding optical flow and the magnitude of the difference between the estimated posture coordinates of the worker.

[0031] The orientation of the item is determined based on the amount of rotation of the item relative to the initial state. For example, when the position of the item is estimated from an image captured initially, the orientation is set for the item. Each time the processing device 20 estimates the position of the item, it calculates the amount of rotation of the estimated position relative to the estimation result of the previous position. Template matching, for example, is used to calculate the amount of rotation. Specifically, an image cut out based on the estimation result of the previous position is used as a template. The similarity between the image cut out based on the estimated position is calculated while rotating the image. The angle at which the maximum similarity is obtained corresponds to the amount of rotation of the item.

[0032] When performing template matching, it is preferable to search for the amount of rotation based on the previous estimation result. This can reduce the amount of calculation. Also, the difference in brightness between corresponding points in the images may be compared with a preset threshold. If the difference is smaller than the threshold, it is determined that no change has occurred between those points. This can reduce erroneous determinations in template matching.

[0033] FIG. 8(a) shows the position of the article estimated at an initial time t11. In the example of FIG. 8(a), a rectangle 110 including corners 111a to 111d is estimated. The directions of "North" (N), "East" (E), "South" (S), and "West" (W) are set for the sides 112a to 112d between the corners 111a to 111d. "North" and "West" are indicated by a thick solid line and a thick dashed line, respectively. FIG. 8(b) shows the result of estimating the position at time t12, which is later than time t11. A rectangle 120 including corners 121a to 121d and sides 122a to 122d is estimated. From the history of the rotation amount of the article calculated between time t11 and time t12, it is estimated that the sides 122a to 122d correspond to the sides 112a to 112d, respectively. As a result, "north," "east," "south," and "west," which correspond to the orientations of sides 112a to 112d, respectively, are set for sides 122a to 122d, respectively.

[0034] Through the above processing, the position and orientation of the article are estimated. Here, an example in which the shape of the article is rectangular has been described. Even if the shape of the article is other than rectangular, the position and orientation of the article can be estimated by a similar method.

[0035] In the example of FIG. 9(a), a worker W is working on a star-shaped (equisceles hexagram) object A2. FIG. 9(b) shows an image obtained by capturing the scene of FIG. 9(a). The processing device 20 estimates an object region 131 shown in FIG. 9(c) using an image 130 shown in FIG. 9(b) and another image. In this example, the object region 131 is a hexagon 132 consisting of six contour points 132a and six sides 132b. As shown in FIG. 9(c), depending on the shape of the object, the movement of the worker, and the movement of the object, the object region may not correspond to the actual shape of the object.

[0036] As shown in FIG. 10(a), a star shape 133a is preset as the shape of the article A2. Furthermore, a hexagon 133b is preset as the shape corresponding to the star shape 133a. The processing device 20 draws the preset hexagon 133b based on each side 132b of the hexagon 132. As a result, six hexagons based on the six sides 132b are drawn. FIGS. 10(b) to 10(d) show some of those hexagons, 134a to 134c, as examples. The processing device 20 calculates the likelihood for each of the six hexagons. The processing device 20 draws the star shape 133a based on the hexagon for which the maximum likelihood is obtained. The processing device 20 adopts the drawn star shape 133a as the shape of the article.

[0037] Thereafter, the position and orientation are estimated using the estimated shape. The number of pieces of information set to indicate the orientation of the item is arbitrary. In the rectangular example shown in Fig. 8(a) and Fig. 8(b), the orientation of the item is indicated using four pieces of information (north, east, south, and west). For the star-shaped item A2 shown in Fig. 9(a), the orientation of the item may be indicated using six directions 135a to 135f, for example, as shown in Fig. 10(a).

[0038] FIG. 11 is a flowchart showing a method for estimating an item position. The processing device 20 estimates the position of the item at time t by processing the flowchart shown in FIG. 11. First, the processing device 20 judges whether an image at time t+d can be acquired (step S40a). That is, the processing device 20 judges whether an image at time t+d has been acquired by the imaging device 10. If an image at time t+d can be acquired, the processing device 20 acquires an image at time t and an image at time t+d (step S40b). The processing device 20 estimates motion information from the image at time t and the image at time t+d (step S40c). The processing device 20 uses this motion information as the motion information at time t. The processing device 20 estimates the item area from the motion information (step S40d). At this time, the result of pose estimation at time t is used as a mask.

[0039] The processing device 20 estimates the center of the object region (step S40e). The processing device 20 estimates N contour points of the object using the estimated center (step S40f). The processing device 20 extracts n contour points from the N contour points (step S40g). The processing device 20 searches for a polygon that is most likely to represent the shape of the object using the n contour points (step S40h). The processing device 20 adopts the coordinates of the center of the polygon obtained by the search as the object position. A value obtained by adding t' to the current time t is set as time t (step S40i). Then, step S40a is executed again. Thereby, each time an image at time t+d is obtained, the estimation result of the object position at time t is repeatedly updated. If it is determined in step S40a that an image at time t+d cannot be obtained, the processing device 20 ends the object position estimation process.

[0040] (Tracking process) In addition to estimating the position using the motion information as described above, the processing device 20 may further perform a tracking process, in which the position in a newly acquired image is tracked using the results of estimating the previous position.

[0041] Specifically, the processing device 20 cuts out a portion of an object from a previous image based on the result of estimating the position in that image. The processing device 20 stores the cut-out image as a template image. When a new image is acquired, the processing device 20 performs template matching and searches for an area in the new image that has the highest similarity. The processing device 20 uses the area obtained by the search as the result of estimating the position in the new image.

[0042] FIG. 12 is a schematic diagram for explaining a result of position estimation when performing tracking processing. In FIG. 12, the horizontal axis indicates time. The vertical axis indicates the number of candidates for the estimated position. For example, at time t, an item position E1 is estimated using motion information between an image at time t and an image at time t+d. The number of candidates for the item position for time t is "1". Similarly, at the next time t+t', an item position E2 is estimated using motion information between an image at time t+t' and an image at time t+t'+d. The processing device 20 further estimates an item position E11 in an image at time t+t'+d using a template image based on the item position E1. As a result, the number of candidates for the estimated item position for time t+t' becomes "2".

[0043] After that, the same process is repeated every time a new image is acquired. For example, at time t+xt′, the tracking process based on the item position E1 is repeated to obtain the item position E1. x By repeating the tracking process based on the item position E2, the item position E2 x-1 is estimated. At each instant, the processing device 20 adopts the most probable article position as the final article position.

[0044] As the likelihood used to narrow down the final article position, for example, the similarity between an image based on each article position and a master image prepared in advance is used. Each image may be input to a model for state classification, and the confidence level for the classification result may be used as the likelihood.

[0045] Alternatively, the likelihood may be calculated using a judgment model. This judgment model includes a deep learning model. The processing device 20 cuts out an image based on the estimation result of the item position, and inputs it to the judgment model. The judgment model judges whether the input image is an image cut out along the outer edge (four sides) of the item. The judgment model outputs a scalar value from 0 to 1 according to the input of the image. The output is closer to 1 as the outer edge of the input image is closer to the outer edge of the item. For example, if a part of the floor surface other than the item is cut out, or if only a part of the item is cut out, the output is smaller. The processing device 20 cuts out an image for each estimated item position, and obtains an output for each image. The processing device 20 acquires these outputs as the likelihood for each item position.

[0046] When calculating the likelihood, the direction of imaging by the imaging device 10 may be taken into consideration. For example, the imaging device 10 captures an image of the worker and the item from a direction inclined relative to the vertical direction. In this case, the item appears differently in the image at a position close to the imaging device 10 and at a position far from the imaging device 10. For example, the side closer to the imaging device 10 appears longer, and the side farther from the imaging device 10 appears shorter. Based on this geometric condition, the length of the side serving as the reference for the inclination is stored in the storage device 30 in advance. The processing device 20 reads the length of the reference side stored in the storage device 30 for the angle θq of each candidate item position being tracked. The processing device 20 uses the difference between the length of the reference side and the length of the side of the item position being tracked as the likelihood.

[0047] FIG. 13, FIG. 14(a) to FIG. 14(d), and FIG. 15(a) to FIG. 15(d) are diagrams for explaining the processing by the processing system according to the embodiment. 13, a worker W is working on an item A3. The imaging device 10 captures images of the worker W and the item A3 from diagonally above. When viewed vertically, the shape of the item A3 is rectangular.

[0048] 14(a) to 14(c), the article A3 appears differently depending on the relative orientation of the article A3 with respect to the imaging device 10. The processing device 20 utilizes this fact to calculate the likelihood of the article position.

[0049] Specifically, the processing device 20 generates a line segment corresponding to the estimated item position according to a preset rule. In the example of Fig. 14(a) to Fig. 14(c), the processing device 20 first determines the short sides of the item based on the estimated item position. The processing device 20 generates a line segment Li connecting the short sides. The processing device 20 calculates the length of the line segment Li. The processing device 20 also calculates the angle between a reference line BL and the line segment Li. In this example, the reference line BL is parallel to the horizontal direction of the image.

[0050] As a result of the calculation, angles θ1 to θ3 and lengths L1 to L3 are calculated for the examples of Figures 14(a) to 14(c). Angle θ1 is greater than angle θ2, and length L1 is shorter than length L2. Angle θ2 is greater than angle θ3, and length L2 is shorter than length L3. That is, as shown in Figure 14(d), the greater the angle, the shorter the length of line segment Li. Such correspondence between angles and lengths is stored in advance in storage device 30.

[0051] Figures 15(a) to 15(c) show rectangles q1 to q3 obtained by the search, respectively. The processing device 20 calculates the lengths and angles of the line segments connecting the short sides of each of the rectangles q1 to q3. As a result of the calculation, angles θq1 to θq3 and lengths Lq1 to Lq3 are calculated for the examples of Figures 15(a) to 15(c), respectively.

[0052] The processing device 20 refers to the correspondence relationship and obtains the length corresponding to the calculated angle. The processing device 20 calculates the difference between the length corresponding to the angle and the calculated length, and calculates the likelihood according to the difference. The greater the difference, the smaller the calculated likelihood.

[0053] For example, as shown in FIG. 15(d), the processing device 20 calculates the difference Dq1 between the length Lq1 and the length corresponding to the angle θq1 for the rectangle q1. Similarly, the processing device 20 calculates the difference Dq2 and the difference Dq3 for each of the rectangles q2 and q3. The processing device 20 uses the differences Dq1 to Dq3 to calculate the likelihoods of the rectangles q1 to q3, respectively. In this example, the difference Dq1 is smaller than the difference Dq3 and larger than the difference Dq2. Therefore, the likelihood of the rectangle q2 is larger than the likelihood of the rectangle q3 and smaller than the likelihood of the rectangle q1.

[0054] The more candidates for the item position, the more accurately the item position can be estimated. On the other hand, if there are too many candidates, the amount of calculation required for the tracking process becomes excessive, and the process may be delayed. For this reason, it is preferable to predefine the number of candidates to be held. In the example shown in FIG. 12, "x+1" is set as the prescribed number. For time t+xt', x+1 item positions are estimated. For time t+(x+1)t', x+2 item positions are estimated. The processing device 20 narrows down the x+2 item positions to x+1 item positions. In the illustrated example, the result of the tracking process based on the item position Ex is excluded, and other item positions are extracted. The above-mentioned certainty can be used to narrow down the item positions. The processing device 20 extracts x+1 item positions in descending order of certainty.

[0055] FIG. 16 is a flowchart showing an outline of the tracking process. The processing device 20 judges whether an image at time t+d can be acquired (step S41a). If an image at time t+d can be acquired, the processing device 20 acquires an image at time t and an image at time t+d (step S41b). The processing device 20 executes a position update process using the image at time t+d (step S41c). A value obtained by adding t' to the current time t is set as time t (step S41d). Then, step S41a is executed again. If it is determined in step S41a that an image at time t+d cannot be acquired, the processing device 20 ends the tracking process.

[0056] FIG. 17 is a flowchart showing the update process in the tracking process. In the position update process, the processing device 20 cuts out a portion corresponding to the most recently estimated position from the image at time t. The processing device 20 acquires the cut-out image as a template image at time t (step S42a). The processing device 20 compares the template image at time t and the image at time t+d in the tracking candidate area (step S42b). The tracking candidate area is a part of the cut-out image, and is set according to parameters set in advance. For example, an area of ​​50% horizontal and 50% vertical of the cut-out image centered on the item position at time t is set as the tracking candidate area. The processing device 20 judges whether the difference in brightness value between the two images exceeds a threshold value (step S42c). If the difference exceeds the threshold value, the processing device 20 searches for the most similar position and orientation in the image at time t+d while changing the position and orientation of the template image (step S42d). The processing device 20 updates the most recently estimated item position to the item position obtained by the search (step S42e). In step S42c, if the difference in luminance value is equal to or less than the threshold, the update process is skipped. If skipped, the estimation result at time td is used. This suppresses drift in template matching.

[0057] FIGS. 18(a) to 18(c) and 19(a) to 19(c) are images for explaining the processing by the processing system according to the embodiment. Fig. 18(a) shows an image It1 captured at time t1. Fig. 18(b) shows an image It2 captured at time t2. Fig. 18(c) shows an article position estimated based on images It1 and It2. In the position estimation, a rectangle 106a is employed.

[0058] FIG. 19(a) shows a template image Tt0 at time t0. Time t0 is before time t1. Using the estimation result of the item position at time t0, a template image is cut out from the image at time t0. FIG. 19(b) shows an image It1 captured at time t1. FIG. 19(c) shows a rectangle 106b obtained by a tracking process using the template image Tt0. The rectangle 106b indicates the item position in the image It1. For example, the final item position is narrowed down from x item positions including the item position shown in FIG. 18(c) and the item position shown in FIG. 19(c).

[0059] (State Estimation) The processing device 20 estimates the state of an item depicted in an image from the image. For example, template matching is used to estimate the state. The processing device 20 compares the image with a plurality of template images prepared in advance. Each template image is associated with a state of the item. The processing device 20 extracts the template image with the highest similarity. The processing device 20 estimates the state associated with the extracted template image as the state of the item depicted in the image.

[0060] Alternatively, the processing device 20 may input the image to a state estimation model. The state estimation model is trained in advance to estimate the state of an item depicted in an image in response to the input of the image. For example, the state estimation model includes a neural network. The state estimation model preferably includes a CNN. The processing device 20 acquires an estimation result by the state estimation model.

[0061] It is preferable that the processing device 20 cuts out a portion from the entire image, which also shows a worker and the like other than the item. The item is shown in a part of the cut-out image. The cut-out may be performed using an estimation result of the position of the item. Cutting out increases the proportion of the area of ​​the item shown in the image. This reduces the influence of elements other than the item on the estimation of the state. As a result, the accuracy of the state estimation can be improved. If the image is not cut out, it is also possible to directly estimate the state of the item from the image acquired by the imaging device 10.

[0062] (Estimation of work location) The processing device 20 estimates the location of the worker's work relative to the item based on the estimated results of the worker's posture, the estimated results of the item's position, and the estimated results of the item's orientation. For example, the processing device 20 obtains the positions of the worker's left and right hands from the estimated results of the posture. The processing device 20 calculates the relative positions and orientations of the left and right hands with respect to the item. Based on the relative positional relationship, the processing device 20 estimates the location of the work relative to the item.

[0063] FIG. 20 is a schematic diagram for explaining a method for estimating a work location. In the example of FIG. 20, the position of the left hand 140a of the worker 140 (x left ,y left ) and the position of the right hand 140b (x right ,y right ) are estimated. As the position of the article 141, the center 142 and the positions (x0, y0), (x1, y1), (x2, y2), and (x3, y3) of the four corners 142a to 142d are estimated. In addition, the orientation of the article, "north", "east", "south", and "west", are estimated. The orientation of the article is classified by boundaries 143a and 143b that pass through the center 142. In this example, the diagonals of the rectangular article 141 are set as the boundaries 143a and 143b. The direction and number of the boundaries are set appropriately according to the shape of the article.

[0064] The processing device 20 sets gates for estimating the work location based on the position and orientation of the article. For example, the processing device 20 sets north, east, south, and west gates along each side of the article 141. As shown by line Li1, the left hand 140a faces the "east" gate. As shown by line Li2, the right hand 140b faces the "north" gate. Lines Li1 and Li2 are extensions of the left lower arm and the right lower arm, respectively. The lower arm is a line segment (bone) connecting the wrist and elbow.

[0065] Based on the positions of the joints and each gate, the processing device 20 estimates that the left hand 140a is located on the east side of the item 141. That is, the location of the work with the left hand is estimated to be on the east side of the item. The processing device 20 also estimates that the right hand 140b is located on the north side of the item 141. That is, the location of the work with the right hand is estimated to be on the north side of the item.

[0066] Any joint may be used to estimate the working location. For example, the positions of a finger, a wrist, or an elbow may be used to estimate the working location depending on the work to be performed. The positions of multiple joints among these may be used to estimate the working location.

[0067] FIG. 21 is a flowchart showing a method for estimating a work location. The processing device 20 sets a gate in each direction of the article based on the estimated position and orientation of the article (step S61). The processing device 20 determines whether the worker's lower arms cross the gate (step S62). If the lower arms cross the gate, the processing device 20 sets the positions of the left hand and right hand as the work position (step S63). If the lower arms do not cross the gate, the processing device 20 sets the intersection of the extension line of the lower arms and the gate as the work position (step S64). The processing device 20 estimates that the gate where the lower arms or the extension line cross is the work location (step S65).

[0068] (Estimation of Work) The processing device 20 generates first graph data based on the estimated posture, the estimated state, and the estimated working location. The graph data has a graph-type data structure. The first graph data includes a plurality of nodes and a plurality of edges. The nodes that are related to each other are connected by the edges.

[0069] 22 to 24 are schematic diagrams illustrating examples of the structure of the first graph data. The first graph data GD1 includes first data D1, second data D2, and third data D3, as shown in Fig. 22. The first data D1, second data D2, and third data D3 are separated from each other and are not connected by edges.

[0070] The first data D1 is generated based on the posture estimation result. The first data D1 includes a plurality of first nodes n1 and a plurality of first edges e1. The plurality of first nodes n1 correspond to a plurality of joints of the worker, respectively. The plurality of first edges e1 correspond to a plurality of skeletons of the worker, respectively. In the example shown in FIG. 22, the positions of the ankles, knees, waist, chest, wrists, elbows, and head are estimated from the image, and a plurality of first nodes n1 corresponding thereto are set. In addition, a plurality of first edges e1 are set corresponding to the skeletons connecting those joints. Therefore, no edge is set between joints that are not actually connected. For example, the wrist node and the elbow node are connected by an edge, but the wrist node and the waist node are not connected.

[0071] The second data D2 is generated based on the state estimation result. The second data D2 includes a plurality of second nodes n2 and a plurality of second edges e2. The plurality of second nodes n2 correspond to a plurality of states that the item can take. Each of the plurality of second edges e2 corresponds to a transition of the state of the item. In the example shown in FIG. 22, eight second nodes n2 are set because the item can take eight states. Nine second edges e2 are set corresponding to the possibility of transition of those states. For example, a transition from the first state to the second state, a transition from the second state to the first state, and a transition from the first state to the third state may occur. The second edges e2 are set corresponding to these transitions.

[0072] The third data D3 is generated based on the estimation result of the work location. The third data D3 includes a plurality of third nodes n3 and a plurality of third edges e3. The plurality of third nodes n3 correspond to a plurality of locations where the item can be worked on. Each of the plurality of third edges e3 indicates the relationship between the locations where the work can be performed. For example, in actual work, the locations where a transition can occur are connected by an edge. In the example shown in FIG. 22, the north node and the east node are connected by an edge. This indicates that the location where the work can be performed can transition from the north location to the east location, or from the east location to the north location. Similarly, an edge is set between the east node and the south node, between the south node and the west node, and between the west node and the north node. This indicates that the location where the work can be performed can transition between the respective nodes. In addition, no edge is set between the north location and the south location, and between the east location and the west location. This indicates that there are no transitions of work locations between the North location and the South location, and between the East location and the West location.

[0073] As shown in FIG. 23, in the first graph data GD1, the first node n1 and the second node n2 may be connected by an edge e, and the first node n1 and the third node n3 may be connected by another edge e. Each second node n2 is connected to the first node n1 having the association. Each third node n3 is connected to the first node n1 having the association. As an example, when the association between the position of the worker's waist and a certain state of the item is higher than the association of other combinations, the first node n1 of the waist and the second node n2 of that state of the item are connected by an edge e. When the association between the position of the worker's left elbow and another state of the item is higher than the association of other combinations, the first node n1 of the left elbow and the second node n2 of that other state of the item are connected by an edge e. When a certain part of the item is mainly worked on by the right hand of the worker, the third node n3 corresponding to that part is connected to the first node n1 corresponding to the right hand by an edge e. When another part of the article is operated on mainly with the worker's left hand, the third node n3 corresponding to that another part is connected to the first node n1 corresponding to the left hand by an edge e.

[0074] As shown in Fig. 24, a second edge e2 connecting the second nodes n2 may be further set in the first graph data GD1 shown in Fig. 23. A third edge e3 connecting the third nodes n3 may be further set. By setting the second edge e2 or the third edge e3 in addition to the edge e connecting the first node n1 and the second node n2 or the edge e connecting the first node n1 and the third node n3, it is possible to further improve the accuracy of the task estimation.

[0075] The first graph data GD1 may include more nodes and edges than the illustrated example. For example, nodes corresponding to the neck, shoulders, fingers, etc., and edges corresponding to these nodes may be further set. More states may be set as possible states of an item. Work locations may be classified more finely according to the size of the item. By increasing the number of nodes, the accuracy of work estimation can be further improved. The number of nodes set can be appropriately changed according to the processing capacity of the processing device 20.

[0076] FIG. 25 is a schematic diagram illustrating the structure of a neural network. For example, graph data is input to a neural network 200 shown in Fig. 25. The neural network 200 includes an input layer 210, a GNN 220, and a fully connected layer 230. The GNN 220 includes a plurality of convolution layers 220a and a pooling layer 220b. Any of the first graph data GD1 shown in Figs. 22 to 24 is input to the input layer 210. The size of the input layer 210 is set according to the number of nodes in the first graph data GD1 and the number of edges in the first graph data GD1.

[0077] Fig. 26(a) is a diagram for explaining a specific structure of the first graph data, Fig. 26(b) is a diagram illustrating an example of a feature vector, and Figs. 27(a) to 27(f) are diagrams illustrating specific examples of the feature vector. Here, an example will be described in which the first graph data GD1 shown in Fig. 23 or 24 is used. As shown in Fig. 26(a), the first graph data includes a set vector V and an adjacency matrix A. The set vector V includes values ​​of each of a plurality of nodes. Each node is represented by a feature vector v. As shown in Fig. 26(b), the feature vector v includes five values: the X coordinate of the joint, the Y coordinate of the joint, a value indicating the state of the item, a value indicating the working location of the right hand, and a value indicating the working location of the left hand.

[0078] As an example, FIG. 27(a) shows a feature vector v1 corresponding to the left wrist node. The feature vector v1 includes the X coordinate of the left wrist and the Y coordinate of the left wrist. The feature vector v1 does not indicate the state of the item or the work location, so these values ​​are set to "0". FIG. 27(b) shows the value of a feature vector v2 corresponding to the left elbow node. The feature vector v2 includes the X coordinate of the left elbow and the Y coordinate of the left elbow. The feature vectors v1 and v2 are examples of the first node n1.

[0079] FIG. 27(c) shows the feature vector v corresponding to the first item state node. 12 The feature vector v 12 In the model, a value between 0 and 1 is set as a value indicating the state of an item, and other values ​​are set to "0". The value indicating the state is set based on the state estimation result. The state estimation model outputs the confidence of each state as the state estimation result. For example, the output confidence is set as the value indicating the state of the item. 12 In the example, the confidence level is set to "0.8" as the value indicating the state of the item. Figure 27(d) shows the feature vector v 13 The feature vector v 13 In the feature vector v, the confidence level is set to "0.2" as the value indicating the state of the item. 12 and v 13 is an example of the second node n2.

[0080] Figure 27(e) shows the feature vector v n-1 The feature vector v n-1 contains a value indicating the right hand working position. FIG. 27(f) shows the feature vector v n The feature vector v n contains a value indicating the working position of the left hand. n-1 and v n is an example of a third node n3.

[0081] In the adjacency matrix A shown in Fig. 26(a), the relevance between nodes is represented by "0" or "1." That is, the existence of each of the first edge e1 to the third edge e3 is represented by a value of "0" or "1." "0" indicates that the edge does not exist. "1" indicates that the edge exists.

[0082] In the GNN 220, each convolutional layer 220a convolves each feature vector v according to the following formula (1). In formula (1), v i is the initial feature vector of the i-th node. conv(t) is the updated feature vector of the i-th node, obtained by t convolutions. A(i) is the set of the feature vector of the i-th node in the set vector V and the feature vectors of the nodes defined as adjacent to the i-th node by the adjacency matrix A. Here, "adjacent" means connected by an edge. W(t) is the weight updated t times.

number

[0083] The output result from the convolution layer 220a is further reduced by the pooling layer 220b. The output result from the pooling layer 220b is converted to one-dimensional data by the fully connected layer 230. The output result from the fully connected layer 230 corresponds to an estimated result of an operation. For example, the fully connected layer 230 outputs a confidence for each class. One class corresponds to one operation. The operation corresponding to the class with the highest confidence is the estimated operation being performed.

[0084] FIG. 28 is a schematic diagram illustrating another structure of a neural network. As shown in FIG. 28, the neural network 200a may include input layers 211 to 213, GNNs 221 to 223, and a fully connected layer 230. The input layer 211 receives first data D1 related to the posture. The GNN 221 includes a convolutional layer 221a and a pooling layer 221b, and processes the first data D1. The input layer 212 receives second data D2 related to the state of the item. The GNN 222 includes a convolutional layer 222a and a pooling layer 222b, and processes the second data D2. The input layer 213 receives third data D3 related to the work location on the item. The GNN 223 includes a convolutional layer 223a and a pooling layer 223b, and processes the third data D3. The convolutional layers 221a to 223a convolute the feature vectors of each node, similar to the convolutional layer 220a. The pooling layers 221b to 223b reduce the convoluted feature vectors in the same manner as the pooling layer 220b. The output results of the GNNs 221 to 223 are combined in the fully connected layer 230. The fully connected layer 230 outputs an estimation result of the task.

[0085] 29, 30(a), and 30(b) are schematic diagrams illustrating other structures of graph data. Changes in node values ​​over time may be used for the estimation. For example, as shown in FIG. 29, first graph data GD1 is generated from an image (first image) obtained at time t1. The first graph data GD1 includes a plurality of first nodes n1, a plurality of first edges e1, a plurality of second nodes n2, and a plurality of third nodes n3. Second graph data GD2 is generated from an image (second image) obtained at time t2 after time t1. The second graph data GD2 includes a plurality of first nodes n1, a plurality of first edges e1, a plurality of second nodes n2, and a plurality of third nodes n3, similar to the first graph data GD1.

[0086] The processing device 20 connects the plurality of nodes of the first graph data GD1 and the plurality of nodes of the second graph data GD2 by a plurality of edges e4. Note that in FIG. 29, only some of the edges e4 are shown, and the other edges e4 are omitted. Specifically, the plurality of first nodes n1 of the first graph data GD1 are connected to the plurality of first nodes n1 of the second graph data GD2, respectively. The plurality of second nodes n2 of the first graph data GD1 are connected to the plurality of second nodes n2 of the second graph data GD2, respectively. The plurality of third nodes n3 of the first graph data GD1 are connected to the plurality of third nodes n3 of the third graph data GD3, respectively. The processing device 20 inputs the first graph data GD1 and the second graph data GD2, which are connected to each other, to the neural network 200.

[0087] In another method, a plurality of first nodes included in the first data D1 based on an image at time t1 and a plurality of first nodes included in the first data D1 based on an image at time t2 are connected to each other. A plurality of second nodes included in the second data D2 based on an image at time t1 and a plurality of second nodes included in the second data D2 based on an image at time t2 are connected to each other. A plurality of third nodes included in the third data D3 based on an image at time t1 and a plurality of third nodes included in the third data D3 based on an image at time t2 are connected to each other. These data are input to the neural network 200a.

[0088] As shown in Fig. 30(a), images may be captured at three or more times t1 to t3, and the first graph data GD1 to third graph data GD3 may be generated from the three images. For example, corresponding nodes between the first graph data GD1 and the second graph data GD2 based on images captured at adjacent times are connected by an edge e4. Corresponding nodes between the second graph data GD2 and the third graph data GD3 based on images captured at adjacent times are connected by an edge e4.

[0089] One graph data may be selected from a plurality of graph data, and nodes of the selected graph data may be connected to nodes of other graph data. For example, the third graph data GD3 may be selected from the first graph data GD1 to the third graph data GD3. As shown in FIG. 30(b), each of the nodes of the first graph data GD1 and the second graph data GD2 may be connected to a node of the third graph data GD3. In this case, the node of the first graph data GD1 and the node of the second graph data GD2 are not connected by an edge. Note that in FIG. 30(a) and FIG. 30(b), only a part of edges e4 is shown, and other edges e4 are omitted.

[0090] By representing the changes in each node over time in a graph structure, the accuracy of task estimation can be improved.

[0091] In the example shown in FIG. 30(a) or FIG. 30(b), all of the temporally consecutive images do not have to be used to generate the graph data. For example, when 10 images are acquired between times t1 and t10, 10 pieces of graph data may be generated from the 10 images, and the nodes of the graph data may be connected to each other. A portion of the 10 images may be selected. In this case, graph data is generated from each of the selected images, and the nodes of the graph data are connected to each other. In either case, a plurality of graph data items connected to each other are input to the neural network 200 or the neural network 200a.

[0092] 31 and 32 are schematic diagrams illustrating other structures of neural networks. In order to use the temporal change of the node for estimation, the neural network may include a Long Short Term Memory (LSTM) network. The neural network 200b shown in FIG. 31 further includes an LSTM network 225, in comparison with the neural network 200 shown in FIG. 25. The data (values) output from the GNN 220 are input to the LSTM network 225. The data output from the LSTM network 225 is input to the fully connected layer 230, and the fully connected layer 230 outputs an estimation result of the task.

[0093] The neural network 200c shown in Fig. 32 further includes LSTM networks 225a to 225c compared to the neural network 200a shown in Fig. 28. The data output from the GNNs 221 to 223 are input to the LSTM networks 225a to 225c, respectively. The data output from the LSTM networks 225a to 225c are input to the fully connected layer 230, and the fully connected layer 230 outputs an estimation result of the task.

[0094] (LSTM network) FIG. 33 is a schematic diagram illustrating a specific structure of an LSTM network. As shown in Figure 33, each neuron N in the LSTM network 300 includes a forget gate 310, an input gate 320, and an output gate 330. In Figure 33, x t represents the input value to neuron N at time t. C t represents the state of neuron N at time t. t represents the output value of the forget gate 310 at time t. t represents the output value of the input gate at time t. t represents the output value of the output gate at time t. t represents the output value of neuron N at time t. t , i t , C t ,o t , and h t are respectively expressed by the following formulas 2 to 6.

number

number

number

number

number

[0095] (study) The neural network used to estimate the work is trained in advance. A plurality of training data are used for training. Each training data includes input data and teaching data (labels). The input data has a graph structure and is generated using an image showing an actual work state. The input data may be prepared using a synthetic image that imitates an actual work state. The above-mentioned graph data generation method can be applied to generate the input data. The specific structure of the prepared input data is appropriately changed according to the structure of the trained neural network. When the neural network 200 shown in FIG. 25 is trained, graph data is used in which a node corresponding to the posture, a node corresponding to the state of the item, and a node corresponding to the work location are selectively connected to each other, as shown in FIG. 23 or FIG. 24. When the neural network 200a shown in FIG. 28 is trained, graph data is used in which a node corresponding to the posture, a node corresponding to the state of the item, and a node corresponding to the work location are separated from each other, as shown in FIG. 22. The teaching data is a label indicating the work corresponding to the input data. Each intermediate layer of the neural network is trained so that teaching data is output in response to the input of input data.

[0096] (Example output) FIG. 34 is a schematic diagram illustrating an output result by the processing system according to the embodiment. In Fig. 34, the horizontal direction indicates time. The vertical direction indicates the estimation results of the item state and the work location based on the image at each time. The top row indicates the work estimated based on the item state and the work location.

[0097] While the work is being performed, images showing the state of the work are repeatedly acquired. The processing device 20 repeatedly estimates the work based on each image. This allows estimation of what work the worker was performing at each time. For example, the processing system 1 estimates the work in real time while the work is being performed.

[0098] In the above example, three pieces of information are used to estimate the work: the posture of the worker, the state of the item, and the work location on the item. The embodiment of the present invention is not limited to this example, and one selected from the state and the work location, and the posture of the worker may be used to estimate the work. For example, if the state (appearance) of the item changes as the work progresses, the work can be accurately estimated even without information on the work location. If the work location changes as the work progresses, the work can be accurately estimated even without information on the state of the item.

[0099] Most preferably, three pieces of information, the worker's posture, the state of the item, and the location of the work on the item, are used to estimate the work, which allows a wide range of work to be estimated with greater accuracy.

[0100] Advantages of the embodiment will be described. Various methods have been used to attempt to estimate the work being performed. Work generally involves the repetition of the same movements or small changes in movements. For this reason, estimating work is often more difficult than estimating physical actions such as running, jumping, and bending. In order to accurately estimate work, there is a method in which multiple sensors are attached to the body of a worker. In this method, the work is estimated by determining the worker's small movements from the data of each sensor. In this case, many expensive sensors are required, which requires high costs. In addition, it is time-consuming to attach sensors to the worker, and the sensors may get in the way of the work.

[0101] Regarding this issue, the processing system 1 according to the embodiment uses at least two pieces of information to estimate the work being performed. The at least two pieces of information include the posture of the worker and one or more pieces selected from the state of the item and the location of the work on the item. These pieces of information change depending on the work being performed. Therefore, by using these pieces of information, it becomes possible to estimate the work.

[0102] Furthermore, in the embodiment of the present invention, in order to further improve the accuracy of the estimation, graph data is generated using at least two pieces of information. The graph data includes a plurality of nodes and a plurality of edges. The edges indicate that the nodes have a relationship with each other. For example, the connection between the joints of the body is represented by an edge. The processing system 1 inputs the graph data into the GNN to obtain an estimation result of the task.

[0103] The joints in some parts of the body are connected to each other by a skeleton, and the movements of these joints are related to each other. On the other hand, the movements of joints that are not connected by a skeleton have little correlation with each other.

[0104] For example, in the body, there are combinations of joints with high correlation in movement, such as a combination of a wrist and an elbow, or a combination of an ankle and a knee. On the other hand, there are combinations of joints with low correlation in movement, such as a combination of an ankle and a wrist. By using graph data, it is possible to express the correlation between nodes. In other words, it is possible to learn a set of nodes connected by edges and a set of nodes not connected independently. When data used for estimating an operation does not have a graph structure, all nodes are comprehensively learned as being related to each other. By using graph data, it is possible to estimate a more detailed movement by taking into account the correlation between nodes.

[0105] In addition, the posture of the worker, the state of the item, and the location of the work can all be estimated from the images. The work can be estimated without attaching sensors to the worker. Therefore, the work can be estimated without interfering with the work performed by the worker. The cost required for estimating the work can also be reduced.

[0106] In particular, complex and detailed motions appear at manufacturing sites where goods are manufactured. Furthermore, similar motions may appear even in different tasks. When a large made-to-order product (indented product) is manufactured, thousands to tens of thousands of parts may be assembled. For this reason, the number of tasks is extremely large, and it is not easy to accurately estimate the tasks based on posture alone. By using one or more selected from the state and the task location in addition to the posture, it is possible to accurately estimate the tasks.

[0107] In addition, the manufacturing process of an item includes tasks that are highly dependent on the worker, such as assembly work. Assembly work is often complex and diverse. In order to flexibly respond to assembly work, the worker's adaptability is required. By estimating the work content of the worker while the work is being performed by the worker, improvements such as efficiency improvement, standardization, and yield improvement can be expected. However, there are many types of assembly work at manufacturing sites. Furthermore, the work content, its complexity, and work time vary depending on the product to be manufactured. The work time also varies widely from task to task, from a few minutes to a few hours, days, or months. In some cases, more than 10,000 parts need to be installed to complete one product. When work estimation is performed for the manufacturing process of these products, the combination of information that is effective for the estimation is different. There may be cases where information that is unnecessary when estimating the assembly work of one product is necessary when estimating the assembly work of another product. Work analysis using various methods or models has been performed on the assembly work of these products. However, these methods and models are effective only for the assembly work of a specific product. In most cases, it is difficult to apply the method to assembly work of different products. Therefore, in the conventional estimation method, it is necessary to build and manage the same number of work analysis methods or models as the number of product types. When analyzing a new assembly process using the conventional estimation method, it is necessary to build a method or model corresponding to the new assembly process from scratch.

[0108] The inventors of the present application have discovered that by combining a plurality of pieces of information using a GNN, it is possible to estimate with high accuracy the work being performed from a plurality of tasks in a generalized manner for the assembly work of various products. For example, by providing an integrated neural network for estimation for a site where various tasks may be performed, it is possible to estimate with high accuracy the work being performed by a worker. According to the embodiment of the present invention, it is possible to estimate the work more simply and with high accuracy.

[0109] In addition, in the processing system 1, auxiliary sensors may be attached to the body of the worker. For example, in addition to the posture of the worker, the state of the item, and the work location, the acceleration and angular velocity of a part of the body may be used to estimate the work. Even in this case, the number of sensors required can be reduced compared to when the work is estimated from the sensors alone.

[0110] FIG. 35 is a schematic diagram showing a hardware configuration. The processing device 20 includes, for example, the hardware configuration shown in Fig. 35. A computer 90 shown in Fig. 35 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0111] The ROM 92 stores a program for controlling the operation of the computer. The ROM 92 stores programs necessary for the computer to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0112] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0113] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs.

[0114] The input interface (I / F) 95 connects the computer 90 and the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0115] The output interface (I / F) 96 connects the computer 90 and the output device 96a. The output I / F 96 is, for example, a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI (registered trademark)). The CPU 91 transmits data to the output device 96a via the output I / F 96 and can cause the output device 96a to display an image.

[0116] A communication interface (I / F) 97 connects the computer 90 to a server 97a external to the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97. The camera 99 photographs an item and stores the image in the server 97a.

[0117] The storage device 94 includes one or more selected from a hard disk drive (HDD) and a solid state drive (SSD). The input device 95a includes one or more selected from a mouse, a keyboard, a microphone (audio input), and a touch pad. The output device 96a includes one or more selected from a monitor, a projector, a speaker, and a printer. A device having the functions of both the input device 95a and the output device 96a, such as a touch panel, may be used.

[0118] The storage device 94 can be used as the storage device 30. The camera 99 can be used as the imaging device 10.

[0119] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0120] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0121] Implementations of the invention may include the following features. (Feature 1) A posture of the worker is estimated from a first image showing the worker and an object; Estimating, from the first image, one or more selected from a state of the item and a location of a task performed on the item by the worker; generating first graph data including a plurality of nodes and a plurality of edges based on the one or more selected from the state and the working location and the posture; A processing system that inputs the first graph data into a neural network, including a graph neural network (GNN), and uses results output from the neural network to infer an operation being performed by the worker. (Feature 2) The state is estimated from the first image; The processing system of feature 1, wherein the first graph data includes a plurality of first nodes each corresponding to a plurality of joints of the worker, a plurality of first edges each corresponding to a plurality of bones of the worker, and a plurality of second nodes each corresponding to a plurality of states that the item can take. (Feature 3) 3. The processing system of claim 2, wherein in the first graph data, each of the second nodes is connected to any of the first nodes by an edge. (Feature 4) The first graph data is First data including the plurality of first nodes and the plurality of first edges; second data including the plurality of second nodes and a plurality of second edges each indicating a relationship between the second nodes, the second data being separated from the first data; Including, 3. The processing system of claim 2, wherein the first data and the second data are input to the neural network. (Feature 5) The GNN includes a first GNN and a second GNN, In the neural network, the first data is input to the first GNN, and the result output from the first GNN and the second data are input to the second GNN, and the result output from the second GNN are input to a fully connected layer; 5. The processing system of claim 4, further comprising: estimating the task using a result output from the fully connected layer. (Feature 6) The work location is estimated from the first image; The processing system of feature 1, wherein the first graph data includes a plurality of first nodes each corresponding to a plurality of joints of the worker, a plurality of first edges each corresponding to a plurality of bones of the worker, and a plurality of third nodes each corresponding to a plurality of locations of the item. (Feature 7) Both the state and the work location are estimated from the first image; The processing system of feature 1, wherein the first graph data includes a plurality of first nodes each corresponding to a plurality of joints of the worker, a plurality of first edges each corresponding to a plurality of bones of the worker, a plurality of second nodes each corresponding to a plurality of states that the item can take, and a plurality of third nodes each corresponding to a plurality of locations of the item. (Feature 8) A posture of the worker is estimated from a second image showing the worker and the object; Estimating, from the second image, one or more selected from a state of the item and a location of a task performed on the item by the worker; generating second graph data including a plurality of nodes and a plurality of edges using the result estimated from the second image; inputting the second graph data to the neural network in addition to the first graph data, and estimating the task using a result output from the neural network; 2. The processing system according to claim 1. (Feature 9) the plurality of nodes of the first graph data and the plurality of nodes of the second graph data are connected to each other by a plurality of edges; 9. The processing system of claim 8, wherein the first graph data and the second graph data are input to the neural network. (Feature 10) the second image is an image obtained after the first image; the second graph data is input to the neural network after the first graph data; 9. The processing system of feature 8, wherein in the neural network, the first graph data is input to the GNN, and the result output from the GNN is input to a Long Short Term Memory (LSTM) network, and then the second graph data is input to the GNN, and the result output from the GNN is input to the LSTM network. (Feature 11) In the neural network, the first graph data is input to the GNN, and a result output from the GNN and the second graph data are input to the GNN, and a result output from the GNN are input to a fully connected layer; 9. The processing system of claim 8, further comprising: estimating the task using a result output from the fully connected layer. (Feature 12) The processing device includes: A posture of the worker is estimated from a first image showing the worker and an object; Estimating, from the first image, one or more selected from a state of the item and a location of a work performed on the item by the worker; generating first graph data including a plurality of nodes and a plurality of edges based on the one or more selected from the state and the working location and the posture; inputting the first graph data into a neural network including a graph neural network (GNN), and estimating the task being performed by the worker using a result output from the neural network; Processing methods. (Feature 13) A program for causing a computer to execute the processing method according to feature 12. (Feature 14) A storage medium storing a program for causing a computer to execute the processing method according to feature 12.

[0122] According to the embodiment described above, a processing system, a processing method, a program, and a storage medium are provided that are capable of estimating an operation more simply and accurately.

[0123] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0124] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0125] 1: processing system, 10: imaging device, 20: processing device, 30: storage device, 40: input device, 50: output device, 90: computer, 91: CPU, 92: ROM, 93: RAM, 94: storage device, 95: input interface, 95a: input device, 96: output interface, 96a: output device, 97: communication interface, 97a: server, 98: system bus, 99: camera, 100: joint, 101: worker area, 102: movement information, 103: item area, 103a: center, 103b: contour point, 104: rectangle, 104a-104d: sides, 105a-105d, 106a, 106b: rectangle, 110: rectangle, 111a-111d: corners, 112a to 112d: edges, 120: rectangle, 121a to 121d: corners, 122a to 122d: edges, 130: image, 131: object region, 132: hexagon, 132a: contour point, 132b: edges, 133a: star, 133b, 134a to 134c: hexagon, 135a to 135f: direction, 140: worker, 140a: left hand, 140b: right hand, 141: object, 142: center, 142a to 142d: corners, 143a, 143b: boundary line, 200, 200a to 200c: neural network, 210, 211 to 213: input layer, 220 to 223: GNN, 220a to 223a: convolution layer, 220b to 223b: pooling layer, 225, 225a to 225c: LSTM network, 230: fully connected layer, D1: first data, D2: second data, D3: third data, GD1: first graph data, GD2: second graph data, GD3: third graph data, e: edge, e1: first edge, e2: second edge, e3: third edge, e4: edge, n1: first node, n2: second node, n3: third node

Claims

1. A posture of the worker is estimated from a first image showing the worker and an object; Estimating, from the first image, one or more selected from a state of the item and a location of a work performed on the item by the worker; generating first graph data including a plurality of nodes and a plurality of edges based on the one or more selected from the state and the working location and the posture; A processing system that inputs the first graph data into a neural network, including a graph neural network (GNN), and uses results output from the neural network to infer an operation being performed by the worker.

2. The state is estimated from the first image; 2. The processing system according to claim 1, wherein the first graph data includes a plurality of first nodes each corresponding to a plurality of joints of the worker, a plurality of first edges each corresponding to a plurality of bones of the worker, and a plurality of second nodes each corresponding to a plurality of states that the article can take.

3. The processing system according to claim 2 , wherein in the first graph data, each of the second nodes is connected to any of the first nodes by an edge.

4. The first graph data is First data including the plurality of first nodes and the plurality of first edges; second data including the plurality of second nodes and a plurality of second edges each indicating a relationship between the second nodes, the second data being separated from the first data; Including, The processing system of claim 2 , wherein the first data and the second data are input to the neural network.

5. The GNN includes a first GNN and a second GNN, In the neural network, the first data is input to the first GNN, and a result output from the first GNN and the second data is input to the second GNN, and a result output from the second GNN are input to a fully connected layer; The processing system of claim 4 , further comprising: a process for estimating the task using results output from the fully connected layer.

6. The work location is estimated from the first image; 2. The processing system according to claim 1, wherein the first graph data includes a plurality of first nodes corresponding respectively to a plurality of joints of the worker, a plurality of first edges corresponding respectively to a plurality of bones of the worker, and a plurality of third nodes corresponding respectively to a plurality of locations of the item.

7. Both the state and the work location are estimated from the first image; 2. The processing system according to claim 1, wherein the first graph data includes a plurality of first nodes corresponding respectively to a plurality of joints of the worker, a plurality of first edges corresponding respectively to a plurality of bone structures of the worker, a plurality of second nodes corresponding respectively to a plurality of states that the item can take, and a plurality of third nodes corresponding respectively to a plurality of locations of the item.

8. A posture of the worker is estimated from a second image showing the worker and the object; Estimating, from the second image, one or more selected from a state of the item and a location of a work performed on the item by the worker; generating second graph data including a plurality of nodes and a plurality of edges using the result estimated from the second image; inputting the second graph data to the neural network in addition to the first graph data, and estimating the task using a result output from the neural network; The processing system of claim 1 .

9. the plurality of nodes of the first graph data and the plurality of nodes of the second graph data are connected to each other by a plurality of edges; The processing system of claim 8 , wherein the first graph data and the second graph data are input to the neural network.

10. the second image is an image obtained after the first image; the second graph data is input to the neural network after the first graph data; 9. The processing system of claim 8, wherein in the neural network, the first graph data is input to the GNN, and the result output from the GNN is input to a Long Short Term Memory (LSTM) network, and then the second graph data is input to the GNN, and the result output from the GNN is input to the LSTM network.

11. In the neural network, the first graph data is input to the GNN, and a result output from the GNN and the second graph data are input to the GNN, and a result output from the GNN are input to a fully connected layer; The processing system of claim 8 , further comprising: a process for estimating the task using results output from the fully connected layer.

12. The processing device includes: A posture of the worker is estimated from a first image showing the worker and an object; Estimating, from the first image, one or more selected from a state of the item and a location of a work performed on the item by the worker; generating first graph data including a plurality of nodes and a plurality of edges based on the one or more selected from the state and the working location and the posture; inputting the first graph data into a neural network including a graph neural network (GNN), and estimating the task being performed by the worker using a result output from the neural network; Processing methods.

13. A program for causing a computer to execute the processing method according to claim 12.

14. A storage medium storing a program for causing a computer to execute the processing method according to claim 12.

Images