Processing system, processing method, program, and storage medium

Abstract

To provide a processing system, a processing method, a program, and a storage medium, which can estimate a task in more detail.SOLUTION: A processing system estimates a pose of a worker, a position of an article, an orientation of the article, and a state of the article on the basis of an image of the worker and the article. The processing system further estimates a work spot of the worker on the article on the basis of an estimation result of the pose, an estimation result of the position, and an estimation result of the orientation. The processing system further estimates a task performed by the worker on the basis of an estimation result of the work spot and an estimation result of the state.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a processing system, a processing method, a program, and a storage medium. [Background technology]

[0002] There are systems that automatically estimate the work being performed. There is a need for technology that can estimate the work in more detail for these systems. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-91249 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a processing system, a processing method, a program, and a storage medium that are capable of estimating work in more detail. [Means for solving the problem]

[0005] A processing system according to an embodiment estimates the posture of the worker, the position of the item, the orientation of the item, and the state of the item from images of the worker and the item. The processing system further estimates the location of the work performed by the worker on the item based on the posture estimation result, the position estimation result, and the orientation estimation result. The processing system further estimates the work being performed by the worker based on the work location estimation result and the state estimation result. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. [Figure 2]Fig. 2(a) is a schematic diagram showing a worker and an object, and Fig. 2(b) shows an example of an image captured by an imaging device. [Figure 3] FIG. 3 is a flowchart showing an example of the operation of the processing system according to the embodiment. [Figure 4] FIG. 4 is a diagram for explaining the processing performed by the processing system according to the embodiment. [Figure 5] FIG. 5 is a diagram for explaining the processing performed by the processing system according to the embodiment. [Figure 6] FIG. 6 is a diagram for explaining the processing performed by the processing system according to the embodiment. [Figure 7] FIG. 7 is a diagram for explaining the processing performed by the processing system according to the embodiment. [Figure 8] FIG. 8 is a diagram for explaining the processing performed by the processing system according to the embodiment. [Figure 9] FIG. 9 is a diagram for explaining the processing performed by the processing system according to the embodiment. [Figure 10] FIG. 10 is a diagram for explaining the processing performed by the processing system according to the embodiment. [Figure 11] FIG. 11 is a flowchart showing a method for estimating an item position. [Figure 12] FIG. 12 is a schematic diagram for explaining the result of position estimation when performing tracking processing. [Figure 13] FIG. 13 is a diagram for explaining the processing performed by the processing system according to the embodiment. [Figure 14] FIG. 14 is a diagram for explaining the processing performed by the processing system according to the embodiment. [Figure 15] FIG. 15 is a diagram for explaining the processing performed by the processing system according to the embodiment. [Figure 16] FIG. 16 is a flowchart showing an outline of the tracking process. [Figure 17] FIG. 17 is a flowchart showing the update process in the tracking process. [Figure 18]18(a) to 18(c) are images for explaining the processing performed by the processing system according to the embodiment. [Figure 19] 19(a) to 19(c) are images for explaining the processing performed by the processing system according to the embodiment. [Figure 20] FIG. 20 is a schematic diagram for explaining a method for estimating a work location. [Figure 21] FIG. 21 is a flowchart showing a method for estimating a work location. [Figure 22] FIG. 22 is an example of a work database. [Figure 23] FIG. 23 is a schematic diagram illustrating an output result by the processing system according to the embodiment. [Figure 24] FIG. 24 is a schematic diagram for explaining the processing by the processing system according to the embodiment. [Figure 25] FIG. 25 is a graph illustrating an estimation result by the processing system according to the embodiment. [Figure 26] FIG. 26 is a graph illustrating an estimation result by the processing system according to the embodiment. [Figure 27] FIG. 27 is a schematic view illustrating a specific configuration of the processing system according to the embodiment. [Figure 28] FIG. 28 is a schematic diagram showing an example of output by the processing system according to the embodiment. [Figure 29] FIG. 29 is a schematic diagram showing an example of output by the processing system according to the embodiment. [Figure 30] FIG. 30 is a schematic diagram showing an example of output by the processing system according to the embodiment. [Figure 31] FIG. 31 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present specification and the drawings, elements similar to those already described are designated by the same reference numerals, and detailed descriptions thereof will be omitted where appropriate.

[0008] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. The processing system according to the embodiment is used to estimate the work being performed by a worker from an image. As shown in Fig. 1, the processing system 1 includes an imaging device 10, a processing device 20, a storage device 30, an input device 40, and an output device 50.

[0009] Fig. 2(a) is a schematic diagram showing a worker and an object, and Fig. 2(b) shows an example of an image captured by an imaging device. Each process performed by the processing system 1 will now be described in detail. For example, as shown in FIG. 2(a), an item A1 is placed on a carrier C. A worker W performs a predetermined task on the item A1. The item A1 may be a semi-finished product, a unit used in a finished product, or the like. An imaging device 10 captures an image of the worker W and the item A1. FIG. 2(b) shows an image IMG captured by the imaging device 10.

[0010] Preferably, the imaging device 10 is attached to a wall or ceiling and captures images of the worker W and the item A1 from above. This makes it easier to capture the worker W and the item A1. The imaging device 10 may capture images directly below or at an angle relative to the vertical. The imaging device 10 repeatedly captures images. Alternatively, the imaging device 10 may capture video. In this case, still images are repeatedly extracted from the video. The imaging device 10 stores the images or video in the storage device 30.

[0011] The processing device 20 accesses the storage device 30 and acquires the image captured by the imaging device 10. From the image, the processing device 20 estimates the posture of the worker W, the position of the item A1, the orientation of the item A1, and the state of the item A1. Furthermore, the processing device 20 estimates the location of the work performed by the worker W on the item A1 based on the posture, position, and orientation. Then, the processing device 20 estimates the work being performed by the worker W based on the location and state of the work.

[0012] The storage device 30 stores images or videos as well as data necessary for processing by the processing device 20. The input device 40 is used by a user to input data to the processing device 20. The processing device 20 outputs the data obtained by processing to the output device 50 so that the data can be recognized by the user.

[0013] FIG. 3 is a flowchart showing an example of the operation of the processing system according to the embodiment. An overview of the operation of the processing system according to the embodiment will be described with reference to FIG. 3. The imaging device 10 captures an image of a worker and an item and acquires a video (step S10). The processing device 20 extracts an image from the video (step S20). The processing device 20 estimates the worker's posture from the image (step S30). The processing device 20 estimates the position and orientation of the item from the image (step S40). The processing device 20 estimates the state of the item from the image (step S50). The processing device 20 estimates the work location relative to the item based on the worker's posture, the position of the item, and the orientation of the item (step S60). The processing device 20 estimates the work being performed based on the state of the item and the work location (step S70). The processing device 20 outputs the estimation result (step S80).

[0014] Each process executed by the processing device 20 will be specifically described below.

[0015] (Posture estimation) The processing device 20 estimates the posture of the worker W from an image in which the worker W is captured. For example, the processing device 20 inputs the image into a posture estimation model prepared in advance. The posture estimation model is trained in advance so as to estimate the posture of a person appearing in an input image. The processing device 20 acquires an estimation result from the posture estimation model. For example, the posture estimation model includes a neural network. It is preferable that the posture estimation model includes a convolutional neural network (CNN). As the posture estimation model, OpenPose, DarkPose, CenterNet, or the like can be used.

[0016] (Position and Orientation Estimation) The processing device 20 extracts two images captured at different times from a plurality of images. The processing device 20 estimates motion information from the two images. The motion information is information that indicates the motion of an object between one image and another image. For example, Dense Optical Flow is calculated as the motion information. Any method can be used to calculate Dense Optical Flow, and methods such as Recurrent All-Pairs Field Transforms (RAFT) or total variation (TV)-L1 can be used.

[0017] 4 to 10 are diagrams for explaining the processing performed by the processing system according to the embodiment. FIG. 4(a) shows image It1 captured at time t1. FIG. 4(b) shows image It2 captured at time t2. Time t2 is after time t1. FIG. 4(c) shows movement information from image It1 to image It2, calculated by the processing device 20. In normal work, mainly workers and items related to the work move. When workers and items related to another work are not captured in the image, the movement information indicates the area in the image where the workers and items are captured. Here, the part of the image including the workers and items indicated by the movement information is called a "partial area."

[0018] The movement information used to estimate the position of an item may include the movement of tools, jigs, and other items other than the worker and the item. However, the shape of the tools and how these tools appear in the movement information are sufficiently different from the shape of the worker and the item and how the worker and the item appear in the movement information. Therefore, as will be described later, by using the "likelihood" of the shape or position of the item, the influence of the movement of tools, jigs, and the like on the estimation of the position of the item can be sufficiently reduced.

[0019] The result of the above-mentioned pose estimation indicates the area in the image where the worker is shown. Here, the area in which the worker is shown, which is indicated by the result of the pose estimation, is called the "worker area." The processing device 20 estimates the worker area in the image from the result of the pose estimation. The processing device 20 uses the worker area as a mask to remove the worker area from the movement information. This allows only the movement information of the item to be obtained. The movement information of the item indicates the area in the image where the item is shown. Here, the area in which the item, which is indicated by the movement information of the item, is called the "item area." The item area is estimated from the movement information of the item.

[0020] FIG. 5(a) shows the result of pose estimation for image It1 shown in FIG. 4(a). The positions of multiple joints 100 are estimated by pose estimation. From the result of pose estimation, a worker area 101 is identified, as shown in FIG. 5(a). FIG. 5(b) shows movement information 102 from time t1 to time t2. By using the worker area 101 as a mask to remove part of FIG. 5(b), the movement information of the item shown in FIG. 5(c) is obtained. The movement information of the item indicates an object area 103 in which the item appears in image It1.

[0021] The processing device 20 copies the movement information shown in FIG. 5(c). The processing device 20 calculates the correlation coefficient with the copied movement information while shifting the position of the movement information up, down, left, and right, respectively, to obtain a two-dimensional correlation coefficient map. The processing device 20 estimates the coordinates at which the maximum correlation coefficient is obtained from the obtained correlation coefficient map as the center of the object region. When calculating the correlation coefficient, preprocessing may be applied to the copied movement information. For example, the copied movement information may be flipped up and down, flipped left and right, or flipped up and down and left and right to obtain the center coordinates of an object that translates up and down, an object that translates left and right, or an object that rotates. The preprocessing applied to the copied movement information is not limited to these.

[0022] The processing device 20 scans in N directions at equal intervals from the center of the article area to estimate the contour points of the article. For example, the point where the value in the correlation coefficient map first drops is adopted as the contour point. In this way, N contour points are obtained. For example, N is set to 36.

[0023] The processing device 20 extracts n contour points from N contour points. The value n is smaller than the value N. For example, the processing device 20 extracts n contour points using a greedy algorithm. In the greedy algorithm, the angle between a contour point of interest and an adjacent contour point is calculated. For each contour point, the processing device 20 calculates the angle between the adjacent contour point. The processing device 20 extracts n contour points in ascending order of angle. For example, if the shape of an article is equal to or can be approximated by an m-polygon when viewed from above, the value n is set to m. Note that if the article is circular, the angles between adjacent contour points will be substantially equal. In this case, the value n may be equal to the value N. In other words, the process of extracting n contour points may be omitted.

[0024] FIG. 6(a) shows the results of estimating the center 103a and setting N contour points 103b for the object region 103 shown in FIG. 5(c). In this example, N is set to 30. Therefore, 30 contour points 103b are set. In the example shown in FIGS. 5(c) and 6(a), the shape of the object A1 is rectangular. At the corners of the rectangle, the angle between adjacent contour points becomes small. As shown in FIG. 6(b), the processing device 20 extracts four contour points 103b corresponding to the corners of the rectangle from the 30 contour points. The contour of the object A1 is estimated by connecting the four contour points 103b.

[0025] The processing device 20 uses the n contour points to search for a polygon that is most likely to represent the shape of the object. Specifically, the processing device 20 depicts a predetermined shape of the object using one of the n sides as a reference. Based on the estimated contour, the processing device 20 calculates the likelihood of the position of the depicted shape. The processing device 20 depicts the shape using each side as a reference and calculates the likelihood for each. The processing device 20 adopts the position of the shape that has the highest likelihood as the position of the shape of the object A1 depicted in the image.

[0026] As shown in Figure 6(c), a rectangle 104 based on the contour point 103b assumed in Figure 6(b) has four sides 104a to 104d. Figure 7(a) shows the result of drawing a predetermined rectangle based on side 104a. Similarly, Figures 7(b) to 7(d) show the result of drawing a predetermined rectangle based on sides 104b to 104d.

[0027] The processing device 20 calculates the likelihood between the rectangle 104 shown in FIG. 6(c) and the rectangles 105a to 105d shown in FIGS. 7(a) to 7(d) as the likelihood of each of the rectangles 105a to 105d. The average value within the rectangle in the obtained correlation coefficient map is used as the likelihood. As an example, for the rectangles 105a to 105d shown in FIGS. 7(a) to 7(d), "0.9," "0.4," "0.2," and "0.8" are calculated as the likelihoods, respectively. The processing device 20 uses the rectangle 105a for which the maximum likelihood is obtained. Alternatively, in calculating the likelihood, the image may be cut out based on each of the rectangles 105a to 105d. The processing device 20 may input each image into a model for state classification, which will be described later, and obtain the confidence level for the classification result as the likelihood.

[0028] The processing device 20 adopts the position of the shape for which the maximum likelihood is obtained as the position of the shape of the article at the time when one of the two images was captured. The processing device 20 calculates coordinates indicating the position of the article from the adopted shape. For example, the processing device 20 sets the center coordinates of the adopted shape as the article position. Alternatively, the processing device 20 may calculate the article position from the adopted shape according to preset conditions. The processing device 20 outputs the coordinates as an estimation result of the article position.

[0029] Furthermore, it is preferable that the capture times of the two images used to estimate the movement information are far enough apart that the movement of the worker or the object is clearly visible. As an example, the image capture device 10 captures video at 25 fps. Therefore, when images captured at adjacent times are extracted, the difference between their capture times is 1 / 25 of a second. The movement of the worker or the object is unlikely to be visible within a 1 / 25 second gap. The influence of image noise, etc., becomes significant, making it easy for erroneous movement information to be generated. For example, it is preferable that the difference between the capture times of the two images used to estimate the movement information be longer than 1 / 20 of a second and shorter than 1 / 2 second.

[0030] Furthermore, the sampling rate of the video captured by the image capture device 10 may be dynamically changed. For example, if the worker or object is moving quickly, the sampling rate is increased. This change in speed can be determined based on the magnitude of the immediately preceding optical flow and the magnitude of the difference between the estimated posture coordinates of the worker.

[0031] The orientation of an item is determined based on the amount of rotation of the item relative to its initial state. For example, when the position of the item is estimated from an image captured initially, the orientation is set for that item. Each time the processing device 20 estimates the position of an item, it calculates the amount of rotation of the estimated position relative to the estimation result of the previous position. Template matching, for example, is used to calculate the amount of rotation. Specifically, an image cut out based on the estimation result of the previous position is used as a template. The similarity between the cut-out image and the template is calculated while rotating the image based on the estimated position. The angle at which the maximum similarity is obtained corresponds to the amount of rotation of the item.

[0032] When performing template matching, it is preferable to search for the amount of rotation based on the most recent estimated result. This reduces the amount of calculation. Alternatively, the difference in brightness between corresponding points in the images may be compared with a preset threshold. If the difference is smaller than the threshold, it is determined that no change has occurred between those points. This reduces erroneous determinations in template matching.

[0033] FIG. 8(a) shows the estimated position of an item at an initial time t11. In the example of FIG. 8(a), a rectangle 110 including corners 111a to 111d is estimated. The directions "north," "east," "south," and "west" are set for sides 112a to 112d between the corners 111a to 111d. "North" and "west" are indicated by a thick solid line and a thick dashed line, respectively. FIG. 8(b) shows the result of estimating the position at time t12, which is later than time t11. A rectangle 120 including corners 121a to 121d and sides 122a to 122d is estimated. From the history of the rotation amount of the item calculated between time t11 and time t12, it is estimated that sides 122a to 122d correspond to sides 112a to 112d, respectively. As a result, "north," "east," "south," and "west," which correspond to the orientations of sides 112a to 112d, are set for sides 122a to 122d, respectively.

[0034] The position and orientation of the article are estimated by the above processing. Here, an example in which the shape of the article is rectangular has been described. Even if the shape of the article is not rectangular, the position and orientation of the article can be estimated by the same method.

[0035] In the example of FIG. 9(a), worker W is working on star-shaped (equisceles six-pointed star) object A2. FIG. 9(b) shows an image obtained by capturing the scene of FIG. 9(a). The processing device 20 estimates an object region 131 shown in FIG. 9(c) using the image 130 shown in FIG. 9(b) and another image. In this example, the object region 131 is a hexagon 132 consisting of six contour points 132a and six sides 132b. As shown in FIG. 9(c), depending on the shape of the object and the movements of the worker and the object, the object region may not correspond to the actual shape of the object.

[0036] As shown in FIG. 10(a), a star 133a is preset as the shape of the item A2. Furthermore, a hexagon 133b is preset as the shape corresponding to the star 133a. The processing device 20 draws the preset hexagon 133b based on each side 132b of the hexagon 132. As a result, six hexagons based on the six sides 132b are drawn. FIGS. 10(b) to 10(d) illustrate some of the hexagons, 134a to 134c. The processing device 20 calculates the likelihood for each of the six hexagons. The processing device 20 draws a star 133a based on the hexagon with the highest likelihood. The processing device 20 adopts the drawn star 133a as the shape of the item.

[0037] Thereafter, the position and orientation are estimated using the estimated shape. The number of pieces of information set to indicate the orientation of an item is arbitrary. In the rectangular example shown in FIGS. 8(a) and 8(b), the orientation of the item is indicated using four pieces of information (north, east, south, and west). For the star-shaped item A2 shown in FIG. 9(a), the orientation of the item may be indicated using six directions 135a to 135f, as shown in FIG. 10(a), for example.

[0038] FIG. 11 is a flowchart showing a method for estimating an item position. The processing device 20 estimates the position of an item at time t by processing the flowchart shown in FIG. 11. First, the processing device 20 determines whether an image at time t+d can be acquired (step S40a). That is, the processing device 20 determines whether an image at time t+d has been acquired by the imaging device 10. If an image at time t+d can be acquired, the processing device 20 acquires an image at time t and an image at time t+d (step S40b). The processing device 20 estimates motion information from the image at time t and the image at time t+d (step S40c). The processing device 20 uses this motion information as the motion information at time t. The processing device 20 estimates an item region from the motion information (step S40d). At this time, the result of pose estimation at time t is used as a mask.

[0039] The processing device 20 estimates the center of the item region (step S40e). The processing device 20 uses the estimated center to estimate N contour points of the item (step S40f). The processing device 20 extracts n contour points from the N contour points (step S40g). The processing device 20 uses the n contour points to search for a polygon that is most likely to represent the shape of the item (step S40h). The processing device 20 uses the coordinates of the center of the polygon obtained by the search as the item position. The value obtained by adding t' to the current time t is set as time t (step S40i). Step S40a is then executed again. As a result, the estimation result of the item position at time t is repeatedly updated each time an image at time t+d is obtained. If it is determined in step S40a that an image at time t+d cannot be acquired, the processing device 20 terminates the item position estimation process.

[0040] (Tracking process) In addition to the above-described position estimation using the motion information, the processing device 20 may further perform a tracking process, in which a position in a newly acquired image is tracked using a result of estimating a previous position.

[0041] Specifically, the processing device 20 cuts out a portion of an image showing an object based on the position estimation result in the previous image. The processing device 20 saves the cut-out image as a template image. When a new image is acquired, the processing device 20 performs template matching and searches for an area in the new image that has the highest similarity. The processing device 20 uses the area obtained by the search as the position estimation result in the new image.

[0042] FIG. 12 is a schematic diagram for explaining the result of position estimation when performing tracking processing. In FIG. 12, the horizontal axis represents time. The vertical axis represents the number of candidates for the estimated position. For example, at time t, an item position E1 is estimated using motion information between the image at time t and the image at time t+d. The number of candidate item positions for time t is "1." Similarly, at the next time t+t', an item position E2 is estimated using motion information between the image at time t+t' and the image at time t+t'+d. The processing device 20 further estimates an item position E11 in the image at time t+t'+d using a template image based on the item position E1. As a result, the number of candidate item positions for time t+t' becomes "2."

[0043] After that, the same process is repeated every time a new image is acquired. For example, at time t+xt′, the tracking process based on the item position E1 is repeated, and the item position E1 x By repeating the tracking process based on the item position E2, the item position E2 x-1 The processing device 20 uses the most probable item position at each time as the final item position.

[0044] The likelihood used to narrow down the final item position is, for example, the similarity between an image based on each item position and a master image prepared in advance. Each image may be input into a model for state classification, and the confidence level for the classification result may be used as the likelihood.

[0045] Alternatively, the likelihood may be calculated using a judgment model. This judgment model includes a deep learning model. The processing device 20 cuts out an image based on the estimation result of the item position and inputs it to the judgment model. The judgment model determines whether the input image is an image cut out along the outer edge (four sides) of the item. The judgment model outputs a scalar value between 0 and 1 according to the input image. The output becomes closer to 1 the closer the outer edge of the input image is to the outer edge of the item. For example, if a part of the floor surface other than the item is cut out, or if only a part of the item is cut out, the output becomes smaller. The processing device 20 cuts out an image for each estimated item position and obtains an output for each image. The processing device 20 obtains these outputs as the likelihood for each item position.

[0046] When calculating the likelihood, the direction of imaging by the imaging device 10 may be taken into consideration. For example, if the imaging device 10 captures an image of a worker and an item from a direction inclined relative to the vertical, the image will appear differently at a position closer to the imaging device 10 than at a position farther from the imaging device 10. For example, sides closer to the imaging device 10 will appear longer, and sides farther from the imaging device 10 will appear shorter. Based on this geometric condition, the length of the side that serves as a reference for the inclination is stored in advance in the storage device 30. The processing device 20 reads the length of the reference side stored in the storage device 30 for the angle θq of each candidate item position being tracked, and uses the difference from the length lq of the side of the item position being tracked as the likelihood.

[0047] 13 to 15 are diagrams for explaining the processing performed by the processing system according to the embodiment. 13, a worker W is working on an item A3. The imaging device 10 captures an image of the worker W and the item A3 from diagonally above. When viewed vertically, the shape of the item A3 is rectangular.

[0048] 14(a) to 14(c), the appearance of the article A3 differs depending on the orientation of the article A3 relative to the imaging device 10. The processing device 20 utilizes this fact to calculate the likelihood of the article position.

[0049] Specifically, the processing device 20 generates a line segment corresponding to the estimated item position according to a preset rule. In the example of FIGS. 14(a) to 14(c), the processing device 20 first determines the short sides of the item based on the estimated item position. The processing device 20 generates a line segment Li connecting the short sides. The processing device 20 calculates the length of the line segment Li. The processing device 20 also calculates the angle between the reference line BL and the line segment Li. In this example, the reference line BL is parallel to the horizontal direction of the image.

[0050] As a result of the calculation, angles θ1 to θ3 and lengths L1 to L3 are calculated for the examples of Figures 14(a) to 14(c). Angle θ1 is greater than angle θ2, and length L1 is shorter than length L2. Angle θ2 is greater than angle θ3, and length L2 is shorter than length L3. That is, as shown in Figure 14(d), the greater the angle, the shorter the length of line segment Li. Such correspondence relationships between angles and lengths are stored in advance in storage device 30.

[0051] 15(a) to 15(c) show rectangles q1 to q3 obtained by the search, respectively. The processing device 20 calculates the lengths and angles of the line segments connecting the short sides of each of the rectangles q1 to q3. As a result of the calculations, angles θq1 to θq3 and lengths Lq1 to Lq3 are calculated for the examples of FIGS. 15(a) to 15(c), respectively.

[0052] The processing device 20 refers to the correspondence relationship and obtains the length corresponding to the calculated angle. The processing device 20 calculates the difference between the length corresponding to the angle and the calculated length, and calculates a likelihood based on the difference. The greater the difference, the smaller the calculated likelihood.

[0053] For example, as shown in FIG. 15(d), the processing device 20 calculates the difference Dq1 between the length Lq1 and the length corresponding to the angle θq1 for rectangle q1. Similarly, the processing device 20 calculates the difference Dq2 and the difference Dq3 for rectangles q2 and q3, respectively. The processing device 20 uses the differences Dq1 to Dq3 to calculate the likelihood of each of rectangles q1 to q3. In this example, the difference Dq1 is smaller than the difference Dq3 and larger than the difference Dq2. Therefore, the likelihood of rectangle q2 is larger than the likelihood of rectangle q3 and smaller than the likelihood of rectangle q1.

[0054] The greater the number of candidate item positions, the more accurately the item positions can be estimated. On the other hand, if there are too many candidates, the amount of calculation required for the tracking process becomes excessive, which may delay the process. For this reason, it is preferable to predetermine the number of candidates to be retained. In the example shown in FIG. 12, the predetermine number is set to "x+1." At time t+xt', x+1 item positions are estimated. At time t+(x+1)t', x+2 item positions are estimated. The processing device 20 narrows down the x+2 item positions to x+1 item positions. In the example shown, the results of the tracking process based on the item position Ex are excluded, and other item positions are extracted. The above-mentioned certainty can be used to narrow down the item positions. The processing device 20 extracts the x+1 item positions in descending order of certainty.

[0055] FIG. 16 is a flowchart showing an outline of the tracking process. The processing device 20 determines whether an image at time t+d can be acquired (step S41a). If an image at time t+d can be acquired, the processing device 20 acquires an image at time t and an image at time t+d (step S41b). The processing device 20 executes a position update process using the image at time t+d (step S41c). The value obtained by adding t' to the current time t is set as time t (step S41d). Then, step S41a is executed again. If it is determined in step S41a that an image at time t+d cannot be acquired, the processing device 20 ends the tracking process.

[0056] FIG. 17 is a flowchart showing the update process in the tracking process. In the position update process, the processing device 20 cuts out a portion of the image at time t that corresponds to the most recently estimated position. The processing device 20 acquires the cut-out image as a template image for time t (step S42a). The processing device 20 compares the template image at time t with the image at time t+d within the tracking candidate region (step S42b). The tracking candidate region is a portion of the cut-out image and is set according to pre-set parameters. For example, a region that is 50% horizontally and 50% vertically of the cut-out image, centered on the item position at time t, is set as the tracking candidate region. The processing device 20 determines whether the difference in brightness between the two images exceeds a threshold (step S42c). If the difference exceeds the threshold, the processing device 20 searches for the position and orientation in the image at time t+d that have the highest similarity while changing the position and orientation of the template image (step S42d). The processing device 20 updates the most recently estimated item position to the item position obtained by the search (step S42e). In step S42c, if the difference in brightness value is equal to or less than the threshold, the update process is skipped. If skipped, the estimation result at time td is used. This suppresses drift in template matching.

[0057] 18(a) to 18(c) and 19(a) to 19(c) are images for explaining the processing by the processing system according to the embodiment. Fig. 18(a) shows an image It1 captured at time t1. Fig. 18(b) shows an image It2 captured at time t2. Fig. 18(c) shows an article position estimated based on images It1 and It2. Rectangle 106a is used in the position estimation.

[0058] FIG. 19(a) shows a template image Tt0 at time t0. Time t0 is before time t1. A template image is cut out from the image at time t0 using the result of estimating the item position at time t0. FIG. 19(b) shows an image It1 captured at time t1. FIG. 19(c) shows a rectangle 106b obtained by a tracking process using the template image Tt0. The rectangle 106b indicates the item position in image It1. For example, the final item position is narrowed down from x item positions including the item position shown in FIG. 18(c) and the item position shown in FIG. 19(c).

[0059] (State estimation) The processing device 20 estimates the state of an item depicted in an image from the image. For example, template matching is used to estimate the state. The processing device 20 compares the image with a plurality of template images prepared in advance. Each template image is associated with the state of the item. The processing device 20 extracts the template image with the highest similarity. The processing device 20 estimates the state associated with the extracted template image as the state of the item depicted in the image.

[0060] Alternatively, the processing device 20 may input the image into a state estimation model. The state estimation model is trained in advance to estimate the state of an item depicted in an image in response to the input image. For example, the state estimation model includes a neural network. Preferably, the state estimation model includes a CNN. The processing device 20 obtains the estimation result obtained by the state estimation model.

[0061] It is preferable that the processing device 20 cuts out a portion from the entire image, which also includes other elements such as a worker other than the item. The item is captured in that portion of the cut-out image. The result of estimating the position of the item may be used for the cut-out. Cutting out increases the proportion of the area of ​​the item captured in the image. This reduces the impact of elements other than the item on the state estimation. As a result, the accuracy of the state estimation can be improved. If the image is not cut out, it is also possible to directly estimate the state of the item from the image acquired by the imaging device 10.

[0062] (Estimation of work location) The processing device 20 estimates the worker's working location relative to the item based on the estimated results of the worker's posture, the estimated results of the item's position, and the estimated results of the item's orientation. For example, the processing device 20 obtains the positions of the worker's left hand and right hand from the estimated results of the posture. The processing device 20 calculates the relative positions and orientations of the left hand and right hand relative to the item. Based on the relative positional relationship, the processing device 20 estimates the working location relative to the item.

[0063] FIG. 20 is a schematic diagram for explaining a method for estimating a work location. In the example of FIG. 20, the position of the left hand 140a of the worker 140 (x left ,y left ) and the position of the right hand 140b (x right ,y right ) are estimated. The positions (x0, y0), (x1, y1), (x2, y2), and (x3, y3) of the center 142 and the four corners 142a to 142d are estimated as the position of the article 141. In addition, the orientation of the article, "north," "east," "south," and "west," are estimated. The orientation of the article is distinguished by boundaries 143a and 143b that pass through the center 142. In this example, the diagonals of the rectangular article 141 are set as boundaries 143a and 143b. The direction and number of boundaries are set appropriately according to the shape of the article.

[0064] The processing device 20 sets gates for estimating the work location based on the position and orientation of the item. For example, the processing device 20 sets gates for "north," "east," "south," and "west" along each side of the item 141. As shown by line Li1, the left hand 140a faces the "east" gate. As shown by line Li2, the right hand 140b faces the "north" gate. Lines Li1 and Li2 are extensions of the left lower arm and the right lower arm, respectively. The lower arm is a line segment (bone) connecting the wrist and elbow.

[0065] Based on the positions of the joints and each gate, the processing device 20 estimates that the left hand 140a is located on the east side of the item 141. That is, the location of work with the left hand is estimated to be on the east side of the item. The processing device 20 also estimates that the right hand 140b is located on the north side of the item 141. That is, the location of work with the right hand is estimated to be on the north side of the item.

[0066] Any joint may be used to estimate the work location. For example, the positions of the fingers, wrist, or elbow may be used to estimate the work location depending on the work being performed. The positions of multiple joints among these may also be used to estimate the work location.

[0067] FIG. 21 is a flowchart showing a method for estimating a work location. The processing device 20 sets gates in the directions of the respective items based on the estimated position and orientation of the items (step S61). The processing device 20 determines whether the worker's lower arms intersect with the gates (step S62). If the lower arms intersect with the gates, the processing device 20 sets the positions of the left and right hands as work positions (step S63). If the lower arms do not intersect with the gates, the processing device 20 sets the intersection of the extension of the lower arms and the gates as the work positions (step S64). The processing device 20 estimates that the gates where the lower arms or extensions intersect are work locations (step S65).

[0068] (Work Estimation) The processing device 20 estimates the work being performed by the worker based on the estimated results of the item state and work location. For example, the storage device 30 stores a work database containing data related to the work. The work database includes a list of work that can be performed. Each work is previously associated with the item state and work location. The processing device 20 refers to the work database and extracts, from multiple work tasks, a work that corresponds to the estimated item state and work location. The processing device 20 estimates that the extracted work is being performed by the worker.

[0069] The task database may store the order in which tasks are to be performed. In this case, the processing device 20 estimates the task being performed based on the tasks estimated up to that point, the estimated state of the item, and the task corresponding to the task location. By referring to the order in which tasks are to be performed, the accuracy of task estimation can be improved.

[0070] FIG. 22 is an example of a work database. The task database 150 shown in FIG. 22 includes a number 151, a major item 152, a medium item 153, task content 154, an item status 155, and a task location 156. The number 151 is a number assigned to each task. For example, the number 151 indicates the order in which the task is to be performed. A number for identifying each task may be registered as the number 151. The major item 152 indicates a rough classification of the task. The medium item 153 indicates a medium classification of the task included in the major item 152. The task content 154 indicates a specific task included in the medium item 153. The item status 155 indicates the status of the item when the task content 154 is being performed. The task location 156 indicates the location where the task is performed when the task content 154 is being performed.

[0071] While the worker is performing the task, images showing the task are repeatedly acquired. The processing device 20 repeatedly estimates the task based on each image. This allows the processing device 20 to estimate what task the worker was performing at each time.

[0072] FIG. 23 is a schematic diagram illustrating an output result by the processing system according to the embodiment. In Fig. 23, the horizontal direction indicates time. The vertical direction indicates the estimation results of the item state and work location based on the image at each time. The top row also indicates the work estimated based on the item state and work location.

[0073] To improve the accuracy of the estimation, the processing device 20 may estimate the work being performed based on the results of work estimation over a predetermined period of time.

[0074] FIG. 24 is a schematic diagram for explaining the processing by the processing system according to the embodiment. FIG. 24 shows the location of work performed by the left hand, the location of work performed by the right hand, and the estimated state of the item, as well as the estimated result of the work based on these. The work location is estimated to be either "east," "west," "south," or "north" of the item. As shown in FIG. 24, the processing device 20 sets a window Wi when estimating the work. The time width (period) P of the window Wi is set based on the standard time of the work that follows the already estimated work. For example, after the start of "Work 2" is estimated, the period P of the window Wi is set based on the standard time of the next work, "Work 3."

[0075] The processing device 20 aggregates the estimated execution time of each task within the window Wi. If the ratio of the duration of any task within the window Wi exceeds a preset threshold, the processing device 20 estimates that the task is being performed. As an example, the threshold is set to 0.5.

[0076] The processing device 20 estimates the work while continuously sliding the window Wi. The sliding amount of the window Wi is set to be sufficiently small with respect to the period of the window Wi.

[0077] When the task being performed is estimated, the processing device 20 estimates that the task is being performed from the starting point of the window Wi. Therefore, when it is estimated that "task 3" is being performed in the illustrated window Wi, it is estimated that "task 3" is being performed from the starting point SP of the window Wi.

[0078] Note that the work location may change during one work, as shown in Fig. 24. The processing device 20 may estimate the work using this change in the work location.

[0079] As shown in Figure 24, in reality, the state of an item may be erroneously estimated. In the illustrated example, during a period when the state should be estimated as "State 1," the state is temporarily estimated as "State 2," "State 3," or "State 4." There are also moments when the work location is in transition. These temporary erroneous estimation results may lead to an erroneous estimation of the work.

[0080] Regarding this problem, as described above, by estimating the work performed in a window Wi based on a plurality of estimation results of the work location and a plurality of estimation results of the state in the window Wi, it is possible to reduce the influence of temporary erroneous estimation results on the work estimation results, thereby improving the accuracy of work estimation.

[0081] 25 and 26 are graphs illustrating the estimation results by the processing system according to the embodiment. In Fig. 25 and Fig. 26, the horizontal direction indicates time. The vertical direction indicates the work to be performed. In Fig. 25 and Fig. 26, the solid lines indicate the results of work estimation by the processing device 20. The dashed lines indicate the progress of the work actually performed. Fig. 25 shows the results of work estimation without using the above-mentioned window. Fig. 26 shows the results of work estimation using the window.

[0082] In the results shown in Figure 25, good estimation results were obtained for "Task 3" to "Task 9." On the other hand, for "Task 1" and "Task 2," incorrect estimation occurred in the area surrounded by the dashed line.

[0083] In the results shown in FIG. 26, good estimation results are obtained for "Task 3" to "Task 9", and good estimation results are also obtained for "Task 1" and "Task 2".

[0084] The processing device 20 may calculate data related to the performed work based on repeated estimation of the work. For example, the processing device 20 may calculate the time (man-hours) from the start to the end of each work. The processing device 20 may compare the performed work with a previously created schedule and calculate the progress against the schedule. The processing device 20 may also calculate whether the performed work is ahead or behind schedule. The processing device 20 may compare the actual man-hours with a preset standard man-hours for each work. If the actual man-hours are longer than the standard man-hours, the processing device 20 extracts that work. This automatically extracts work that has room for improvement.

[0085] The above-described estimation of the work is repeated until the end condition is satisfied. For example, the work ends at a preset time. The end time of the work may be set to the end of the work day, or a time when a preset time has elapsed since the start of the work. The work may also end when it is estimated that the last work has been completed. Whether the work that has been performed is the last can be determined based on the work database. An end instruction may also be input from the user or a higher-level system.

[0086] (System Configuration) FIG. 27 is a schematic view illustrating a specific configuration of the processing system according to the embodiment. The above-described estimation-related processing may be executed by one processing device 20 (computer), or may be executed by cooperation of multiple processing devices 20. To estimate work in real time, it is preferable that the amount of calculation per processing device 20 is small. For this reason, it is preferable that the estimation-related processing is executed by multiple processing devices 20.

[0087] For example, as shown in FIG. 27, the processing system 1 includes multiple processing devices 20a to 20g. When the imaging device 10 acquires video data D1, it stores the data in the storage device 30. The processing device 20a monitors the storage device 30, and when the video data D1 is saved, it extracts image data D2 from the video data D1. The processing device 20a saves the image data D2 in the storage device 30. The processing device 20b monitors the storage device 30, and when the image data D2 is saved, it estimates the worker's posture from the image. The processing device 20b saves posture data D3 indicating the posture estimation results in the storage device 30. The processing device 20c monitors the storage device 30, and when the image data D2 and posture data D3 are saved, it estimates the position and orientation of the item. The processing device 20c saves item data D4 indicating the estimation results of the position and orientation of the item in the storage device 30. The processing device 20d monitors the storage device 30, and when the item data D4 is saved, it estimates the state of the item. The processing device 20d stores status data D5 indicating the status estimation result in the storage device 30. The processing device 20e monitors the storage device 30, and when the image data D2, posture data D3, and item data D4 are stored, it estimates the work location. The processing device 20e stores the work location estimation result in work location data D6 in the storage device 30. The processing device 20f monitors the storage device 30, and when the image data D2, posture data D3, item data D4, status data D5, and work location data D6 are stored, it estimates the work being performed. The processing device 20f stores work data D7 indicating the work being performed in the storage device 30. The processing device 20g generates data to be output to the output device 50 based on the work data D7. The processing device 20g may also perform other operations such as calculating man-hours and comparing with a schedule.

[0088] The specific processing method in the processing system 1 according to the embodiment is not limited to the above-described example. As described above, the next process may be executed in response to the generation of a file. Alternatively, the next process may be executed in response to the storage of data corresponding to each file in memory without the file being generated. Data may be communicated between processing devices, and the next process may be executed in response to the transmission and reception of the data.

[0089] (User Interface) 28 to 30 are schematic diagrams showing examples of outputs from the processing system according to the embodiment. The processing device 20 displays, for example, a user interface (UI) 200 shown in Fig. 28 on the output device 50. The UI 200 displays an image 201, a posture estimation result 202, a tracking result 203, a state estimation result 204, a work location estimation result 205, a standard time 206, a measurement time 207, a seek bar 208, and a time chart 209 related to the estimation results.

[0090] Image 201 indicates an image cut out from a video captured by imaging device 10. Pose estimation result 202 includes estimated skeleton 202a and estimated number of people 202b. Skeleton 202a is the result of pose estimation for image 201. Tracking result 203 indicates whether an item captured in image 201 can be tracked and the orientation of the item. In this example, the orientation is indicated by an angle relative to a preset reference line. State estimation result 204 indicates the estimation result of the state of the item based on image 201. Work location estimation result 205 indicates the estimation result of the work location based on image 201. Standard time 206 indicates the standard time (man-hours) for each task. Measured time 207 indicates the time (man-hours) measured for each task from the estimation result of the task.

[0091] The seek bar 208 includes a slider 208a and a bar 208b. The slider 208a indicates the time when the image 201 was captured. The user can display the estimation result for the image at any time by sliding the slider 208a on the bar 208b. The time chart 209 includes charts 209a to 209e. The chart 209a indicates whether or not pose estimation was possible at each time. The period during which the processing device 20 succeeded in pose estimation and the period during which the processing device 20 failed in pose estimation are displayed in different modes (colors). The chart 209b indicates whether or not the item was tracked. The period during which the processing device 20 succeeded in pose estimation and the period during which the processing device 20 failed in pose estimation are displayed in different colors. The chart 209c indicates the result of state estimation. The chart 209c displays colors corresponding to each state. The chart 209d indicates the estimation result for the work location. Chart 209d displays a color corresponding to each task location. Chart 209e shows the estimated results of the tasks. Chart 209e displays a color corresponding to each task.

[0092] In order to prepare data necessary for learning state estimation, the processing device 20 may display a UI 210 shown in Fig. 29. The UI 210 displays an image 211, a button 212, an input field 213, a field 214, a button 215, and a seek bar 216.

[0093] Image 211 indicates an image cut out from a video captured by imaging device 10. Button 212 is a button for setting the end of image cutout. In input field 213, the state of the item shown in the cut-out image is input. In field 214, the number of images to be cut out is displayed. Button 215 is a button for setting the start of image cutout. Seek bar 216 includes slider 216a and bar 216b. Slider 216a indicates the timing at which image 211 was captured. The user can display the estimation result for an image at any time by sliding slider 216a on bar 216b.

[0094] The processing device 20 cuts out images at predetermined intervals from the period set by the buttons 212 and 215. The number of images cut out from the set period is displayed in the field 214. The processing device 20 associates the cut-out images with the state set in the input field 213. The position and orientation of the item may also be estimated for the image. The processing device 20 cuts out a portion of the image based on the estimation result and associates it with the set state.

[0095] A user can easily prepare an image associated with the state of an item through the UI 210. The prepared image can be used for learning a model for estimating the state of an item, or as a template image in template matching.

[0096] The processing device 20 may display a UI 220 shown in Fig. 30 on the output device 50. The UI 220 displays an image 221, fields 222 to 227, and a seek bar 228.

[0097] The seek bar 228 includes a slider 228a and a bar 228b. The slider 228a indicates the timing at which the image 221 was captured. The position (time) of the slider 228a is displayed near the slider 228a. The user can display an image of any time by sliding the slider 228a on the bar 228b.

[0098] Marks 221a are displayed on the image 221. The marks 221a correspond to the vertices of a polygon. The user can move the marks 221a on the image 221 by dragging and dropping using the input device 40. When the processing device 20 accepts the movement of the mark 221a, it renders a polygon corresponding to the moved mark 221a. In the example shown in FIG. 30, four marks 221a are displayed. Also, rectangles 221b corresponding to the four marks 221a are displayed, with "north" and "east" in the rectangles 221b indicated by thick solid and thick dashed lines, respectively. The user moves the marks 221a to display rectangles 221b corresponding to the shapes of objects shown in the image 221.

[0099] An arrow 221c is also displayed on the image 221. The arrow 221c indicates the direction of the rectangle 221b. The user can move the start point and end point of the arrow 221c by dragging and dropping. For example, the direction perpendicular to the side intersecting with the arrow 221c and pointing from the start point of the arrow 221c to the end point is set as "north." The directions "east," "south," and "west" are set clockwise from "north." In the illustrated example, of the four sides of the rectangle 221b, the two sides facing "north" and "east" are indicated by thick lines. Also, as shown in FIG. 30, a symbol 221c indicating the direction of the rectangle 221b is set as "north." d In the illustrated example, a compass needle is displayed as a symbol indicating the direction.

[0100] The processing device 20 calculates data related to the set rectangle 221b and displays it in columns 222 to 227. For example, in columns 225 to 227, "width" is the distance between the short sides of the rectangle 221b. "height" is the distance between the long sides of the rectangle 221b. "angle" is the angle of the "north" direction of the rectangle 221b relative to a preset reference line.

[0101] A user can easily prepare images associated with the position and orientation of an object through the UI 220. The prepared images can be used to train a decision model to obtain a certainty.

[0102] The advantages of the embodiment will be described. Various methods have been attempted to estimate the work being performed. For example, there is a demand for technology that can estimate the work in more detail or more accurately. One example is a method of estimating the work being performed from the state of an object captured in an image. This method allows the work to be easily analyzed without using expensive sensors or the like. However, if the state of the object does not change, it is not possible to distinguish between the work being performed even if the work changes.

[0103] To address this issue, the processing system 1 according to the embodiment estimates the work being performed based not only on the state of the item but also on the location of the work on the item. By using the location of the work on the item for estimation, the work being performed can be accurately estimated even during periods when the state of the item does not change. Furthermore, even when there are multiple tasks that require the same location to be performed on an item, these tasks can be distinguished based on the state of the item. According to the embodiment, the work can be estimated in more detail and with higher accuracy.

[0104] FIG. 31 is a schematic diagram showing a hardware configuration. The processing device 20 includes, for example, the hardware configuration shown in Fig. 31. A computer 90 shown in Fig. 31 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0105] The ROM 92 stores programs that control the operation of the computer. The ROM 92 stores programs necessary for the computer to execute each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0106] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a work memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During program execution, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0107] The storage device 94 stores data necessary for executing the program and data obtained by executing the program.

[0108] The input interface (I / F) 95 connects the computer 90 and the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0109] The output interface (I / F) 96 connects the computer 90 and the output device 96a. The output I / F 96 is, for example, a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI (registered trademark)). The CPU 91 can transmit data to the output device 96a via the output I / F 96 and cause the output device 96a to display an image.

[0110] A communication interface (I / F) 97 connects the computer 90 to a server 97a external to the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97. The camera 99 photographs an item and stores the image in the server 97a.

[0111] The storage device 94 includes one or more selected from a hard disk drive (HDD) and a solid state drive (SSD). The input device 95a includes one or more selected from a mouse, a keyboard, a microphone (voice input), and a touchpad. The output device 96a includes one or more selected from a monitor, a projector, a speaker, and a printer. A device having the functions of both the input device 95a and the output device 96a, such as a touch panel, may also be used.

[0112] The storage device 94 can be used as the storage device 30. The camera 99 can be used as the imaging device 10.

[0113] The various data processing operations described above may be recorded as a computer-executable program on a magnetic disk (such as a flexible disk or hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, or DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0114] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, a computer reads a program from the recording medium and causes a CPU to execute instructions written in the program based on the program. The computer may acquire (or read) the program via a network.

[0115] According to the embodiments described above, a processing system, a processing method, a program, and a storage medium are provided that are capable of estimating an operation in more detail.

[0116] Although several embodiments of the present invention have been described above, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. Furthermore, the above-described embodiments can be implemented in combination with each other. [Explanation of symbols]

[0117] 1: Processing system, 10: Imaging device, 20, 20a to 20g: Processing device, 30: Storage device, 40: Input device, 50: Output device, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 95a: Input device, 96: Output interface, 96a: Output device, 97: Communication interface, 97a: Server, 98: System bus, 99: Camera, 100: Joint, 101: Worker area, 102: Movement information, 103: Item area, 103a: Center, 103b: Contour point, 104: Rectangle, 104a to 104d: Side, 105a to 105d: Rectangle, 106a,106b: rectangle, 110: rectangle, 111a to 111d: corners, 112a to 112d: edges, 120: rectangle, 121a to 121d: corners, 122a to 122d: edges, 130: image, 131: item area, 132: hexagon, 132a: contour points, 132b: edges, 133a: star, 133b: hexagon, 134a to 134c: hexagon, 135a to 135f: direction, 140: worker, 140a: left hand, 140b: right hand, 141: item, 142: center, 142a to 142d: corners, 143a: boundary line, 150: work database, 152: major item, 153: Medium item, 154: Work content, 155: Item state, 156: Work location, 201: Image, 202: Posture estimation result, 202a: Skeleton, 202b: Number of people, 203: Tracking result, 204: State estimation result, 205: Work location estimation result, 206: Standard time, 207: Measurement time, 208: Seek bar, 208a: Slider, 208b: Bar, 209: Time chart, 209a to 209e: Chart, 211: Image, 212: Button, 213: Input field, 214: Field, 215: Button, 216: Seek bar, 216a: Slider, 216b: Bar, 221: Image, 221a: Mark, 221b: Rectangle, 221c: arrow, 221d: symbol, 222 to 227: column, 228: seek bar, 228a: slider, 228b: bar, A1 to A3: item, BL: reference line, C: carrier, D1: video data, D2: image data, D3: posture data, D4: item data, D5: status data, D6: work location data, D7: work data, IMG: image, It1, It2: images, Li: line segment, P: period, SP: base point, Tt0: template image, W: worker, Wi: window,

Claims

1. Estimating the posture of the worker, the position of the item, the orientation of the item, and the state of the item from images of the worker and the item; estimating a location of work performed by the worker on the object based on the posture estimation result, the position estimation result, and the orientation estimation result; estimating the work being performed by the worker based on the work location estimation result and the state estimation result; 1. A processing system comprising: In estimating the work location, setting a plurality of gates for the article based on the position estimation result and the orientation estimation result; a processing system that estimates the work location based on a positional relationship between a part of the worker and the plurality of gates, which is indicated by the posture estimation result.

2. In estimating the position and the orientation, extracting a partial area including the worker and the item from the image; identifying a worker area in which the worker is photographed based on the posture estimation result; estimating an item area in which the item is captured by excluding the worker area from the partial area; estimating the position and the orientation using the object region; The processing system of claim 1 .

3. In estimating the position and the orientation, estimating an outer edge of the article using the article region; estimating the position and the orientation using at least a portion of the outer perimeter; The processing system of claim 2 .

4. 4. The processing system according to claim 3, wherein an area where there is movement among a plurality of said images is extracted as said partial area.

5. Extracting a plurality of the partial regions from a plurality of the images; estimating a plurality of outer edges using the plurality of partial regions; The processing system of claim 3 or 4, wherein one of the outer edges is used to estimate the position and the orientation.

6. 6. The processing system according to claim 1, wherein a part of the image including the article is cut out, and the state is estimated using the cut-out part of the image.

7. Estimating the posture of the worker, the position of the item, the orientation of the item, and the state of the item from images of the worker and the item; estimating a location of work performed by the worker on the object based on the posture estimation result, the position estimation result, and the orientation estimation result; estimating the work being performed by the worker based on the work location estimation result and the state estimation result; 1. A processing system comprising: In estimating the position and the orientation, extracting a partial area including the worker and the item from the image; identifying a worker area in which the worker is photographed based on the posture estimation result; estimating an item area in which the item is captured by excluding the worker area from the partial area; estimating an outer edge of the article using the article region; estimating the position and the orientation using at least a portion of the outer perimeter; Extracting a plurality of the partial regions from among the plurality of the images; estimating a plurality of outer edges using the plurality of partial regions; A processing system that estimates the position and the orientation using one of the plurality of outer edges.

8. 8. The processing system according to claim 1, wherein the work at each of the times is estimated based on a plurality of the images taken at different times.

9. The processing system according to claim 8 , wherein the estimation of the work involves estimating the work performed during a predetermined period based on a plurality of estimation results of the work location and a plurality of estimation results of the state during the predetermined period.

10. On the computer, Estimating the posture of the worker, the position of the item, the orientation of the item, and the state of the item from images of the worker and the item; estimating a location of work performed by the worker on the object based on the posture estimation result, the position estimation result, and the orientation estimation result; estimating the work being performed by the worker based on the work location estimation result and the state estimation result; A processing method comprising: In the estimation of the work location, the computer setting a plurality of gates for the article based on the position estimation result and the orientation estimation result; estimating the work location based on a positional relationship between a part of the worker and the plurality of gates, which is indicated by the posture estimation result; Processing method.

11. A computer comprising: Estimating the posture of the worker, the position of the item, the orientation of the item, and the state of the item from images of the worker and the item; estimating a location of work performed by the worker on the object based on the posture estimation result, the position estimation result, and the orientation estimation result; estimating the work being performed by the worker based on the work location estimation result and the state estimation result; A processing method comprising: In estimating the position and the orientation, the computer extracting a partial area including the worker and the item from the image; Identifying a worker area in which the worker is photographed based on the posture estimation result; By excluding the worker area from the partial area, an item area in which the item is captured is estimated; estimating an outer edge of the article using the article region; estimating the position and the orientation using at least a portion of the outer edge; The computer, extracting a plurality of the partial regions from among the plurality of the images; a plurality of the outer edges are estimated using the plurality of partial regions; using one of the plurality of outer edges to estimate the position and the orientation; Processing method.

12. A program that causes the computer to execute the processing method according to claim 10 or 11.

13. A storage medium storing the program according to claim 12.

Images