PROGRESS DETERMINATION SYSTEM, PROGRESS DETERMINATION METHOD, PROGRAM, AND STORAGE MEDIUM

Abstract

To provide a progress determination system, a progress determination method, a program and a storage medium which can improve the progress determination accuracy.SOLUTION: A progress determination system according to an embodiment comprises a first acquisition unit and a second acquisition unit. The first acquisition unit acquires area data indicating an area value of each of plural colors from an image in which an article related to the work is imaged. The second acquisition unit inputs the area data to a classifier and acquires a classification result indicating a progress level from the classifier.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a progress determination system, a progress determination method, a program, and a storage medium. [Background technology]

[0002] Various techniques have been developed to automatically determine the progress of work. There is a demand for improved accuracy in determining the progress. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-71566 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a progress determination system, a progress determination method, a program, and a storage medium that are capable of improving the accuracy of progress determination. [Means for solving the problem]

[0005] A progress determination system according to an embodiment includes a first acquisition unit and a second acquisition unit. The first acquisition unit acquires area data indicating area values ​​of each of a plurality of colors from an image of an item related to a task. The second acquisition unit inputs the area data to a classifier and acquires a classification result indicating a progress degree from the classifier. [Brief description of the drawings]

[0006] [Figure 1] 1 is a schematic diagram illustrating a progress determination system according to an embodiment. [Diagram 2] 1 is a schematic diagram for explaining a process of a progress determination system according to an embodiment; [Diagram 3]1 is a schematic diagram for explaining a process of a progress determination system according to an embodiment; [Figure 4] 1 is a schematic diagram for explaining a process of a progress determination system according to an embodiment; [Diagram 5] 1 is a schematic diagram for explaining a process of a progress determination system according to an embodiment; [Figure 6] 5A to 5C are schematic diagrams illustrating an example of an output by the progress determination system according to the embodiment. [Figure 7] 13 is a schematic diagram illustrating another example of output by the progress determination system according to the embodiment. FIG. [Figure 8] FIG. 13 is a schematic diagram for explaining a method for calculating man-hours. [Figure 9] 1 is a flowchart showing a process for learning a classifier in an embodiment. [Figure 10] 4 is a flowchart showing a process performed by the progress determination system according to the embodiment. [Figure 11] 13 is a flowchart showing a process during learning of a classifier in a first modified example of the embodiment. [Figure 12] 11 is a flowchart showing a process performed by a progress determination system according to a first modified example of the embodiment. [Figure 13] FIG. 11 is a schematic diagram illustrating a progress determination system according to a second modified example of the embodiment. [Figure 14] FIG. 11 is a schematic diagram illustrating a work site to which a progress determination system according to a second modified example of an embodiment is applied. [Figure 15] 11 is a graph illustrating an example of area data. [Figure 16] 13 is a flowchart showing a process performed by a progress determination system according to a second modified example of the embodiment. [Figure 17] FIG. 2 is a schematic diagram illustrating a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. In this specification and each drawing, the same elements as those already described are designated by the same reference numerals, and detailed description thereof will be omitted as appropriate.

[0008] FIG. 1 is a schematic diagram illustrating a progress determination system according to an embodiment. The progress determination system according to the embodiment is used to determine the progress of work from images of items related to the work. The work is a predetermined job in manufacturing, logistics, construction, inspection, etc. The items are products or equipment that are the subject of the work, or equipment, parts, tools, etc. that are used in the work.

[0009] 1, the progress determination system 1 includes a first acquisition unit 11, a second acquisition unit 12, a memory unit 15, a photographing unit 20, an input unit 21, and a display unit 22. The photographing unit 20 photographs an item related to the work and generates an image. The photographing unit 20 repeatedly photographs the item. The photographing unit 20 may generate a video by photographing. The photographing unit 20 stores the image or video in the memory unit 15.

[0010] Each image is associated with item identification data indicating the photographed item, task identification data indicating the task related to the item, and the time of photographing. The item identification data and task identification data are set in advance by a user before photographing begins. The user may be a worker, a task supervisor, or an administrator who manages the progress determination system 1.

[0011] The first acquisition unit 11 acquires an image stored in the storage unit 15. When a video is stored in the storage unit 15, the first acquisition unit 11 extracts a still image from the video. The first acquisition unit 11 calculates the area value of each of a plurality of colors in the image. Specifically, the colors to be extracted from the image are set in advance by the user and stored in the storage unit 15. The user sets in advance the range of pixel values ​​corresponding to each color.

[0012] For example, if the pixel values ​​of each pixel in the image are based on the RGB color space, the pixel values ​​include the luminance of each of R, G, and B. Upper and lower limits for the luminance of each of R, G, and B are set as ranges. If the pixel values ​​of each pixel in the image are based on the Lab color space, upper and lower limits for the values ​​of L, a, and b are set as ranges. A range is set for each color. For example, if four colors are used to determine the progress, a range is set for each of the four colors.

[0013] The first acquisition unit 11 compares the pixel value of each pixel with a plurality of ranges. When the pixel value is included in any of the ranges, the first acquisition unit 11 determines that the color of the pixel having that pixel value is a color corresponding to that range. By comparing each pixel value with the plurality of ranges, it is determined whether the color of each pixel is any of the colors set in advance. The first acquisition unit 11 calculates the number of pixels of each color based on the comparison result. The first acquisition unit 11 uses the number of pixels of each color as the area value of each color. The first acquisition unit 11 transmits area data including the area value of each color to the second acquisition unit 12. The first acquisition unit 11 also stores the area data in the storage unit 15 in association with the item identification data, the task identification data, and the shooting time.

[0014] When the second acquisition unit 12 acquires the area data, it inputs the area data to a classifier. When the area data is input, the classifier outputs a classification result indicating the progress of the work. The classifier is trained in advance by a user and stored in the storage unit 15. For example, a classifier trained by a random forest, a Bayesian classifier, or the like is used as the classifier.

[0015] When the area data is input, the classifier outputs a classification result of the progress degree. For example, the classifier outputs a conformance rate of the area data to each progress degree. The second acquisition unit 12 selects the progress degree with the highest conformance rate from the classification result. The second acquisition unit 12 acquires the selected progress degree as the progress degree corresponding to the input area data. The second acquisition unit 12 stores the acquired progress degree in the memory unit 15 as the progress degree of the work at the shooting time linked to the area data.

[0016] The input unit 21 is used when the user inputs the above-mentioned various data to the progress determination system 1. The display unit 22 displays the data output from the second acquisition unit 12 so that the data can be visually recognized by the user.

[0017] 2 to 5 are schematic diagrams for explaining the process of the progress determination system according to the embodiment. The processing performed by the progress determination system 1 will be described with reference to a specific example.

[0018] (study) The user trains the classifier in advance. The user prepares training data. For example, the training data includes a plurality of training images and a plurality of progress levels respectively linked to the plurality of training images. For example, the user uses the photographing unit 20 to photograph an item in a state corresponding to each progress level and generate a training image. The photographing conditions of the training image are set to be the same as the photographing conditions when determining the progress level. For example, the position and angle of the photographing unit 20 with respect to the item are set to be the same when preparing the training image and when determining the progress level. Instead of a photographed image, a CAD drawing, a 3D model image, or a hand-drawn illustration may be used as the training image.

[0019] In this example, the first acquisition unit 11 functions as a learning unit that trains a classifier. The first acquisition unit 11 acquires area data from each training image. The first acquisition unit 11 trains the classifier using the area data as input data and the progress levels associated with the training images as labels.

[0020] FIG. 2(a) is an example of a learning image prepared by a user. Learning image TI1 shows a shelf 50. Parts 51 to 53 used in work are stored on shelf 50. The color of shelf 50 is white (WH). The color of part 51 is yellow (YL). The color of part 52 is black (BK). The color of part 53 is green (GR). A progress level of "0%" is associated with learning image TI1. The progress level may be expressed as a percentage as in this example, or may be indicated by another numerical value.

[0021] In this case, four colors, white, yellow, green, and black, can be used to determine the progress level. The user sets the brightness ranges of white, yellow, green, and black. For example, when each pixel value is based on the RBG color space and is expressed in 256 gradations, the range of (R:G:B)=(245-255:245-255:245-255) is set as the white range. The range of (R:G:B)=(245-255:245-255:0-10) is set as the yellow range. The range of (R:G:B)=(0-10:245-255:0-10) is set as the green range. The range of (R:G:B)=(0-10:0-10:0-10) is set as the black range.

[0022] The first acquisition unit 11 compares the pixel value of each pixel in the learning image TI1 in Fig. 2(a) with the respective color ranges set by the user. Fig. 2(b) shows an example of area data of the learning image TI1 in Fig. 2(a) obtained from the comparison result. The first acquisition unit 11 performs supervised learning on the classifier using the area data in Fig. 2(b) as input data and 0% as a label.

[0023] 3(a) to 3(d) show other learning images TI2 to TI5 and the progress degrees associated with each learning image. In this example, part 51 is taken out first in the work. Next, part 52 is taken out. Finally, part 53 is taken out. The more parts are taken out, the smaller the area of ​​the color of that part becomes. Also, the larger the area of ​​the color of shelf 50 becomes. The first acquisition unit 11 acquires area data from each of learning images TI2 to TI5, similar to learning image TI1. The first acquisition unit 11 sequentially trains a classifier using multiple area data and multiple labels (progress degrees). This allows the classifier to train so that it can output a progress degree in response to the input of area data.

[0024] The classifier may output a classification result of the process included in the work instead of the progress degree. For example, when one work includes multiple processes, the process corresponds to the progress degree. In this case, the learning image and 、The corresponding training image Process name are prepared as learning data. The first acquisition unit 11 uses the area data as input data and the process names as labels to train a classifier. When the second acquisition unit 12 acquires the process names that are the classification results, it acquires the progress degree associated with the process name as the progress degree corresponding to the input area data.

[0025] Alternatively, unsupervised learning may be performed on the classifier. The first acquisition unit 11 acquires a plurality of area data from a plurality of prepared training images. The first acquisition unit 11 sequentially inputs the plurality of area data to the classifier and performs unsupervised learning. This allows the classifier to learn so that it can classify the plurality of area data. The first acquisition unit 11 stores the learned classifier in the storage unit 15.

[0026] When unsupervised learning is performed, when area data is input to the classifier, the classifier outputs a classification result of the area data. The second acquisition unit 12 refers to a training image belonging to the output classification, and acquires the progress degree associated with the training image as the progress degree corresponding to the area data input to the classifier.

[0027] As described above, the classifier may output a classification result that directly indicates the degree of progress through supervised learning. The classifier may output a classification result that indirectly indicates the degree of progress through unsupervised learning. In either case, the second acquisition unit 12 can obtain the degree of progress of the task at the time of photographing based on the classification result that indicates the degree of progress.

[0028] FIG. 4(a) to FIG. 4(d) are examples of other learning images. A parts box 60 is shown in learning images TI6 to TI9. A cable 62 with a tag 61 attached is stored in the parts box 60. The color of the top surface of the parts box 60 is blue (BL). The color of the inside of the parts box 60 and the color of the tag 61 are white (WH). The color of the cable 62 is black (BK). Progress degrees "0%, " "25%, " "50%, " "100%" are linked to the learning images TI6 to TI9, respectively.

[0029] The first acquisition unit 11 calculates area data including area values ​​of blue, white, and black from each of the learning images TI6 to TI9. The first acquisition unit 11 sequentially trains a classifier using the multiple area data and multiple progress levels.

[0030] (judgement) Fig. 5(a) is an example of an image IM1 captured by the capturing unit 20 to determine the degree of progress. In the image IM1, all parts 51 are taken out and some parts 52 are taken out. The first acquiring unit 11 calculates the area value of each color from the image IM1. Fig. 5(b) shows the area value of each color calculated from the image IM1 of Fig. 5(a).

[0031] The second acquisition unit 12 inputs the area data of FIG. 5(b) to the classifier. For example, the classifier outputs a classification result indicating that the degree of progress corresponding to the input area data is 50%. The second acquisition unit 12 acquires the degree of progress "50%" as the degree of progress of the work at the time when the image IM1 was captured. Alternatively, the classifier outputs a classification result indicating that the input area data belongs to the same classification as the area data of the learning image TI3 of FIG. 3(b). The second acquisition unit 12 acquires the degree of progress "50%" linked to the learning image TI3 as the degree of progress of the work at the time when the image IM1 was captured.

[0032] The second acquisition unit 12 may further calculate the actual man-hours. For example, the second acquisition unit 12 calculates the actual man-hours from the time when the photographing unit 20 starts photographing to the time when the image is obtained. The second acquisition unit 12 associates the actual man-hours with the progress degree and stores them in the storage unit 15.

[0033] (Pretreatment) The first acquisition unit 11 may cut out a part of the image generated by the photographing unit 20. The user specifies in advance the area in the image in which the object appears. For example, the coordinates of the four corners are specified, and a rectangular image in which the object appears is cut out. The cutout is performed on both the learning image and the image for progress assessment. By cutting out the image, it is possible to suppress the influence of the colors of the equipment, floors, walls, people, etc. around the object on the learning and assessment.

[0034] Furthermore, the first acquisition unit 11 may remove an area in which a person is shown from the cut-out image. For example, a classifier for identifying a person in an image is prepared in advance. The classifier includes a neural network. Preferably, a convolutional neural network (CNN) is used. Supervised learning is performed in advance on the classifier so that the person can be identified from the image. When the classifier identifies a person in the image, the first acquisition unit 11 removes the identified area. This makes it possible to suppress the influence of the color of the person's clothing, the color of an item carried by the person, etc. on the judgment.

[0035] The first acquisition unit 11 may normalize at least one of the brightness and contrast of the image. For example, the first acquisition unit 11 normalizes the luminance and the luminance contrast. The normalization is performed on both the learning image and the image for progress assessment. The normalization can reduce the influence of changes in the brightness of the work site, changes in the camera settings, and the like on each pixel value.

[0036] (display) The display unit 22 displays the data output from the second acquisition unit 12. For example, the second acquisition unit 12 outputs the progress rate and the work results to the display unit 22. A work plan may be stored in the memory unit 15. The second acquisition unit 12 further acquires the work plan, and outputs the progress rate, the work results, and the work plan to the display unit 22.

[0037] FIG. 6 is a schematic diagram illustrating an example of an output by the progress determination system according to the embodiment. The second acquisition unit 12 displays a judgment result screen 100 shown in Fig. 6 on the display unit 22. On the judgment result screen 100, a target time 110, a progress degree 120, a target man-hour 130, an actual man-hour 140, and an actual time 150 are displayed.

[0038] The work plan includes a target time 110 and a target man-hours 130. The target time 110 is the target time for reaching each progress level 120. The target man-hours 130 is the target man-hours for that work. In this example, the target man-hours 130 is expressed as the comparison between the target time 110 and the length of the bar. Specifically, the bar extends from the target time 13:00 to 17:00, indicating that the target man-hours is four hours.

[0039] The work results include actual man-hours 140 and actual time 150. Actual time 150 is the time when each progress level 120 was actually reached. It is the actual man-hours required to reach each progress level. In this example, actual man-hours 140 is expressed as the comparison between actual time 150 and the length of the bar. Specifically, the bar extends from the target time 13:00 to 17:00, indicating that the actual man-hours are 4 hours.

[0040] The actual man-hours 140 and actual time 150 are determined based on the time when the image was captured and the degree of progress acquired by the second acquisition unit 12. For example, the time when the image capture unit 20 starts capturing is regarded as the start time of the work. The time required to reach each degree of progress is calculated as the actual man-hours.

[0041] In the example of FIG. 6, the latest progress assessment result was obtained at 17:00. In the work plan, task A is started at 13:00, and the targets are set to reach 10%, 20%, and 30% progress at 14:00, 15:30, and 17:00, respectively. In the results, task A is started at 13:00, and the progress reaches 10% and 20% at 15:00 and 17:00, respectively. The progress has not yet reached 30%. The actual times when the progress reaches 10% and 20% are later than the target times.

[0042] For example, when an actual time is delayed from a target time for a certain degree of progress, the second acquisition unit 12 displays the actual time in a manner that is distinguishable from other actual times. In the example of FIG. 6, an actual time 152 of "15:00" and an actual time 153 of "17:00" are displayed in a manner that is distinguishable from an actual time 151 of "13:00". This allows the user to easily confirm which degree of progress is delayed from the target. In addition, the actual man-hours 140 may display a difference 145 between the target man-hours 130 and the actual man-hours 140. This makes it easier for the user to intuitively understand the difference between the target man-hours 130 and the actual man-hours 140.

[0043] The second acquisition unit 12 may predict the time to reach any one of the progress levels. The time to reach is calculated using the difference between the target time and the actual time for the latest progress level. The second acquisition unit 12 adds the difference between the target time for the latest progress level and the target time for the next progress level to the actual time for the latest progress level. This calculates the time to reach the next progress level.

[0044] 6, the difference between the target time at a progress rate of 20% and the target time at a progress rate of 30% is 1.5 hours. The second acquisition unit 12 adds 1.5 hours to the actual time at a progress rate of 20% to calculate the arrival time 160 as 18:30.

[0045] Alternatively, the arrival time may take into consideration the speed of the work up to the latest progress level. The second acquisition unit 12 compares the target progress level and the actual progress level at the time when the progress level was last determined. The second acquisition unit 12 calculates a ratio of the actual progress level to the target progress level. The second acquisition unit 12 multiplies the difference between the target time for the latest progress level and the target time for the progress level for which the arrival time is calculated by the ratio. The second acquisition unit 12 adds the difference multiplied by the ratio to the actual time when the progress level was last determined.

[0046] For example, in the example of FIG. 6, the second acquisition unit 12 compares the target progress of 30% at 17:00 when the progress was last determined with the actual progress of 20%. The second acquisition unit 12 calculates a ratio of 0.67 of the actual progress of 20% to the target progress of 30%. The second acquisition unit 12 multiplies the difference of 1.5 hours between the target time of 15:30 at the latest progress of 20% and the target time of 17:00 at the progress of 30% for which the arrival time is calculated, by the ratio 0.67. The second acquisition unit 12 adds the product of 1.5 hours and 0.67 to the actual time of 17:00 when the progress was last determined. As a result, 19:15 is calculated as the arrival time.

[0047] Using the above method, the second acquisition unit 12 may predict the time when the degree of progress reaches 100%. In other words, the time when the degree of progress reaches 100% is the expected time when the task is to be completed.

[0048] The arrival time is predicted based on the most recent progress determination result, thereby improving user convenience.

[0049] 7(a) and 7(b) are schematic diagrams illustrating another example of output by the progress determination system according to the embodiment. When one task includes multiple steps, the progress degree and the steps may be linked in the work plan. For example, the second acquisition unit 12 acquires the progress degree and also acquires the step name associated with the progress degree.

[0050] 7(a), task A includes processes a and b. For example, the progress rate of task A from 0% to 10% is linked to process a. The progress rate of task A from 10% to 30% is linked to process b. In target man-hours 130, target man-hours 131 for process a and target man-hours 132 for process b are displayed. In actual man-hours 140, target man-hours 141 for process a and target man-hours 142 for process b are displayed.

[0051] A comparison between the target and actual progress in each process may be displayed. For example, the user can move the pointer 170 displayed on the display unit 22 by operating the input unit 21. When the user moves the pointer 170 to any process in the target result 130 and clicks it, details of that process are displayed as shown in FIG. 7(b). In the detail screen 200 shown in FIG. 7(b), a progress degree 220, a target progress degree 230, and an actual progress degree 240 are displayed. The target progress degree 230 indicates the progress degree that should be achieved by the time the latest image is captured. The actual progress degree 240 indicates the progress degree that has been achieved by the time the latest image is captured. By displaying the detail screen 200, the user can easily grasp the details of each process even when there are many processes.

[0052] The man-hours for each process may be calculated based on the result of the progress determination, the process linked to the progress, and the photographing time. In the example shown in Fig. 7(a), the two hours from 13:00 to 15:00 can be calculated as the actual man-hours for process a. The two hours from 15:00 to 17:00 can be calculated as the actual man-hours for process b.

[0053] FIG. 8 is a schematic diagram for explaining a method for calculating the number of processes. A more detailed calculation method of man-hours will be described with reference to Fig. 8. One progress level is linked to one process. In the example of Fig. 7(a), a progress level between 0% and 10% is linked to process a. A progress level between 10% and 30% is linked to process b.

[0054] The horizontal axis in FIG. 8 represents time. The dashed line in FIG. 8 represents the timing of photographing by the photographing unit 20. The photographing unit 20 starts photographing at timing t1. The start time of photographing is treated as the start time of process a. Until timing t2, the progress rate is determined to be less than 10%. Therefore, it is determined that process a is being performed from timing t1 to t2. From timing t3 to t4, the progress rate is determined to be 10% or more and less than 30%. Therefore, it is determined that process b is being performed from timing t3 to t4. At timing t5, the progress rate is determined to be 30% or more. Therefore, it is determined that another process is being performed at timing t5.

[0055] In the example of FIG. 8, the following two methods can be applied as a method for calculating the actual man-hours. In the first method, the period from time t1 to time t2 is calculated as the actual man-hours for process a, and the period from time t3 to time t4 is calculated as the actual man-hours for process b. In the second method, the period from t1 to t3 is calculated as the actual man-hours of process a, and the period from t3 to t4 is calculated as the actual man-hours of process b. Alternatively, the period from t1 to t2 is calculated as the actual man-hours of process a, and the period from t2 to t4 is calculated as the actual man-hours of process b.

[0056] According to the first method, a difference occurs between the end time of process a and the start time of process b. As a result, the calculated actual man-hours are shorter than the actual man-hours. As in the second method, by matching the start time of the previous process with the end time of the next process for two consecutive processes, the difference between the calculated actual man-hours and the actual man-hours can be reduced.

[0057] FIG. 9 is a flowchart showing a process for learning a classifier in the embodiment. First, the user sets data required for determining progress using the input unit 21 (step S1) and stores it in the storage unit 15. For example, the range of each color, the position of the item in the captured image, etc. are set. In addition, the user appropriately stores the work plan, the classifier to be trained, the discriminator, etc. in the storage unit 15. The user prepares training data (step S2) and stores it in the storage unit 15. The training data includes multiple pairs of training images and progress degrees. The first acquisition unit 11 acquires area data from each training image (step S3). The first acquisition unit 11 trains the classifier using the multiple area data (step S4). As described above, the training may be performed by either supervised learning or unsupervised learning. The first acquisition unit 11 stores the trained classifier in the storage unit 15.

[0058] FIG. 10 is a flowchart showing the processing performed by the progress determination system according to the embodiment. The photographing unit 20 photographs an item related to the work and generates an image (step S11). The first acquisition unit 11 performs pre-processing on the image (step S12). The first acquisition unit 11 acquires area data from the image (step S13). The second acquisition unit 12 inputs the area data to a classifier (step S14) and obtains a classification result. The second acquisition unit 12 acquires a progress degree corresponding to the classification result (step S15). The second acquisition unit 12 outputs a progress degree determination result (step S16).

[0059] Advantages of the embodiment will be described. In the progress determination system 1 according to the embodiment, area data is used to determine the progress. The color area in an image is not easily affected by the state of the item (orientation, position, detailed structure, etc.). For example, even if the state of the item during work changes from a previously assumed state of the item, the change in the color area value is not easily affected. By using the area data, it is possible to reduce the impact of the state of the item on the progress determination result, and improve the accuracy of the progress determination.

[0060] The progress determination system 1 is applicable to a wide range of work, such as manufacturing, logistics, construction, and inspection, so long as there is a correlation between the progress of work and the area value of a color in an image.

[0061] (First Modification) In addition to the area data, edge data may be used to determine the degree of progress. The first acquisition unit 11 acquires the area data from an image of the item and performs edge detection on the image. The Canny Edge method or the Sobel method may be used for the edge detection. A threshold value for a luminance change during edge detection is set in advance by the user and stored in the storage unit 15. By the edge detection, multiple edges are extracted from the image and edge data is acquired. The first acquisition unit 11 outputs the edge data to the second acquisition unit 12 and stores it in the storage unit 15.

[0062] Even when training the classifier, the first acquisition unit 11 acquires area data and edge data from training images. The first acquisition unit 11 acquires pairs of area data and edge data from a plurality of training images, and trains the classifier. In supervised learning, the classifier is trained to output a classification result indicating a degree of progress in response to an input of a pair of area data and edge data.

[0063] FIG. 11 is a flowchart showing a process during learning of a classifier in the first modified example of the embodiment. The user sets data required for determining the progress degree, similarly to the flowchart shown in FIG. 9 (step S1). At this time, the user sets a threshold for edge detection in addition to each color range. The user prepares learning data (step S2). The first acquisition unit 11 acquires area data and edge data from each learning image (step S3a). The first acquisition unit 11 trains a classifier using the area data and edge data (step S4a).

[0064] FIG. 12 is a flowchart showing a process performed by the progress determination system according to the first modified example of the embodiment. Steps S11 and S12 are executed in the same manner as in the flowchart shown in Fig. 10. The first acquisition unit 11 acquires area data and edge data from the image (step S13a). The second acquisition unit 12 inputs the area data and edge data to a classifier (step S14a) and obtains a classification result. The second acquisition unit 12 acquires a progress degree corresponding to the classification result (step S15). The second acquisition unit 12 outputs a determination result of the progress degree (step S16).

[0065] By using edge data in addition to area data, the accuracy of determining the progress level can be further improved when there is a correlation between the progress level of a task and the shape of an item.

[0066] (Second Modification) FIG. 13 is a schematic diagram illustrating a progress determination system according to a second modified example of the embodiment. The progress determination system 2 according to the second modified example further includes an integration unit 13. The progress determination system 2 also includes a plurality of image capture units 20.

[0067] Each photographing unit 20 photographs the same item at the same time from different positions and angles to generate multiple images. Each photographing unit 20 photographs the item repeatedly and stores the images in the storage unit 15. Each image is associated with item identification data indicating the photographed item, task identification data indicating the task to which the item is related, and the photographing time.

[0068] The timing of photographing by each photographing unit 20 may be offset to the extent that no substantial difference in the progress level occurs. For example, in determining the progress level of a task that takes one day, the timing of photographing by each photographing unit 20 may be offset by less than one minute. Even in such a case, each photographing unit 20 can be considered to photograph the item at substantially the same timing.

[0069] The first acquisition unit 11 acquires area data from each image, associates the item identification data, task identification data, and shooting time of the original image with the area data, and stores the area data in the storage unit 15. The first acquisition unit 11 also transmits the area data, the item identification data, the task identification data, and the shooting time to the integration unit 13.

[0070] The integration unit 13 selects multiple area data based on multiple images captured at the same time. The integration unit 13 integrates the selected multiple area data into one area data. For example, the integration unit 13 averages the area values ​​of each color of the multiple area data. Alternatively, the integration unit 13 may integrate the multiple area data based on the accuracy of each area data.

[0071] As the degree of accuracy, one or more selected from the following first to fourth degrees of accuracy can be used. The first accuracy is preset by a user in accordance with the reliability of each area data. The more accurate the area data calculated from the captured image, the higher the reliability of the area data and the larger the first accuracy is set.

[0072] The second accuracy is the size of the object in the image. When a part of the captured image is cut out by pre-processing, the size of the cut-out image corresponds to the size of the object. The integration unit 13 sets the second accuracy based on the size of the cut-out image. Alternatively, the size of the object in the image depends on the distance between the object and the imaging unit 20. A value corresponding to the distance between the object and the imaging unit 20 may be preset as the second accuracy by the user.

[0073] The third accuracy is the angle of the photographing unit 20 with respect to the article. For example, if the area of ​​the article where the color changes according to the progress of the work faces upward, the color change is likely to be more accurately shown in an image of the article photographed from above. A value corresponding to the angle of the photographing unit 20 with respect to the article is preset by the user as the third accuracy.

[0074] The fourth accuracy is based on the size of the area of ​​the person in the image. If a part of an object in the image is hidden by a person, the area value cannot be calculated accurately. For example, the first acquisition unit 11 cuts out a part of the captured image. The first acquisition unit 11 identifies a person in the cut-out image and removes the area in which the person appears. The integration unit 13 reduces the fourth accuracy as the size of the removed area increases. Alternatively, the integration unit 13 may reduce the fourth accuracy as the ratio of the size of the removed area to the size of the cut-out image increases.

[0075] The integration unit 13 calculates the accuracy for each area data. When two or more of the four accuracy levels are used, the accuracy levels are added together to calculate one accuracy level. A weight may be set for each of the two or more accuracy levels, and one accuracy level may be calculated by the weighted sum.

[0076] The integration unit 13 integrates multiple area data into one piece of area data using multiple probabilities. For example, the integration unit 13 normalizes the multiple probabilities so that the sum of the multiple probabilities is "1". The integration unit 13 integrates the multiple area data and the normalized multiple probabilities, respectively, and adds up the integrated values. This results in one integrated piece of area data.

[0077] The second acquisition unit 12 inputs the integrated area data to a classifier and acquires the classification result.

[0078] The processing performed by the progress determination system 2 will be described with reference to a specific example. FIG. 14 is a schematic diagram illustrating a work site to which a progress determination system according to a second modified example of the embodiment is applied. In the work site shown in Fig. 14, photographing units 20a and 20b are installed. The photographing units 20a and 20b photograph a wagon 70 from different positions and angles. Parts 71 and 72 are placed on the top surface of the wagon 70. Worker O takes out parts 71 and 72 one by one from the wagon 70 and installs them into equipment placed in another location. For example, the color of the top surface of the wagon 70 is white (WH). The color of part 71 is yellow (YL). The color of part 72 is green (GR).

[0079] 15(a) to 15(c) are graphs showing an example of area data. The photographing units 20a and 20b photograph the top surface of the wagon 70, the part 71, and the part 72 at the same time. The photographing units 20a and 20b generate a first image and a second image, respectively. The first acquisition unit 11 acquires first area data from the first image, and acquires second area data from the second image. FIGS. 15(a) and 15(b) respectively illustrate the first area data and the second area data. The integration unit 13 integrates the first area data and the second area data. The integration unit 13 refers to the first accuracy to the fourth accuracy.

[0080] For example, the reliability of the first area data and the reliability of the second area data are equal for the first accuracy, and the user sets the first accuracy for the first area data and the first accuracy for the second area data to the same value.

[0081] For the second accuracy, the distance between the wagon 70 and the photographing unit 20a is shorter than the distance between the wagon 70 and the photographing unit 20b. The user sets the second accuracy for the first area data to be greater than the second accuracy for the second area data.

[0082] Regarding the third accuracy, the photographing unit 20a is provided directly above the wagon 70 and faces the top surface of the wagon 70. The photographing unit 20b photographs the wagon 70 from diagonally above. The photographing unit 20a can more easily photograph the entire wagon 70, the parts 71, and the parts 72 than the photographing unit 20b. The user sets the third accuracy for the first area data to be greater than the third accuracy for the second area data.

[0083] Regarding the fourth accuracy, for example, the first image does not show the worker O. The second image shows the worker O. In this case, in pre-processing, the first acquisition unit 11 removes the area showing the worker O from the second image. The integration unit 13 reduces the fourth accuracy for the second area data in accordance with the ratio of the size of the removed area to the size of the second image.

[0084] The integrating unit 13 adds the first to fourth accuracies for the first area data to calculate one accuracy. The integrating unit 13 adds the first to fourth accuracies for the second area data to calculate one accuracy. The integrating unit 13 normalizes each accuracy so that the sum of the accuracy for the first area data and the accuracy for the second area data is "1". The integrating unit 13 multiplies each area value of the first area data by the normalized accuracy. The integrating unit 13 multiplies each area value of the second area data by the normalized accuracy. The integrating unit 13 adds the integrated value of the first area data and the accuracy to the integrated value of the second area data and the accuracy for each color. In this way, multiple area data are integrated into one.

[0085] Fig. 15(c) is an example of a result of integrating the first area data and the second area data shown in Fig. 15(a) and Fig. 15(b). In this example, the accuracy for the first area data is greater than the accuracy for the second area data. Therefore, the difference between the integrated area data and the first area data is smaller than the difference between the integrated area data and the second area data. The second acquisition unit 12 inputs the area data shown in Fig. 15(c) into a classifier and acquires the classification result.

[0086] FIG. 16 is a flowchart showing a process performed by the progress determination system according to the second modified example of the embodiment. A plurality of photographing units 20 photograph an item at the same time to generate a plurality of images (step S11b). The first acquiring unit 11 performs pre-processing on each image (step S12b). The first acquiring unit 11 acquires area data from each image (step S13b). The second acquiring unit 12 integrates the plurality of area data into one (step S20). Thereafter, steps S14 to S16 are executed in the same manner as in the flowchart shown in FIG. 10.

[0087] In the second modification, edge data may be used as in the first modification. The first acquisition unit 11 acquires area data and edge data from each of images captured at the same timing. The integration unit 13 integrates a plurality of area data into one area data. The integration unit 13 integrates a plurality of edge data into one edge data. For example, the first acquisition unit 11 generates one integrated edge data by overlapping a plurality of edge data. Alternatively, the first acquisition unit 11 may synthesize a plurality of edge data by Poisson Image Editing to generate a single integrated edge data. The second acquisition unit 12 inputs the area data and the edge data to a classifier and acquires a classification result.

[0088] FIG. 17 is a schematic diagram showing a hardware configuration. The progress determination system 1 according to the embodiment can be realized by the hardware configuration shown in Fig. 17. The processing device 90 shown in Fig. 17 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0089] The ROM 92 stores a program for controlling the operation of the computer. The ROM 92 stores programs necessary for the computer to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0090] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0091] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs.

[0092] The input interface (I / F) 95 connects the processing device 90 and the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0093] The output interface (I / F) 96 connects the processing device 90 and the display device 96a. The output I / F 96 is, for example, a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI: registered trademark). The CPU 91 can transmit data to the display device 96a via the output I / F 96 and cause the display device 96a to display an image.

[0094] A communication interface (I / F) 97 connects the processing device 90 to a server 97a external to the processing device 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97. The camera 99 photographs an item and stores the image in the server 97a.

[0095] The storage device 94 includes one or more selected from a hard disk drive (HDD) and a solid state drive (SSD). The input device 95a includes one or more selected from a mouse, a keyboard, a microphone (audio input), and a touch pad. The display device 96a includes one or more selected from a monitor and a projector. A device having both the functions of the input device 95a and the display device 96a, such as a touch panel, may be used.

[0096] The processing device 90 functions as a first acquisition unit 11, a second acquisition unit 12, and an integration unit 13. The storage device 94 and the server 97a function as a storage unit 15. The input device 95a functions as an input unit 21. The display device 96a functions as a display unit 22. The camera 99 functions as an imaging unit 20.

[0097] The above-described progress determination system or progress determination method can be used to improve the accuracy of determining the progress degree. Also, a similar effect can be obtained by using a program for operating a computer as the progress determination system.

[0098] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0099] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0100] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0101] 1, 2: progress determination system, 11: first acquisition unit, 12: second acquisition unit, 13: integration unit, 15: memory unit, 20, 20a, 20b: photography unit, 21: input unit, 22: display unit, 50: shelf, 51-53: parts, 60: parts box, 61: tag, 62: cable, 70: cart, 71, 72: parts, 90: processing device, 91: CPU, 92: ROM, 93: RAM, 94: storage device, 95: input interface, 95a: input device, 96: output interface, 96a: display device, 97: communication interface, 97a: server, 98: system bus, 99: camera, 100: determination result, 110: target time, 120: progress degree, 130: target progress degree, 140: Actual progress, 141: Difference, 150-153: Actual time, 160: Arrival time, 170: Pointer, A: Work, O: Worker

Claims

1. a first acquisition unit that acquires a plurality of images of an item related to a work taken at the same time from different angles, and acquires area data indicating area values ​​of each of a plurality of colors from each of the plurality of images; an integration unit that integrates the plurality of area data into one based on respective probabilities for the plurality of area data; a second acquisition unit that inputs the integrated area data into a classifier and acquires a classification result indicating a progress degree from the classifier; A progress assessment system equipped with the above.

2. The progress determination system according to claim 1 , wherein the first acquisition unit determines a color of each pixel included in the image based on a range of pixel values ​​corresponding to the plurality of colors, and calculates the number of pixels of each of the colors as the area value.

3. The first acquisition unit further acquires edge data indicating an edge of the article from the image, The progress determination system according to claim 1 , wherein the second acquisition unit inputs the area data and the edge data to the classifier and acquires the classification result.

4. the accuracy of each of the plurality of area data is based on one or more selected from a first accuracy, a second accuracy, a third accuracy, and a fourth accuracy; the first accuracy is set corresponding to a reliability of each of the area data, the second accuracy is set corresponding to a size of the object in each of the images; The third accuracy is set in accordance with an angle of an image capturing unit that captures each of the images with respect to the object, 4. The progress determination system according to claim 1, wherein the fourth accuracy is set in accordance with a size of the person in each of the images.

5. Further comprising an imaging unit for imaging the item, The progress determination system according to any one of claims 1 to 4, wherein the first acquisition unit cuts out an area in which the item is captured from an image captured by the photographing unit, and acquires the area data from the cut-out image.

6. 6. The progress determination system according to claim 1, wherein the first acquisition unit acquires the area data after removing an area in which a person is photographed from the image.

7. The progress determination system according to any one of claims 1 to 6, further comprising a display unit that displays the degree of progress, a performance of the work calculated based on the time the image was taken, and a predetermined plan for the work.

8. The classifier is trained using area data obtained from a training image in which the article is captured as input data and a progress level corresponding to the training image as a label; The progress determination system according to claim 1 , wherein the area data obtained from the image is input to the trained classifier.

9. Acquire multiple images of work-related items taken at the same time from different angles, acquiring area data indicating area values ​​of each of a plurality of colors from each of the plurality of images; Integrating the plurality of area data into one based on the respective probabilities of the plurality of area data; The method of determining progress includes inputting the integrated area data into a classifier and obtaining a classification result indicating a degree of progress from the classifier.

10. On the computer, Acquire multiple images of work-related items taken at the same time from different angles, obtaining area data indicating area values ​​of each of a plurality of colors from each of the plurality of images; Integrating the plurality of area data into one based on the respective probabilities for the plurality of area data; inputting the integrated area data into a classifier, and obtaining a classification result from the classifier that indicates a degree of progress; program.

11. A storage medium storing the program according to claim 10.

Images