Progress determination system, progress determination method, and storage medium

Abstract

The present invention provides a progress determination system, a progress determination method, a program, and a storage medium that can improve the determination accuracy of progress. The progress determination system according to an embodiment includes a first acquisition unit and a second acquisition unit. The first acquisition unit acquires area data related to the area values of a plurality of colors from an image of an object related to an operation. The second acquisition unit inputs the area data to a classifier and acquires a classification result indicating progress from the classifier.

Description

Technical Field

[0001] Embodiments of the present invention relate to a progress determination system, a progress determination method, and a storage medium. Background Art

[0002] In order to automatically determine the progress of work, various techniques have been developed. Regarding the determination of progress, an improvement in accuracy is desired.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2020-71566 Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a progress determination system, a progress determination method, and a storage medium capable of improving the determination accuracy of progress.

[0007] The progress determination system according to the embodiment includes a first acquisition unit and a second acquisition unit. The first acquisition unit acquires area data related to the area values of multiple colors from an image capturing an article related to work. The second acquisition unit inputs the area data to a classifier and acquires a classification result indicating the progress from the classifier. Brief Description of the Drawings

[0008] Figure 1 It is a schematic diagram showing the progress determination system according to the embodiment.

[0009] Figure 2 It is a schematic diagram for explaining the processing of the progress determination system according to the embodiment.

[0010] Figure 3 It is a schematic diagram for explaining the processing of the progress determination system according to the embodiment.

[0011] Figure 4 It is a schematic diagram for explaining the processing of the progress determination system according to the embodiment.

[0012] Figure 5 It is a schematic diagram for explaining the processing of the progress determination system according to the embodiment.

[0013] Figure 6 It is a schematic diagram showing an output example of the progress determination system according to the embodiment.

[0014] Figure 7 It is a schematic diagram showing another output example of the progress determination system according to the embodiment.

[0015] Figure 8 It is a schematic diagram for explaining a method of calculating man-hours.

[0016] Figure 9 It is a flowchart showing the processing during the learning of the classifier in the embodiment.

[0017] Figure 10 It is a flowchart showing the processing of the progress determination system of the embodiment.

[0018] Figure 11 It is a flowchart showing the processing during the learning of the classifier in the first modification of the embodiment.

[0019] Figure 12 It is a flowchart showing the processing of the progress determination system of the first modification of the embodiment.

[0020] Figure 13 It is a schematic diagram showing the progress determination system of the second modification of the embodiment.

[0021] Figure 14 It is a schematic diagram of a work site where the progress determination system of the second modification of the embodiment is applied.

[0022] Figure 15 It is a curve graph showing an example of area data.

[0023] Figure 16 It is a flowchart showing the processing of the progress determination system of the second modification of the embodiment.

[0024] Figure 17 It is a schematic diagram showing the hardware structure.

[0025] Explanation of reference numerals

[0026] 1, 2: Progress determination system, 11: First acquisition unit, 12: Second acquisition unit, 13: Merging unit, 15: Storage unit, 20, 20a, 20b: Photographing unit, 21: Input unit, 22: Display unit, 50: Shelf, 51 - 53: Components, 60: Fitting box, 61: Label, 62: Cable, 70: Trolley, 71, 72: Components, 90: Processing device, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 95a: Input device, 96: Output interface, 96a: Display device, 97: Communication interface, 97a: Server, 98: System bus, 99: Camera, 100: Determination result, 110: Target time, 120: Progress, 130: Target progress, 140: Actual progress, 141: Difference, 150 - 153: Actual time, 160: Arrival time, 170: Indicator, A: Operation, O: Operator Detailed implementation manners

[0027] Hereinafter, each embodiment of the present invention will be described with reference to the accompanying drawings. In the specification of the present application and each figure, the same reference numerals are assigned to the same elements as those already described, and detailed descriptions are appropriately omitted.

[0028] Figure 1 It is a schematic diagram showing a progress determination system of an embodiment.

[0029] The progress determination system of the embodiment is used to determine the progress of the operation based on an image of an article related to the operation. The operation is a specified work in manufacturing, logistics, construction, inspection, etc. The article is a product or equipment that becomes the object of the operation, equipment, parts, or tools used in the operation, etc.

[0030] As Figure 1 shown, the progress determination system 1 includes a first acquisition unit 11, a second acquisition unit 12, a storage unit 15, a photographing unit 20, an input unit 21, and a display unit 22. The photographing unit 20 photographs an article related to the operation and generates an image. The photographing unit 20 repeatedly photographs the article. The photographing unit 20 may also generate a moving image by photographing. The photographing unit 20 saves the image or the moving image in the storage unit 15.

[0031] The image is associated with article identification data indicating the photographed article, operation identification data indicating the operation associated with the article, and the photographing time. The article identification data and the operation identification data are preset by the user before the start of photographing. The user is an operator, a supervisor of the operation, or a manager who manages the progress determination system 1, etc.

[0032] The first acquisition unit 11 acquires the image saved in the storage unit 15. When a moving image is saved in the storage unit 15, the first acquisition unit 11 cuts out a still image from the moving image. The first acquisition unit 11 calculates the area value of each of a plurality of colors in the image. Specifically, the colors extracted from the image are preset by the user and saved in the storage unit 15. The user presets the range of pixel values corresponding to each color.

[0033] For example, when the pixel value of each pixel in the image is based on the RGB color space, the pixel value includes the brightness of R, G, and B respectively. The upper and lower limits of the brightness of R, G, and B respectively are set as a range. When the pixel value of each pixel in the image is based on the Lab color space, the upper and lower limits of the values of L, a, and b respectively are set as a range. The range is set for each color. For example, when four colors are used in the determination of the progress, ranges are set for the four colors respectively.

[0034] The first acquisition unit 11 compares the pixel value of each pixel with a plurality of ranges. When the pixel value is included in a certain range, the first acquisition unit 11 determines that the color of the pixel having the pixel value is the color corresponding to the range. By comparing each pixel value with a plurality of ranges, it is determined whether the color of each pixel is a certain color set in advance. The first acquisition unit 11 calculates the number of pixels of each color based on the comparison result. The first acquisition unit 11 uses the number of pixels of each color as the area value of each color. The first acquisition unit 11 sends area data related to the area value of each color to the second acquisition unit 12. The area data includes, for example, the area value of each color. The area data may also include the ratio or distribution of the area values of each color. The first acquisition unit 11 associates the area data with the article identification data, the operation identification data, and the shooting time and stores them in the storage unit 15.

[0035] When the second acquisition unit 12 acquires area data including one or more selected from the area value, ratio, and distribution of each color, it inputs the area data to the classifier. If the classifier is input with area data, it outputs a classification result indicating the progress of the operation. The classifier is pre-learned by the user and stored in the storage unit 15. For example, a classifier learned by random forest or a Bayesian classifier, etc. is used as the classifier.

[0036] Alternatively, as the area data, a histogram representing the area value, the ratio, or the distribution may also be used. That is, the area data may also be image data representing information related to the area value of each color. In this case, a neural network for classifying the image data is used as the classifier. Preferably, the neural network includes a convolutional neural network (CNN: Convolutional Neural Network).

[0037] If the classifier is input with area data, it outputs a classification result of the progress. For example, the classifier outputs the suitability rate of the area data for each progress. The second acquisition unit 12 selects the progress with the highest suitability rate according to the classification result. The second acquisition unit 12 acquires the selected progress as the progress corresponding to the input area data. The second acquisition unit 12 stores the acquired progress as the progress of the operation at the shooting time associated with the area data in the storage unit 15.

[0038] The input unit 21 is used when the user inputs the above various data to the progress determination system 1. The display unit 22 displays the data output from the second acquisition unit 12 in a visually confirmable manner for the user.

[0039] Figures 2 to 5 It is a schematic diagram for explaining the processing of the progress determination system of the embodiment.

[0040] With reference to specific examples, the processing of the progress determination system 1 will be described. Here, an example in which the area values of each color in an image are used as area data will be described.

[0041] (Learning)

[0042] The user pre-learns the classifier. The user prepares learning data. For example, the learning data includes a plurality of learning images and a plurality of progress levels respectively associated with the plurality of learning images. For example, the user captures an item in a state corresponding to each progress level by the imaging unit 20 to generate a learning image. The imaging conditions of the learning image are set in the same manner as the imaging conditions at the time of progress determination. For example, when preparing the learning image and determining the progress, the position and angle of the imaging unit 20 with respect to the item are set to be the same. As the learning image, a CAD drawing, a 3D model image, or an illustration drawn by a person may be used instead of the captured image.

[0043] In this example, the first acquisition unit 11 functions as a learning unit for learning the classifier. The first acquisition unit 11 acquires area data from each learning image. The first acquisition unit 11 uses the area data as input data and the progress level associated with the learning image as a label to make the classifier learn.

[0044] Figure 2 FIG. (a) is an example of a learning image prepared by the user. In the learning image TI1, a shelf 50 is captured. Components 51 to 53 used in the operation are stored on the shelf 50. The color of the shelf 50 is white (WH). The color of the component 51 is yellow (YL). The color of the component 52 is black (BK). The color of the component 53 is green (GR). The learning image TI1 is associated with the progress level "0%". The progress level can be expressed as a percentage as in this example, or can be expressed by other values.

[0045] In this case, in the determination of the progress level, the four colors white, yellow, green, and black can be used. The user sets the brightness range of each of white, yellow, green, and black. For example, when each pixel value is represented by 256 gray levels based on the RGB color space, the range of (R:G:B)=(245 - 255:245 - 255:245 - 255) is set as the range of white. The range of (R:G:B)=(245 - 255:245 - 255:0 - 10) is set as the range of yellow. The range of (R:G:B)=(0 - 10:245 - 255:0 - 10) is set as the range of green. The range of (R:G:B)=(0 - 10:0 - 10:0 - 10) is set as the range of black.

[0046] The first acquisition unit 11 compares Figure 2 the pixel values of each pixel of the learning image TI1 in FIG. (a) with the ranges of the respective colors set by the user.Figure 2 The (b) is obtained based on the comparison result Figure 2 An example of the area data of the learning image TI1 in (a). The first acquisition unit 11 will Figure 2 The area data of (b) is used as input data, 0% is used as a label, and supervised learning is performed on the classifier.

[0047] Figure 3 The (a) to Figure 3 The (d) shows other learning images TI2 to TI5 and the progress associated with each learning image. In this example, during the operation, first, the component 51 is taken out. Next, the component 52 is taken out. Finally, the component 53 is taken out. The more components are taken out, the smaller the area of the color of that component becomes. The area of the color of the shelf 50 becomes larger. The first acquisition unit 11 obtains area data from the learning images TI2 to TI5 in the same manner as the learning image TI1. The first acquisition unit 11 uses a plurality of area data and a plurality of labels (progress) to sequentially learn the classifier. Thereby, the classifier is learned to be able to output the progress based on the input of the area data.

[0048] The classifier may also output the classification result of the processes included in the operation instead of the progress. For example, when one operation includes a plurality of processes, the processes correspond to the progress. In this case, a learning image and the process name corresponding to the learning image are prepared as learning data. The first acquisition unit 11 uses the area data as input data and uses the process name as a label to learn the classifier. When the second acquisition unit 12 obtains the process name as the classification result, it obtains the progress associated with the process name as the progress corresponding to the input area data.

[0049] Alternatively, unsupervised learning may be performed on the classifier. The first acquisition unit 11 obtains a plurality of area data from the plurality of prepared learning images. The first acquisition unit 11 sequentially inputs the plurality of area data to the classifier and performs unsupervised learning. Thereby, the classifier is learned to be able to classify the plurality of area data. The first acquisition unit 11 stores the learned classifier in the storage unit 15.

[0050] In the case of performing unsupervised learning, when area data is input to the classifier, the classifier outputs the classification result of the area data. The second acquisition unit 12 refers to the learning image belonging to the output classification and obtains the progress associated with the learning image as the progress corresponding to the area data input to the classifier.

[0051] As described above, the classifier may also output the classification result directly representing the progress through supervised learning. The classifier may also output the classification result indirectly representing the progress through unsupervised learning. In any case, the second acquisition unit 12 can obtain the progress of the operation at the time of shooting based on the classification result representing the progress.

[0052] Figure 4 of (a) to Figure 4 of (d) is an example of other learning images. In the learning images TI6 to TI9, a parts box 60 is photographed. A cable 62 with a label 61 is stored in the parts box 60. The color of the upper surface of the parts box 60 is blue (BL). The colors of the inside of the parts box 60 and the label 61 are white (WH). The color of the cable 62 is black (BK). The learning images TI6 to TI9 are respectively associated with progress “0%”, “25%”, “50%”, and “100%”.

[0053] The first acquisition unit 11 calculates area data including area values of blue, white, and black for each of the learning images TI6 to TI9. The first acquisition unit 11 uses the plurality of area data and the plurality of progress values to sequentially train the classifier.

[0054] (Determination)

[0055] Figure 5 of (a) is an example of an image IM1 photographed by the photographing unit 20 for determining progress. In the image IM1, a situation where all the components 51 have been taken out and a part of the component 52 has been taken out is photographed. The first acquisition unit 11 calculates the area value of each color based on the image IM1. Figure 5 of (b) indicates Figure 5 the area values of each color calculated from the image IM1 of (a).

[0056] The second acquisition unit 12 inputs Figure 5 the area data of (b) into the classifier. For example, the classifier outputs a classification result indicating that the progress corresponding to the input area data is 50%. The second acquisition unit 12 acquires the progress “50%” as the progress of the operation when the image IM1 is photographed. Alternatively, the classifier outputs a classification result indicating that the input area data belongs to the same classification as the area data of the learning image TI3 of (b). The second acquisition unit 12 acquires the progress “50%” associated with the learning image TI3 as the progress of the operation when photographing the image IM1. Figure 3 the area data of (b) into the classifier. For example, the classifier outputs a classification result indicating that the progress corresponding to the input area data is 50%. The second acquisition unit 12 acquires the progress “50%” as the progress of the operation when the image IM1 is photographed. Alternatively, the classifier outputs a classification result indicating that the input area data belongs to the same classification as the area data of the learning image TI3 of (b). The second acquisition unit 12 acquires the progress “50%” associated with the learning image TI3 as the progress of the operation when photographing the image IM1.

[0057] The second acquisition unit 12 can also calculate the actual working hours. For example, the second acquisition unit 12 calculates the time from the time when the photographing unit 20 starts photographing to the time when the image is obtained as the actual working hours. The second acquisition unit 12 associates the actual working hours with the progress and stores them in the storage unit 15.

[0058] (Pre - processing)

[0059] The first acquisition unit 11 may also cut out a part from the image generated by the imaging unit 20. The user designates in advance an area in the image where the item is imaged. For example, the coordinates of the four corners are designated, and a quadrilateral image in which the item is imaged is cut out. Cutting out is performed for both the learning image and the image for progress determination. By cutting out the image, it is possible to suppress the influence of the colors of equipment, the floor, walls, people, etc. around the item on learning and determination.

[0060] Furthermore, the first acquisition unit 11 may also remove from the cut-out image an area in which a person is imaged. For example, an identifier for identifying a person in the image is prepared in advance. The identifier includes a neural network. A convolutional neural network (CNN) is preferably used. In the identifier, supervised learning is performed in advance so that a person can be identified from the image. When the first acquisition unit 11 identifies a person in the image by the identifier, the identified area is removed. Thereby, it is possible to suppress the influence of the color of the person's clothes, the color of the item carried by the person, etc. on determination.

[0061] The first acquisition unit 11 may normalize at least one of the brightness and contrast of the image. For example, the first acquisition unit 11 normalizes the brightness and the brightness contrast. Normalization is performed for both the learning image and the image for progress determination. By normalization, it is possible to reduce the influence of changes in the brightness of the work site, changes in the settings of the camera, etc. on each pixel value.

[0062] (Display)

[0063] The display unit 22 displays the data output from the second acquisition unit 12. For example, the second acquisition unit 12 outputs the progress and the work results to the display unit 22. The work plan may also be saved in the storage unit 15. The second acquisition unit 12 also acquires the work plan and outputs the progress, the work results, and the work plan to the display unit 22.

[0064] Figure 6 It is a schematic diagram showing an output example of the progress determination system according to the embodiment.

[0065] The second acquisition unit 12 causes the display unit 22 to display Figure 6 the determination result screen 100 shown. In the determination result screen 100, a target time 110, a progress 120, a target man-hour 130, an actual man-hour 140, and an actual time 150 are displayed.

[0066] The work plan includes a target time 110 and a target man-hour 130. The target time 110 is the target of the time when each progress 120 is reached. The target man-hour 130 is the target man-hour for the work. In this example, the target man-hour 130 is represented by the comparison of the target time 110 and the length of the bar. Specifically, the bar extends from the target time 13:00 to 17:00, indicating that the target man-hour is 4 hours.

[0067] The operation results include the actual working hours 140 and the actual time 150. The actual time 150 is the time when each progress 120 is actually reached. It is the actual working hours required to reach each progress. In this example, the actual working hours 140 are represented by comparing the actual time 150 with the length of the bar. Specifically, the bar extends from the target time of 13:00 to 17:00, indicating that the actual working hours are 4 hours.

[0068] The actual working hours 140 and the actual time 150 are determined based on the time when the image is captured and the progress obtained by the second acquisition unit 12. For example, the time when the shooting unit 20 starts shooting is regarded as the start time of the operation. The time until each progress is reached is calculated as the actual working hours.

[0069] In Figure 6 the example, the determination result of the latest progress is obtained at 17:00. In the operation plan, operation A starts at 13:00, and the progress is set to reach 10%, 20%, and 30% at 14:00, 15:30, and 17:00 respectively as the target. In the actual situation, operation A starts at 13:00, and the progress reaches 10% and 20% at 15:00 and 17:00 respectively. The progress does not reach 30%. The actual time when the progress reaches 10% and the actual time when the progress reaches 20% are delayed compared to the target time.

[0070] For example, regarding a certain progress, when the actual time is delayed compared to the target time, the second acquisition unit 12 displays the actual time in a way that can be distinguished from other actual times. In Figure 6 the example, the actual time 152 at "15:00" and the actual time 153 at "17:00" have been displayed in a way that is distinguishable from the actual time 151 at "13:00". Thus, the user can easily confirm in which progress there is a delay compared to the target. In the actual working hours 140, the difference 145 between the target working hours 130 and the actual working hours 140 can also be shown. Thus, the user can easily and intuitively understand the difference between the target working hours 130 and the actual working hours 140.

[0071] The second acquisition unit 12 can also predict the arrival time when a certain progress is reached. The arrival time is calculated using the difference between the target time and the actual time of the latest progress. The second acquisition unit 12 adds the difference between the target time of the latest progress and the target time of the next progress to the actual time in the latest progress. Thus, the arrival time when reaching the next progress is calculated.

[0072] In Figure 6 the example, the difference between the target time of the 20% progress and the target time of the 30% progress is 1.5 hours. The second acquisition unit 12 adds 1.5 hours to the actual time of the 20% progress as the arrival time 160, and calculates 18:30.

[0073] Alternatively, regarding the arrival time, the lead time of the operations up to the latest progress may also be considered. The second acquisition unit 12 compares the target progress at the time when the progress was last determined with the actual progress. The second acquisition unit 12 calculates the ratio of the actual progress to the target progress. The second acquisition unit 12 multiplies the difference between the target time of the latest progress and the target time of the progress at which the arrival time is calculated by the said ratio. The second acquisition unit 12 adds the actual time at which the progress was last determined to the difference after multiplying by the said ratio.

[0074] For example, in Figure 6 the example of, the second acquisition unit 12 compares the target progress of 30% at 17:00 when the progress was last determined with the actual progress of 20%. The second acquisition unit 12 calculates the ratio of 0.67 of the actual progress of 20% to the target progress of 30%. The second acquisition unit 12 multiplies the difference of 1.5 hours between the target time of 15:30 at the latest progress of 20% and the target time of 17:00 at the progress of 30% at which the arrival time is calculated by the said ratio 0.67. The second acquisition unit 12 adds the actual time of 17:00 at which the progress was last determined to the product of 1.5 hours and 0.67. Thus, 19:15 is calculated as the arrival time.

[0075] By the above method, the second acquisition unit 12 can also predict the arrival time when the progress reaches 100%. The arrival time when the progress reaches 100% is, in other words, the estimated time when the operation ends.

[0076] By predicting the arrival time based on the determination result of the latest progress, the convenience for the user can be improved.

[0077] Figure 7 of (a) and Figure 7 of (b) are schematic diagrams showing other output examples of the progress determination system of the embodiment.

[0078] In the case where one operation includes a plurality of processes, in the operation plan, the progress and the processes may also be associated. For example, the second acquisition unit 12 acquires the progress and also acquires the process name associated with the progress.

[0079] In Figure 7 the example shown in (a) of, operation A includes processes a and b. For example, the progress of 0% to 10% of operation A is associated with process a. The progress of 10% to 30% of operation A is associated with process b. In the target man-hours of 130, the target man-hours 131 of process a and the target man-hours 132 of process b are shown. In the actual man-hours of 140, the target man-hours 141 of process a and the target man-hours 142 of process b are shown.

[0080] It is also possible to display the comparison between the target and the actual progress in each process. For example, the user can move the indicator 170 displayed on the display unit 22 by operating the input unit 21. If the user matches the indicator 170 with any one of the processes in the target man-hour 130 and clicks, the details of the process as shown in Figure 7 as shown in (b) are displayed. In Figure 7 In the detailed screen 200 shown in (b), the progress 220, the target progress 230, and the actual progress 240 are displayed. The target progress 230 indicates the progress that should be achieved before the moment when the latest image is captured. The actual progress 240 indicates the progress that can be achieved before the moment when the latest image is captured. Through the display of the detailed screen 200, even in the case of a large number of processes, the user can easily grasp the details in each process.

[0081] It is also possible to calculate the man-hour of each process based on the determination result of the progress, the process associated with the progress, and the shooting time. In Figure 7 In the example shown in (a), the 2 hours from 13:00 to 15:00 can be calculated as the actual man-hour of process a. The 2 hours from 15:00 to 17:00 can be calculated as the actual man-hour of process b.

[0082] Figure 8 is a schematic diagram for explaining the calculation method of man-hour.

[0083] Refer to Figure 8 to explain the more detailed calculation method of man-hour. One process is associated with one progress. In Figure 7 In the example of (a), for a progress of 0% or more and less than 10%, process a is associated. For a progress of 10% or more and less than 30%, process b is associated.

[0084] Figure 8 The horizontal axis in Figure 8 represents time.

[0085] In Figure 8 In the example, the following two methods can be applied as the calculation method of the actual man-hour.

[0086] In the first method, the actual working hours of process a are calculated from time t1 to t2, and the actual working hours of process b are calculated from time t3 to t4.

[0087] In the second method, the actual working hours of process a are calculated from time t1 to t3, and the actual working hours of process b are calculated from time t3 to t4. Alternatively, the actual working hours of process a are calculated from time t1 to t2, and the actual working hours of process b are calculated from time t2 to t4.

[0088] According to the first method, a difference is generated between the end time of process a and the start time of process b. Therefore, the calculated actual working hours are shorter than the actual working hours. As in the second method, for two consecutive processes, by making the start time of the previous process and the end time of the subsequent process consistent, the difference between the calculated actual working hours and the actual working hours can be reduced.

[0089] Figure 9 It is a flowchart showing the processing during the learning of the classifier in the embodiment.

[0090] First, the user sets the data required for progress determination (step S1) using the input unit 21 and saves it in the storage unit 15. For example, the ranges of each color, the positions of the items in the captured image, etc. are set. In addition, the user appropriately saves the operation plan, the classifier to be learned, the recognizer, etc. in the storage unit 15. The user prepares the learning data (step S2) and saves it in the storage unit 15. The learning data includes a group of multiple learning images and progress. The first acquisition unit 11 acquires the area data from each learning image (step S3). The first acquisition unit 11 uses the multiple area data to make the classifier learn (step S4). As described above, the learning can perform either supervised learning or unsupervised learning. The first acquisition unit 11 saves the learned classifier in the storage unit 15.

[0091] Figure 10 It is a flowchart showing the processing of the progress determination system in the embodiment.

[0092] The imaging unit 20 images the items related to the operation and generates an image (step S11). The first acquisition unit 11 performs preprocessing on the image (step S12). The first acquisition unit 11 acquires the area data from the image (step S13). The second acquisition unit 12 inputs the area data into the classifier (step S14) to obtain a classification result. The second acquisition unit 12 acquires the progress corresponding to the classification result (step S15). The second acquisition unit 12 outputs the determination result of the progress (step S16).

[0093] The advantages of the embodiment are described.

[0094] In the progress determination system 1 of the embodiment, area data is used in the determination of progress. The area of the color in the image is not easily affected by the state of the article (orientation, position, structure of the detailed part, etc.). For example, even when the state of the article during operation changes from the pre-conceived state of the article, the change in the area value of the color is not easily affected. By using area data, the influence of the state of the article on the determination result of the progress can be reduced, and the determination accuracy of the progress can be improved.

[0095] If there is a correlation between the progress of the operation and the area value of the color in the image, the progress determination system 1 can be applied to a wide range of operations such as manufacturing, logistics, construction, inspection, etc.

[0096] (First variant example)

[0097] In order to determine the progress, in addition to the area data, edge data can also be used. The first acquisition unit 11 acquires area data from the image of the article and performs edge detection on the image. The Canny Edge method or the Sobel method can be used for edge detection. The threshold for the brightness change during edge detection is preset by the user and stored in the storage unit 15. Through edge detection, a plurality of edges are extracted from the image to obtain edge data. The first acquisition unit 11 outputs the edge data to the second acquisition unit 12 and stores it in the storage unit 15.

[0098] During the learning of the classifier, the first acquisition unit 11 also acquires area data and edge data from the learning images. The first acquisition unit 11 acquires a set of area data and edge data from each of the plurality of learning images and makes the classifier learn. In supervised learning, the classifier learns in such a way that it outputs a classification result indicating the progress based on the input of the set of area data and edge data.

[0099] Figure 11 It is a flowchart showing the processing during the learning of the classifier in the first variant example of the embodiment.

[0100] The user and Figure 9 As shown in the flowchart, the data required for the determination of progress is set (step S1). At this time, in addition to setting the range of each color, the user also sets the threshold for edge detection. The user prepares the learning data (step S2). The first acquisition unit 11 acquires area data and edge data from each learning image (step S3a). The first acquisition unit 11 uses the area data and edge data to make the classifier learn (step S4a).

[0101] Figure 12 It is a flowchart showing the processing of the progress determination system in the first variant example of the embodiment.

[0102] And Figure 10Similarly for the flowchart shown, steps S11 and S12 are executed. The first acquisition unit 11 acquires area data and edge data from the image (step S13a). The second acquisition unit 12 inputs the area data and the edge data into the classifier (step S14a) to obtain a classification result. The second acquisition unit 12 acquires the progress corresponding to the classification result (step S15). The second acquisition unit 12 outputs the determination result of the progress (step S16).

[0103] By using the edge data based on the area data, when there is a correlation between the progress of the operation and the shape of the article, the determination accuracy of the progress can be further improved.

[0104] (Second modification example)

[0105] Figure 13 It is a schematic diagram showing a progress determination system according to a second modification example of the embodiment.

[0106] The progress determination system 2 of the second modification example further includes a merging unit 13. The progress determination system 2 includes a plurality of photographing units 20.

[0107] Each photographing unit 20 photographs the same article from different positions and angles at the same timing to generate a plurality of images. Each photographing unit 20 repeatedly photographs the article and stores the images in the storage unit 15. Each image is associated with article identification data indicating the photographed article, operation identification data indicating the operation associated with the article, and the photographing time.

[0108] The photographing timing of each photographing unit 20 may deviate within a range where there is no substantial difference in progress. For example, in the determination of the progress of an operation that takes one day, the photographing timing of each photographing unit 20 may deviate by less than one minute. In such a case, each photographing unit 20 can also be regarded as photographing the article at substantially the same timing.

[0109] The first acquisition unit 11 acquires area data from each image, associates the article identification data, operation identification data, and photographing time of the basic image with the area data, and stores them in the storage unit 15. The first acquisition unit 11 sends the area data, article identification data, operation identification data, and photographing time to the merging unit 13.

[0110] The merging unit 13 selects a plurality of area data based on the plurality of images photographed at the same timing. The merging unit 13 merges the selected plurality of area data into one area data. For example, the merging unit 13 averages the area values of each color of the plurality of area data. Alternatively, the merging unit 13 may also merge the plurality of area data based on the accuracy of each area data.

[0111] As the accuracy, one or more selected from the following first accuracy to fourth accuracy can be used.

[0112] The first accuracy corresponds to the reliability of each area data and is preset by the user. The more accurate the area data calculated from the captured image, the higher the reliability of the area data, and the larger the first accuracy is set.

[0113] The second accuracy is the size of the article in the image. In the case where a part of the captured image is cut out through preprocessing, the size of the cut-out image corresponds to the size of the article. The merging unit 13 sets the second accuracy based on the size of the cut-out image. Alternatively, the size of the article in the image depends on the distance between the article and the photographing unit 20. A value corresponding to the distance between the article and the photographing unit 20 may also be preset by the user as the second accuracy.

[0114] The third accuracy is the angle of the photographing unit 20 relative to the article. For example, in the case where the area where the color of the article changes according to the progress of the operation faces upward, in the image obtained by photographing the article from above, the possibility that the change in color appears more accurately is high. A value corresponding to the angle of the photographing unit 20 relative to the article is preset by the user as the third accuracy.

[0115] The fourth accuracy is based on the size of the area of the person photographed in the image. In the image, if a part of the article is blocked by a person, the area value cannot be accurately calculated. For example, the first acquisition unit 11 cuts out a part of the captured image. The first acquisition unit 11 identifies the person in the cut-out image and removes the area where the person is photographed. The larger the size of the removed area, the smaller the merging unit 13 makes the fourth accuracy. Alternatively, it may be that the larger the ratio of the size of the removed area to the size of the cut-out image, the smaller the merging unit 13 makes the fourth accuracy.

[0116] The merging unit 13 calculates the accuracy for each area data. When using two or more of the four accuracies, these accuracies are added together to calculate one accuracy. It is also possible to set weights for two or more accuracies respectively and calculate one accuracy through a weighted sum.

[0117] The merging unit 13 merges a plurality of area data into one area data using a plurality of accuracies. For example, the merging unit 13 normalizes the plurality of accuracies so that the sum of the plurality of accuracies becomes "1". The merging unit 13 accumulates the plurality of area data and the normalized plurality of accuracies respectively, and adds these accumulated values together. Thus, one merged area data is obtained.

[0118] The second acquisition unit 12 inputs the merged area data to the classifier and obtains a classification result.

[0119] With reference to a specific example, the processing of the progress determination system 2 will be described.

[0120] Figure 14 It is a schematic diagram of a work site showing a progress determination system to which a second modification of the embodiment is applied.

[0121] At Figure 14 In the work site shown, imaging units 20a and 20b are provided. The imaging units 20a and 20b image the trolley 70 from different positions and angles. Components 71 and 72 are placed on the top surface of the trolley 70. The operator O sequentially takes out the components 71 and 72 from the trolley 70 and assembles them into equipment placed at other locations. For example, the color of the top surface of the trolley 70 is white (WH). The color of the component 71 is yellow (YL). The color of the component 72 is green (GR).

[0122] Figure 15 of (a) to Figure 15 of (c) is a graph showing an example of area data.

[0123] The imaging units 20a and 20b image the top surface of the trolley 70, the component 71, and the component 72 at the same timing. The first image and the second image are respectively generated by the imaging units 20a and 20b. The first acquisition unit 11 acquires first area data from the first image and second area data from the second image. Figure 15 of (a) and Figure 15 of (b) respectively illustrate the first area data and the second area data. The merging unit 13 merges the first area data and the second area data. The merging unit 13 refers to the first accuracy to the fourth accuracy.

[0124] For example, regarding the first accuracy, the reliability of the first area data and the reliability of the second area data are the same. The user sets the first accuracy for the first area data and the first accuracy for the second area data to the same value.

[0125] Regarding the second accuracy, the distance between the trolley 70 and the imaging unit 20a is shorter than the distance between the trolley 70 and the imaging unit 20b. The user sets the second accuracy for the first area data to be larger than the second accuracy for the second area data.

[0126] Regarding the third accuracy, the imaging unit 20a is provided directly above the trolley 70 and faces the top surface of the trolley 70. The imaging unit 20b images the trolley 70 from an oblique upper direction. Compared with the imaging unit 20b, the imaging unit 20a is more likely to image the entire trolley 70, the component 71, and the component 72. The user sets the third accuracy for the first area data to be larger than the third accuracy for the second area data.

[0127] Regarding the fourth accuracy, for example, the operator O is not captured in the first image. The operator O is captured in the second image. In this case, in the preprocessing, the first acquisition unit 11 removes the area in which the operator O is captured from the second image. The merging unit 13 reduces the fourth accuracy for the second area data according to the ratio of the size of the removed area to the size of the second image.

[0128] The merging unit 13 adds the first accuracy to the fourth accuracy for the first area data to calculate one accuracy. The merging unit 13 adds the first accuracy to the fourth accuracy for the second area data to calculate one accuracy. The merging unit 13 normalizes each accuracy so that the sum of the accuracies for the first area data and the accuracies for the second area data becomes "1". The merging unit 13 multiplies each area value of the first area data by the normalized accuracy. The merging unit 13 multiplies each area value of the second area data by the normalized accuracy. The merging unit 13 adds the cumulative value of the first area data and the accuracy and the cumulative value of the second area data and the accuracy for each color. Thus, multiple area data are merged into one.

[0129] Figure 15 of (c) is Figure 15 of (a) and Figure 15 An example of the result after merging the first area data and the second area data shown in (b) of. In this example, the accuracy for the first area data is greater than the accuracy for the second area data. Therefore, the difference between the merged area data and the first area data is smaller than the difference between the merged area data and the second area data. The second acquisition unit 12 inputs Figure 15 the area data shown in (c) of to the classifier and obtains a classification result.

[0130] Figure 16 It is a flowchart showing the processing of the progress determination system according to the second modification of the embodiment.

[0131] Multiple imaging units 20 image an object at the same timing to generate multiple images (step S11b). The first acquisition unit 11 performs preprocessing on each image (step S12b). The first acquisition unit 11 obtains area data from each image (step S13b). The second acquisition unit 12 merges multiple area data into one (step S20). Thereafter, steps S14 to S16 are executed in the same manner as the Figure 10 flowchart shown in.

[0132] In the second modification example, the edge data can also be used in the same manner as in the first modification example. The first acquisition unit 11 acquires area data and edge data from each of the images captured at the same timing. The merging unit 13 merges a plurality of area data into one area data. The merging unit 13 merges a plurality of edge data into one edge data. For example, the first acquisition unit 11 generates one merged edge data by overlapping a plurality of edge data. Alternatively, the first acquisition unit 11 can also synthesize a plurality of edge data by Poisson Image Editing to generate one merged edge data. The second acquisition unit 12 inputs the area data and the edge data to the classifier and acquires a classification result.

[0133] Figure 17 It is a schematic diagram showing a hardware configuration.

[0134] The progress determination system 1 of the embodiment can be implemented by Figure 17 the hardware configuration shown. Figure 17 The processing device 90 shown includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0135] The ROM 92 stores a program for controlling the operation of the computer. Programs required for the computer to implement the above-described various processes are stored in the ROM 92. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0136] The CPU 91 includes a processing circuit. The CPU 91 uses the RAM 93 as a working memory and executes a program stored in at least any one of the ROM 92 or the storage device 94. During the execution of the program, the CPU 91 controls each component via the system bus 98 and executes various processes.

[0137] The storage device 94 stores data required for the execution of the program and data obtained by the execution of the program.

[0138] The input interface (I / F) 95 connects the processing device 90 to the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0139] The output interface (I / F) 96 connects the processing device 90 to the display device 96a. The output I / F 96 is, for example, a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI: registered trademark). The CPU 91 can send data to the display device 96a via the output I / F 96 to cause the display device 96a to display an image.

[0140] The communication interface (I / F) 97 connects the server 97a outside the processing device 90 to the processing device 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97. The camera 99 photographs an object and saves the image in the server 97a.

[0141] The storage device 94 includes one or more selected from a hard disk drive (HDD: Hard Disk Drive) and a solid state drive (SSD: Solid State Drive). The input device 95a includes one or more selected from a mouse, a keyboard, a microphone (voice input), and a touchpad. The display device 96a includes one or more selected from a monitor and a projector. A device having the functions of both the input device 95a and the display device 96a, such as a touch panel, may also be used.

[0142] The processing device 90 functions as a first acquisition unit 11, a second acquisition unit 12, and a merging unit 13. The storage device 94 and the server 97a function as a storage unit 15. The input device 95a functions as an input unit 21. The display device 96a functions as a display unit 22. The camera 99 functions as a photographing unit 20.

[0143] By using the progress determination system or the progress determination method described above, the determination accuracy of the progress can be improved. The same effect can be obtained by using a program for causing a computer to operate as the progress determination system.

[0144] The processing of the above various data may also be recorded as a program executable by a computer in a magnetic disk (such as a floppy disk and a hard disk), an optical disk (such as a CD-ROM, a CD-R, a CD-RW, a DVD-ROM, a DVD±R, a DVD±RW, etc.), a semiconductor memory, or other non-transitory computer-readable recording media (non-transitory computer-readable storagemedium).

[0145] For example, the information recorded in the recording medium can be read by a computer (or an embedded system). In the recording medium, the recording format (storage format) is arbitrary. For example, the computer reads a program from the recording medium and causes the CPU to execute the instructions described in the program based on the program. In the computer, the acquisition (or reading) of the program can also be performed via a network.

[0146] The embodiments may also include the following technical solutions.

[0147] (Technical solution 1)

[0148] A progress determination system includes:

[0149] A first acquisition unit that acquires area data related to the area values of multiple colors from an image of an object related to an operation; and

[0150] A second acquisition unit that inputs the area data into a classifier and acquires a classification result indicating progress from the classifier.

[0151] (Technical solution 2)

[0152] The progress determination system according to Technical solution 1, wherein the area data includes the area values of the multiple colors, the ratios of the area values of the multiple colors, or the distributions of the area values of the multiple colors.

[0153] (Technical solution 3)

[0154] The progress determination system according to Technical solution 2, wherein the first acquisition unit determines the color of each pixel included in the image based on the ranges of multiple pixel values corresponding to the multiple colors, and calculates the number of pixels of each color as the area value.

[0155] (Technical solution 4)

[0156] The progress determination system according to any one of Technical solutions 1 to 3, wherein the first acquisition unit further acquires edge data indicating the edge of the object from the image,

[0157] The second acquisition unit inputs the area data and the edge data into the classifier and acquires the classification result.

[0158] (Technical solution 5)

[0159] The progress determination system according to any one of Technical solutions 1 to 4, further including a merging unit,

[0160] The first acquisition unit respectively acquires multiple pieces of area data from multiple images obtained by photographing the object from different angles at the same timing,

[0161] The merging unit merges the plurality of area data into one based on the respective accuracies of the plurality of area data.

[0162] The second acquisition unit inputs the merged area data into the classifier and obtains the classification result.

[0163] (Technical solution 6)

[0164] The progress determination system according to Technical solution 5, wherein the accuracy for each of the plurality of area data is based on one or more selected from a first accuracy, a second accuracy, a third accuracy, and a fourth accuracy.

[0165] The first accuracy is set corresponding to the reliability of each of the area data.

[0166] The second accuracy is set corresponding to the size of the article in each of the images.

[0167] The third accuracy is set corresponding to the angle of the imaging unit that captures each of the images with respect to the article.

[0168] The fourth accuracy is set corresponding to the size of the person in each of the images.

[0169] (Technical solution 7)

[0170] The progress determination system according to any one of Technical solutions 1 to 6, further comprising an imaging unit that images the article.

[0171] The first acquisition unit cuts out the area in which the article is imaged from the image captured by the imaging unit, and obtains the area data from the cut-out image.

[0172] (Technical solution 8)

[0173] The progress determination system according to any one of Technical solutions 1 to 7, wherein the first acquisition unit obtains the area data after removing the area in which a person is imaged from the image.

[0174] (Technical solution 9)

[0175] The progress determination system according to any one of Technical solutions 1 to 8, further comprising a display unit that displays the progress, the actual performance of the operation calculated based on the time when the image was captured, and the plan of the operation set in advance.

[0176] (Technical solution 10)

[0177] The progress determination system according to any one of Technical Solutions 1 to 9, wherein the classifier uses the area data obtained from the learning image of the item as input data and uses the progress corresponding to the learning image as a label for learning.

[0178] The area data obtained from the image is input to the classifier that has completed learning.

[0179] Above, several embodiments of the present invention have been illustrated, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, changes, etc. can be made without departing from the gist of the invention. These embodiments and their variations are included in the scope or gist of the invention and are included in the invention described in the claims and its equivalents. The foregoing embodiments can be implemented in combination with each other.

Claims

1. A progress determination system, comprising: A first acquisition unit that acquires area data related to the area values of multiple colors from an image capturing an item related to an operation; A second acquisition unit that inputs the area data into a classifier and acquires a classification result indicating the progress from the classifier; and A merging unit, The first acquisition unit respectively acquires multiple pieces of the area data from multiple images obtained by capturing the item from mutually different angles at the same timing, The merging unit merges the multiple pieces of area data into one based on the accuracy of each of the multiple pieces of area data, The second acquisition unit inputs the merged area data into the classifier and acquires the classification result.

2. The progress determination system according to claim 1, wherein, The area data includes the area values of the multiple colors, the ratios of the area values of the multiple colors, or the distribution of the area values of the multiple colors.

3. The progress determination system according to claim 2, wherein, The first acquisition unit determines the color of each pixel included in the image based on the range of multiple pixel values respectively corresponding to the multiple colors, and calculates the number of pixels of each color as the area value.

4. The progress determination system according to claim 1 or 2, wherein, The first acquisition unit further acquires edge data indicating the edge of the item from the image, The second acquisition unit inputs the area data and the edge data into the classifier and acquires the classification result.

5. The progress determination system according to claim 1, wherein, The accuracy of each piece of the multiple pieces of area data is based on one or more selected from a first accuracy, a second accuracy, a third accuracy, and a fourth accuracy, The first accuracy is set corresponding to the reliability of each piece of the area data, The second accuracy is set corresponding to the size of the item in each of the images, The third accuracy is set corresponding to the angle of the imaging unit capturing each of the images with respect to the item, The fourth accuracy is set corresponding to the size of a person in each of the images.

6. The progress determination system according to claim 1 or 2, wherein, It further comprises an imaging unit for imaging the item, The first acquisition unit cuts out the area capturing the item from the image captured by the imaging unit, and acquires the area data from the cut-out image.

7. The progress determination system according to claim 1 or 2, wherein, The first acquisition unit acquires the area data after removing the area capturing a person from the image.

8. The progress determination system according to claim 1 or 2, wherein, The progress determination system further comprises a display unit that displays the progress, the actual performance of the operation calculated based on the time when the image was captured, and the pre-set plan of the operation.

9. The progress determination system according to claim 1 or 2, wherein, The classifier uses the area data obtained from the learning image of the item as input data and learns using the progress corresponding to the learning image as a label. The learned classifier is input with the area data obtained from the image.

10. A progress determination method, comprising the following steps: Obtain area data related to the area values of multiple colors from an image of an item related to an operation; Input the area data into a classifier and obtain a classification result representing the progress from the classifier; Respectively obtain multiple area data from multiple images obtained by photographing the item from mutually different angles at the same timing; Based on the respective accuracies of the multiple area data, merge the multiple area data into one; And Input the merged area data into the classifier and obtain the classification result.

11. A storage medium storing a program that causes a computer to execute the following processing: Obtain area data related to the area values of multiple colors from an image of an item related to an operation; Input the area data into a classifier and obtain a classification result representing the progress from the classifier; Respectively obtain multiple area data from multiple images obtained by photographing the item from mutually different angles at the same timing; Based on the respective accuracies of the multiple area data, merge the multiple area data into one; And Input the merged area data into the classifier and obtain the classification result.

Images