Information processing device, classification method, classification program, and control system

Abstract

To detect waste plastic.SOLUTION: An information processing device (1) includes: a similarity calculation part (107) for calculating a similarity between a histogram of a brightness value in a target area of a difference image generated from images obtained by capturing the inside a waste pit at a predetermined time interval and a histogram of a brightness value generated from images in which a bagged waste is captured; and a classification part (108) for classifying the waste in the image in the target area based on the calculated similarity.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device for classifying waste stored in a waste pit. [Background technology]

[0002] Waste incineration facilities and the like are provided with waste pits to temporarily store collected waste. Since the waste in the waste pits may contain waste of various properties, proper classification before processing is required in order to properly process the waste.

[0003] An example of a document disclosing prior art related to such classification is the following Patent Document 1. Patent Document 1 discloses a method for operating an automatic crane that identifies heterogeneous waste based on a waste color map that indicates the color tones of the waste piled up in a waste pit, and controls the crane to agitate the identified heterogeneous waste. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-126246 A Summary of the Invention [Problem to be solved by the invention]

[0005] Here, waste plastic is known as a waste that requires attention when incinerating. Waste plastic is a waste that is mainly composed of plastic, and if a large amount of waste plastic is put into an incinerator at once, the combustion state may become unstable. For this reason, when waste plastic is put into the waste pit, it is preferable to mix it with other waste so that a large amount of waste plastic is not put into the incinerator at once.

[0006] However, since many waste plastics have no color, there is a possibility that the presence of waste plastics will not be reflected in the waste color map of Patent Document 1. In other words, the technology of Patent Document 1 has a problem in that it is difficult to reflect waste plastics that should be detected as heterogeneous waste in the waste color map.

[0007] An object of one aspect of the present invention is to provide an information processing device or the like capable of detecting waste plastic in a waste pit. [Means for solving the problem]

[0008] In order to solve the above problems, an information processing device according to one embodiment of the present invention includes a similarity calculation unit that calculates the similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside a waste pit that stores waste, and a histogram of brightness values ​​generated from an image that is known to contain bagged waste or waste plastic, and a classification unit that classifies the waste appearing in the target area based on the similarity.

[0009] In order to solve the above problems, a classification method according to one embodiment of the present invention is a classification method executed by one or more information processing devices, and includes a similarity calculation step of calculating a similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside a waste pit that stores waste, and a histogram of brightness values ​​generated from an image that is known to contain bagged waste or waste plastic, and a classification step of classifying the waste appearing in the target area based on the similarity.

[0010] In order to solve the above problems, a control system according to one embodiment of the present invention includes an information processing device that classifies waste appearing in a target area based on the similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside a waste pit that stores waste, and a histogram of brightness values ​​generated from an image that is known to contain bagged waste or waste plastic, and a control device that controls the operation of a crane placed in the waste pit based on the results of the classification by the information processing device. Effect of the Invention

[0011] According to one aspect of the present invention, it becomes possible to detect waste plastic in a waste pit. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of a configuration of a main part of an information processing device according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a diagram illustrating an example of the configuration of a control system according to an embodiment of the present invention. [Diagram 3] 13 is a diagram showing an example of a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken of the inside of a waste pit at a predetermined time interval. FIG. [Figure 4] FIG. 4 is a diagram illustrating an example of processing executed by the information processing device. [Diagram 5] FIG. 13 is a diagram illustrating a method for setting a target region. [Figure 6] 4 is a flowchart illustrating an example of a process executed by the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] [System Configuration] The configuration of a control system 5 according to one embodiment of the present invention will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the control system 5. As shown in the figure, the control system 5 includes an information processing device 1, a control device 2, and a crane 3. Fig. 2 also shows a cross section of a waste pit P that stores waste.

[0014] The crane 3 is a machine arranged in the waste pit P, and is used to transport waste in the waste pit P. The components of the crane 3 include a girder 31, a traverse cart 32, a wire 33, and a bucket 34. The girder 31 is arranged so as to span between rails (extending in the depth direction of the figure) provided on opposing walls of a building (not shown) of the waste treatment plant, and can be moved in the depth direction of the figure along the rails. The traverse cart 32 is provided on the girder 31, and can be moved on the girder 31 in the left-right direction of the figure (direction perpendicular to the movement direction of the girder 31). A wire 33 is attached to the traverse cart 32, and a bucket 34 for grabbing waste is provided at the tip of the wire 33. The bucket 34 can be opened and closed.

[0015] The control device 2 controls the operation of the crane 3. For example, the control device 2 drives the girder 31 and the traverse cart 32 to move the bucket 34 to a desired position, opens the bucket 34 at that position and lowers it onto the waste in the waste pit P, and closes the bucket 34 to wind up the wire 33. Through this series of controls, the waste at that position can be picked up by the bucket 34.

[0016] The information processing device 1 classifies the waste stored in the waste pit P. More specifically, the information processing device 1 classifies the waste captured in the target area based on the similarity between a histogram of brightness values ​​in the target area of ​​a difference image generated from images captured at a predetermined time interval inside the waste pit P and a histogram of brightness values ​​generated from an image known to include bagged waste or waste plastic. Then, the control device 2 controls the operation of the crane 3 based on the result of the classification by the information processing device 1.

[0017] As described above, the control system 5 includes an information processing device 1 that classifies waste appearing in a target area based on the similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside the waste pit P where waste is stored, and a histogram of brightness values ​​generated from an image that is known to contain bagged waste or waste plastic, and a control device 2 that controls the operation of a crane 3 placed in the waste pit P based on the results of the classification by the information processing device 1.

[0018] Although details will be described later, the above configuration makes it possible to detect waste plastic in the waste pit P. Furthermore, the above configuration also makes it possible to appropriately dispose of the waste plastic by the crane 3 when the waste plastic is detected.

[0019] The information processing device 1 can be applied to classifying various types of waste, but an example of classifying waste to be incinerated in an incinerator will be described below. In this case, the bagged waste may be, for example, combustible garbage generated by general households. There is no significant problem if bagged waste is put into an incinerator en masse, but since putting waste plastic into an incinerator en masse can significantly change the combustion state of the incinerator, it is preferable to detect waste plastic separately from bagged waste. The information processing device 1 makes such detection possible.

[0020] [Device configuration] The configuration of the information processing device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the main configuration of the information processing device 1. As shown in the figure, the information processing device 1 includes a control unit 10 that controls each unit of the information processing device 1, and a storage unit 11 that stores various data used by the information processing device 1. The information processing device 1 also includes a communication unit 12 that allows the information processing device 1 to communicate with other devices, an input unit 13 that accepts input of various data to the information processing device 1, and an output unit 14 that allows the information processing device 1 to output various data.

[0021] The control unit 10 includes a difference image generating unit 101, a binary image generating unit 102, an expansion processing unit 103, a target region setting unit 104, an average value calculating unit 105, a histogram generating unit 106, a similarity calculating unit 107, a classification unit 108, and a crane control unit 109. The storage unit 11 stores a comparison histogram 111 and a classification model 112.

[0022] The difference image generating unit 101 generates a difference image from images taken at a predetermined time interval of the interior of the waste pit P where waste is stored. When the captured image is an RGB color image, the difference image generating unit 101 calculates the difference between each RGB component for each pixel included in an image captured at a certain time and each pixel in an image captured a predetermined time later. The difference image generating unit 101 then sets the sum of the differences calculated for each RGB component as the luminance value of each pixel in the difference image. The above-mentioned predetermined time may be set appropriately depending on the frequency at which waste is dumped into the waste pit P, etc.

[0023] The binary image generating unit 102 generates a binary image from the difference image generated by the difference image generating unit 101. When generating the binary image, the binary image generating unit 102 first extracts pixels whose brightness values ​​are equal to or greater than a predetermined threshold value from the difference image. When bagged waste or waste plastic is dumped into the waste pit P, the brightness value of the dumped area is often high because the waste is often light gray to dark gray. Therefore, by extracting pixels whose brightness values ​​are equal to or greater than the threshold value, it is possible to extract pixels that are likely to include newly dumped bagged waste or waste plastic.

[0024] Then, the binary image generating unit 102 generates a binary image by setting the luminance value of each extracted pixel in the difference image to a predetermined value (e.g., 255) and the luminance value of each non-extracted pixel to another predetermined value (e.g., 0). In the following, an example will be described in which the luminance value of each extracted pixel is set to 255 and the luminance value of each non-extracted pixel is set to 0. In this case, the extracted pixels are white and the non-extracted pixels are black.

[0025] The binary image generating unit 102 may extract pixels having an absolute brightness value equal to or greater than a predetermined threshold value from the difference image generated by the difference image generating unit 101. In this case, pixels that have changed to lower brightness values ​​are also extracted. This makes it possible to extract pixels from an area where black or similar colored waste, such as pruning waste and sludge waste, which will be described later, has been dumped. In addition, the brightness value is also lowered when gray waste plastic is dumped into an area where white bagged waste is piled up. The binary image generating unit 102 can extract pixels from such an area by extracting pixels having an absolute brightness value equal to or greater than a predetermined threshold value from the difference image. An example of extracting pixels having an absolute brightness value equal to or greater than a predetermined threshold value from the difference image to generate a binary image will be described below.

[0026] The expansion processing unit 103 performs expansion processing on the pixels extracted by the binary image generating unit 102. The expansion processing is a process in which if any pixel around the pixel of interest is white, the pixel of interest is made a white pixel. By performing the expansion processing, the subdivided pixel group can be made into a single region.

[0027] The target area setting unit 104 sets the target area based on the extraction result obtained by extracting pixels having a luminance value equal to or greater than a predetermined threshold value from the difference image. More specifically, the target area setting unit 104 sets the target area using the binary image generated by the binary image generating unit 102. Since the binary image generated by the binary image generating unit 102 indicates the above extraction result, the target area is set based on the above extraction result by setting the target area using the binary image. The method of setting the target area will be described in detail later.

[0028] The average value calculation section 105 calculates the average value of the luminance in the target region of the difference image generated by the difference image generation section 101 .

[0029] The histogram generating unit 106 generates a histogram of luminance values ​​in the target region of the difference image generated by the difference image generating unit 101. The range of luminance values ​​and the class width in the generated histogram may correspond to the comparison histogram 111 described later. For example, if the comparison histogram 111 is a histogram with a luminance value range of -255 to 255 and a class width of 10, the histogram generating unit 106 may generate a histogram with the same range and the same class width. In addition, the histogram generating unit 106 normalizes the generated histogram. The normalization is performed in order to calculate the similarity with the comparison histogram 111.

[0030] The similarity calculation unit 107 calculates the similarity between the histogram generated by the histogram generation unit 106 and the comparison histogram 111. The comparison histogram 111 is a histogram of brightness values ​​generated from an image known to include bagged waste or waste plastic. The comparison histogram 111 is a normalized histogram like the histogram generated by the histogram generation unit 106. Note that the method of calculating the similarity between histograms is not particularly limited. For example, the similarity calculation unit 107 may calculate the similarity by applying a well-known similarity calculation method.

[0031] The classification unit 108 classifies the waste appearing in the target area based on the similarity calculated by the similarity calculation unit 107. The classification unit 108 can classify based only on the similarity, but can also classify the waste taking into consideration the average value calculated by the average value calculation unit 105 in addition to the similarity. Furthermore, when the waste appearing in the target area is highly likely to be bagged waste or waste plastic but it is difficult to determine which it is based on the similarity and the average value, the classification unit 108 classifies the waste using the classification model 112.

[0032] The classification model 112 is a classification model that has been trained to be able to classify bagged waste and waste plastic. The classification model 112 can be generated by machine learning using images of bagged waste and images of waste plastic as training data. The machine learning algorithm is not particularly limited. For example, the classification model 112 may be a convolutional neural network with high image classification accuracy.

[0033] The crane control unit 109 controls the crane 3 via the control device 2 by communication through the communication unit 12. For example, when waste in a target area is classified as waste plastic, the crane control unit 109 notifies the control device 2 of the location of the target area and causes the crane 3 to transport the waste at that location and mix it with other waste. For example, the crane control unit 109 may disperse waste plastic along the movement path of the bucket 34 by opening the bucket 34 while moving the bucket 34 horizontally.

[0034] As described above, the information processing device 1 includes a similarity calculation unit 107 that calculates the similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside the waste pit P where waste is stored, and a comparison histogram 111, which is a histogram of brightness values ​​generated from an image that is known to contain bagged waste or waste plastic, and a classification unit 108 that classifies the waste appearing in the target area based on the calculated similarity.

[0035] As a result of experiments and studies by the inventors of the present application, it was found that when a difference image is created from images taken of the inside of the waste pit P at a predetermined time interval and a histogram of the brightness values ​​is created, the histogram has characteristics that correspond to the type of waste. In particular, it was found that a histogram of brightness values ​​generated from an image known to show bagged waste or waste plastic has different characteristics from a histogram of brightness values ​​generated from an image showing other types of waste.

[0036] For example, Fig. 3 shows four histograms, h1 to h4. These histograms have in common that they are histograms of brightness values ​​in a target area set in a difference image generated from images taken at a specified time interval inside the waste pit P, but the types of waste captured in the target area are different. In each histogram, the horizontal axis is the brightness value and the vertical axis is the frequency.

[0037] Specifically, the histogram h1 is a histogram of brightness values ​​in a target area in which pruning waste is captured. Pruning waste is waste generated by pruning vegetation and mainly contains fragments of vegetation. Pruning waste may contain materials that are difficult to burn, such as live wood, so it is preferable to avoid throwing all pruning waste into an incinerator.

[0038] Histogram h2 is a histogram of brightness values ​​in the target area in which sludge is captured. Sludge is muddy waste generated during the process of wastewater treatment, etc. Since sludge is also difficult to burn, it is preferable to avoid throwing all sludge into an incinerator.

[0039] Histogram h3 is a histogram of brightness values ​​in the target area in which bagged waste is captured, and histogram h4 is a histogram of brightness values ​​in the target area in which waste plastic is captured.

[0040] Of histograms h1 to h4, histograms h3 and h4 both have shapes that resemble normal curves overall, and it can be said that the shapes of these histograms are similar. On the other hand, the target area showing pruning waste, which is mostly green and brown, has a high proportion of pixels with negative brightness values, and histogram h1 is a histogram with a strong bias toward negative values. Furthermore, the target area showing sludge waste that is black or a color close to black has an even higher proportion of pixels with negative brightness values, and histogram h2 is a histogram with an even stronger bias toward negative values.

[0041] From the above, when the similarity between the histogram generated for a target area showing a certain waste and histogram h3 or h4 is high, it can be said that the waste shown in the target area is likely to be bagged waste or waste plastic. On the other hand, when the similarity is low, it can be said that the waste shown in the target area is likely to be waste other than bagged waste and waste plastic.

[0042] Therefore, by classifying the waste appearing in the target area based on its similarity to a comparison histogram 111, which is a histogram of brightness values ​​generated from an image known to contain bagged waste or waste plastic, it becomes possible to detect waste plastic within the waste pit P.

[0043] The information processing device 1 also includes an average value calculation unit 105 that calculates an average value of luminance in the target area. The classification unit 108 can also classify waste appearing in the target area based on the similarity calculated by the similarity calculation unit 107 and the average value calculated by the average value calculation unit 105.

[0044] As a result of experiments and studies by the inventors of the present application, it was found that the average brightness value in the above-mentioned difference image also shows different tendencies depending on the type of waste. For example, in the example of Figure 3, the average brightness value in histogram h1 is -41.93, and the average brightness value in histogram h2 is -104.62. In contrast, the average brightness value in histogram h3 is -5.97, and the average brightness value in histogram h4 is 38.23.

[0045] As such, since the average brightness value shows different trends depending on the type of waste, by classifying the waste appearing in the target area based on the average brightness value in addition to the similarity, it is possible to improve the detection accuracy of waste plastic in the waste pit P.

[0046] [Example of processing] An example of processing executed by the information processing device 1 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of processing executed by the information processing device 1. In the example of Fig. 4, a difference image A3 is generated from an image A1 taken of the inside of a waste pit P at a certain time and an image A2 taken of the inside of the waste pit P a predetermined time later.

[0047] Of the areas appearing in images A1 and A2, areas in which the luminance value did not change during the above-mentioned predetermined time period are black areas with a luminance value of zero in the difference image A3. On the other hand, areas in which the luminance value has changed during the above-mentioned predetermined time period, such as when new waste is put into the waste pit P, are areas in which the luminance value is non-zero in the difference image A3. For example, in the difference image A3, the area A31 directly below the waste input port is an overall white area with a high luminance value, which indicates that waste has been put into area A31 during the above-mentioned predetermined time period.

[0048] FIG. 4 shows a histogram h5 generated with the region A31 as the target region. The similarity calculation unit 107 calculates the similarity between the histogram h5 thus generated and a histogram h6, which is a comparison histogram 111. The histogram h6 is a histogram of brightness values ​​in the target region in which bags of waste are captured. The histogram h6 may be generated statistically based on histograms generated from a plurality of images in which various bags of waste are captured. The waste captured in the region A31 is classified based on the similarity thus calculated.

[0049] [Classification method] A method for classifying waste based on similarity will be described below. If the comparison histogram 111 is a histogram generated from an image showing waste in a bag, and the similarity is high, it is highly likely that the waste in the target area is waste in a bag. Similarly, if the comparison histogram 111 is a histogram generated from an image showing waste plastic, and the similarity is high, it is highly likely that the waste in the target area is waste plastic.

[0050] For this reason, the classification unit 108 may classify waste appearing in the target area as bagged waste on the condition that the degree of similarity with the comparison histogram 111 generated from an image known to include bagged waste is equal to or greater than a predetermined threshold. The classification unit 108 may also classify waste appearing in the target area as waste plastic on the condition that the degree of similarity with the comparison histogram 111 generated from an image known to include waste plastic is equal to or greater than a predetermined threshold. In the following, the threshold is referred to as the "upper limit value."

[0051] According to the former configuration, waste in bags can be detected separately from waste plastic. According to the latter configuration, waste plastic can be detected separately from waste in bags. From the viewpoint of preventing misclassification, it is preferable to set the threshold value relatively high. For example, in an experiment conducted by the inventor of the present application, it was confirmed that sufficient classification accuracy can be achieved by setting the threshold value to 0.87 or more when the similarity is expressed as a value between 0 and 1 (the closer to 1, the higher the similarity).

[0052] The above conditions may include a condition that the average value calculated by the average value calculation unit 105 is within a predetermined numerical range. This numerical range is set so that if the waste in the target area is waste plastic or bagged waste, it falls within the range, and if not, it falls outside the range. For example, the above numerical range may be set based on the brightness value in an image known to include bagged waste or an image known to include waste plastic. For example, in an experiment by the inventor of the present application, it was confirmed that the brightness value of the area in which bagged waste and waste plastic are shown falls within the range of approximately -20 to 85, so the above numerical range may be set to -20 to 85.

[0053] On the other hand, if the similarity with the comparison histogram 111 is low, it is highly likely that the waste reflected in the target area is neither bagged waste nor waste plastic. For this reason, the classification unit 108 may classify the waste reflected in the target area as a different category from bagged waste and waste plastic, provided that the similarity is less than a predetermined threshold. This allows waste that is highly likely to be waste other than bagged waste and waste plastic to be appropriately classified. In the following, the above threshold is referred to as the "lower limit value." Also, waste that is highly likely to be waste other than bagged waste and waste plastic is referred to as heterogeneous waste. In general, like waste plastic, it is not preferable to incinerate heterogeneous waste together, and it is preferable to disperse them in the waste pit P by stirring with the crane 3.

[0054] In addition, by setting the lower limit value low, it is possible to reduce the possibility that bagged waste and waste plastics are erroneously classified as heterogeneous waste. Therefore, the lower limit value may be appropriately determined according to the required classification accuracy, etc. For example, in an experiment conducted by the inventor of the present application, when the similarity is expressed as a value from 0 to 1 (the closer to 1, the higher the similarity), it has been confirmed that by setting the lower limit value to 0.50 or less, the possibility that bagged waste and waste plastics are erroneously classified as heterogeneous waste is low.

[0055] Furthermore, the classification unit 108 may classify waste appearing in the target area as heterogeneous waste if the average value calculated by the average value calculation unit 105 is outside a predetermined numerical range. This allows waste that is likely to be heterogeneous waste in terms of the average brightness value to be appropriately classified. For example, if the above numerical range is set to -20 to 85, the classification unit 108 classifies waste with an average value calculated by the average value calculation unit 105 greater than 85 and waste with an average value less than -20 as heterogeneous waste.

[0056] The classification unit 108 may also classify waste into heterogeneous waste using both the similarity and the average value. For example, the classification unit 108 may classify waste appearing in the target area as heterogeneous waste when the similarity is less than a predetermined threshold value and the average value calculated by the average value calculation unit 105 is outside a predetermined numerical range. For example, the classification unit 108 may classify waste appearing in the target area as heterogeneous waste when at least one of the conditions that the similarity is less than a predetermined threshold value and that the average value calculated by the average value calculation unit 105 is outside a predetermined numerical range is satisfied.

[0057] Here, if the similarity is higher than the lower limit and lower than the upper limit, it is highly likely that the waste in the target area is either bagged waste or waste plastic. The same is true if the average value calculated by the average value calculation unit 105 is within the above-mentioned predetermined numerical range. However, it is difficult to determine whether the waste in the target area is bagged waste or waste plastic from only the similarity and the average value.

[0058] For this reason, the classification unit 108 classifies the waste appearing in the target area using a classification model 112 that has been trained to be able to classify bagged waste and waste plastic, provided that the similarity is higher than a predetermined lower limit and less than a predetermined upper limit.

[0059] According to this configuration, it is possible to appropriately classify bagged waste and waste plastic and detect waste plastic with high accuracy. In addition, the above configuration, in which the classification objects are limited to two types, bagged waste and waste plastic, is advantageous in that the learning cost of the classification model can be minimized compared to a configuration in which all various types of waste are classified by the classification model.

[0060] The classification model 112 may output information indicating the degree of certainty that the waste reflected in the target area is bagged waste (e.g., a numerical value between 0 and 1) and information indicating the degree of certainty that the waste is waste plastic. In this case, the classification unit 108 may classify the waste reflected in the target area as waste plastic when the degree of certainty that the waste is waste plastic is equal to or greater than a predetermined threshold, and may classify the waste reflected in the target area as bagged waste when the degree of certainty is less than the threshold.

[0061] The conditions for using the classification model 112 may include a condition that the average value calculated by the average value calculation unit 105 is within the above-mentioned predetermined numerical range. This makes it possible to avoid classification using the classification model 112 when the average value calculated by the average value calculation unit 105 makes it difficult to determine that the waste reflected in the target area is bagged waste or waste plastic.

[0062] As described above, various methods can be applied to classify waste captured in a target area, but an example of classification based on the following conditions will be described below. (First condition): If the similarity is ≧0.87, the waste appearing in the target area is classified as bagged waste. (Second condition): If 0.5<similarity<0.87 and −20<average value<85, classification is performed using the classification model 112. (Third condition): If the classification model 112 classifies the item as waste plastic and the probability that the item is waste plastic is 0.95 or more, the item is classified as waste plastic, and if the probability is less than 0.95, the item is classified as bagged waste. (Fourth condition): If any of the following conditions is satisfied: similarity ≦ 0.5, average value ≦ -20, or average value ≧ 85, it is classified as heterogeneous waste.

[0063] [How to set the target area] The target area may be set manually, for example, by a user of the information processing device 1 specifying it via the input unit 13, but since the information processing device 1 is provided with the target area setting unit 104, the target area can be set automatically. Below, a method for setting the target area by the target area setting unit 104 will be described with reference to Fig. 5. Fig. 5 is a diagram for explaining the method for setting the target area.

[0064] 5 shows the difference image A4. A total of seven areas, areas A41a to A41g, are set in the difference image A4. Each of the areas A41a to A41g is set at a position corresponding to a position directly below an inlet for throwing waste into the waste pit P. That is, in this example, seven inlets for throwing waste are provided along the longitudinal direction of the waste pit P. In the following, when there is no need to distinguish between the areas A41a to A41g, they will simply be referred to as area A41.

[0065] The area A41 may be set according to the number and positions of the waste inlets provided in the waste pit P. Since newly inserted waste is reflected in the area A41, the target area setting unit 104 can set the target area in an appropriate position by setting the target area within the area A41.

[0066] When waste is thrown in through the throw-in opening, the position on the image where the newly thrown in waste appears will differ depending on the height of the waste already piled up at the throw-in position. Therefore, area A41 may be dynamically set according to the pile height of waste at each location in the waste pit P. This makes it possible to set the target area within an appropriate area according to the pile height of waste at each location in the waste pit P.

[0067] When setting the target area within the area A41, the target area setting unit 104 may set the target area based on the extraction result of extracting pixels having a brightness value equal to or greater than a predetermined threshold from the difference image. When waste is thrown into the waste pit P, an area consisting of pixels having a high brightness value is generated within the area A41. Therefore, by setting the target area based on the extraction result of extracting pixels having a brightness value equal to or greater than a predetermined threshold from the difference image, it is possible to set a valid target area.

[0068] For example, the target area setting unit 104 may obtain a rectangular area surrounding a pixel group consisting of white pixels in the binary image generated by the binary image generating unit 102 and subjected to expansion processing by the expansion processing unit 103, and set the rectangular area as the target area in the difference image. In FIG. 5, the target area set in this manner within the area A41a is shown as A42a. The target areas are set in the same manner for the areas A41b to A41g. This method of setting the target area is also applicable to the case where the area A41 is not set.

[0069] As described above, the binary image generating unit 102 may extract pixels whose "absolute value" of brightness is equal to or greater than a predetermined threshold value from the difference image. In this case, the target area setting unit 104 sets the target area based on the result of the extraction. This makes it possible to set both areas into which waste such as waste plastics with high brightness values ​​have been dumped and areas into which waste such as pruning waste and sludge with low brightness values ​​have been dumped as target areas.

[0070] Moreover, the area A41 may include an area into which no waste has been thrown. Since it is not necessary to set a target area in an area into which no waste has been thrown, the target area setting unit 104 may set a target area in an area of ​​the area A41 into which waste is highly likely to have been thrown.

[0071] For example, the target area setting unit 104 may set the target area to an area that contains a predetermined ratio or more of pixels having a brightness value equal to or greater than a predetermined threshold value among a plurality of areas A41 in the difference image that are preset at positions corresponding to a plurality of input ports through which waste is input into the waste pit P. This makes it possible to set the target area by narrowing it down to the area A41 where there is a high possibility that waste has been input.

[0072] For example, in the binary image generated by the binary image generating unit 102 and expanded by the expansion processing unit 103, if the ratio of white pixels in a region corresponding to a certain region A41 is equal to or greater than a predetermined threshold, the target region setting unit 104 may set the target region in that region A41. In other words, in the binary image after expansion processing, if the ratio of white pixels in a region corresponding to a certain region A41 is less than a predetermined threshold, the target region setting unit 104 may not set the target region in that region A41. The above ratio may be determined as appropriate, and may be, for example, 40%.

[0073] [Processing flow] The flow of the process executed by the information processing device 1 will be described with reference to Fig. 6. Fig. 6 is a flow chart showing an example of the process executed by the information processing device 1. Note that it is assumed here that the inside of the waste pit P is photographed at a predetermined time interval during the time period when waste is carried into the waste pit P. The process of Fig. 6 is repeated every time a new image is photographed and acquired by the information processing device 1.

[0074] In S1, the differential image generating unit 101 acquires images of the inside of the waste pit P taken at a predetermined time interval, and generates a differential image from those images.

[0075] In S2, the binary image generating unit 102 generates a binary image from the difference image generated in S1. More specifically, the binary image generating unit 102 generates a binary image by extracting pixels whose absolute brightness values ​​are equal to or greater than a threshold value from the difference image. Note that the binary image generating unit 102 may generate a binary image by extracting pixels whose brightness values ​​are equal to or greater than a threshold value from the difference image.

[0076] In S3, the expansion processing unit 103 performs expansion processing on the pixels extracted in S2. Then, in S4, the target area setting unit 104 sets a target area based on the extraction result of S2. For example, the target area setting unit 104 obtains a rectangular area surrounding a pixel group consisting of white pixels in the binary image that has been subjected to expansion processing, and sets the rectangular area as the target area in the difference image. As described with reference to FIG. 5, the target area setting unit 104 may set the target area within the area A41 that is set at a position corresponding to directly below the input port for inputting waste into the waste pit P. Note that multiple target areas may be set, and in the case where multiple target areas are set, the processing from S5 onwards is performed for each target area.

[0077] In S5, the average value calculation unit 105 calculates the average value of luminance in the target region set in S4 of the difference image generated in S1. Then, in S6, the histogram generation unit 106 generates a histogram of luminance values ​​in the target region set in S4 of the difference image generated in S1. The histogram generation unit 106 also normalizes the generated histogram. Note that the process of S6 may be performed before S5, or the processes of S5 and S6 may be performed in parallel.

[0078] In S7 (similarity calculation step), the similarity calculation unit 107 calculates the similarity between the histogram generated in S6 and the comparison histogram 111. The comparison histogram 111 is a histogram of brightness values ​​generated from an image known to include bagged waste. Note that the comparison histogram 111 may also be a histogram of brightness values ​​generated from an image known to include waste plastic. After S7 ends, the process proceeds to S8. S8 to S12 are classification steps for classifying waste in the target area based on the similarity calculated in S7.

[0079] In S8, the classification unit 108 judges whether the similarity calculated in S7 satisfies a first condition. The first condition is that the similarity calculated in S7 is equal to or greater than a predetermined upper limit (e.g., 0.87). If the result of the judgment in S8 is YES, the process proceeds to S9, and if the result of the judgment in S8 is NO, the process proceeds to S10.

[0080] In S9, the classification unit 108 classifies the waste in the target area as bagged waste, and the process in Fig. 6 ends. If the comparison histogram 111 used in S7 is a histogram of brightness values ​​generated from an image known to include waste plastic, the waste in the target area is classified as waste plastic in S9.

[0081] In S10, the classification unit 108 judges whether the average value calculated in S5 and the similarity calculated in S7 satisfy a second condition. The second condition is that the similarity calculated in S7 is less than a predetermined upper limit and greater than a predetermined lower limit, and the average value calculated in S5 is less than a predetermined upper limit and greater than a predetermined lower limit. For example, the classification unit 108 may judge that the second condition is satisfied if 0.5<similarity<0.87 and -20<average value<85. If the judgment in S10 is YES, the process proceeds to S12, and if the judgment in S10 is NO, the process proceeds to S11. If the above-mentioned fourth condition is satisfied, the judgment in S10 is NO.

[0082] In S11, the classification unit 108 classifies the waste appearing in the target area as heterogeneous waste, and the process in Fig. 6 ends. For example, assume that a histogram like histogram h1 (pruning waste) or histogram h2 (sludge waste) in Fig. 3 is generated in S6. In this case, since neither the first condition nor the second condition is satisfied, NO is determined in S8 and S10, and the waste appearing in the target area is classified as heterogeneous waste in S11.

[0083] In S12, the classification unit 108 classifies the waste in the target area using the classification model 112 that has been trained to be able to classify bagged waste and waste plastic. The image input to the classification model 112 may be one that includes the target area. For example, when the classification unit 108 classifies the target area A42a in FIG. 5, the classification unit 108 may cut out a portion corresponding to the area A41a from an image that was taken later among the images that were the basis of the difference image generated in S1, and input the cutout portion to the classification model 112. The classification result in S12 is either bagged waste or waste plastic. In S12, the classification unit 108 may classify the waste in the target area based on the third condition described above. When the processing of S12 ends, the processing of FIG. 6 ends.

[0084] The processing of Fig. 6 described above includes the classification method according to this embodiment. That is, the classification method according to this embodiment includes a similarity calculation step (S7) for calculating the similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside a waste pit P storing waste, and a histogram of brightness values ​​generated from an image known to include bagged waste or waste plastic, and a classification step (S8 to S12) for classifying waste in the target area based on the calculated similarity. This makes it possible to detect waste plastic in the waste pit P.

[0085] Although not shown in FIG. 6, if the classification by the classification unit 108 results in waste plastic or heterogeneous waste, the crane control unit 109 notifies the control device 2 of the target area in which the waste was detected, and has the crane 3 transport the waste and mix it with other waste.

[0086] [Use of multiple imaging modalities] In the above embodiment, an example has been described in which waste in a target area in the waste pit P is classified based on an image captured by one photographing device, but multiple photographing devices may be used. The waste in the waste pit P is basically not flat, and depending on the photographing angle, blind spots may occur due to other waste or the crane 3, and the appearance of the waste in the photographed image may change depending on the way the light hits it, etc. In this regard, by using multiple photographing devices and photographing the inside of the waste pit P from multiple angles, it becomes possible to stably perform highly accurate classification.

[0087] For example, if the waste pit P is rectangular in top view, one image capturing device may be placed near one apex to overlook the waste pit P, and another image capturing device may be placed near the diagonal apex to overlook the waste pit P. This allows one image capturing device to capture an area that is a blind spot for the other image capturing device.

[0088] The method for classifying waste in a target area from images captured by each imaging device is as described in the above embodiment. In this case, multiple classification results are obtained for the same target area, and the crane control unit 109 may control the crane 3 based on those classification results. For example, when at least one classification result for a certain target area indicates that the waste in that target area is waste plastic or heterogeneous waste, the crane control unit 109 may cause the crane 3 to transport that waste and mix it with other waste.

[0089] [Modifications] The execution subject of each process described in the above embodiment is arbitrary and is not limited to the above example. In other words, an information processing system having the same function as the information processing device 1 can be constructed by a plurality of information processing devices capable of communicating with each other. For example, each process shown in FIG. 6 can be shared and executed by a plurality of information processing devices (which can also be called computers). In other words, the execution subject of the classification method according to this embodiment may be one information processing device or a plurality of information processing devices.

[0090] [Software implementation example] The functions of the information processing device 1 can be realized by a program for causing a computer to function as the information processing device 1, and a program (classification program) for causing a computer to function as each control block of the information processing device 1 (particularly each part included in the control unit 10).

[0091] In this case, the information processing device 1 includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the above program. The control device and storage device execute the above program to realize each function described in the above embodiment.

[0092] The above program may be recorded in one or more computer-readable recording media, not temporarily. The recording media may or may not be included in the information processing device 1. In the latter case, the above program may be supplied to the information processing device 1 via any wired or wireless transmission medium.

[0093] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0094] 〔summary〕 An information processing device according to aspect 1 of the present invention includes a similarity calculation unit that calculates the similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside a waste pit that stores waste, and a histogram of brightness values ​​generated from an image that is known to contain bagged waste or waste plastic, and a classification unit that classifies the waste appearing in the target area based on the similarity.

[0095] An information processing device according to aspect 2 of the present invention is, in the above-mentioned aspect 1, provided with an average value calculation unit that calculates an average value of brightness in the target area, and the classification unit classifies waste appearing in the target area based on the similarity and the average value calculated by the average value calculation unit.

[0096] In an information processing device according to aspect 3 of the present invention, in the case of aspect 1 or 2, the classification unit classifies waste appearing in the target area using a classification model that has been trained to be able to classify bagged waste and waste plastic, provided that the similarity is higher than a predetermined lower limit and lower than a predetermined upper limit.

[0097] In an information processing device according to aspect 4 of the present invention, in any of aspects 1 to 3, the classification unit classifies waste appearing in the target area as bagged waste, provided that the similarity with the histogram generated from an image known to contain bagged waste is equal to or greater than a predetermined threshold, or classifies waste appearing in the target area as waste plastic, provided that the similarity with the histogram generated from an image known to contain waste plastic is equal to or greater than a predetermined threshold.

[0098] In an information processing device according to aspect 5 of the present invention, in any of aspects 1 to 4, the classification unit classifies waste appearing in the target area as a different category from bagged waste and waste plastic, provided that the similarity is less than a predetermined threshold.

[0099] An information processing device according to aspect 6 of the present invention, in any of aspects 1 to 5, is provided with a target area setting unit that sets the target area based on the extraction result of extracting pixels having brightness values ​​equal to or greater than a predetermined threshold value from the difference image.

[0100] In an information processing device of aspect 7 of the present invention, in aspect 6, the target area setting unit sets the target area to an area in the differential image that contains a predetermined percentage or more of pixels whose brightness values ​​are above a predetermined threshold value, out of a plurality of areas in advance set at positions corresponding to each of a plurality of input ports through which waste is input into the waste pit.

[0101] A classification method according to aspect 8 of the present invention is a classification method executed by one or more information processing devices, and includes a similarity calculation step of calculating a similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside a waste pit that stores waste, and a histogram of brightness values ​​generated from an image that is known to contain bagged waste or waste plastic, and a classification step of classifying the waste appearing in the target area based on the similarity.

[0102] A classification program according to a ninth aspect of the present invention is a classification program for causing a computer to function as the information processing device according to the first aspect, and causes a computer to function as the similarity calculation unit and the classification unit.

[0103] A control system according to aspect 10 of the present invention includes an information processing device that classifies waste appearing in a target area based on the similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside a waste pit for storing waste, and a histogram of brightness values ​​generated from an image known to contain bagged waste or waste plastic, and a control device that controls the operation of a crane placed in the waste pit based on the results of the classification by the information processing device.

[0104] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0105] 1. Information processing device 104 Target area setting section 105 Average value calculation section 107 Similarity calculation unit 108 Classification Department 111 Comparison Histogram 112 Classification Model 2. Control device 3. Crane 5. Control System

Claims

1. a similarity calculation unit that calculates the similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside a waste pit that stores waste, and a histogram of brightness values ​​generated from an image that is known to include bagged waste or waste plastic; A classification unit that classifies the waste captured in the target area based on the similarity.

2. an average value calculation unit that calculates an average value of luminance in the target region; The information processing apparatus according to claim 1 , wherein the classification unit classifies the waste appearing in the target area based on the similarity and the average value calculated by the average value calculation unit.

3. The information processing device according to claim 1 or 2, wherein the classification unit classifies the waste appearing in the target area using a classification model that has been trained to be able to classify bagged waste and waste plastic, provided that the similarity is higher than a predetermined lower limit and less than a predetermined upper limit.

4. The classification unit is classifying waste in the target area as bagged waste if the similarity to the histogram generated from an image known to contain bagged waste is equal to or exceeds a predetermined threshold; or The information processing device according to claim 1 or 2, wherein waste appearing in the target area is classified as waste plastic on the condition that the similarity with the histogram generated from an image known to contain waste plastic is equal to or greater than a predetermined threshold.

5. The information processing apparatus according to claim 1 , wherein the classification unit classifies the waste captured in the target area into a different category from bagged waste and waste plastic, on condition that the similarity is less than a predetermined threshold.

6. The information processing apparatus according to claim 1 , further comprising a target region setting unit that sets the target region based on an extraction result obtained by extracting pixels having a luminance value equal to or greater than a predetermined threshold value from the difference image.

7. The information processing device described in claim 6, wherein the target area setting unit sets the target area to an area in the difference image that contains a predetermined percentage or more of pixels whose brightness values ​​are equal to or greater than a predetermined threshold value among a plurality of areas in advance set at positions corresponding to each of a plurality of input ports through which waste is input into the waste pit.

8. A classification method executed by one or more information processing devices, comprising: a similarity calculation step of calculating a similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside a waste pit that stores waste, and a histogram of brightness values ​​generated from an image that is known to include bagged waste or waste plastic; A classification method comprising: a classification step of classifying waste appearing in the target area based on the similarity.

9. 2. A classification program for causing a computer to function as the information processing device according to claim 1, the classification program causing a computer to function as the similarity calculation unit and the classification unit.

10. an information processing device that classifies waste appearing in a target area based on a similarity between a histogram of brightness values ​​in a target area of ​​a difference image generated from images taken at a predetermined time interval inside a waste pit that stores waste, and a histogram of brightness values ​​generated from an image that is known to include bagged waste or waste plastic; A control system including a control device that controls the operation of a crane arranged in the waste pit based on the classification results by the information processing device.

Images