Methods, computer program products, apparatuses, and systems related to anomaly sensing

Abstract

An abnormality sensing method includes: acquiring a judgment image that is an image of a judgment object that should be regularly arranged in a state in which a tubular container is erected; creating a difference image in which a difference between an image of a normal judgment object that is regularly arranged and the judgment image is emphasized; calculating a multi-division error of the difference image; determining a maximum value of the multi-division error; and determining normality or abnormality of the judgment object based on the maximum value.

Description

Technical Field

[0001] This invention relates to anomaly sensing. Background Technology

[0002] Generally, during the container manufacturing process, containers under manufacturing or finished products are sometimes arranged in a regular manner. For example, when finished containers are shipped, they are stacked on pallets in a regular arrangement. For example, Japanese Patent Application Publication No. 2011-111319 discloses a technology related to a palletizer for stacking cans on pallets. Summary of the Invention

[0003] In processes where containers should be arranged regularly, it is generally necessary to eliminate the following anomalies: missing containers, containers that should be upright but are tilted, or containers that are arranged haphazardly. Therefore, it is required to properly sense such anomalies.

[0004] The purpose of this invention is to properly sense abnormalities in the state of a container.

[0005] According to one aspect of the present invention, an anomaly sensing method includes: acquiring a judgment image, which is an image of judgment objects that should be regularly arranged in a cylindrical container in an upright state; creating a difference image that emphasizes the difference between the image of the regularly arranged normal judgment objects and the judgment image; calculating a multi-segmentation error of the difference image, determining a maximum value of the multi-segmentation error; and determining whether the judgment objects are normal or abnormal based on the maximum value.

[0006] According to the present invention, abnormalities in the state of the container can be appropriately sensed. Attached Figure Description

[0007] Figure 1 This is a schematic diagram illustrating a configuration example of a stacking machine system according to one implementation method.

[0008] Figure 2 This is a diagram showing examples of input images to the autoencoder, output images from the autoencoder, and their differential images, respectively, related to the normal situation, the abnormal situation where a leaking tank was produced, and the abnormal situation where a neat arrangement resulted in disorder.

[0009] Figure 3 This is a flowchart outlining the decision-making process using the first decision-making method.

[0010] Figure 4 This is a diagram illustrating an example of evaluation values ​​obtained by applying a first decision method to multiple images being judged.

[0011] Figure 5This is a diagram showing an example of the evaluation values ​​obtained by using a comparison example method for multiple judged images.

[0012] Figure 6 This is a flowchart outlining the decision-making process using the second decision-making method.

[0013] Figure 7 This is a diagram illustrating an example of evaluation values ​​obtained by using a second decision method on multiple judged images. Detailed Implementation

[0014] One embodiment will be described with reference to the accompanying drawings. This embodiment relates to a stacking machine system. This stacking machine system is used in the final stage of the manufacturing process of cans, which are one type of bottomed cylindrical container, to arrange and stack the manufactured cans regularly on pallets for shipment. The stacking machine system of this embodiment is equipped with an anomaly sensing system that senses any issues that may occur during stacking, such as can leakage, can tipping, or disordered arrangement, to prevent can leakage, tipping, or disorder during shipment.

[0015] [Composition and Operation of Stacking Machine System]

[0016] <Stacking Machine>

[0017] Figure 1 This is a schematic diagram illustrating a configuration example of the stacking machine system 1 according to this embodiment. The stacking machine system 1 includes: a stacking machine 10 for stacking manufactured empty cans onto a pallet for shipment; and a control device 30 for controlling the operation of the stacking machine 10. The stacking machine 10 arranges cans 90, upright with their bottoms facing down, in a regular and orderly manner, layer by layer. The stacking machine 10 stacks the neatly arranged cans 90 on a pallet a predetermined number of layers by sandwiching separator plates between each layer. Figure 1 In the process, the manufactured cans 90 are conveyed from right to left. The stacking machine 10 has a supply section 11, a tidying section 12, a transfer section 13, and a stacking section 14 in sequence from the upstream side to the downstream side.

[0018] The manufactured cans 90 are supplied from the supply section 11 to the stacking machine 10. The supply section 11 is provided with partition plates 22 corresponding to the number of rows of cans 90 to be neatly arranged, and the cans 90 are neatly arranged in a predetermined number of rows and supplied to the arranging section 12. In the arranging section 12, the cans 90 are filled without gaps, thereby the cans 90 are regularly and neatly arranged.

[0019] The stacking machine 10 is equipped with an ejector 25 that reciprocates between the arranging section 12 and the transfer section 13. When a layer or more of cans 90 to be placed on a pallet are neatly arranged in the arranging section 12, the ejector 25 pushes the neatly arranged layer of cans 90 toward the transfer section 13. The layer of cans 90 waits in the transfer section 13 until it is stacked.

[0020] A lifting device is provided in the stacking section 14. A pallet is placed on the lifting device. The lifting device aligns the height of the dividers on which the cans 90 are placed with the height of the transfer section 13. A reciprocating transfer machine 28 is provided in the stacking machine 10 between the transfer section 13 and the stacking section 14. The transfer machine 28 places a layer of cans 90 waiting in the transfer section 13 onto the dividers in the stacking section 14. Then, the lifting device lowers the pallet by one layer. Dividers are placed on the cans 90, with the height of the dividers aligned with the height of the transfer section 13.

[0021] When the cans 90 are stacked in the stacking section 14, freeing up the transfer section 13, the ejector 25 repeatedly pushes a single layer of neatly arranged cans 90 from the stacking section 12 to the transfer section 13. The transfer machine 28 repeatedly loads a single layer of cans 90 waiting in the transfer section 13 onto the dividers of the stacking section 14 according to a predetermined number of layers. As described above, the cans 90, neatly arranged regularly and without gaps, are placed on pallets according to a predetermined number of layers. When the predetermined number of layers of cans 90 have been stacked, these cans 90 are strapped together, wrapped in shrink film, and shipped out.

[0022] <Anomaly Detection System>

[0023] In this embodiment, the stacking machine 10 of the stacking machine system 1 is equipped with an anomaly sensing system 50. The anomaly sensing system 50 detects the following anomalies in relation to the cans 90 to be stacked on a pallet: missing cans (where a gap exists between neatly arranged cans 90, resulting in an insufficient number of cans 90); tipped cans (cans 90 tilting but remaining stacked as is); or disordered neat arrangement (cans 90 being stacked as is). Before stacking on the pallet, the anomaly sensing system 50 detects the occurrence of missing cans, tipped cans, disordered neat arrangement, etc. This prevents the cans 90 from being stacked on the pallet while anomalies such as missing cans, tipped cans, or disordered neat arrangement are present.

[0024] The anomaly sensing system 50 senses the occurrence of anomalies based on images of the cans 90 neatly arranged in the tidying section 12. Therefore, the anomaly sensing system 50 includes an imaging device 52 provided in the tidying section 12. The imaging device 52 is located on the upper part of the downstream side of the tidying section 12 and is configured to capture images of all cans 90 pushed out by the pusher 25 from above. As an example, in this embodiment, three imaging devices 52 are arranged in the width direction of the tidying section 12. Each imaging device 52 continuously or intermittently captures images of the cans 90 passing below it, so as to capture images of all cans 90 moved by the pusher 25. The imaging device 52 is not limited to this; for example, it may be connected to a PoE (Power over Ethernet) hub 54, powered from the PoE hub 54, and the captured images may be transmitted via the PoE hub 54.

[0025] The anomaly sensing system 50 includes an anomaly sensing device 56. The anomaly sensing device 56 is a computer comprising integrated circuits such as a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), or a Central Processing Unit (CPU), and may also include a Graphics Processing Unit (GPU). The anomaly sensing device 56 acquires images captured by the imaging device 52 via a PoE hub 54. Using the acquired images, the anomaly sensing device 56 senses anomalies such as leaking containers, tipped-over containers, or disordered arrangement.

[0026] An anomaly sensing device 56 is connected to a control device 30 that controls the operation of the stacking machine 10. When the anomaly sensing device 56 detects an anomaly, it sends a signal indicating that an anomaly has been detected to the control device 30. Upon receiving the anomaly sensing signal, the control device 30 stops the stacking of the cans 90. Furthermore, at this time, the control device 30 causes a notification device 62, such as a display device or a speaker, to output an alarm indicating that an anomaly has occurred.

[0027] [Anomaly Detection Method]

[0028] The method of anomaly sensing performed by the anomaly sensing device 56 will be described. The anomaly sensing device 56 acquires an image of the cans 90, which should be regularly arranged in the neat arrangement section 12 and is to be judged, from the imaging device 52 as the judged image. The anomaly sensing device 56 generates a difference image that emphasizes the difference between the image of the regularly arranged normal cans 90 and the judged image acquired from the imaging device 52.

[0029] As an example, in this embodiment, the anomaly sensing device 56 utilizes an autoencoder employing a deep neural network. The autoencoder in this embodiment is configured, for example, to output an image of a normal can 90 as input by untaught machine learning using images of cans 90 arranged normally and regularly (i.e., normal images). The anomaly sensing device 56 inputs the determined image acquired from the imaging device 52 into the autoencoder to obtain a reference image as the output of the autoencoder. Then, the anomaly sensing device 56 calculates the difference between the determined image acquired from the imaging device 52 and the reference image as the output of the autoencoder to create a difference image.

[0030] Figure 2 Examples of input images, output images, and their differential images to the autoencoder are shown, respectively, relating to (a) a normal situation, (b) an abnormal situation resulting in a leaking can, and (c) an abnormal situation resulting in disordered arrangement. Under normal conditions, the difference between the input image to the autoencoder configured as in this embodiment and the output reference image is small, resulting in a small error calculated from the differential image. Conversely, under abnormal conditions, the difference between the input image to the autoencoder and the output reference image is large, resulting in a large error calculated from the differential image. Therefore, the abnormality sensing device 56 can determine whether the object being judged is normal or abnormal based on the error calculated from the obtained differential image, that is, determine whether abnormalities such as leaking cans, tipped cans, or disordered arrangement have occurred.

[0031] In this embodiment, the autoencoder generated through machine learning is imported into a small computer equipped with an FPGA, thereby enabling the anomaly sensing device 56 to be configured as an edge device in a factory equipped with a stacking machine 10. It should be noted that the anomaly sensing device 56 can be connected to a network or configured to allow for remote operation of rewriting, relearning, and reinstalling the autoencoder.

[0032] In this embodiment, the manufactured can 90 is used as the evaluation object. The mouth portion of the can 90 is slightly narrower than its body portion; therefore, when the image is taken from above by the imaging device 52, the portion of the can 90 that slopes from its body to its mouth, referred to as the shoulder, is also captured in the image. Furthermore, for the manufactured can 90, including its body and shoulder, a printing corresponding to the product is applied to its outer periphery. This printing varies depending on the position of the outer periphery. Regarding the cans 90 neatly arranged in the aligning section 12 of the stacking machine 10, their circumferential orientation is not specifically controlled; therefore, in the evaluation image captured by the imaging device 52, the orientation of the shoulder printing differs for each can 90.

[0033] Furthermore, in this embodiment, the image of the can 90 moving on the ejector 25 is acquired by the imaging device 52 as the judgment image. Therefore, the position of the can 90 in the judgment image may be different in each image.

[0034] Due to the characteristics of these determined images, in this embodiment, even if observed... Figure 2 As is known under normal circumstances (a), the noise in the differential image is high. Therefore, in this embodiment, the error of the differential image, which increases under abnormal conditions, is easily buried in the noise. Therefore, in this embodiment, a determination method is used that can appropriately detect anomalies even when the noise in the differential image is high.

[0035] [First determination method]

[0036] The determination of whether an object is normal or abnormal using the first determination method is explained. Figure 3 This is a flowchart illustrating a general anomaly sensing method using the first determination method. The processing associated with this anomaly sensing method is performed by the anomaly sensing device 56 through a series of procedures.

[0037] In step S11, the anomaly sensing device 56 acquires an image obtained using the imaging device 52 as the image to be judged. In step S12, the anomaly sensing device 56 inputs the acquired image to the autoencoder configured as described above to obtain a reference image as the output image. In step S13, the anomaly sensing device 56 creates a difference image between the image to be judged acquired in step S11 and the reference image acquired in step S12. In step S14, the anomaly sensing device 56 calculates the multi-segment squared error associated with the difference image. Here, the multi-segment squared error is the squared error of the difference image calculated by dividing the difference image into multiple regions and calculating it for each segmented region. The squared error is calculated as the sum of the squared values ​​of the differences between the brightness values ​​of each pixel in the image to be judged and the reference image. In step S15, the anomaly sensing device 56 determines the maximum value of the squared error of each segmented region calculated in step S14 as an evaluation value. In step S16, the anomaly sensing device 56 determines whether the image being judged is normal or abnormal based on the maximum value determined in step S15, which serves as the evaluation value. For example, if the determined maximum value is greater than a predetermined threshold, the anomaly sensing device 56 determines that there is an anomaly, and if the maximum value is below the predetermined threshold, the anomaly sensing device 56 determines that it is normal.

[0038] 〈Judgment Example〉

[0039] A reference was made for 30,198 normal images and 6 abnormal images. Figure 3 The above-described steps S11 to S15 describe the processing. In step S12, an autoencoder is used, constructed through untaught learning, which employs a deep neural network using 36,222 normal images as learning data. Furthermore, in step S14, as a multi-segment squared error, a squared error is calculated for each region formed by dividing the image into six segments (vertical bisection and horizontal trisection). In step S15, the maximum value of this squared error is determined as the evaluation value.

[0040] The processing results are shown below. Figure 4 .exist Figure 4 In the figure, the vertical axis represents the maximum value of the six-segment squared error, which serves as the evaluation value. The circular markers on the left represent the evaluation values ​​of images with normal conditions, while the square markers on the right represent the evaluation values ​​of images with anomalies. The anomalies marked with square markers (a) to (d) correspond to the values ​​of images (a) to (d) shown below the figure.

[0041] As a comparative example, for this image, the squared error of the entire difference image is calculated as the evaluation value without segmenting the region, replacing the processing in steps S14 and S15. The result is shown below. Figure 5 .

[0042] like Figure 5 As shown, even after calculating the squared error associated with the entire difference image, it is impossible to distinguish between normal and abnormal situations based on the value of this squared error. This has been confirmed as follows: Figure 4 As shown, by using the maximum value of the six-segment squared error, for example, it is possible to set such... Figure 4 The thresholds shown by the dashed lines in the chart can distinguish between normal and abnormal situations.

[0043] [Second determination method]

[0044] The determination of whether an object is normal or abnormal when using the second determination method is explained. Figure 6 This is a flowchart outlining an anomaly detection method using the second determination method. Compared to the first determination method, the second determination method differs in the following ways: it processes the generated difference image using a smoothing filter, and calculates the multi-segment squared error for the filtered image.

[0045] Steps S21 to S23 of the second determination method are the same as steps S11 to S13 of the first determination method. That is, in step S21, the anomaly sensing device 56 acquires the image to be determined from the imaging device 52; in step S22, the anomaly sensing device 56 inputs the acquired image to be determined into the autoencoder to obtain a reference image; and in step S13, the anomaly sensing device 56 creates a difference image between the input image and the reference image.

[0046] In step S24, the anomaly sensing device 56 performs smoothing filtering on the difference image. The smoothing filter used can be, for example, an averaging filter, a median filter, a Gaussian filter, or other noise-removing filters.

[0047] Steps S25 to S27 of the second determination method are the same as steps S14 to S16 of the first determination method. That is, in step S25, the anomaly sensing device 56 calculates the multi-segment squared error associated with the differential image after smoothing filtering; in step S26, the anomaly sensing device 56 determines the maximum value of the multi-segment squared error; and in step S27, the anomaly sensing device 56 determines whether the image being determined is normal or abnormal based on the determined maximum value.

[0048] 〈Judgment Example〉

[0049] With Figure 4 and Figure 5 The results were processed similarly, with reference to 30,198 normal images and 6 abnormal images. Figure 6The above-described steps S21 to S26 are described. Here, for the smoothing filter in step S24, an averaging filter is used. The other parts are the same as those shown as a determination example related to the first determination method described above.

[0050] The processing results are shown below. Figure 7 .and Figure 4 Similarly, in Figure 7 In the figure, the vertical axis represents the maximum value of the six-segment squared error, which serves as the evaluation value. The circular markers on the left represent the evaluation values ​​of images with normal conditions, while the square markers on the right represent the evaluation values ​​of images with anomalies. The anomalies marked with square markers (a) to (d) correspond to the values ​​of images (a) to (d) shown below the figure.

[0051] like Figure 7 As shown, it was confirmed that, according to the second determination method including smoothing filtering, the maximum value of the obtained six-segment squared error is significantly different under normal and abnormal conditions. The second determination method can make a more accurate determination of normal or abnormal conditions than the first determination method.

[0052] As described above, the stacking machine system 1 of this embodiment, equipped with the anomaly sensing system 50, can detect anomalies with high precision before stacking the cans 90 onto the pallet. As a result, the generation of defective products can be prevented. In particular, the stacking of cans 90 by the stacking machine 10 is generally mechanized, and there are many situations where human intervention is not possible. The stacking machine system 1 of this embodiment has the advantage of being able to detect anomalies with high precision without human intervention. The stacking of cans 90 by the stacking machine 10 is the final stage of the manufacturing process and is the process immediately before product shipment. Therefore, anomaly sensing at this stage is particularly important in order to prevent the generation of defective products, and the anomaly sensing system 50 of this embodiment is particularly effective.

[0053] In the stacking machine system 1 of this embodiment, anomalies are sensed based on images captured by the imaging device 52. Alternatively, various sensors such as ultrasonic sensors, light sensors, and pressure sensors could be used to detect tipped or leaking containers. However, in such cases, many sensors would be required to confirm the condition of all containers 90 arranged in the width direction. In contrast, as in this embodiment, image-based analysis allows for the acquisition of a wide range of information using only a few cameras, resulting in high efficiency.

[0054] Furthermore, in detection using the sensors described above, for example, when switching to the manufacture of different types of cans 90, the sensors need to be adjusted whenever the conditions of the can 90 change. In particular, such adjustments are required for all the sensors that need to be installed in large numbers as described above. In contrast, according to this embodiment, it is easier to generate an autoencoder for detecting anomalies in different types of cans 90, and even if the can 90 to be manufactured is changed, it can be handled simply by changing the autoencoder fed into the anomaly sensing device 56.

[0055] It should be noted that the same analysis as shown in the above-mentioned judgment example was performed on the cans 90 with different diameters, confirming that the anomaly sensing system 50 of this embodiment also functions for cans 90 with different diameters.

[0056] Furthermore, the anomaly sensing device 56 of this embodiment utilizes an autoencoder employing machine learning, thus making it easier to achieve high-precision detection related to a wide variety of states, which is difficult to achieve with conventional rule-based image analysis. In particular, in this embodiment, the anomaly sensing device 56 is configured to: segment the image into multiple regions, calculate the squared error, use the maximum value of the squared error as the evaluation value for judgment, or perform processing using a smoothing filter midway, thereby making appropriate judgments even for images that include the shoulder of the can 90 or images acquired while in motion, resulting in a lot of noise in the differential image. Accurate judgment representation can be made even for images including the shoulder of the can 90: the anomaly sensing device 56 of this embodiment has a very high degree of freedom related to the object being judged. In addition, accurate judgment representation can be made even for images acquired while in motion: the anomaly sensing device 56 of this embodiment has few constraints related to image acquisition, including physical aspects such as the performance of the imaging device, the location where the imaging device is set, and the shooting conditions, resulting in a very high degree of freedom related to system design.

[0057] [other]

[0058] To prevent abnormal leakage detection, in addition to the aforementioned abnormality sensing system 50, ultrasonic sensors, optical sensors, pressure sensors, etc., for detecting leaking or tipped containers may also be provided, thus constructing the abnormality sensing system in a multi-layered manner. For example, consider installing a tipping sensor at the outlet of the supply section 11, since tipped containers are prone to occur there, or installing a leakage sensor at the outlet of the tidying section 12.

[0059] The above describes the preferred embodiments of the invention, but the invention is not limited to the above embodiments, and various modifications can be made within the scope of the invention.

[0060] In the above embodiments, the following example is shown: to obtain a difference image that emphasizes the difference between the image of the regularly arranged normal object to be judged, i.e., can 90, and the image of the object to be judged acquired by the imaging device 52, a deep learning-based autoencoder is used. However, it is not limited to this; any method can be used as long as it can obtain a difference image that emphasizes the difference between the image of the normal object to be judged and the image of the object to be judged.

[0061] Furthermore, as an example of the error used for evaluation, the sum of the squared differences of the pixel values ​​of each segmented region is given, but it is not limited to this. Any value representing the error in the difference image of each segmented region, i.e., the multi-segmentation error, is acceptable, and other values ​​may also be used.

[0062] In the above embodiments, an example of applying the anomaly sensing system 50 to a stacking machine 10 that places manufactured cans onto pallets is shown, but the application is not limited thereto. For example, the anomaly sensing system 50 can also be applied to processes related to other containers, such as resin containers and other types of containers, and is not limited to processes related to cans. These containers may be empty or filled with contents. Furthermore, the anomaly sensing system 50 can also be applied to other processes where containers are arranged regularly, and is not limited to applications to stacking machines.

[0063] The entire contents of the documents described in this specification and the contents of the Japanese application specification, which forms the basis of the Paris Convention priority claim of this application, are incorporated herein by reference.

Claims

1. An anomaly sensing method, comprising: Acquire the image of the object to be judged, which should be arranged regularly in a cylindrical container standing upright; A difference image is created that emphasizes the difference between an image of a normally arranged object to be judged and the image to be judged. Calculate the multi-segmentation error of the difference image and determine the maximum value of the multi-segmentation error; as well as Based on the maximum value, the normality or abnormality of the object being judged is determined.

2. The anomaly sensing method according to claim 1, wherein, The anomaly detection method further includes: performing a smoothing filter on the difference image before calculating the multi-segmentation error. For the differential image that has undergone the smoothing filtering process, the multi-segmentation error is calculated.

3. The anomaly sensing method according to claim 1 or 2, wherein, Creating the difference image includes: The image to be judged is input to an autoencoder to obtain a reference image as the output of the autoencoder, wherein the autoencoder is configured to output an image of a normal object to be judged in response to an input image of a normal object by learning from images of normally arranged normal objects; and The difference image is created by calculating the difference between the image being judged and the reference image.

4. The anomaly sensing method according to claim 3, wherein, The items to be determined are multiple containers to be stacked on a pallet. The autoencoder is generated by learning from normal images of multiple containers that are to be neatly arranged on a tray.

5. The anomaly sensing method according to claim 4, wherein, The image to be determined includes an image obtained by photographing the object being determined while it is moving.

6. The anomaly sensing method according to claim 1 or 2, wherein, The determination includes: comparing the maximum value with a predetermined threshold, and determining that there is an anomaly if the maximum value is greater than the predetermined threshold.

7. A computer program product comprising an anomaly sensing program that causes a computer to execute the anomaly sensing method according to any one of claims 1 to 6.

8. An anomaly sensing device, wherein, The system includes a computer that performs the anomaly sensing method according to any one of claims 1 to 6.

9. An anomaly sensing system for a stacking machine, comprising: The camera device is configured to photograph multiple containers neatly arranged in front of a tray; and An anomaly sensing device, comprising a computer for performing the anomaly sensing method according to claim 4 or 5, wherein the anomaly sensing device acquires an image obtained by the imaging device as a judgment image, and determines whether a plurality of containers to be stacked on the tray are normal or abnormal.

10. The anomaly sensing system for a stacking machine according to claim 9, wherein, It also has: A tilt sensor is configured to detect a tilted container; and The missing sensor is configured to detect the missing parts of the container.

Images