Detection device, detection method, and detection program

Abstract

To provide a detection device, a detection method, and a detection program that can reduce the possibility of wrong detection of the position of an abnormal part occurring in a belt that is driven in an ironmaking raw material conveyance facility.SOLUTION: A detection device comprises: an image division unit that divides time-series images created by one or more cameras photographing a belt that is driven in an ironmaking raw material conveyance facility at a fixed field of view so as to set the dimension in a width direction of the belt to be 500 millimeters or less, thereby creating a plurality of division images; a representative value determination unit that determines a representative value of a first feature quantity of a normal image; a feature quantity extraction unit that extracts a second feature quantity from a plurality of division images created from second time-series images; a distance estimation unit that estimates, for each division image, the distance between the second feature quantity and the representative value; and an abnormality detection unit that detects the position of an abnormal part occurring in the belt based on the specified position in the division image.SELECTED DRAWING: Figure 1

Description

[[Technical Field]]

[0001] The present invention relates to a detection device, a detection method, and a detection program. [[Background Art]]

[0002] Conveying equipment for iron-making raw materials (belt conveyors) is required to achieve stable operation and reduce inspection load. Further, the belt may break due to the occurrence of abnormal portions such as scratches on the belt surface of the iron-making raw material conveying equipment. If the belt breaks, or if a significant scratch immediately before breaking is found on the belt, the sintering machine in the subsequent process may stop. In order to avoid such a shutdown of the sintering machine, it is necessary to detect abnormal portions occurring in the conveying equipment at an early stage.

[0003] Therefore, for the purpose of early detection of abnormal portions occurring in conveying equipment, Patent Document 1 discloses a system that determines the presence or absence of an abnormal portion based on an image captured of the conveying equipment. Further, Patent Document 2 discloses an abnormality monitoring device that determines the deterioration level of an end portion of a belt of conveying equipment based on an image. [[Prior Art Documents]] [[Patent Documents]]

[0004] [[Patent Document 1]] Japanese Unexamined Patent Publication No. 2020-128286 [[Patent Document 2]] Japanese Unexamined Patent Publication No. 2021-017296 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0005] However, there is a problem that the possibility of erroneously detecting the position of an abnormal portion occurring on a belt of conveying equipment cannot be reduced.

[0006] In view of the above circumstances, the present invention aims to provide a detection device, a detection method, and a detection program that can reduce the possibility of falsely detecting the location of an abnormal part in a belt driven by a conveying equipment for steelmaking raw materials. [Means for solving the problem]

[0007] One aspect of the present invention is an image division unit that generates a plurality of divided images by dividing a time-series image generated by one or more cameras that image a belt driven in a conveying equipment for steelmaking raw materials with a fixed field of view, such that the width dimension of the belt is 500 millimeters or less; a representative value determination unit that determines a representative value of a first feature of a normal image based on one or more normal images in which the belt without abnormalities is imaged from the plurality of divided images generated from the first time-series image; and a plurality of divided images generated from the second time-series image. The detection device comprises: a feature extraction unit that extracts a second feature from a divided image; a distance estimation unit that estimates the distance between the second feature and the representative value for each divided image generated from the second time-series image; an anomaly detection unit that identifies the position of the divided image in the fixed field of view where the distance is greater than or equal to a threshold for each divided image generated from the second time-series image, and detects the position of the anomaly that has occurred in the belt based on the identified position of the divided image; and a notification unit that, when the position of the anomaly is detected, notifies the divided image in which the distance is greater than or equal to the threshold and the position of the anomaly that has occurred in the belt.

[0008] One aspect of the present invention is a detection method performed by a detection device, comprising the steps of: generating a plurality of divided images by dividing a time-series image generated by one or more cameras that image a belt driven in a conveying facility for steelmaking raw materials with a fixed field of view, such that the width dimension of the belt is 500 millimeters or less; determining a representative value of a first feature of a normal image based on one or more normal images from which the belt without abnormalities is imaged among the plurality of divided images generated from the first time-series image; and determining a representative value of a first feature of a normal image based on the plurality of divided images generated from the second time-series image. The detection method includes the steps of: extracting a second feature from a plurality of the divided images; estimating the distance between the second feature and the representative value for each divided image generated from the second time-series image; identifying the position of the divided image in the fixed field of view where the distance is greater than or equal to a threshold for each divided image generated from the second time-series image, and detecting the position of the abnormal part that occurred in the belt based on the identified position of the divided image; and, if the position of the abnormal part is detected, notifying the divided image where the distance is greater than or equal to the threshold and the position of the abnormal part that occurred in the belt.

[0009] One aspect of the present invention involves a computer that generates a plurality of divided images by dividing a time-series image generated by one or more cameras that image a belt driven in a steelmaking raw material conveying facility with a fixed field of view, such that the width dimension of the belt is 500 millimeters or less; a computer that determines a representative value of a first feature of a normal image based on one or more normal images from which the belt without abnormalities is imaged, among the plurality of divided images generated from the first time-series image; and a computer that determines a plurality of divided images generated from the second time-series image. This detection program is for causing the following to be executed: a procedure for extracting a second feature from the divided images; a procedure for estimating the distance between the second feature and the representative value for each divided image generated from the second time-series images; a procedure for identifying the position of the divided image in the fixed field of view where the distance is greater than or equal to a threshold for each divided image generated from the second time-series images, and detecting the position of the abnormal part that occurred in the belt based on the identified position of the divided image; and a procedure for notifying the divided image in which the distance is greater than or equal to the threshold and the position of the abnormal part that occurred in the belt when the position of the abnormal part is detected. [Effects of the Invention]

[0010] This invention makes it possible to reduce the possibility of falsely detecting the location of an abnormal part in a belt driven by a conveying equipment for steelmaking raw materials. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the configuration of the detection system in the embodiment. [Figure 2] This figure shows an example of the configuration of the conveying equipment in the embodiment. [Figure 3] This figure shows an example of an image (frame) in the embodiment that includes an image of a belt captured with a fixed field of view. [Figure 4] This flowchart shows an example of the operation of the learning device in the embodiment. [Figure 5]This flowchart shows an example of the operation of the process for determining representative values ​​in the embodiment. [Figure 6] This flowchart shows an example of the operation of the process for detecting the location of an abnormal part in the embodiment. [Figure 7] This figure shows a first example of the false detection rate in the embodiment. [Figure 8] This figure shows a second example of the false detection rate in the embodiment. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described in detail with reference to the drawings. Figure 1 shows an example of the configuration of the detection system 1 in an embodiment. The detection system 1 is a system that detects the location (presence or absence) of abnormal parts on a belt driven in a conveying equipment (belt conveyor) for transporting ironmaking raw materials. Ironmaking raw materials are, for example, iron ore, sintered ore, coal, coke, limestone, mixtures of two or more of these, compounded products of two or more of these, or granulated products of the mixture or compounded products. Abnormal parts are, for example, scratches (damaged parts). The detection system 1 detects the location (presence or absence) of abnormal parts based on the captured image of the belt. The detection system 1 may also estimate the name of the type of abnormal part detected.

[0013] The detection system 1 comprises one or more cameras 2, a communication line 3, a learning device 4, a detection device 5, and a notification device 6. The learning device 4 comprises a learning communication unit 40, a learning storage device 41, a learning memory unit 42, and a learning processing unit 43. The detection device 5 (estimation device) comprises a detection communication unit 50, a detection storage device 51, a detection memory unit 52, and a detection processing unit 53. The detection processing unit 53 comprises an image splitting unit 530, a representative value determination unit 531, a feature extraction unit 532, a distance estimation unit 533, an anomaly detection unit 534, a type estimation unit 535, and a notification unit 536. Note that the learning device 4 and the detection device 5 may be integrated. That is, the detection device 5 may include the learning device 4.

[0014] The communication line 3 is a line such as a LAN (Local Area Network) or a WAN (Wide Area Network).

[0015] Some or all of the learning device 4, the detection device 5, and the notification device 6 are implemented as software when a processor such as a CPU (Central Processing Unit) executes a program expanded from a storage device, which is a non-volatile recording medium (non-transitory recording medium), into a storage unit. The program may be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include portable media such as flexible disks, magneto-optical disks, ROM (Read Only Memory), CD-ROM (Compact Disc Read Only Memory), and non-transitory recording media such as storage devices like hard disks built into computer systems.

[0016] Some or all of the learning device 4, the detection device 5, and the notification device 6 may be implemented using hardware including an electronic circuit (electronic circuit or circuitry) using, for example, LSI (Large Scale Integration circuit), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array).

[0017] Figure 2 is a diagram (side view) showing a configuration example of the conveying facility 10 according to the embodiment. The conveying facility 10 includes a belt conveyor. The conveying facility 10 includes a belt 11, a tail pulley 12, and a tension pulley 13. The width dimension (length) of the belt 11 is not limited to a specific dimension, and is, for example, approximately 600 millimeters to 2500 millimeters. The conveying facility 10 conveys iron-making raw materials in the longitudinal direction of the belt 11 by driving the belt 11 carrying the iron-making raw materials.

[0018] For example, iron ore, which is one of the raw materials for iron making, is discharged from a storage yard (not shown) by a reclaimer (not shown). The iron ore is transferred via a plurality of conveying facilities 10 and conveyed to a sintering machine (not shown). Abnormal portions such as scratches may occur on the belt 11 of the conveying facility 10. For this reason, the belt 11 needs to be repaired before it breaks due to the abnormal portions.

[0019] The camera 2 is installed at a position where the surface of the belt 11 can be imaged, for example, in the vicinity of the tail pulley 12 or the tension pulley 13. In the following description, the camera 2 is installed at a position that overlooks the surface of the belt 11 in the vicinity of the tail pulley 12. The camera 2 continuously images the belt 11 driven in the conveying facility 10 within a fixed field of view 20. The fixed field of view 20 is predetermined so that the entire width of the belt 11 in the width direction of the belt 11 can be accommodated. The camera 2 generates moving image data (time-series images) obtained by imaging the belt 11 within the fixed field of view 20.

[0020] Note that within the fixed field of view 20, the iron-making raw material 14 does not need to be placed on the belt 11. Further, the iron-making raw material 14 remaining on the surface of the belt 11 may adhere to the surface of the belt 11 so as to form a streaky pattern in the longitudinal direction of the belt 11.

[0021] FIG. 3 is a diagram illustrating an example of an image 100 (frame) including a partial image of the belt 11 captured in the fixed field of view 20 according to the embodiment. The image 100 is a still image at an arbitrary time among time-series images generated by the camera 2. The image 100 includes an image of the upper surface of the belt 11 in the fixed field of view 20. Further, the image 100 includes an image of a part of the conveying facility 10 in the fixed field of view 20 as a background image for the image of the upper surface of the belt 11 in the fixed field of view 20.

[0022] During the machine learning training phase, images of abnormal areas captured by camera 2 are used as training images. In this case, camera 2 transmits time-series images to the learning device 4. In the learning device 4, the learning processing unit 43 divides the training image into a mesh-like structure, generating multiple divided images of the training image. Alternatively, instead of the images generated by camera 2, pre-prepared images of abnormal areas may be used as training images.

[0023] In the estimation stage (detection stage) following the learning stage, the image of the belt 11 is used as a detection image by the detection device 5. In this case, the camera 2 transmits a time-series of images to the detection device 5. The detection device 5 generates multiple divided images of the detection image by dividing the detection image in a mesh-like manner using the image division unit 530.

[0024] Furthermore, it is not limited to generating multiple divided images from a single image captured at a predetermined time by one camera 2; for example, each of multiple images captured simultaneously by multiple cameras 2 may be used as a divided image. In this case, the image dividing unit 530 does not have to divide each of the multiple images captured simultaneously by multiple cameras 2. Alternatively, the image dividing unit 530 may divide at least some of the multiple images captured simultaneously by multiple cameras 2.

[0025] Returning to Figure 1, let's explain the learning device 4. During the machine learning training phase, the learning communication unit 40 acquires images of the belt 11 in the fixed field of view 20 from the camera 2 as training images. The learning communication unit 40 then transmits the trained model generated by the learning processing unit 43 to the detection communication unit 50.

[0026] The learning memory device 41 stores the learning program executed by the learning processing unit 43. The learning program stored in the learning memory device 41 is loaded into the learning memory unit 42 when the learning device 4 is started.

[0027] The learning memory device 41 stores training data used in machine learning (supervised learning), such as deep learning. The training data includes multiple training images (training data (explanatory variables)) and multiple ground truth data (target variable). In the training data, the training images and the ground truth data are associated. The training images are images of abnormal parts that have occurred on the belt. The ground truth data are, for example, the names of the types of abnormal parts that have been previously captured in the training images (e.g., dents or cracks). The training data is created in advance by, for example, a creator (not shown).

[0028] The learning processing unit 43 acquires training data, which includes multiple training images and multiple correct answer data, from the learning storage device 41. The learning processing unit 43 generates a trained model by machine learning (supervised learning), using images of abnormal parts that have occurred on the belt 11 driven in the steelmaking raw material conveying equipment 10 as explanatory variables and the names of the types of abnormal parts as the objective variable.

[0029] Next, we will describe the detection device 5. During the learning or estimation phase (detection period), the detection communication unit 50 acquires a pre-trained model generated by the learning processing unit 43 from the learning communication unit 40. During the estimation phase, the detection communication unit 50 acquires time-series images of the belt 11 in the fixed field of view 20 (imaging field of view) from the camera 2 as detection images.

[0030] The detection memory device 51 stores the detection program (estimation program) executed by the detection processing unit 53 and the trained model. The detection program and trained model stored in the detection memory device 51 are loaded into the detection storage unit 52 when the detection device 5 is started.

[0031] Hereafter, the time series before the estimation stage (learning stage, etc.) will be referred to as the "first time series." The time series during the estimation stage (detection period) will be referred to as the "second time series."

[0032] During the learning phase, the image splitting unit 530 acquires training images (first time-series images) from the camera 2. During the estimation phase (detection period), the image splitting unit 530 acquires detection images (second time-series images) from the camera 2.

[0033] During the learning and estimation phases, the image segmentation unit 530 segments the time-series images generated by the camera 2, which captures the belt 11 in a fixed field of view 20, so that the width dimension of the belt 11 is 500 millimeters or less. If, for example, five segmented images are generated in the width direction of the belt 11, the number of edges (widthwise ends) of the belt 11 captured in each segmented image is 0 or 1. In other words, the average number of edges of the belt 11 captured in the segmented images is 0.4 (=(1+0+0+0+1) / 5), which is significantly less than the number of edges of the belt 11 captured in the image 100 of the fixed field of view 20 (2 edges).

[0034] Furthermore, the image division unit 530 generates divided images such that the length (dimension) of each divided image is predetermined in the longitudinal direction of the belt 11. For example, if the length of the detection range in the longitudinal direction of the belt 11 is predetermined to be 100 millimeters based on the length of the abnormal part, the length of the divided image in the longitudinal direction may also be predetermined to be 100 millimeters.

[0035] Before the estimation stage, the representative value determination unit 531 acquires a normal image (an image of the undamaged part) of the belt 11 in which no abnormalities have occurred. The representative value determination unit 531 or the feature extraction unit 532 extracts features from one or more (e.g., 950) normal images in which the belt 11 without abnormalities has occurred, from among the multiple segmented images generated from the images before the estimation stage (first time series). A known tool may be used for feature extraction. For example, a pre-trained model that has been trained (supervised learning) using a set of images in which dogs are the subject, a set of images in which cats are the subject, and combinations of the type of subject (dog or cat) in each image is publicly available as a known tool for extracting features from images. This pre-trained model outputs the name of the subject (dog or cat) in an image that is separately input as the target for estimation. For this pre-trained model, the optimal hierarchical conditions in the neural network may be predetermined based on the results of estimation processing performed in advance using sample images. Image features input to one or more hidden layers in a neural network may be extracted and output from those hidden layers in a hierarchical order based on optimal hierarchical conditions. Note that the trained model in this embodiment is not limited to such a model.

[0036] Hereinafter, the feature quantities extracted from normal images without abnormalities will be referred to as "first features." The first features represent, for example, the image features of a region on the surface of the belt 11 where no abnormalities (scratches) have occurred. The representative value determination unit 531 determines a representative value (for example, mean, centroid, or mode) of the first features based on one or more normal images. The representative value determination unit 531 records the determined representative value in the detection storage device 51 or the detection storage unit 52.

[0037] During the estimation phase, the feature extraction unit 532 extracts the features of multiple segmented images generated from the images captured during the estimation phase (hereinafter referred to as "second features") from the multiple segmented images generated from the images of the estimation phase (second time series).

[0038] During the estimation phase, the distance estimation unit 533 estimates the distance between the second feature and the representative value for each segmented image generated from the second time-series image. The distance is, for example, the Mahalanobis distance, the Euclidean distance, or the Manhattan distance.

[0039] During the estimation phase, the anomaly detection unit 534 identifies the location (coordinates) of each segmented image generated from the second time-series images in the fixed field of view 20 where the distance between the second feature and the representative value is greater than or equal to a threshold. The threshold is predetermined based on experiments, etc. Based on the identified location of the segmented image, the anomaly detection unit 534 detects the location of the anomaly that occurred in the belt 11.

[0040] The method for detecting the location of an anomaly based on the position of the segmented image identified in the fixed field of view 20 is not limited to a specific method. For example, the location of an anomaly may be detected based on the relative position of the belt 11 from a predetermined reference position. The reference position is, for example, the position of a joint in the belt, or the position of a colored area (not shown) that has been marked on the belt 11 in advance. Alternatively, the location where the anomaly occurred on the belt 11 (relative position from the reference position) may be detected based on the time elapsed from the time the reference position was captured by the camera 2 (reference time) to the time the time series of images in which an anomaly was determined to be present was captured, and the driving speed (conveying speed) of the belt 11.

[0041] In the estimation stage, the type estimation unit 535 estimates the type (name) of the anomaly at the detected location using a trained model. For example, the type estimation unit 535 inputs a segmented image (a segmented image in which an anomaly was captured) in which the distance between the second feature and the representative value is greater than or equal to a threshold into the trained model. The type estimation unit 535 obtains the name of the type of anomaly at the detected location from the trained model. In this way, the type estimation unit 535 estimates the name of the type of anomaly at the detected location.

[0042] When the notification unit 536 detects the location (presence or absence) of an abnormality, it notifies the notification device 6 of the segmented image whose distance is greater than or equal to a threshold, and the location of the abnormality that occurred on the belt 11. When the type estimation unit 535 estimates the type (name) of the abnormality, it notifies the notification device 6 of the name of the type of abnormality.

[0043] Next, we will explain an example of the operation of detection system 1. Figure 4 is a flowchart showing an example of the operation of the learning device 4 in the embodiment. During the learning phase, the learning processing unit 43 acquires an image of the abnormal part of the belt 11 as an explanatory variable (training image) (step S101). The learning processing unit 43 acquires the name of the type of abnormal part as the target variable (ground truth data) (step S102). The learning processing unit 43 generates a trained model using machine learning (e.g., supervised learning) with the explanatory variable and the target variable (step S103). The learning processing unit 43 outputs the trained model to the detection device 5 via the learning communication unit 40 (step S104).

[0044] Figure 5 is a flowchart illustrating an example of the operation of the process for determining representative values ​​in the embodiment. Before the estimation stage, the representative value determination unit 531 acquires a normal image in which the belt 11 without abnormalities is captured (step S201). The representative value determination unit 531 or the feature extraction unit 532 extracts the features (first features) of a normal image from one or more normal images in which the belt 11 without abnormalities is captured, from among a plurality of segmented images generated from the images before the estimation stage (first time series) (step S202).

[0045] The representative value determination unit 531 determines representative values ​​for the feature quantities of normal images based on one or more normal images (step S203). The representative value determination unit 531 records the determined representative values ​​in the detection storage device 51 or the detection storage unit 52 (step S204).

[0046] Figure 6 is a flowchart illustrating an example of the operation of the process for detecting the location of an anomaly in the embodiment. During the learning stage or estimation stage (detection period), the detection communication unit 50 acquires a learned model from the learning device 4 (step S301). The image splitting unit 530 acquires a time series (second time series) of images from the camera 2 during the estimation stage (step S302). The image splitting unit 530 splits the time series images generated by the camera 2, which images the belt 11 in a fixed field of view 20, so that the width dimension of the belt 11 is 500 millimeters or less (step S303).

[0047] The feature extraction unit 532 extracts second features from multiple segmented images generated from the second time-series image (step S304). The distance estimation unit 533 estimates the distance between the second features and the representative value for each segmented image generated from the second time-series image (step S305). The anomaly detection unit 534 detects the location of the anomaly in the belt 11 based on the location of the segmented image identified in the fixed field of view 20 (step S306).

[0048] The notification unit 536 determines whether or not the location of the abnormal part has been detected (whether or not the distance is greater than or equal to a threshold) for each segmented image generated from the second time-series image (step S307). If it is determined that the location of the abnormal part has not been detected (step S307: NO), the notification unit 536 proceeds to step S311.

[0049] If it is determined that the location of an anomaly has been detected (step S307: YES), the notification unit 536 notifies the notification device 6 of the segmented image whose distance is greater than or equal to the threshold, and the location of the anomaly that occurred on the belt 11 (step S308). The type estimation unit 535 inputs the segmented image whose distance is greater than or equal to the threshold into the trained model. The type estimation unit 535 obtains the name of the type of anomaly at the detected location from the trained model (step S309). The type estimation unit 535 notifies the notification device 6 of the name of the type of anomaly (output of the trained model) (step S310).

[0050] The detection processing unit 53 determines whether or not to terminate the detection process (step S311). If it is determined to continue the detection process (for example, if the belt 11 is in motion) (step S311: NO), the detection processing unit 53 returns to step S302. If it is determined to terminate the detection process (for example, if the belt 11 is stopped) (step S311: NO), the detection processing unit 53 terminates the detection process.

[0051] As described above, the image division unit 530 generates multiple divided images by dividing a time-series image (frame) generated by a camera 2 that captures the belt 11 driven in the steelmaking raw material conveying equipment 10 with a fixed field of view 20, so that the width dimension of the belt 11 is 500 millimeters or less.

[0052] The representative value determination unit 531 determines a representative value of the first feature of a normal image based on one or more normal images from among the multiple segmented images generated from the first time series images in which the belt 11 without abnormalities is captured. The feature extraction unit 532 extracts the second feature from the multiple segmented images generated from the second time series images.

[0053] The distance estimation unit 533 estimates the distance between the second feature and the representative value for each segmented image generated from the second time-series images. The anomaly detection unit 534 identifies the position of the segmented image in the fixed field of view where the distance is greater than or equal to a threshold for each segmented image generated from the second time-series images. Based on the identified position of the segmented image, the anomaly detection unit 534 detects the position of the anomaly that occurred in the belt 11. If the position of the anomaly is detected, the notification unit 536 notifies the segmented image where the distance is greater than or equal to a threshold, and the position of the anomaly that occurred in the belt 11. The type estimation unit 535 may estimate the type (name) of the anomaly at the detected position.

[0054] This makes it possible to reduce the possibility of falsely detecting the location of an abnormality in the belt driven by the steelmaking raw material conveying equipment.

[0055] Before image 100 is divided so that the width dimension is 500 millimeters or less, the total number of edges and patterns contained in image 100 is 2 or more. In contrast, after the captured image 100 is divided, the number of edges contained in the divided images of image 100 becomes less than 2. Also, after image 100 is divided so that the width dimension is 500 millimeters or less, the number of streaky patterns contained in the divided images of image 100 becomes less than the number of streaky patterns contained in image 100 of the fixed field of view 20. The edges of the belt 11 and the patterns on the surface of the belt 11 are noise for detecting abnormalities. The less such noise there is, the lower the possibility of false detection of the location of the abnormality.

[0056] The shorter the dimensions of the segmented images in the width direction of the belt 11, the higher the accuracy of detecting the location of abnormalities. However, if the dimensions of the segmented images in the width direction of the belt 11 are, for example, 10 millimeters or less, the effect of reducing false detections saturates, and the rate of false detections is not reduced significantly. Therefore, a lower limit is set for the dimensions of the segmented images in the width direction of the belt 11 based on the relationship between the accuracy of detecting the location of abnormalities and the effect of reducing false detections. Furthermore, the dimensions of the segmented images in the width direction of the belt 11 (optimal value) are set so as not to be lower than the lower limit.

[0057] Next, I will explain some examples of its effects. Figure 7 shows a first example of the false detection rate in the embodiment. In the following, the belt width (dimensions) is, for example, 450 mm, 900 mm, and 1200 mm. The number of detection images in which no abnormality is captured is, for example, 950. The number of detection images in which an abnormality is captured is, for example, 50. The "false detection rate" shown in Figure 7 represents the rate (percentage) of false detections regarding the presence or absence of abnormalities (scratches). This false detection refers to cases where a detection image was judged to have an abnormality (scratches) but actually did not have any scratches. There were no cases where a detection image was judged to have no abnormality (no scratches) but actually did have scratches.

[0058] (1) Comparison between the first comparative example and the first example In the first comparison example, the width of the belt is 450 millimeters. The width of the fixed field of view image is 500 millimeters. Furthermore, the fixed field of view image is not segmented; that is, no segmented images are generated. The number of normal images is 950, as an example.

[0059] In the first embodiment, the width of the belt is 900 millimeters. The width of the fixed field of view image is 500 millimeters. Two split images are generated in the width direction of the belt. The number of normal images is 950, as an example.

[0060] When determining the presence or absence of an abnormality in 950 normal images (detection images in which no abnormalities are captured) and 50 abnormal images (detection images in which abnormalities are captured), the detection accuracy was improved in the first embodiment compared to the first comparative example. Even though the width of the fixed field of view in the first comparative example and the width of the divided image in the first embodiment were the same, the possibility of false detection of the location (presence or absence) of the abnormality was reduced by reducing the number of edges included in the divided image to one.

[0061] (2) Comparison of the second comparative example and the first example In the second comparison example, the width of the belt is 900 millimeters. The width of the fixed field of view image is 1000 millimeters. Furthermore, the fixed field of view image is not segmented; that is, no segmented images are generated. The number of normal images is 950, as an example.

[0062] When determining the presence or absence of abnormalities in 950 normal images and 50 abnormal images, the false detection rate in the second comparative example was high, at approximately 80%. In the first embodiment, detection accuracy was improved compared to the second comparative example. Even though the belt width in the second comparative example and the belt width in the first embodiment were the same, the possibility of false detection of the location (presence or absence) of abnormalities was reduced by decreasing the number of edges included in the segmented image to one.

[0063] (3) Comparison of the third comparative example with the first example In the third comparison example, the width of the belt is 1200 millimeters. The width of the divided image is 650 millimeters. The fixed field of view image was divided in the width direction, generating two divided images in the width direction. The number of normal images is 950 in this example.

[0064] When determining the presence or absence of abnormalities in 950 normal images and 50 abnormal images, the false detection rate in the third comparative example was high, at approximately 40%. In the first embodiment, detection accuracy was improved compared to the third comparative example. When the width of the divided images exceeded 500 millimeters, the number of streaky patterns contained in such divided images increased, making false detection of the location of abnormalities more likely.

[0065] (4) Comparison of the first and second embodiments In the second embodiment, the width of the belt is 1200 millimeters. The width of the divided image is 250 millimeters. The fixed field of view image is divided in the width direction, and five divided images are generated in the width direction. The number of normal images is 950 as an example.

[0066] When determining the presence or absence of abnormalities in 950 normal images and 50 abnormal images, the false detection rate in the second embodiment is low, at approximately 10%. In the second embodiment, the detection accuracy was further improved compared to the first comparative example. Since the number of streaky patterns in the segmented images of the second embodiment is less than the number of streaky patterns in the segmented images of the first embodiment, the possibility of false detection of the location (presence or absence) of abnormalities was further reduced in the second embodiment.

[0067] (5) Comparison of the fourth comparative example with the first and second examples In (1) through (4) above, representative values ​​of the features of normal images (first features) were determined using normal images. In other words, training data including training images in which abnormalities were captured (explanatory variables) and ground truth data indicating the presence of abnormalities (dependent variable) were not used.

[0068] In contrast, in the fourth comparative example, supervised learning was performed using training data that included training images (explanatory variables) in which the anomaly was captured, and ground truth data (dependent variable) indicating the presence of the anomaly. The number of training images was 50 as an example. In the fourth comparative example, the detection accuracy decreased compared to the first and second embodiments.

[0069] For example, if 1000 time-series training images in a fixed field of view 20 are prepared, in reality, there are often 5 or fewer training images in which anomalies are captured. In the first and second embodiments, even if the number of training images in which anomalies are captured is small (5 or fewer per 1000 images), since the training images in which anomalies are captured are not used in the machine learning for anomaly detection, anomalies can be detected with high accuracy.

[0070] In contrast, in the fourth comparative example, when the number of training images containing anomalies was small (5 or fewer per 1000 images), sufficient training was not performed, and the accuracy of the trained model could not be improved, making it impossible to detect anomalies.

[0071] Figure 8 shows a second example of the false detection rate (detection accuracy of the type of abnormality) in the embodiment. In the third embodiment and the fifth comparative example, the width (dimension) of the belt is 1200 millimeters as an example. The width of the divided image is 250 millimeters as an example. The image of the fixed field of view was divided in the width direction, and five divided images were generated in the width direction. Normal images do not need to be used. Instead of using normal images, supervised learning was performed using training data that included training images in which abnormalities were captured (explanatory variables) and ground truth data (dependent variables) representing the names of the types of abnormalities (e.g., large indentation, small indentation, large crack, small crack, ear tear, etc.). The number of training images is 50 as an example.

[0072] In the third embodiment, the type estimation unit 535 acquired 55 detection images as an example. Of these 55 detection images, 50 images captured abnormal parts (scratches), and the detection result for the location (presence or absence) of the abnormal parts was correct. In contrast, the remaining 5 detection images did not capture abnormal parts (scratches), and the detection result for the location (presence or absence) of the abnormal parts was incorrect. The type estimation unit 535 estimated the type (name) of the abnormal parts at the detected locations in this way.

[0073] As a result, for the 50 detection images in which the detection result for the location (presence or absence) of the abnormality was correct, the accuracy rate for identifying the type of abnormality was 100%. In other words, for the 50 detection images in which the detection result for the location (presence or absence) of the abnormality was correct, the error rate for identifying the type of abnormality was 0% (=0 images). In contrast, for the 5 detection images in which the detection result for the location (presence or absence) of the abnormality was incorrect to begin with, the accuracy rate for identifying the type of abnormality was 0%. In other words, for the 5 detection images in which the detection result for the location (presence or absence) of the abnormality was incorrect to begin with, the error rate for identifying the type of abnormality was 100% (=5 images). Therefore, the occurrence rate of false detections (the percentage of incorrect identification of the type of abnormality) in the third embodiment was approximately 10%, similar to the second embodiment.

[0074] In the fifth comparative example, the trained model generated by supervised learning in the third embodiment was used. For example, 50 detection images were used, and 950 detection images were used, without any abnormalities. A total of 1000 detection images were input into the trained model.

[0075] As a result, for the 50 detection images in which the anomaly was captured, the trained model outputted the correct name of the anomaly type with a 50% accuracy rate. Conversely, for the 50 detection images in which the anomaly was captured, the trained model outputted the incorrect name of the anomaly type with a 50% error rate (25 images = 50 × 0.5).

[0076] In contrast, for the 950 detection images in which no abnormalities were captured, the trained model outputted the correct name for the type of abnormality, "No Name," with a correct answer rate of "60%." That is, for the 950 detection images in which no abnormalities were captured, the trained model outputted the incorrect name for the type of abnormality with a wrong answer rate of "40%" (380 images = 950 × 0.4). Therefore, the rate of false detection (wrong answer rate for the name of the type of abnormality) in the fifth comparative example is high at approximately 40% (= (25 + 380) / 1000). Thus, the detection accuracy (correct answer rate) in the fifth comparative example was lower compared to the first, second, and third examples.

[0077] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Explanation of Symbols]

[0078] 1...Detection system, 2...Camera, 3...Communication line, 4...Learning device, 5...Detection device, 6...Notification device, 10...Conveying equipment, 11...Belt, 12...Tail pulley, 13...Tension pulley, 14...Steelmaking raw materials, 20...Fixed field of view, 40...Learning communication unit, 41...Learning memory device, 42...Learning memory unit, 43...Learning processing unit, 50...Detection communication unit, 51...Detection memory device, 52...Detection memory unit, 53...Detection processing unit, 100...Image, 530...Image segmentation unit, 531...Representative value determination unit, 532...Feature extraction unit, 533...Distance estimation unit, 534...Anomaly detection unit, 535...Type estimation unit, 536...Notification unit

Claims

1. An image division unit generates multiple divided images by dividing a time-series image generated by one or more cameras that drive a belt to transport steelmaking raw materials in a fixed field of view, such that the width dimension of the belt is 500 millimeters or less, in a conveying equipment for steelmaking raw materials. A representative value determination unit determines a representative value of the first feature quantity of the normal image based on one or more normal images in which the belt without abnormalities is captured, from among the multiple divided images generated from the first time-series image, A feature extraction unit extracts a second feature from a plurality of divided images generated from the second time-series images, A distance estimation unit that estimates the distance between the second feature quantity and the representative value for each of the divided images generated from the second time-series image, An anomaly detection unit identifies the position of the segmented image in the fixed field of view where the distance is greater than or equal to a threshold, for each segmented image generated from the second time-series image, and detects the position of the anomaly in the belt based on the identified position of the segmented image. When the location of the abnormal part is detected, a notification unit notifies the divided image in which the distance is greater than or equal to the threshold, and the location of the abnormal part that occurred on the belt. A detection device equipped with the following features.

2. The detection device according to claim 1, wherein the distance is the Mahalanobis distance, the Euclidean distance, or the Manhattan distance.

3. Type estimation unit for estimating the type of abnormal part at the detected location. The detection device according to claim 1 or claim 2, further comprising the above.

4. A learning processing unit generates a trained model using machine learning, where an image of the abnormal part occurring in the belt driven by the steelmaking raw material conveying equipment is used as the explanatory variable, and the name of the type of abnormal part is used as the dependent variable. The detection device according to claim 3, further comprising the following:

5. A detection method performed by a detection device, The process involves generating multiple divided images by dividing a time-series image generated by one or more cameras that drive a belt to transport steelmaking raw materials in a steelmaking raw material transporting facility, with a fixed field of view, so that the width dimension of the belt is 500 millimeters or less. The steps include determining a representative value of the first feature of a normal image based on one or more normal images from among the multiple segmented images generated from the first time-series image in which the belt without abnormalities is captured, The steps include extracting a second feature quantity from a plurality of segmented images generated from a second time-series image, The steps include: estimating the distance between the second feature and the representative value for each segmented image generated from the second time-series image; For each of the segmented images generated from the second time-series images, the position of the segmented image where the distance is greater than or equal to a threshold is identified in the fixed field of view, and based on the identified position of the segmented image, the position of the abnormal part that occurred in the belt is detected. If the location of the abnormal part is detected, the step of notifying the divided image in which the distance is greater than or equal to the threshold, and the location of the abnormal part that occurred on the belt, A detection method that includes this.

6. A step to generate a trained model by machine learning, using images of the abnormal part that occurred on the belt driven by the steelmaking raw material conveying equipment as explanatory variables and the name of the type of abnormal part as the dependent variable. The detection method according to claim 5, further comprising:

7. On the computer, A procedure for generating multiple divided images by dividing a time-series image generated by one or more cameras that drive a belt to transport steelmaking raw materials in a steelmaking raw material conveying equipment and capture the belt in a fixed field of view, such that the width dimension of the belt is 500 millimeters or less. A procedure for determining a representative value of the first feature of a normal image based on one or more normal images in which the belt without abnormalities is captured, from among a plurality of divided images generated from the first time-series image, A procedure for extracting a second feature quantity from a plurality of segmented images generated from a second time-series image, A procedure for estimating the distance between the second feature and the representative value for each segmented image generated from the second time-series image, A procedure for identifying the position of each segmented image generated from the second time-series image in the fixed field of view where the distance is greater than or equal to a threshold, and detecting the position of the abnormal part in the belt based on the identified position of the segmented image, When the location of the abnormal part is detected, a procedure is provided to notify the divided image in which the distance is greater than or equal to the threshold, and the location of the abnormal part that occurred on the belt. A detection program to execute.

8. A procedure for generating a trained model by machine learning, using images of abnormal parts on the belt that drives and transports steelmaking raw materials in a steelmaking raw material transporting equipment as explanatory variables, and the name of the type of abnormal part as the dependent variable. The detection program according to claim 7 for further execution.

Images