Judgment device, anomaly detection system, judgment method, and program

Abstract

To accurately detect an anomaly without decreasing detection speed.SOLUTION: A determination device includes: an image acquisition section that has an opening / closing mechanism for opening / closing an opening / closing section and acquires an image obtained by imaging a moving body into which an object flows or from which the object flows out, in a case where the opening / closing section is in an open state; an identification section that identifies coordinates of a bounding box surrounding a region of the opening / closing section among the acquired images; and a determination section that determines whether or not there is an anomaly in the opening / closing section present within a range of the identified bounding box.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a determination device, an abnormality detection system, a determination method, and a program.

Background Art

[0002] Conventionally, a technique is known in which a monitoring camera installed above a vehicle door such as a train door is used to image the state before and after opening and closing in the vicinity of the vehicle door, and it is determined whether or not a foreign object is sandwiched according to the difference in the imaged images (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to such a conventional technique, since the images before and after the opening and closing of the vehicle door are used as comparison targets, for example, a person walking on the platform or a passenger inside the vehicle may be reflected in the image. In such a method of comparing the images before and after the opening and closing of the vehicle door, even if a difference is recognized between the images before and after the opening and closing, it is not always the case that a foreign object is sandwiched in the door, and false detection may occur. In addition, although it is conceivable to perform a process to prevent false detection by image processing, there is a concern that the detection speed of foreign objects will be slow. In the technical field of foreign object detection for vehicle doors such as trains, a slow detection speed increases the risk of the train departing with a foreign object sandwiched. That is, according to the conventional technique, there is a problem that the detection speed becomes slow when trying to improve the detection accuracy.

[0005] Therefore, the present invention has been made in view of these circumstances, and aims to provide a technology that can detect anomalies with high accuracy without reducing the detection speed. [Means for solving the problem]

[0006] A determination device according to one aspect of the present invention has an opening / closing mechanism for opening and closing an opening / closing section, and an image acquisition unit that acquires an image of a moving object that is flowing in or out of the opening / closing section when the opening / closing section is in the open state, The image acquisition unit includes a first neural network trained to predict the coordinates of the bounding box surrounding the area of ​​the opening / closing part in the image acquired by the image acquisition unit, A unit for identifying the coordinates of the bounding box in the acquired image, A neural network trained to predict whether or not there is an abnormality in the opening / closing part within the bounding box range, comprising a second neural network different from the first neural network included in the specific part, A determination unit that determines whether or not there is an abnormality in the opening / closing part located within the range of the identified bounding box. The identification unit, based on the image acquired by the image acquisition unit, repeatedly processes the image using the first neural network until it is determined that the opening / closing unit is in a closed state. If the determination unit is determined to be in a closed state, it uses the second neural network to determine whether or not there is an abnormality in the opening / closing unit that is within the range of the identified bounding box. .

[0008] According to a determination device according to one aspect of the present invention, the moving body moves along a predetermined movement path, the identification unit repeats processing until the moving body stops moving along the movement path, and when the determination unit determines that the moving body has stopped moving along the movement path, it determines whether or not there is an abnormality in the opening / closing part located within the range of the identified bounding box.

[0011] According to one aspect of the present invention, the determination unit processes the image information within the bounding box identified by the identification unit from the image, and predicts whether or not an abnormality exists based on the processed information.

[0012] According to a determination device according to one aspect of the present invention, the image is a color image, and the determination unit processes the image information within the bounding box identified by the identification unit into a grayscale image, and predicts whether or not an abnormality exists based on the information processed into a grayscale image.

[0013] According to one aspect of the present invention, the determination device is a railway vehicle, and the opening / closing part is a door for a railway vehicle.

[0014] An abnormality detection system according to one aspect of the present invention comprises any of the above-described determination devices, an imaging device that captures an image and provides the captured image to the determination device, and an alarm device that issues an alarm when the determination device determines that an abnormality exists in the opening / closing part.

[0015] A determination method according to one aspect of the present invention includes an opening and closing mechanism for opening and closing an opening and closing section, and an image acquisition step of acquiring an image of a moving object in which an object flows in or out when the opening and closing section is in the open state, Using a first neural network trained to predict the coordinates of the bounding box surrounding the area of ​​the opening / closing part in the image acquired by the image acquisition step, A process of identifying the coordinates of the bounding box in the acquired image, A neural network trained to predict whether or not there is an abnormality in the opening / closing part within the bounding box range, using a second neural network different from the first neural network used in the specific process, A determination step of determining whether or not there is an abnormality in the opening / closing part located within the range of the identified bounding box. The identification step involves using the first neural network to process the image acquired by the image acquisition step until it is determined that the opening / closing part is in a closed state. If the opening / closing part is determined to be in a closed state, the determination step uses the second neural network to determine whether or not there is an abnormality in the opening / closing part that is within the range of the identified bounding box. .

[0016] A program according to one aspect of the present invention includes an image acquisition step in which a computer acquires an image of a moving object that has an opening / closing mechanism for opening and closing an opening / closing part, and when the opening / closing part is in the open state, an object flows into the interior or an object flows out to the exterior. Using a first neural network trained to predict the coordinates of the bounding box surrounding the area of ​​the opening / closing part in the image acquired by the image acquisition step, A selection step of identifying the coordinates of the bounding box in the acquired image, A neural network trained to predict whether or not there is an abnormality in the opening / closing part within the bounding box range, using a second neural network different from the first neural network used in the specific step, A determination step of determining whether or not there is an abnormality in the opening / closing part located within the range of the identified bounding box. A program to perform the following steps: The identification step involves using the first neural network to repeatedly process the image acquired by the image acquisition step until it is determined that the opening / closing part is in a closed state; and the determination step, if it is determined that the opening / closing part is in a closed state, uses the second neural network to determine whether or not there is an abnormality in the opening / closing part that is within the range of the identified bounding box. . [Effects of the Invention]

[0017] According to the present invention, anomalies can be detected with high accuracy without reducing the detection speed. [Brief explanation of the drawing]

[0018] [Figure 1] This is a diagram illustrating the overview of the anomaly detection system according to the embodiment. [Figure 2] This is a functional configuration diagram showing an example of the functional configuration of the anomaly detection system according to the embodiment. [Figure 3] It is a functional configuration diagram showing an example of a determination device according to an embodiment. [Figure 4] It is a diagram for explaining a series of operations in the inference stage of the determination device according to the embodiment. [Figure 5] It is a diagram for explaining an example of an image acquired by the image acquisition unit according to the embodiment, which is an image without abnormality. [Figure 6] It is a diagram for explaining an example of a bounding box specified by the specifying unit according to the embodiment in an image without abnormality. [Figure 7] It is a diagram for explaining an example of an image determined to be normal by the determination unit according to the embodiment. [Figure 8] It is a diagram for explaining an example of an image acquired by the image acquisition unit according to the embodiment, which is an image with abnormality. [Figure 9] It is a diagram for explaining an example of a bounding box specified by the specifying unit according to the embodiment in an image with abnormality. [Figure 10] It is a diagram for explaining an example of an image determined to be abnormal by the determination unit according to the embodiment. [Figure 11] It is a diagram showing an example of teacher data used for learning of the specifying unit according to the embodiment. [Figure 12] It is a diagram for explaining the prediction of the opening / closing state of the opening / closing part by the specifying unit according to the embodiment. [Figure 13] It is a diagram showing an example of teacher data used for learning of the determination unit according to the embodiment. [Figure 14] It is a diagram for explaining a first modification example of the abnormality detection system according to the embodiment. [Figure 15] It is a diagram for explaining a second modification example of the abnormality detection system according to the embodiment.

Embodiments for Carrying Out the Invention

[0019] Embodiments of the present invention will be described below with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below.

[0020] [Anomaly detection system] First, the anomaly detection system 1 will be explained with reference to Figures 1 and 2. Figure 1 is a diagram illustrating the overview of the anomaly detection system according to the embodiment. The overview of the anomaly detection system 1 will be explained with reference to this figure.

[0021] Anomaly detection system 1 is applied to a moving body having an opening / closing mechanism that opens and closes an opening / closing part such as a sliding door. When the opening / closing part is in the open state (for example, when the door is open), an object flows into the moving body or an object flows out of the moving body. In this embodiment, "moving body" may be, for example, a railway car, an elevator car, an automobile, or the like. "An object flows into the moving body or an object flows out of the moving body" may be, for example, the entry or exit of a person or luggage.

[0022] The moving object preferably moves along a predetermined path. The predetermined path may be, for example, a railway track if the moving object is a railway vehicle, an elevator shaft if the moving object is an elevator car, or a road if the moving object is a vehicle. In the following description, we will explain an example in which the anomaly detection system 1 is used on a railway platform to detect foreign objects caught in the doors of a railway vehicle.

[0023] Anomaly detection system 1 is used on a railway platform PH. In one example shown in Figure 1, a railway vehicle 50 is stopped. The railway vehicle 50 is equipped with at least one door 51. The railway vehicle 50 is equipped with a pair of doors 51A and 51B as the door 51. Doors 51A and 51B open and close by sliding in the left and right directions around a meeting section 52. In the following description, door 51 may be referred to as a railway vehicle door or an opening / closing section.

[0024] Platform PH is equipped with platform doors 60. The platform doors 60 consist of a pocket section 61 and a door section 62. The door section 62 opens and closes by sliding in the left and right directions. The state in which the door section 62 is closed is sometimes described as the locked state, and the state in which the door section 62 is open is sometimes described as the open state. The platform doors 60 consist of a pair of pocket sections 61A and 61B as the pocket section 61, and a pair of door sections 62A and 62B as the door section 62.

[0025] The door pocket section 61A supports the door section 62A so that it can be opened and closed. The door pocket section 61A houses the door section 62A inside when the door section 62A is in the open position. The door pocket section 61B supports the door section 62B so that it can be opened and closed. The door pocket section 61B houses the door section 62A inside when the door section 62B is in the open position.

[0026] The door section 62 is positioned opposite the position of the door 51 when the railway vehicle 50 is stopped. Since multiple doors 51 are provided along the railway vehicle 50, multiple door pocket sections 61 and door sections 62 are provided corresponding to the doors 51. Each of the multiple door sections 62 is controlled to be in an open state when the railway vehicle 50 is stopped, and controlled to be in a locked state when the railway vehicle 50 is not stopped.

[0027] The anomaly detection system 1 includes one or more imaging devices 20. The imaging device 20 has a field of view α and is positioned to capture at least a portion of the meeting section 52. Preferably, the imaging device 20 is positioned at an angle that captures at least the lower half of the meeting section 52. More preferably, the imaging device 20 is positioned at an angle that captures at least the lower three-quarters of the meeting section 52. More preferably, it is positioned at an angle that captures the entire meeting section 52. Note that a wider field of view may be achieved by providing multiple imaging devices 20 and capturing at different fields of view. The imaging device 20 captures an image of the meeting section 52 and provides the captured image to a determination device 10 (not shown).

[0028] The imaging device 20 is installed, for example, in the door pocket 61 of the platform door 60. The imaging device 20 may be installed in a location other than the door pocket 61, as long as it is in a position that allows imaging of the meeting section 52. For example, the imaging device 20 may be installed on a pillar, ceiling, or railway vehicle 50 side (not shown) instead of the platform door 60 side. Multiple imaging devices 20 may be provided depending on the number of doors 51.

[0029] Furthermore, the abnormality detection system 1 may also be applied to platforms PH that do not have platform doors 60. When the abnormality detection system 1 is applied to platforms PH that do not have platform doors 60, the imaging device 20 may be installed in a predetermined position on the ceiling or floor of the platform PH, for example, where it can image the meeting portion 52 of the door 51.

[0030] Figure 2 is a functional configuration diagram showing an example of the functional configuration of an anomaly detection system according to the embodiment. An example of the functional configuration of the anomaly detection system 1 will be described with reference to this figure. The anomaly detection system 1 comprises a determination device 10, an imaging device 20, and an alarm device 30.

[0031] The imaging device 20 has its field of view and imaging angle fixed in advance to a suitable position. The imaging device 20 takes an image at a predetermined timing and outputs the captured result as image information IP to the determination device 10. The image information IP is obtained by applying predetermined image processing to the image taken by the imaging device 20, but it may also be a composite of multiple images. The predetermined timing at which the imaging device 20 takes an image may be, for example, a predetermined period, and the predetermined period may be a 1-second period. Furthermore, the imaging device 20 may perform imaging when it receives a predetermined trigger signal from a control unit (not shown), or it may start imaging at a predetermined cycle when it receives a predetermined trigger signal from a control unit (not shown), and end imaging when it receives another predetermined trigger signal from the control unit (not shown).

[0032] The determination device 10 acquires image information IP from the imaging device 20. The determination device 10 determines whether or not an abnormality exists in the image contained in the acquired image information IP. An abnormality may be, for example, a foreign object such as a passenger's luggage or clothing caught in the door 51. The determination device 10 determines that an abnormality exists if a foreign object such as luggage or clothing is present in the image taken from the outside of the railway vehicle 50. The determination device 10 outputs the result of its determination as determination information IJ to the alarm device 30. The determination device 10 may be installed in the same housing as the imaging device 20 or in the vicinity of the platform door 60. The higher the resolution of the image acquired by the imaging device 20, the larger the data volume. Therefore, if an external server or the like is used as the determination device 10, sufficient communication bandwidth is required. By installing the determination device 10 in the vicinity of the imaging device 20 and processing on the edge device side, the amount of communication can be reduced.

[0033] The alarm device 30 acquires judgment information IJ from the judgment device 10. Based on the acquired judgment information IJ, the alarm device 30 makes a predetermined alarm if it determines that an abnormality exists. The predetermined alarm may be a notification to a control unit (not shown) provided in the railway vehicle 50, an alarm to the conductor or station staff by sound or illumination of a light unit, or an alarm that displays an image or text information on a display unit (not shown).

[0034] [Judgment device] Figure 3 is a functional configuration diagram showing an example of a determination device according to the embodiment. An example of the functional configuration of the determination device 10 will be described with reference to this figure. The determination device 10 comprises an image acquisition unit 11, a identification unit 12, a determination unit 13, and an output unit 14. The determination device 10 includes a CPU (Central Processing Unit), a storage device such as ROM (Read-only memory) or RAM (Random access memory), etc., connected by a bus, and functions as a device comprising an image acquisition unit 11, a identification unit 12, a determination unit 13, and an output unit 14 by executing a determination program. Furthermore, all or part of the functions of the determination device 10 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field-Programmable Gate Array). The determination program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The determination program may also be transmitted via a telecommunications line.

[0035] The image acquisition unit 11 acquires information about the image of the railway vehicle 50. Specifically, the image acquisition unit 11 acquires information about the image of the door 51's meeting portion 52 taken from the outside of the railway vehicle 50 by the imaging device 20. The image acquisition unit 11 outputs the acquired image as image P1 to the identification unit 12. Note that when outputting image P1 to the identification unit 12, processing to enlarge or reduce it to a specific size may be performed.

[0036] The identification unit 12 acquires image P1 from the image acquisition unit 11. The identification unit 12 identifies the coordinates of the bounding box BB that surrounds the area of ​​door 51 from the acquired image P1. Specifically, the identification unit 12 identifies the coordinates of the bounding box BB using a pre-trained machine learning model. That is, the identification unit 12 is a neural network that has been pre-trained to predict the coordinates of the bounding box BB that surrounds the area of ​​door 51 from the acquired image P1. The identification unit 12 outputs the image P1 surrounded by the identified bounding box BB as the region image P2 to the determination unit 13. The region image P2 is determined by cutting out the region based on the bounding box BB from image P1, but it is not necessary to use the image P1 that the identification unit 12 used to detect the bounding box BB. For example, images P1 acquired chronologically after the coordinates of the bounding box BB have been identified may be used. Also, it is not necessary for the region of region image P2 and the region of the bounding box BB to necessarily match. For example, the region of region image P2 may be a predetermined shape such as a square that includes the region of bounding box BB, or the region of region image P2 may be made wider than the region of bounding box BB. Furthermore, the specific unit 12 may be trained to predict the likelihood of the bounding box BB. Furthermore, when outputting the region image P2 to the determination unit 13, processing may be performed to enlarge or reduce it to a specific size.

[0037] The determination unit 13 acquires a region image P2. Based on the acquired region image P2, the determination unit 13 determines whether or not an anomaly exists within the bounding box BB identified by the identification unit 12. Specifically, the determination unit 13 determines whether or not an anomaly exists within the bounding box BB using a pre-trained machine learning model. That is, the determination unit 13 is a neural network pre-trained to predict whether or not an anomaly exists in door 51 located within the bounding box BB. Furthermore, the neural network of the determination unit 13 is a different neural network from the neural network of the identification unit 12, differing in one or more aspects, such as the network structure, number of channels, inference parameters such as weights and quantization parameters, or the training data used for training. The determination unit 13 outputs the result of its determination regarding the presence or absence of an anomaly as determination information IJ to the output unit 14. The determination unit 13 may be trained to predict the class and likelihood of a foreign object. The class of a foreign object may be a part of the human body, luggage, an umbrella, clothing, etc. By applying scaling to image P1 and region image P2, the learning efficiency of the machine learning model can be improved, thereby increasing the accuracy of identification or judgment.

[0038] The output unit 14 acquires judgment information IJ from the determination unit 13. Based on the judgment information IJ, if the output unit 14 determines that there is an abnormality in the door 51, it outputs the judgment information IJ to the alarm device 30.

[0039] [Inference stage] Next, an example of the operation of the determination device 10 during the inference stage will be described with reference to Figures 4 to 10. Figure 4 is a diagram illustrating a series of operations during the inference stage of the determination device according to this embodiment. The series of operations during the inference stage of the determination device 10 will be explained with reference to this figure.

[0040] (Step S110) The image acquisition unit 11 acquires image information IP from the imaging device 20. The image acquisition unit 11 outputs the image P1 included in the acquired image information IP to the identification unit 12. (Step S120) The identification unit 12 acquires image P1 from the image acquisition unit 11. The identification unit 12 identifies the position coordinates of the door 51 included in image P1.

[0041] (Step S130) When the image acquisition unit 11 acquires an image P1 at a predetermined interval, for example, the railway vehicle 50 may or may not be visible in the image P1. Therefore, if the image acquisition unit 11 cannot determine the position coordinates of the door 51 (i.e., step S130; NO), it does not perform any further processing on the image P1 in which the position coordinates of the door 51 cannot be determined, and returns to step S110. If the image acquisition unit 11 can determine the position coordinates of the door 51 (i.e., step S130; YES), it proceeds to step S140.

[0042] (Step S140) Even if the identification unit 12 identifies the position coordinates of the door 51, the railway vehicle 50 may not be stopped. If the railway vehicle 50 captured in image P1 is not stopped, no further processing is performed on image P1. That is, the identification unit 12 repeats the process until it is determined that the railway vehicle 50 has stopped moving on the tracks. In step S140, the position coordinates of the bounding box BB identified in the current process are compared with the position coordinates of the bounding box BB identified in the previous process. If the difference is within a predetermined range (i.e., step S140; YES), it is determined that the railway vehicle 50 is stopped, and the process proceeds to step S160. If the difference is outside the predetermined range (i.e., step S140; NO), it is determined that the railway vehicle 50 is not stopped, and the process proceeds to step S150. (Step S150) The identification unit 12 stores the position coordinates of the identified bounding box BB and returns the process to step S110.

[0043] (Step S160) Even if the identification unit 12 determines that the railway vehicle 50 is stopped, the door 51 may be open. If the door 51 of the railway vehicle 50 shown in image P1 is open, no further processing is performed on image P1. That is, the identification unit 12 repeats the process until it is determined that the door 51 is closed. If the door is closed (i.e., step S160; YES), the identification unit 12 proceeds to step S170. If the door is open (i.e., step S160; NO), the identification unit 12 returns to step S110. Furthermore, since the specific unit 12 includes a neural network, it may be trained to predict the likelihood of the open / closed state of the door 51 being treated as a class.

[0044] (Step S170) The determination unit 13 determines whether or not an abnormality exists within the bounding box BB (i.e., region image P2) identified by the identification unit 12. That is, if the determination unit 13 determines that the railway vehicle 50 has stopped moving on the tracks and that the door 51 is in the closed state, it determines whether or not an abnormality exists in the door 51 that is within the range of the identified bounding box BB. If the determination unit 13 determines that an abnormality exists (i.e., step S170; YES), it proceeds to step S180. If the determination unit 13 determines that no abnormality exists (i.e., step S170; NO), it terminates the process. (Step S180) The output unit 14 issues an alarm to the alarm device 30 indicating that an abnormality exists.

[0045] Next, we will describe an example of an image captured by the imaging device 20 and judged by the judgment device 10. Referring to Figures 5 to 7, we will describe an example where no abnormalities are present. Figure 5 is a diagram illustrating an example of an image acquired by the image acquisition unit according to this embodiment, which is free of abnormalities. An example of an image P1 captured by the imaging device 20 will be described with reference to this figure. Among the images P1, the image P1 in which no abnormalities are present will be referred to as image P1A.

[0046] Image P1A is an image taken between the railway vehicle 50 and the platform door 60, facing in the direction of extension of the railway vehicle 50 and the track. The left side of the image shows the door pocket section 61A and the door section 62A of the platform door 60. The right side of the image shows the door 51 and the meeting section 52 of the railway vehicle 50. Furthermore, in order to more accurately determine the position coordinates of door 51, it is preferable that image P1 be a color image.

[0047] Figure 6 is a diagram illustrating an example of a bounding box identified by the identification unit according to the embodiment in an image without abnormalities. An example of a bounding box BB identified by the identification unit 12 will be described with reference to this figure. The bounding box BB surrounds the area of ​​door 51. In the example shown in the figure, door 51 is located on the right side of the image, and the bounding box BB is shown to surround the area of ​​door 51. The identification unit 12 outputs coordinate information that allows for the determination of the position and size of the identified bounding box BB to the determination unit 13.

[0048] Figure 7 is a diagram illustrating an example of an image determined to be free of abnormalities by the determination unit according to the embodiment. An example of a region image P2 determined by the determination unit 13 will be explained with reference to the same figure. Among the region images P2, the region image P2 in which no abnormalities are present will be referred to as region image P2A. Region image P2A shows an image of the area within the bounding box BB identified by the identification unit 12. Region image P2A shows at least a portion of door 51A, door 51B, and the meeting section 52 where door 51A and door 51B meet.

[0049] Here, reflections from the door 51 may occur in the region image P2A. Also, passengers of the railway vehicle 50 may be reflected in the transparent portion 53 of the door 51. Therefore, it is preferable that the amount of information contained in the region image P2A is less than the amount of information contained in the image within the bounding box BB of image P1. For example, the resolution of the region image P2A may be lower than the resolution of image P1. In other words, the determination unit 13 may process the image information within the bounding box BB identified by the identification unit 12 in the image P1, and predict whether or not an abnormality exists based on the processed information.

[0050] Furthermore, while image P1 is a color image, region image P2 may be a grayscale image. That is, the determination unit 13 may process the image information within the bounding box BB identified by the identification unit 12 in image P1 and predict whether or not an abnormality exists based on the processed information. The pattern of the door 51 shown in region image P2 may vary depending on the type of railway vehicle 50. When detecting foreign objects in railway vehicle 50 that have a different pattern from the training image used in the learning stage, it is particularly effective to make region image P2 a grayscale image.

[0051] Furthermore, the shape of the door 51 as captured in the region image P2 may change depending on the imaging angle of the imaging device 20 and the stopping position of the railway vehicle 50. Therefore, in order to compensate for the change in the shape of the door 51, the determination unit 13 may correct the shape of the door 51 by image processing. An example of image processing may be trapezoidal correction.

[0052] Next, we will explain an example of a situation where an abnormality exists, referring to Figures 8 to 10. Figure 8 is a diagram illustrating an example of an image acquired by the image acquisition unit according to this embodiment, which contains an abnormality. An example of an image P1 captured by the imaging device 20 will be described with reference to this figure. Among the images P1, the image P1 in which an abnormality is found will be referred to as image P1B.

[0053] Image P1B was captured under the same imaging conditions as Image P1A, so the position of the platform doors 60 is the same. On the other hand, there may be errors in the stopping position of the railway vehicle 50. Image P1B differs from Image P1A in that it includes an object (OBJ).

[0054] Figure 9 is a diagram illustrating an example of a bounding box identified by the identification unit according to the embodiment in an image having an abnormality. An example of a bounding box BB identified by the identification unit 12 will be described with reference to this figure. The stopping position of the railway vehicle 50 may vary depending on the vehicle's entry speed, vehicle weight, braking timing, etc. Therefore, the position coordinates of the bounding box BB, which are determined by the specific unit 12, will differ depending on the image P1.

[0055] Figure 10 is a diagram illustrating an example of an image that is determined to have an abnormality by the determination unit according to the embodiment. An example of a region image P2 determined by the determination unit 13 will be explained with reference to the same figure. Among the region images P2, the region image P2 in which an abnormality exists will be referred to as region image P2B. Image P2B differs from image P2A in that it includes an object (OBJ).

[0056] Here, image P1 has a large number of pixels and is a color image, so it contains a lot of information. On the other hand, region image P2 is a grayscale image with a smaller area cropped from image P1, so it has fewer pixels and contains less information. Therefore, the neural network included in the determination unit 13 can prevent false detections due to overfitting and can detect object OBJ with high accuracy.

[0057] [Learning Stage] Next, an example of the learning stage of the judgment device 10 will be described with reference to Figures 11 to 13. Figure 11 shows an example of training data used for training a specific unit according to this embodiment. An example of training data used for training the specific unit 12 will be explained with reference to this figure.

[0058] The training data used for learning the specific unit 12 is based on images captured by the imaging device 20. According to the images captured by the imaging device 20, the positions of the platform and tracks are constant in the images, but the stopping positions of the railway vehicle 50 differ. Because the stopping positions of the railway vehicle 50 differ, the position of the doors 51 in the images differs from image to image.

[0059] During the learning phase, the identification unit 12 learns from multiple images of the door 51 in different positions, with the door's position coordinates associated with each image, as training data. Figures 11(A) to 11(F) show examples of training data used for learning the identification unit 12. Figures 11(A) to 11(C) show examples of image P1. Figures 11(D) to 11(F) show examples of the position coordinates of the bounding box BB corresponding to image P1. Figures 11(A) and 11(D), 11(B) and 11(E), and 11(C) and 11(F) correspond to each other.

[0060] The identification unit 12 may also predict the open / closed state of the door 51. When predicting the open / closed state of the door 51, the training data used for learning the identification unit 12 may have the open / closed state of the door 51 associated as a class. By associating the open / closed state of the door 51 with the training data, the identification unit 12 learns the open / closed state of the door 51. For example, Figures 11(A) to 11(C) all show the door in the closed state.

[0061] Figure 12 is a diagram illustrating how a specific unit according to an embodiment predicts the opening and closing state of an opening / closing section. Referring to this figure, an example of how the specific unit 12 predicts the opening and closing state of the door 51 will be described.

[0062] Figures 12(A) to 12(C) are all examples of image P1 and are examples of training data used for training the specific unit 12. Figure 12(A) is an example of the closed state, and Figures 12(B) and 12(C) are examples of the open state. During the training phase, the specific unit 12 learns the open / closed states as classes in association with image P1.

[0063] Figures 12(D) to 12(F) show examples of predicting the open / closed state during the inference stage. Figure 12(D) is an example of predicting the open / closed state of Figure (A). According to this figure, the likelihood of being in the open state is 0.5%, and the likelihood of being in the closed state is 99.5%. Figure 12(E) is an example of predicting the open / closed state of Figure (B). According to this figure, the likelihood of being in the open state is 70%, and the likelihood of being in the closed state is 30%. Figure 12(F) is an example of predicting the open / closed state of Figure (C). According to this figure, the likelihood of being in the open state is 91%, and the likelihood of being in the closed state is 9%.

[0064] Figure 13 shows an example of training data used for learning the determination unit according to the embodiment. An example of training data used for learning the determination unit 13 will be explained with reference to this figure. The training data used for learning the determination unit 13 is based on an image (i.e., region image P2) of the bounding box BB surrounding the door 51 identified by the identification unit 12. According to region image P2, the positions of the door 51 and the meeting part 52 are approximately the same, enabling more accurate foreign object detection. Note that "approximately the same range" may mean a range that is similar enough that the determination unit 13 does not make false detections.

[0065] During the learning phase, the determination unit 13 learns to associate region images P2 with the presence or absence of abnormalities. Figure 13(A) shows an example of a case where there are no abnormalities, while Figures 13(B) and 13(C) show an example of a case where there are abnormalities. The determination unit 13 may also learn by associating the region image P2 with the position coordinates of the bounding box BB2 surrounding the foreign object OBJ. In Figure 13(B), the tip of the umbrella, and in Figure 13(C), the passenger's hand, are present as object OBJs, so the bounding box BB2, which is the region surrounding the object OBJ, is learned in association with them.

[0066] The determination unit 13 may also predict the type of object OBJ. When predicting the type of object OBJ, the training data used for training the identification unit 12 may be associated with the type of object OBJ as a class. By associating the type of object OBJ with the training data, the determination unit 13 learns the type of object OBJ. For example, in Figure 13(A), there is no object OBJ, in Figure 13(B) the object OBJ is an umbrella, and in Figure 13(C) the object OBJ is a hand.

[0067] When the determination unit 13 learns the position coordinates of an object OBJ, the determination unit 13 may output the type of object OBJ and its likelihood during the inference stage. The output unit 14 may determine the degree of danger according to the type of object OBJ and output different information accordingly. Furthermore, the alarm device 30 may determine the degree of danger according to the type of object OBJ and vary the type and content of the alarm accordingly. For example, the importance of the alarm may be increased if a part of the human body such as a hand is detected, and decreased if clothing or the like is detected. Furthermore, the alarm device 30 may vary the type and content of the alarm depending on the size of the bounding box BB surrounding the object OBJ. For example, if clothing is detected as the object OBJ, the alarm's importance may be increased if the bounding box BB is larger than a predetermined size, and decreased if it is smaller than the predetermined size.

[0068] [Differentiation] Next, a modified example of the anomaly detection system 1 will be described with reference to Figures 14 and 15. Figure 14 is a diagram illustrating a first modified example of the anomaly detection system according to the embodiment. Anomaly detection system 1A, which is a modified example of anomaly detection system 1, will be described with reference to the same figure. Anomaly detection system 1 is provided with one anomaly detection system 1 for one door 51, whereas anomaly detection system 1A is provided with one anomaly detection system 1 for multiple doors 51. Specifically, anomaly detection system 1 is composed of an imaging device 20, a determination device 10 and an alarm device 30, whereas anomaly detection system 1A is configured such that a combination of multiple imaging devices 20 and determination devices 10 is connected to the alarm device 30 via a predetermined communication network NW.

[0069] The anomaly detection system 1A comprises imaging devices 20-1 through 20-n (where n is a natural number greater than or equal to 1), determination devices 10-1 through 10-n, and an alarm device 30. The imaging device 20 outputs image information IP to the corresponding determination device 10, and the determination device 10 makes a determination on the image captured by the corresponding imaging device 20. Multiple determination devices 10 output determination information IJ to a common alarm device 30 via a predetermined communication network NW. When the alarm device 30 receives information from any of the determination devices 10 indicating the presence of an anomaly, it issues an alarm using a predetermined method.

[0070] According to the anomaly detection system 1A, since multiple judgment devices 10 use a common alarm device 30, the number of alarm devices 30 can be reduced.

[0071] Figure 15 is a diagram illustrating a second modified example of the anomaly detection system according to the embodiment. Anomaly detection system 1B, which is a modified example of anomaly detection system 1, will be described with reference to this figure. Anomaly detection system 1B is similar to anomaly detection system 1A in that it has one anomaly detection system 1 for multiple doors 51. On the other hand, anomaly detection system 1A has one determination device 10 for one door 51, whereas anomaly detection system 1B differs from anomaly detection system 1A in that it has one determination device 10 for multiple doors 51.

[0072] The anomaly detection system 1B comprises imaging devices 20-1 through 20-n (where n is a natural number greater than or equal to 1), a determination device 10, and an alarm device 30. The imaging device 20 outputs image information IP to the common determination device 10 via a predetermined communication network NW. The determination device 10 outputs determination information IJ to the common alarm device 30. When the alarm device 30 receives information from the determination device 10 indicating the presence of an anomaly, it issues an alarm using a predetermined method.

[0073] According to the anomaly detection system 1B, the common determination device 10 uses images acquired from multiple imaging devices 20, which increases the processing load of the determination device 10. On the other hand, according to the anomaly detection system 1B, since multiple imaging devices 20 use a common determination device 10 and alarm device 30, the number of determination devices 10 and alarm devices 30 can be reduced.

[0074] [Summary of Embodiments] According to the embodiment described above, the determination device 10 includes an image acquisition unit 11 to acquire an image of the opening and closing part of the moving object, an identification unit 12 to identify the coordinates of the bounding box BB surrounding the area of ​​the opening and closing part from the acquired image, and a determination unit 13 to determine whether or not there is an abnormality in the opening and closing part that is within the range of the identified bounding box BB. In other words, the determination device 10 has a functional division between the identification unit 12, which identifies the position of the door 51, and the determination unit 13, which determines whether or not there is an abnormality in the door 51. Therefore, the determination device 10 can determine the presence or absence of an abnormality without processing reflections of people, etc., on the glass by image processing, and can detect abnormalities with high accuracy without reducing the detection speed.

[0075] Furthermore, according to the embodiment described above, the identification unit 12 repeats processing until it is determined that the door 51 is in a closed state, and when it is determined that the door 51 is in a closed state, the determination unit 13 determines whether or not there is an abnormality in the door 51 that is within the range of the identified bounding box BB. In other words, according to this embodiment, the determination device 10 performs abnormality detection after it has determined that the door 51 is in a closed state. Therefore, the determination device 10 can reduce the processing load on the determination unit 13. Thus, abnormality detection can be performed without reducing the detection speed.

[0076] Furthermore, according to the embodiment described above, the railway vehicle 50 moves along a predetermined movement path, the identification unit 12 repeats processing until the railway vehicle 50 stops moving along the movement path, and the determination unit 13 determines whether or not there is an abnormality in the door 51 located within the range of the identified bounding box BB when it is determined that the railway vehicle 50 has stopped moving along the movement path. In other words, according to this embodiment, the determination device 10 performs abnormality detection after determining that the railway vehicle 50 is stopped. Therefore, according to this embodiment, abnormality detection can be performed at a fast detection speed.

[0077] Furthermore, according to the embodiment described above, the identification unit 12 includes a neural network trained to predict the coordinates of the bounding box BB surrounding the area of ​​the door 51 in the acquired image P1. Therefore, according to this embodiment, anomaly detection can be performed with high accuracy.

[0078] Furthermore, according to the embodiment described above, the determination unit 13 includes a neural network trained to predict whether or not an abnormality exists in the door 51 located within the bounding box BB, and is a different neural network from the neural network included in the identification unit 12. Therefore, according to this embodiment, the detection accuracy of the identification unit 12 and the detection accuracy of the determination unit 13 can be made different, thereby suppressing false detections due to overfitting. Thus, according to this embodiment, abnormalities can be detected with high accuracy.

[0079] Furthermore, according to the embodiment described above, the determination unit 13 processes the image information within the bounding box BB identified by the identification unit 12 in the image P1 through image processing, and predicts whether or not an abnormality exists based on the processed information. In other words, according to this embodiment, the determination unit 13 performs learning and inference based on information with a deliberately reduced amount of information. Therefore, according to this embodiment, false detection due to overfitting can be suppressed.

[0080] Furthermore, according to the embodiment described above, image P1 is a color image, and the determination unit 13 processes the image information within the bounding box BB identified by the identification unit 12 from image P1 into a grayscale image, and predicts whether or not an abnormality exists based on the information processed into a grayscale image. In other words, according to this embodiment, the determination unit 13 performs learning and inference based on information with a deliberately reduced amount of information. Therefore, according to this embodiment, false detection due to overfitting can be prevented.

[0081] Furthermore, according to the embodiment described above, the moving body is a railway vehicle 50, and the opening / closing part is a railway vehicle door 51. Therefore, according to this embodiment, if a passenger's luggage, clothing, or body is caught in the door 51 before the railway vehicle departs, the abnormality can be detected and the alarm device 30 can be alerted, thereby preventing a dangerous situation.

[0082] Furthermore, according to the embodiment described above, the abnormality detection system 1 comprises a determination device 10, an imaging device 20, and an alarm device 30. Therefore, according to this embodiment, even if an abnormality occurs in each of the multiple doors 51 provided on the railway vehicle 50, the alarm device 30 will sound an alarm, thereby preventing a dangerous situation.

[0083] Furthermore, while the embodiments described above illustrate a process in which the identification unit 12 and the determination unit 13 use progressively trained machine learning models, the invention is not limited to this. For example, it may be a process in which three or more trained machine learning models are used progressively. The determination unit 13 may also be a process in which different trained machine learning models are used in parallel. Furthermore, the functions of each part of the anomaly detection system 1 in the above-described embodiment, or a part thereof, may be realized by recording a program for realizing these functions on a computer-readable recording medium, having a computer system read the program recorded on this recording medium, and executing it. The term "computer system" here includes hardware such as an operating system and peripheral devices.

[0084] Furthermore, "computer-readable recording media" refers to portable media such as magneto-optical disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. In addition, "computer-readable recording media" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over a network such as the Internet, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. Moreover, the above-mentioned program may be for the purpose of realizing some of the functions described above, and may also be able to realize the above-mentioned functions in combination with programs already recorded in the computer system.

[0085] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0086] 1...Anomaly detection system, 10...Determination device, 11...Image acquisition unit, 12...Identification unit, 13...Determination unit, 14...Output unit, 20...Imaging device, 30...Alarm device, 50...Railway vehicle, 51...Door, 52...Meeting section, 60...Platform door, 61...Door pocket section, 62...Door section, P1...Image, P2...Region image, BB...Bounding box

Claims

1. An image acquisition unit having an opening / closing mechanism for opening and closing an opening / closing section, and acquiring an image of a moving object when the opening / closing section is in the open state, in which case an object flows in or out of the unit. The image acquisition unit includes a first neural network trained to predict the coordinates of the bounding box surrounding the area of ​​the opening / closing part in the image acquired by the image acquisition unit, and a specification unit that identifies the coordinates of the bounding box in the acquired image, A neural network trained to predict whether or not there is an abnormality in the opening / closing part located within the bounding box range, comprising a second neural network different from the first neural network included in the identification unit, and a determination unit that determines whether or not there is an abnormality in the opening / closing part located within the identified bounding box range, The identifying unit, based on the image acquired by the image acquisition unit, repeatedly processes the image using the first neural network until it is determined that the opening / closing unit is in a closed state. If the determination unit determines that the opening / closing part is in a closed state, it uses the second neural network to determine whether or not there is an abnormality in the opening / closing part that is within the range of the identified bounding box. Judgment device.

2. The moving body moves along a predetermined path, The specified unit repeats the process until the moving body stops moving along the movement path. If the determination unit determines that the moving body has stopped moving along the movement path, it determines whether or not there is an abnormality in the opening / closing part located within the range of the identified bounding box. The determination device according to claim 1.

3. The determination unit processes the image information within the bounding box identified by the identification unit from the image, and predicts whether or not an abnormality exists based on the processed information. The determination device according to claim 1 or claim 2.

4. The aforementioned image is a color image, The determination unit processes the image information within the bounding box identified by the identification unit into a grayscale image and predicts whether or not an abnormality exists based on the information processed into the grayscale image. The determination device according to claim 3.

5. The aforementioned mobile vehicle is a railway vehicle. The aforementioned opening / closing mechanism is a door for a railway vehicle. A determination device according to any one of claims 1 to 4.

6. A determination device according to any one of claims 1 to 5, An imaging device that captures the aforementioned image and provides the captured image to the determination device, If the determination device determines that there is an abnormality in the opening / closing part, an alarm device will sound an alarm. An anomaly detection system equipped with the following features.

7. The device has an opening / closing mechanism for opening and closing an opening / closing section, and when the opening / closing section is in the open state, it acquires an image of a moving object that is flowing in or out of the device. A selection step involves identifying the coordinates of the bounding box in the acquired image using a first neural network trained to predict the coordinates of the bounding box surrounding the area of ​​the opening / closing part in the image acquired in the image acquisition step, A neural network trained to predict whether or not there is an abnormality in the opening / closing part within the bounding box range, comprising a determination step which uses a second neural network different from the first neural network used in the specified step to determine whether or not there is an abnormality in the opening / closing part within the specified bounding box range, The aforementioned identification step involves repeatedly processing the image obtained by the image acquisition step using the first neural network until it is determined that the opening / closing part is in a closed state. If the determination step determines that the opening / closing part is in a closed state, the second neural network is used to determine whether or not there is an abnormality in the opening / closing part that is within the range of the identified bounding box. Judgment method.

8. On the computer, The device has an opening / closing mechanism for opening and closing an opening / closing section, and when the opening / closing section is in the open state, it acquires an image of a moving object that is flowing in or out of the device. A selection step involves identifying the coordinates of the bounding box in the acquired image using a first neural network trained to predict the coordinates of the bounding box surrounding the area of ​​the opening / closing part in the image acquired in the image acquisition step, A program that causes a neural network trained to predict whether or not there is an abnormality in the opening / closing part located within the bounding box range, and to perform a determination step of determining whether or not there is an abnormality in the opening / closing part located within the specified bounding box range, using a second neural network different from the first neural network used in the specified step, The aforementioned identification step involves repeatedly performing a process using the first neural network based on the image acquired in the image acquisition step until it is determined that the opening / closing part is in a closed state. If the determination step determines that the opening / closing part is in a closed state, the second neural network is used to determine whether or not there is an abnormality in the opening / closing part that is within the range of the identified bounding box. program.

Images