State determination device and state determination method

Abstract

The present invention identifies, with high accuracy, the state of a traffic signal installed on a track being travelled on even when there is no accurate positional information of the traffic signal. To this end, one representative state determination device comprises: a traffic signal database in which information relating to a traffic signal included in an image of a traveling direction of a traveling train is stored on the basis of the image; a sensor unit attached to the train; and a traffic signal identification unit for identifying a traffic signal from an image acquired by the sensor unit. The traffic signal database includes movement locus information of the traffic signal calculated from an image of the traffic signal included in the image of the travel direction, and / or appearance area information of the traffic signal calculated from distance information from a line on which the train travels to the traffic signal. The traffic signal identification unit references the traffic signal database to identify the traffic signal from the image acquired by the sensor unit and determine the state of the traffic signal.

Description

State determination device and state determination method

[0001] The present invention relates to a state determination device and a state determination method.

[0002] Driverless operation of railways is attracting attention as a means of reducing costs and improving services in the rail transport industry. To realize driverless operation, it is necessary to reliably determine the status of signals without human intervention.

[0003] For this reason, conventional driverless operation has been implemented on routes where there is little possibility of people entering the area and where traffic light indicators change little each time the vehicle travels, such as subways and new transportation systems.

[0004] On the other hand, on normal railway lines, there are many railroad crossings, people are pedestrians, and traffic light indicators are prone to change, so it is necessary to be able to detect and recognize traffic lights from a long distance.It is also necessary to be able to reliably detect and recognize the status of traffic lights even in dark places such as at night or inside tunnels.

[0005] For example, Patent Document 1 discloses a technology in which "the driving assistance ECU 20 acquires, from the forward camera, the position of a traffic light candidate that is a lighting device with a probability of being a traffic light of a predetermined value or more as the observed lighting position. The driving assistance ECU 20 also acquires the position of a traffic light in front of the vehicle as the map information traffic light position by referring to map data in the vicinity of the current position of the vehicle. The driving assistance ECU 20 then determines that the lighting device is a traffic light based on the fact that the proximity δ, which is the difference between the traffic light position on the map and the position of the observed lighting device, is less than a predetermined deemed threshold Dth."

[0006] Japanese Patent Application Laid-Open No. 2022-30770

[0007] However, while the invention described in Patent Document 1 makes it possible to detect traffic signals even in dark places such as at night or inside tunnels, it requires the creation of a database in advance that accurately records the positions of traffic signals. Therefore, creating the database requires a large amount of work. Therefore, the present invention aims to provide a technology that can accurately identify the status of traffic signals installed on the tracks on which the train is running, even when accurate position information of the traffic signals is not available.

[0008] In order to solve the above problem, one representative status determination device is a status determination device that includes a traffic light database that stores information about traffic lights included in an image based on an image of the traveling direction of a traveling train, a sensor unit attached to the train, and a traffic light identification unit that identifies traffic lights from the image acquired by the sensor unit, wherein the traffic light database has traffic light movement trajectory information calculated from the image of the traffic light included in the image of the traveling direction, and / or traffic light appearance area information calculated from distance information from the track on which the train is traveling to the traffic light, and the traffic light identification unit refers to the traffic light database to identify traffic lights from the image acquired by the sensor unit and determine the status of the traffic light.

[0009] According to the present invention, even when accurate position information of a traffic signal installed on a railway line is not available, the state of the traffic signal can be identified with high accuracy.

[0010] FIG. 1 shows an example of a traffic light detection system according to a first embodiment. FIG. 2 shows an example of a traffic light database. FIG. 3 is a flowchart of a process for creating a database of information related to traffic lights. FIG. 4 is an example of a screen showing the results of traffic light detection processing based on a movement trajectory. FIG. 5 is a flowchart for performing traffic light detection processing based on a movement trajectory. FIG. 6 shows an example of a traffic light detection system according to a second embodiment. FIG. 7 is an example of a screen showing the results of traffic light detection processing based on tracks. FIG. 8 is a flowchart for performing traffic light detection processing based on tracks. FIG. 9 shows an example of a traffic light detection system according to a third embodiment.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to these embodiments. In the drawings, identical parts are denoted by the same reference numerals. When multiple components having the same or similar functions are present, they may be described using the same reference numerals with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted. Furthermore, although terms such as "first," "second," and "third" may be used to describe various elements or components in this disclosure, it should be understood that these elements or components should not be limited by these terms. These terms are used only to distinguish one element or component from another. Therefore, a first element or component discussed below may also be referred to as a second element or component without departing from the teachings of the inventive concept. The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings. The traffic light detection system of the present invention also functions as a state determination device that determines the state of a traffic light and indicates the traffic light aspect.

[0012] First Embodiment First, a traffic light detection system 1 according to a first embodiment will be described with reference to Figures 1 and 2. Figure 1 shows an example of the traffic light detection system 1. The traffic light detection system 1 is a system that runs a train equipped with an imaging device that captures images of surrounding objects, constructs a traffic light database, and performs traffic light detection processing based on the constructed traffic light database. The traffic light detection system 1 mainly includes a sensor unit 110, a display unit 120, a calculation unit 130, a memory 140, and a storage unit 150.

[0013] <Sensor Unit> The sensor unit 110 mainly captures images in the direction of train travel and can acquire images of traffic signals located in the direction of train travel. In the embodiment of the present disclosure, a camera that captures images in the direction of train travel is used as the sensor unit, but other types of devices may also be used. For example, LiDAR, radar, or a 3D sensor may be used.

[0014] <Display Unit> The display unit 120 is a display device that displays the image captured by the sensor unit 110 and the detection results output by the detection result output unit 158 ​​(described later). The display unit 120 is, for example, a monitor that displays the image captured by the sensor unit 110, and in addition to the image, displays the position and information of the traffic light recognized by the traffic light detection system 1.

[0015] <Calculation Unit> The calculation unit 130 is a device that controls other devices and circuits in the computer, performs data calculations, etc. In this embodiment, it functions as a calculation device that executes programs installed in the storage unit 150.

[0016] <Memory> The memory 140 is a storage device directly connected to the main bus of the computer, etc. The memory 140 temporarily stores information necessary for the calculation unit 130 to execute processing. The memory 140 is, for example, an SRAM (Static Random Access Memory) or an SDRAM (Synchronous Dynamic Random Access Memory).

[0017] <Storage Unit> The storage unit 150 is a storage device that stores a program to be calculated by the calculation unit 130. In the embodiment of the present disclosure, the storage unit 150 has installed therein, in the form of a software program, programs that cause the calculation unit 130 to function as an image acquisition unit 151, a traffic light identification unit 152, a position acquisition unit 153, a traffic light database 154, a traffic light determination unit 155, a database writing unit 156, a database reading unit 157, and a detection result output unit 158.

[0018] Furthermore, the program installed in the storage unit 150 may be in a format other than the program executed by the calculation unit 130. For example, it may be in a format that can be executed as hardware by an application-specific integrated circuit such as an ASIC.

[0019] <<Image Acquisition Unit>> The image acquisition unit 151 acquires a plurality of images captured by the sensor unit 110 and transmits the images to the traffic light identification unit 152 in the order in which they were captured.

[0020] <<Traffic Light Identification Unit>> The traffic light identification unit 152 refers to the traffic light database 154 to identify a traffic light from an image acquired by the sensor unit, and determines the state of the traffic light. In this embodiment, the traffic light identification unit 152 extracts a movement trajectory of a traffic light candidate from multiple images acquired by the sensor unit 110, and identifies the traffic light based on the comparison result between the movement trajectory and the movement trajectory of a traffic light stored in the traffic light database. The traffic light identification unit 152 includes a traffic light candidate detection unit 1521, a traffic light candidate tracking unit 1522, and a comparison unit 1523.

[0021] <<<<Traffic Light Candidate Detection Unit>>> The traffic light candidate detection unit 1521 detects traffic light candidates included in the image acquired by the image acquisition unit 151. Specifically, the traffic light candidate detection unit 1521 uses an algorithm of a known object recognition technology to estimate traffic light candidates included in the acquired image and acquire the coordinates of the estimated traffic light candidates. The algorithm of a known object detection method may be, for example, an algorithm such as YOLO, or in the case of nighttime, an algorithm for detecting illuminants. Furthermore, the traffic light candidate detection unit 1521 detects traffic light candidates from the entire acquired image, but may also use a method of detecting traffic light candidates from a partial area of ​​the acquired image.

[0022] Here, a traffic light candidate is an object that is estimated to be a traffic light and is included in an image in which a traffic light appears. Such an object may not be determined to be a traffic light because the image is captured from a long distance, the traffic light appears small, there is an obstruction between the object and the traffic light, or the object is not clearly captured due to weather or time of day. Note that if an object that is not a traffic light is recognized as a traffic light, the accuracy of the traffic light detection process deteriorates. Therefore, the traffic light candidate detection unit 1521 estimates an object that is estimated to be a traffic light and transmits the estimated object to the traffic light determination unit 155, which separately determines whether the object is a traffic light based on specific criteria.

[0023] Furthermore, the traffic light candidate detection unit 1521 may detect multiple traffic light candidates from one image.

[0024] <<<Traffic Light Candidate Tracking Unit>>> The traffic light candidate tracking unit 1522 tracks and calculates the movement trajectories of traffic light candidates from multiple images acquired by the sensor unit 110. Specifically, the traffic light candidate tracking unit 1522 first associates the traffic light candidates present in an input image with the traffic light candidates included in the image immediately preceding that image. Here, associating traffic light candidates means associating the same traffic light candidates included in different images so that they can be recognized as the same traffic light candidate.

[0025] In this case, a traffic light candidate that cannot be associated with a traffic light candidate included in a different image will have a low determination value calculated by the traffic light determination unit 155, as will be described later. Also, a traffic light candidate that is not included in the input image but exists in the image immediately before the input image is considered to be a traffic light that has been passed by a running train.

[0026] Next, the traffic light candidate tracking unit 1522 calculates a movement trajectory from the change in the position of the associated traffic light candidate on the image. At this time, the movement trajectory indicates the position information of the traffic light candidate on the image using the values ​​of coordinate information on the image.

[0027] <<<Comparison Unit>>> The comparison unit 1523 compares the movement trajectory calculated by the traffic light candidate tracking unit 1522 with the traffic light movement trajectory information held by the traffic light database 154. If the movement trajectory calculated by the traffic light candidate tracking unit 1522 matches the traffic light movement trajectory information held by the traffic light database 154, the comparison unit 1523 identifies the traffic light candidate of the movement trajectory calculated by the traffic light candidate tracking unit 1522 as a traffic light, and determines the state of the traffic light.

[0028] <<Position Acquisition Unit>> The position acquisition unit 153 acquires train position information of the train. In an embodiment of the present disclosure, the position acquisition unit 153 acquires information from an external system mounted on the train, and acquires the current position as train position information. The external system used by the position acquisition unit 153 is, for example, GPS, but may be any other system as long as it can acquire the current position of the train.

[0029] Furthermore, the location information acquired by the location acquisition unit 153 is expressed in kilometers, but may be expressed in other information, such as latitude, longitude, and altitude.

[0030] <<Traffic Signal Database>> Figure 2 shows an example of the traffic signal database 154. The traffic signal database 154 is a database that stores information about traffic signals included in an image based on an image of the direction of travel of a traveling train. The traffic signal database 154 stores a traffic signal ID 1541 for identifying a traffic signal, a location 1542 of the traffic signal, a height 1543 of the traffic signal from the tracks, a distance 1544 from the tracks, and a movement trajectory 1545 of the traffic signal as seen by a traveling train. In this embodiment, the location 1542 of the traffic signal is expressed in kilometers, but may also be expressed by other information, such as latitude, longitude, and altitude.

[0031] <<Traffic Signal Determination Unit>> The traffic signal determination unit 155 determines whether a traffic signal candidate detected by the traffic signal candidate detection unit 1521 is a traffic signal and further determines whether the traffic signal candidate is a traffic signal that can be identified by a train. The traffic signal determination unit 155 determines that the traffic signal candidate is a traffic signal when a determination value for determining the determination exceeds a threshold. Here, the determination value is determined based on, for example, a confidence score indicating the likelihood that the traffic signal candidate output by the traffic signal candidate detection unit 1521 is a traffic signal, an estimated distance from the sensor unit 110 to the traffic signal candidate (the size of the traffic signal candidate in the image), and an estimated position of the traffic signal candidate (height from the ground, distance from the tracks). Alternatively, the determination value may be a value calculated based on the estimated distance and confidence score from the sensor unit 110 to the traffic signal candidate, the estimated position of the traffic signal candidate, and the position of the traffic signal recorded in the traffic signal database 154.

[0032] The confirmation value is set to exceed the threshold value, for example, when the estimated distance from the sensor unit 110 to the traffic light candidate is sufficiently close, the certainty score is sufficiently high, and the estimated position of the traffic light candidate is sufficiently close to the position of the traffic light recorded in the traffic light database 154.

[0033] <<Database Writing Unit>> The database writing unit 156 writes information about a traffic light that has been determined to be a traffic light by the traffic light determination unit 155 to the traffic light database 154. Note that the database writing unit 156 may write information to the traffic light database 154 only during a preliminary drive to establish the traffic light database 154, or may write information continuously during subsequent normal driving as well.

[0034] <<Database Reading Unit>> The database reading unit 157 reads information from the traffic light database 154. For example, when the current position of a train is input as train position information of the train to the traffic light database 154, the database reading unit 157 reads information about the traffic light that the train closest to the train position information will pass before.

[0035] <<Detection Result Output Unit>> The detection result output unit 158 ​​outputs the position of the traffic light, the movement trajectory of the traffic light that was used as the basis for judgment, and the like, and transmits them to the display unit 120. At this time, for example, position information of the traffic light (height from the ground 1543, distance from the track 1544) may also be transmitted.

[0036] <Process for creating a database of information related to traffic lights> Next, the process for creating a database of information related to traffic lights will be described with reference to Fig. 3. Fig. 3 is a flowchart of the process for creating a database of information related to traffic lights. As described above, the traffic light database 154 may be created during pre-driving, or may be created continuously during normal driving.

[0037] (Step S101) In step S101, the image acquisition unit 151 acquires an image from the sensor unit 110 and inputs the image to the traffic light identification unit 152. At this time, if the image acquisition unit 151 acquires multiple images from the sensor unit 110, the multiple images are input in the order in which they were captured.

[0038] (Step S102) In step S102, the position acquisition unit 153 acquires current position information of the running train.

[0039] (Step S103) In step S103, the traffic light candidate detection unit 1521 detects all traffic light candidates included in the images input in order.

[0040] In step S104, the traffic light candidate tracking unit 1522 tracks and calculates the movement trajectories of the traffic light candidates. At this time, a traffic light candidate that cannot be tracked is considered to be a newly detected traffic light candidate or a traffic light candidate that has disappeared from the image after a train has passed.

[0041] (Step S105) In step S105, the traffic light determination unit 155 determines whether the traffic light candidate is a writing target for storing information in the traffic light database 154. If it is a writing target, the process returns to S106, and if it is not a writing target, the process proceeds to S107.

[0042] The traffic light candidate included in the input image is determined to be a traffic light if the determination value for determining determination exceeds a threshold value.

[0043] Next, the traffic light determination unit 155 detects whether there is a traffic light candidate that is not included in the input image but is included in the image immediately before the input image. If there is a traffic light candidate, it is considered that the train has passed through the traffic light, and no movement trajectory will be generated in subsequent images, and it is determined to be a target for writing. Furthermore, if there is a traffic light candidate in the image immediately before the input image but not in the current image, the traffic light candidate included in the current image is considered to have been passed by the train, and is therefore a target for writing.

[0044] (Step S106) In step S106, the database writing unit 156 writes information about the traffic light candidate that has been determined to be a traffic light by the traffic light determination unit 155 into the traffic light database 154.

[0045] (Step S107) In step S107, the traffic light identification unit 152 determines whether or not there are any other images that have not been subjected to the database processing among the images input by the image acquisition unit 151. If there are any other images, the process returns to S102; if there are no other images, the process ends.

[0046] <Traffic Signal Detection Processing Based on Movement Trajectory> Next, the traffic signal detection processing based on movement trajectory will be described with reference to Fig. 4. Fig. 4 shows an example of a screen showing the results of the traffic signal detection processing based on movement trajectory.

[0047] In the traffic light detection process based on a movement trajectory, as shown in Fig. 4, the traffic light detection result is displayed on the image captured by the sensor unit 110. Here, the traffic light detection result is displayed as a traffic light candidate 310 shown inside an appearance area 320. The traffic light candidate 310 inside the appearance area 320 moves in accordance with the movement of the train, thereby generating a movement trajectory 330. The image captured by the sensor unit 110 also includes objects other than the traffic light candidate 310 that is the detection target, such as a railroad track 340 and a station platform 350, but these objects are not related to the detection of the traffic light candidate 310 or the traffic light.

[0048] Next, the traffic light detection process based on the movement trajectory will be described with reference to Fig. 5. Fig. 5 is a flowchart of the traffic light detection process based on the movement trajectory. (Step S201) In step S201, similar to step S101 in Fig. 3, the image acquisition unit 151 acquires an image from the sensor unit 110 and inputs the image to the traffic light identification unit 152. At this time, if the image acquisition unit 151 acquires multiple images from the sensor unit 110, the multiple images are input to the traffic light identification unit 152 in the order in which they were captured.

[0049] (Step S202) In step S202, similar to step S102 in FIG. 3, the position acquisition unit 153 acquires current position information of a running train.

[0050] (Step S203) In step S203, the database reading unit 157 inputs the train's current position information into the traffic light database 154, reads information about the traffic light that the train closest to the train's position information will pass before, and transmits it to the comparison unit 1523.

[0051] (Step S204) In step S204, the traffic light candidate detection unit 1521 detects all traffic light candidates 310 contained in the image that was input earliest among the images input in sequence and that have not been subjected to the traffic light detection process.

[0052] (Step S205) In step S205, the traffic light candidate tracking unit 1522 tracks the traffic light candidate 310 and calculates the movement trajectory 330.

[0053] (Step S206) In step S206, the comparison unit 1523 compares the movement trajectory of the traffic light as seen from the traveling train, which is included in the information about traffic lights stored in the traffic light database 154, with the calculated movement trajectory 330 to determine whether they match. If the comparison shows that the movement trajectory of the traffic light as seen from the traveling train matches the calculated movement trajectory 330, the process proceeds to S207; if they do not match, the process proceeds to S208.

[0054] (Step S207) In step S207, the comparison unit 1523 identifies the traffic light candidate 310 on the movement trajectory 330 as a traffic light.

[0055] (Step S208) In step S208, the traffic light identification unit 152 determines whether there are any other images that have not been subjected to the traffic light detection process among the images input by the image acquisition unit 151. If there are any other images, the process returns to S102; if there are no other images, the process ends.

[0056] <Actions and Effects> The above describes the traffic light detection system 1 according to the embodiment. The traffic light detection system 1 of the present disclosure mainly includes a sensor unit 110, a traffic light candidate detection unit 1521, a traffic light candidate tracking unit 1522, a comparison unit 1523, and a traffic light database 154. The system can create a database of information about traffic lights based on an image of a traveling train in the direction of travel, and can detect traffic lights using the database. This allows for automatic creation of a database of information about traffic lights by traveling along the track, even in cases where there is no database that accurately records the locations of traffic lights. The database can be used to detect traffic lights and determine the signal aspects with high accuracy. Furthermore, even when traffic light recognition accuracy is reduced, such as at night, the traffic light recognition accuracy can be improved by using the traffic light database 154 created during daytime travel, when traffic light recognition accuracy is high.

[0057] Second Embodiment Next, a traffic light detection system 2 according to a second embodiment will be described with reference to FIG. 6 . FIG. 6 illustrates an example of the traffic light detection system 2. The traffic light detection system 2 combines multiple images acquired by the sensor unit 110 with an appearance area based on the positional relationship between tracks and traffic lights stored in the traffic light database 154. When a traffic light candidate is detected within the appearance area, the traffic light detection system 2 identifies the traffic light candidate as a traffic light. The traffic light detection system 2 according to this embodiment differs from the first embodiment in that software for causing the calculation unit 130 to function as a traffic light identification unit 252 is installed in the storage unit 150. The traffic light detection system 2 according to this embodiment also differs from the first embodiment in that the traffic light identification unit 252 includes a track detection unit 2524, but does not include a traffic light candidate tracking unit 1522 or a comparison unit 1523. In the following description, components identical or equivalent to those in the first embodiment described above are denoted by the same reference numerals, and their description will be simplified or omitted.

[0058] <<<Railway Detection Unit>>> The railway detection unit 2524 first detects the railways in the image captured by the sensor unit 110, calculates the appearance area 320 of the traffic light based on distance information from the railway along which the train runs to the traffic light, which information is stored in the traffic light database 154, and transmits the appearance area information to the traffic light candidate detection unit 1521. At this time, the distance information from the railway along which the train runs to the traffic light, which information is stored in the traffic light database 154, is the height of the traffic light from the railway and the distance from the railway. Furthermore, the traffic light candidate detection unit 1521 combines the appearance area information transmitted from the railway detection unit 2524 with the image captured by the sensor unit 110, and detects traffic light candidates included in the combined image.

[0059] <Railway-Based Traffic Light Detection Processing> Next, rail-based traffic light detection processing will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is an example of a screen showing the results of rail-based traffic light detection processing.

[0060] In the railway-based traffic light detection process, as shown in Fig. 7, the traffic light detection result is displayed on the image captured by the sensor unit 110. Here, the traffic light detection result is displayed as a traffic light candidate 310 shown inside an appearance area 320. Since the appearance area 320 is based on the distance from the railway, it is shown as an area surrounded by a dashed line in Fig. 7.

[0061] 8 is a flowchart of a traffic light detection process based on tracks. (Step S301) In step S301, similar to step S101 in Fig. 3, the image acquisition unit 151 acquires an image from the sensor unit 110 and inputs the image to the traffic light identification unit 152. At this time, if the image acquisition unit 151 acquires multiple images from the sensor unit 110, the multiple images are input to the traffic light identification unit 152 in the order in which they were captured.

[0062] (Step S302) In step S302, similar to step S102 in FIG. 3, the position acquisition unit 153 acquires current position information of a running train.

[0063] (Step S303) In step S303, the database reading unit 157 inputs the train's train position information (current position) into the signal database 154, reads information about the signal that the train closest to the train position information will pass before, and transmits it to the comparison unit 1523.

[0064] (Step S304) In step S304, the railroad detection unit 2524 detects the railroad 340 included in the image captured by the sensor unit 110.

[0065] (Step S305) In step S305, the traffic light candidate detection unit 1521 calculates the appearance area 320 based on the height of the traffic light from the track 340 detected by the track detection unit 2524 and the distance from the track.

[0066] (Step S306) In step S306, the traffic light candidate detection unit 1521 combines the calculated appearance region information of the appearance region 320 with the captured image, and detects and identifies the traffic light candidate 310 from the appearance region 320 in the combined image.

[0067] (Step S307) In step S307, the traffic light identification unit 152 determines whether there are any other images that have not been subjected to the traffic light detection process among the images input by the image acquisition unit 151. If there are any other images, the process returns to S302; if there are no other images, the process ends.

[0068] <Actions and Effects> The above describes the traffic light detection system 2 according to the second embodiment. The traffic light detection system 2 according to the present disclosure mainly includes a traffic light candidate detection unit 1521, a track detection unit 2524, and a traffic light database 154, and can identify a traffic light based on the traffic light appearance area calculated from information about the distance from the track on which the train travels to the traffic light. As a result, by designating a portion of the image captured by the sensor unit 110 as the traffic light appearance area, the traffic light candidate detection unit 1521 can detect traffic light candidates only in that appearance area, which reduces the load on image processing and improves detection accuracy.

[0069] [Third Embodiment] Next, a traffic light detection system 3 according to a third embodiment will be described with reference to Fig. 9. Fig. 9 shows an example of the traffic light detection system 3. The traffic light detection system 3 according to this embodiment differs from the traffic light identification unit 152 according to the first embodiment in that the traffic light identification unit 352 includes a track detection unit 2524, and differs from the traffic light identification unit 252 according to the second embodiment in that the traffic light identification unit 352 includes a traffic light candidate tracking unit 1522 and a comparison unit 1523. In the following description, components that are the same as or equivalent to those in the first embodiment and the like will be designated by the same reference numerals, and their description will be simplified or omitted.

[0070] In the traffic light detection process, the traffic light detection system 3 performs both traffic light detection process based on movement trajectories and traffic light detection process based on tracks. For example, after detecting a traffic light using traffic light detection process based on movement trajectories, the state of the detected traffic light may not be confirmed, but the traffic light state may be determined again using traffic light detection process based on tracks, and the traffic light state that matches both processes may be determined as the final traffic light state. This allows for improved traffic light detection accuracy by detecting traffic lights using two different methods, traffic light detection process based on movement trajectories and traffic light detection process based on track areas. Furthermore, by performing traffic light detection process based on tracks and then detecting traffic lights using traffic light detection process based on movement trajectories, the appearance area of ​​traffic light candidates is calculated in advance, and traffic light detection process is performed based on the movement trajectories included in the appearance area to detect and identify traffic lights. This allows for improved detection accuracy while reducing the load of image processing related to traffic light detection by performing traffic light detection process based on movement trajectories while focusing on identifying traffic lights only in appearance areas where traffic light candidates are likely to appear.

[0071] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present invention.

[0072] The present invention can also take the following forms. (Aspect 1) A state determination device comprising: a traffic light database storing information about traffic lights included in an image of a traveling train based on the image taken in the direction of travel of the train; a sensor unit attached to the train; and a traffic light identification unit that identifies a traffic light from an image acquired by the sensor unit, wherein the traffic light database has traffic light movement trajectory information calculated from an image of the traffic light included in the image taken in the direction of travel and / or traffic light appearance area information calculated from distance information from the track on which the train is traveling to the traffic light, and the traffic light identification unit refers to the traffic light database to identify a traffic light from the image acquired by the sensor unit and determine the state of the traffic light. (Aspect 2) A state determination device according to Aspect 1, wherein the traffic light identification unit extracts movement trajectories of candidate traffic lights from a plurality of images acquired by the sensor unit, and identifies the traffic light based on a comparison result between the movement trajectory and a movement trajectory of the traffic light stored in the traffic light database. (Aspect 3) A state determination device according to Aspect 1 or 2, wherein the traffic light identification unit combines an appearance area stored in the traffic light database with an image acquired by the sensor unit, and identifies a traffic light candidate as a traffic light when the traffic light candidate is detected within the appearance area. (Aspect 4) A state determination method according to any one of Aspects 1 to 3, comprising: a first step of storing information about traffic lights included in an image in a traffic light database based on an image of a traveling direction of a traveling train; a second step of calculating a movement trajectory of the traffic light candidate from a plurality of images acquired by the sensor unit; a third step of comparing the movement trajectory of the traffic light candidate with a movement trajectory of the traffic light stored in the traffic light database; and a fourth step of identifying the traffic light candidate as a traffic light when the movement trajectory of the traffic light candidate matches the movement trajectory of the traffic light stored in the traffic light database.(Aspect 5) The state determination method according to Aspect 4, wherein the state determination device includes a track detection unit in the traffic light identification unit, and the state determination method comprises, after the third step, a fifth step of combining an appearance area stored in the traffic light database with an image acquired by the sensor unit, and a sixth step of identifying the traffic light candidate as a traffic light when a movement trajectory of the traffic light candidate matches a movement trajectory of the traffic light stored in the traffic light database and the traffic light candidate is detected within the appearance area. (Aspect 6) The state determination method according to any one of Aspects 1 to 3, wherein the state determination device includes a seventh step of combining an appearance area stored in the traffic light database with an image acquired by the sensor unit, and an eighth step of identifying the traffic light candidate as a traffic light when a movement trajectory of the traffic light candidate matches a movement trajectory of the traffic light stored in the traffic light database and the traffic light candidate is detected within the appearance area.

[0073] 1, 2, 3 Traffic light detection system 110 Sensor unit 120 Display unit 130 Calculation unit 140 Memory 150 Storage unit 151 Image acquisition unit 152, 252, 352 Traffic light identification unit 1521 Traffic light candidate detection unit 1522 Traffic light candidate tracking unit 1523 Comparison unit 153 Position acquisition unit 154 Traffic light database 155 Traffic light determination unit 156 Database writing unit 157 Database reading unit 158 ​​Detection result output unit 2524 Track detection unit

Claims

1. A status determination device comprising: a traffic light database storing information about traffic lights included in an image based on an image of a traveling train in the direction of travel; a sensor unit attached to the train; and a traffic light identification unit that identifies a traffic light from an image acquired by the sensor unit, wherein the traffic light database has traffic light movement trajectory information calculated from an image of the traffic light included in the image in the direction of travel and / or traffic light appearance area information calculated from distance information from the track along which the train is traveling to the traffic light, and the traffic light identification unit refers to the traffic light database to identify a traffic light from the image acquired by the sensor unit and determine the status of the traffic light.

2. A status determination device as described in claim 1, wherein the traffic light identification unit extracts a movement trajectory of a candidate traffic light from multiple images acquired by the sensor unit, and identifies the traffic light based on a comparison result between the movement trajectory and a movement trajectory of a traffic light stored in the traffic light database.

3. A status determination device as described in claim 1, wherein the traffic light identification unit combines an appearance area stored in the traffic light database with the image acquired by the sensor unit, and when a traffic light candidate is detected within the appearance area, identifies the traffic light candidate as a traffic light.

4. A state determination method for the state determination device described in claim 1, comprising: a first step of storing information about traffic signals contained in an image in the direction of travel of a traveling train in a traffic light database; a second step of calculating a movement trajectory of a traffic light candidate from multiple images acquired by the sensor unit; a third step of comparing the movement trajectory of the traffic light candidate with the movement trajectory of the traffic light stored in the traffic light database; and a fourth step of identifying the traffic light candidate as a traffic light when the movement trajectory of the traffic light candidate matches the movement trajectory of the traffic light stored in the traffic light database.

5. A state determination method as claimed in claim 4, wherein the state determination device includes a track detection unit in the traffic light identification unit, and the state determination method comprises, after the third step, a fifth step of combining an appearance area stored in the traffic light database with the image acquired by the sensor unit, and a sixth step of identifying the traffic light candidate as a traffic light when the movement trajectory of the traffic light candidate matches the movement trajectory of the traffic light stored in the traffic light database and the traffic light candidate is detected within the appearance area.

Images