Signal recognition device, signal recognition method, and program

Abstract

Provided is a signal recognition device comprising: an image acquisition unit 101 that acquires an image captured by a camera which is disposed on a mobile body; a light-emitting body detection unit 102 that detects a light-emitting body in the acquired image; a self-position estimation unit 103 that estimates the position of the camera; a signal apparatus determination unit 108 that identifies a light of a signal apparatus from the detected light-emitting body on the basis of the estimated position of the camera; and a signal indication recognition unit 109 that reads the indication of the signal apparatus from the identified light. Thus, it is possible to read the indication of a signal apparatus not only in bright areas, but also in dark areas.

Description

[Technical Field]

[0001] The present invention relates to a signal recognition device, a signal recognition method, and a program, and more particularly to a signal recognition device suitable for recognizing railway signals. [Background technology]

[0002] For example, driverless vehicle operation is attracting attention in transportation businesses such as railways as a means of reducing costs and improving services. Railway operators incur significant personnel and training costs for drivers, and so they have high hopes for reducing driver costs through driverless operations. In terms of services, the elimination of driver allocation constraints will increase operational freedom and improve user convenience through more frequent operations. Driverless railway operation has already been realized on subway and elevated railway lines that do not have level crossings. However, there are no practical examples of this on lines where people may be present, such as lines with level crossings or lines in depots where workers are present. To realize driverless operation on such lines, a means is needed to detect traffic lights that issue instructions to drivers, such as stopping or speed limits, and to recognize the signal aspect. For this reason, it is considered to detect traffic lights and recognize the signal aspect using image processing based on images captured by cameras. General object recognition technology is a technology that recognizes and detects objects using image processing. When using image processing to recognize an object, the clearer the shape of the object is in the image, the easier it is to recognize it.

[0003] Patent Document 1 describes an automatic detection device. The automatic detection device acquires road images, vehicle position data at the time of image capture, and vehicle attitude data at the time of image capture, and then acquires information data on target facilities, which are traffic safety facilities located in the vehicle's direction of travel at the time of image capture, from an information database based on the vehicle position data and vehicle attitude data. The automatic detection device extracts a limited range from the road image that captures the target facility based on the vehicle position data, vehicle attitude data, position data included in the target facility information data, and size data included in the target facility information data. The automatic detection device detects the target facility by comparing the limited range with feature data included in the target facility information data. Patent Document 2 describes a traffic signal recognition method, which acquires a captured image using a vehicle camera, searches for a target traffic light facing the vehicle from among multiple traffic lights, calculates an area including the target traffic light as an ROI from the captured image based on map information, the vehicle's state, and the camera's state, calculates a prior probability distribution indicating the probability of the target traffic light being present at each position within the ROI, generates a contrast-updated image by updating the contrast of the ROI image according to the prior probability distribution, identifies the area within the ROI where the target traffic light is lit as an lit area by extracting feature points from the contrast-updated image, and recognizes the lit color of the target traffic light based on the color of the identified lit area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-76884 [Patent Document 2] Japanese Patent Application Publication No. 2018-63580 Summary of the Invention [Problem to be solved by the invention]

[0005] To safely slow down or stop a vehicle, it is necessary to apply the brakes at an early timing, and to do so, it is desirable to be able to detect and recognize traffic lights at a long distance, not only in bright places but also in dark places such as at night or in tunnels. However, traffic lights are difficult to recognize in dark places because their shapes are not clearly visible. On the other hand, traffic light illuminants are light sources, so although their position can be recognized at night, it is difficult to recognize their size. In addition, a mechanism is needed to distinguish traffic light illuminants from other light sources. An object of the present invention is to provide a signal recognition device, a signal recognition method, and a program that can read the indications of a traffic light not only in bright places but also in dark places. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention provides a traffic light recognition device that includes an image acquisition unit that acquires an image captured by a photographing device disposed on a moving body, an illuminant detection unit that detects illuminants from the acquired image, a position estimation unit that estimates the position of the photographing device, an identification unit that identifies traffic light signals from the detected illuminants based on the estimated position of the photographing device, and a reading unit that reads the traffic light indication from the identified light signals.

[0007] The present invention also provides a signal recognition method that acquires an image captured by a camera installed on a moving object, detects light-emitting objects from the acquired image, estimates the position of the camera, identifies traffic light signals from the detected light-emitting objects based on the estimated position of the camera, and reads the traffic light indication from the identified light signals.

[0008] Furthermore, the present invention provides a program for enabling a computer to realize an image acquisition function for acquiring an image captured by a photographing device disposed on a moving object, an illuminant detection function for detecting an illuminant from the acquired image, a position estimation function for estimating the position of the photographing device, an identification function for identifying a traffic light from the detected illuminants based on the estimated position of the photographing device, and a reading function for reading the traffic light indication from the identified light. [Effects of the Invention]

[0009] According to the invention described in claim 1, it is possible to provide a signal recognition device that can read the indications of a traffic light not only in bright places but also in dark places. According to the invention described in claim 2, even if a traffic light indicates a signal using multiple lights, it can be identified as belonging to a single traffic light. According to the inventions set forth in claims 3 to 5, it is easier to identify that a plurality of lights belong to one traffic light. According to the invention of claim 6, it is possible to identify that a plurality of lights belong to one traffic light without using the distance between the image capturing device and the traffic light. According to the invention described in claim 7, traffic light can be identified from among the photographed light-emitting objects. According to the inventions of claims 8 and 9, even if there is a measurement error in the position or direction of the photographing device or traffic light, the traffic light can be identified more accurately. According to the inventions set forth in claims 10 and 11, traffic light signals can be identified more accurately. According to the invention described in claim 12, it is possible to read the traffic light indication even if multiple lamps are lit or flashing on one traffic light. According to the invention described in claim 13, it becomes easier to read the indications on traffic signals. According to the invention described in claim 14, it is possible to provide a signal recognition method that can read the indications of a traffic light not only in bright places but also in dark places. According to the invention described in claim 15, a function that can provide a signal recognition device that can read the indications of traffic lights not only in bright places but also in dark places can be realized by a computer. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing the hardware configuration of a traffic light detection system according to an embodiment of the present invention; [Figure 2] 1 is a block diagram showing the functional configuration of a traffic light detection system according to an embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of a traffic light determination unit. [Figure 4] 10 is a flowchart illustrating the operation of the traffic light detection system. [Figure 5] FIG. 5 is a diagram showing a specific example of the process performed in step S409 of FIG. [Figure 6] 10(a) and 10(b) are diagrams showing a specific example of the process performed in step S412. [Figure 7] FIG. 10 is a diagram showing a flow in which a signal aspect recognition unit recognizes the aspect of a traffic light. [Figure 8] (a) is a diagram showing the horizontal pattern of the light, and (b) is a diagram showing the vertical pattern of the light. [Figure 9] 10(a) to 10(f) are diagrams showing examples of lighting patterns of the lights. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Here, we will explain the traffic light detection system 1 as an example. The system is attached to a railway vehicle, receives images captured by a camera monitoring the road ahead, detects traffic lights that the train itself should read, and recognizes the traffic light aspects.

[0012] <Overall explanation of traffic light detection system 1> FIG. 1 is a diagram showing the hardware configuration of a traffic light detection system 1 according to the present embodiment. The illustrated traffic light detection system 1 is an example of a traffic light recognition device that recognizes the aspects of traffic lights. The traffic light detection system 1 includes a processor 11 that controls each component through the execution of a program, an output unit 12 that outputs images and other information, an input unit 13 that inputs characters and the like, a camera 14 that captures images, a communication module 15 used for communication with external devices, an internal memory 16 that stores system data and internal data, and an external memory 17 as an auxiliary storage device.

[0013] The processor 11 executes programs such as an OS (operating system), application software, etc. The processor 11 is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). In this embodiment, the internal memory 16 is a semiconductor memory. The internal memory 16 has a ROM (Read Only Memory) in which a BIOS (Basic Input Output System) and the like are stored, and a RAM (Random Access Memory) used as a main storage device. The processor 11 and the internal memory 16 constitute a computer. The processor 11 uses the RAM as a workspace for programs. The external memory 17 is a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) in which firmware, application software, and the like are stored.

[0014] The output unit 12 is, for example, an output device such as a liquid crystal display or an organic EL (Electro Luminescent) display. In this case, images and other information are displayed on the output unit 12. The output unit 12 may also be a speaker or the like, which outputs sounds such as voices and notification sounds.

[0015] The input unit 13 is an input device used by the user to input characters and the like, and is, for example, a keyboard. The input unit 13 is also an input device used to move a cursor displayed on the output unit 12, scroll the screen, and the like, and is, for example, a touchpad. Note that a mouse, a trackball, or the like may be used instead of the touchpad. The input unit 13 may also be a device such as a button or a lever. The camera 14 is an example of an imaging device and captures video or still images. In this case, the camera 14 is installed in the driver's seat or the like of the vehicle and captures the scenery ahead in the direction of travel of the vehicle. The camera 14 includes an optical system that converges light from an object to be photographed, and an image sensor that detects the light converged by the optical system. The optical system is composed of a single lens or a combination of multiple lenses. For example, a twin lens is used, in which two hemispherical lenses are used with their spherical sides facing each other. The image sensor is composed of an array of CCDs (Charge Coupled Devices), CMOSs ​​(Complementary Metal Oxide Semiconductors), etc. The communication module 15 is a communication interface for communicating with the outside.

[0016] FIG. 2 is a block diagram showing the functional configuration of the traffic light detection system 1 according to this embodiment. The traffic light detection system 1 shown in the figure includes an image acquisition unit 101, a light-emitting element detection unit 102, a self-position estimation unit 103, a traffic light position acquisition unit 104, a light-emitting element tracking unit 105, a traffic light information acquisition unit 106, a trajectory detection unit 107, a traffic light determination unit 108, a traffic light aspect recognition unit 109, and a result output unit 110.

[0017] The image acquisition unit 101 acquires images taken by a camera 14 arranged on the vehicle. The light-emitting object detection unit 102 detects light-emitting objects from within the image acquired by the image acquisition unit 101. Light-emitting objects can be detected using object detection technology based on image processing, a method of extracting pixels with high brightness within the image, or the like. The self-position estimation unit 103 is an example of a position estimation unit, and estimates the position of the camera 14. Here, "position" refers to the position of the camera 14 on the Earth. Note that this position does not necessarily have to be on the Earth's surface, but may be near the Earth's surface. That is, it may be a position on an elevated road, on a bridge pier, inside a tunnel, underground, or the like. Specifically, the self-position estimation unit 103 acquires the position coordinates of the camera 14 on a map from a GPS (Global Positioning System), a trip meter, or the like, and estimates this position.

[0018] The traffic light position acquisition unit 104 acquires the position coordinates of the traffic light to be read next by making an inquiry to a map information DB (database), etc. For example, high-precision three-dimensional map data (HD map) can be used as the map. The light-emitting object tracking unit 105 stores the position of the light-emitting object detected by the light-emitting object detection unit 102 in the image for each image frame, tracks the light-emitting object over multiple consecutive frames, and records its movement trajectory. The signal information acquisition unit 106 acquires information on the shape of the traffic light to be read and the possible display patterns by, for example, querying the map information DB. In other words, the lighting patterns of each traffic light are stored in the map information DB, and the signal information acquisition unit 106 acquires this information. The "lighting pattern" refers to information on how the lights appear, such as the color of the signal lights, whether they flash, the flashing frequency, and, if multiple lights are flashing, the spacing and arrangement of the lights. Note that the possible display patterns may also be acquired or estimated based on operation information for the line section on which the train runs.

[0019] The trajectory detection unit 107 detects the trajectory in the image by image processing. The traffic light determination unit 108 determines which of the light-emitting objects detected by the light-emitting object detection unit 102 is a light-emitting object (light) that belongs to the traffic light that should be read. At this time, the traffic light determination unit 108 uses at least the estimated position of the camera 14 as information acquired by the self-position estimation unit 103 to the trajectory detection unit 107. The traffic light determination unit 108 functions as an identification unit that identifies the light of a traffic light from among the detected light-emitting objects based on the estimated position of the camera 14. At this time, the traffic light determination unit 108 can use the camera parameters of the camera 14. The camera parameters include external parameters that convert world coordinates into camera coordinates and internal parameters that convert camera coordinates into image coordinates. The signal aspect recognition unit 109 compares the patterns of the light emitting body (light), such as color, arrangement, and flashing state, determined by the traffic light determination unit 108 with the possible traffic light aspects acquired by the signal information acquisition unit 106, and recognizes the pattern that matches as the aspect of the signal. The signal aspect recognition unit 109 functions as a reading unit that reads the traffic light aspect from the identified light. The result output unit 110 outputs the aspect recognition result, which is the result of the signal aspect recognized by the signal aspect recognition unit 109.

[0020] Next, the traffic light determination unit 108 will be described in detail. FIG. 3 is a block diagram showing the functional configuration of the traffic light determination unit 108. The traffic light determination unit 108 includes a self-position acquisition unit 201, a traffic light position acquisition unit 202, a traffic light information storage unit 203, an in-image signal light movement trajectory estimation unit 204, an in-image signal light interval estimation unit 205, a traffic light flashing frequency estimation unit 206, a light-emitting object tracking result storage unit 207, a signal light frequency measurement unit 208, and a signal light / light-emitting object matching unit 209.

[0021] The self-position acquisition unit 201 acquires position coordinates (longitude, latitude, altitude) that change from moment to moment from the self-position estimation unit 103, and stores a certain number of frames of images captured by the camera 14. The traffic light position acquisition unit 202 acquires from the traffic light position acquisition unit 104 the position coordinates (latitude, longitude, altitude) of traffic lights that may appear in the image, and stores them. The traffic light information storage unit 203 acquires from the traffic light information acquisition unit 106 information on the shape of the traffic light and possible indication patterns for each traffic light stored in the traffic light position acquisition unit 202, and stores the information.

[0022] The in-image signal light movement trajectory estimation unit 204 estimates the movement trajectory of the traffic light in the image for a certain number of frames based on the self-position coordinates for a certain number of frames stored in the self-position acquisition unit 201 and the traffic light position coordinates stored in the traffic light position acquisition unit 202. The in-image signal light interval estimation unit 205 estimates the interval between lights in an image when one traffic light has two or more lights turned on, based on the self-position coordinates for a certain number of frames stored in the self-position acquisition unit 201, the position coordinates of the traffic light stored in the signal position acquisition unit 202, and the signal information stored in the signal information storage unit 203. The signal light blinking frequency estimation unit 206 estimates the frequency component of the blinking of the light based on the signal information acquired by the signal information storage unit 203. At this time, the signal information used to estimate the frequency component may include the cycle at which the light blinks during the signal phase, the flicker frequency when the signal light is an LED (Light-Emitting Diode), and the like.

[0023] The light-emitting object tracking result storage unit 207 acquires and stores the movement trajectories of all light-emitting objects in the images recorded by the light-emitting object tracking unit 105 for a certain number of frames. The signal light frequency measurement unit 208 measures the frequency components of the light brightness for each of all light-emitting objects stored in the light-emitting object tracking result storage unit 207. To measure the frequency components of the light brightness, a possible method is to perform a Fourier transform on the luminance value of a specific color component that is close to the color of the light. The signal light / light-emitting object matching unit 209 compares the estimated values ​​of how a traffic light appears on the screen, stored in the intra-image signal light movement trajectory estimation unit 204, intra-image signal light interval estimation unit 205, and signal light flashing frequency estimation unit 206, with the appearance of each light-emitting object on the screen, stored in the light-emitting object tracking result storage unit 207 and signal light frequency measurement unit 208. Then, the signal light / light-emitting object matching unit 209 calculates the similarity between the two, selects the light-emitting object that appears most similar to a traffic light, and outputs this as the traffic light detection result.

[0024] <Explanation of the operation of traffic light detection system 1> Next, the operation of the traffic light detection system 1 will be described. FIG. 4 is a flowchart illustrating the operation of the traffic light detection system 1. First, the image acquisition unit 101 acquires an image from the camera 14 (step S401). Next, the light-emitting body detection unit 102 detects light-emitting bodies from the image acquired by the image acquisition unit 101 (step S402). Furthermore, the self-position estimation unit 103 estimates the position of the camera 14 (step S403). Furthermore, the traffic light position acquisition unit 104 acquires the position coordinates of the traffic light to be read (step S404). On the other hand, the light-emitting object tracking unit 105 tracks the position of the light-emitting object detected by the light-emitting object detection unit 102 within the image, and records the movement trajectory (step S405).

[0025] Furthermore, the traffic light information acquisition unit 106 acquires information on the shape of the traffic light to be read and information on the possible indication patterns (step S406). Furthermore, the traffic light determination unit 108 acquires the relative speed between the camera 14 and the traffic light to be read (step S407). This can be obtained from the change between the position of the camera 14 estimated in step S403 and the position coordinates of the traffic light acquired in step S404. Furthermore, the traffic light determination unit 108 acquires the relative position between the camera 14 and the traffic light to be read (step S408). This can be obtained from the position of the camera 14 estimated in step S403 and the position coordinates of the traffic light acquired in step S404. The traffic light determination unit 108 can obtain the distance between the two from this relative position. Within the traffic light determination unit 108, a self-position acquisition unit 201 acquires the position of the camera 14. Furthermore, a traffic light position acquisition unit 202 acquires the position coordinates of the traffic light. Then, the above-mentioned relative speed and the above-mentioned relative position are calculated.

[0026] Next, the traffic light determination unit 108 determines whether or not all of the light-emitting objects detected in step S402 match the expected traffic light movement (step S409). This can be determined based on the movement trajectory recorded by the light-emitting object tracking unit 105. Furthermore, the traffic light movement can be determined based on the relative speed determined in step S407 and the relative position determined in step S408, as will be described in detail later. Inside the traffic light determination unit 108, the intra-image signal light movement trajectory estimation unit 204 performs the process of step S409 based on the movement trajectory stored in the light-emitting object tracking result storage unit 207.

[0027] As a result, if there is a matching light-emitting object (Yes in step S410), the traffic light determination unit 108 determines whether there are two or more matching light-emitting objects (step S411). If there are two or more matching light-emitting elements (Yes in step S411), the traffic light determination unit 108 determines whether there is a pair of light-emitting elements that matches the expected light spacing (step S412). This determination is made based on the traffic light shape information acquired in step S406, as will be described in detail later. At this time, the determination can also be made based on the display pattern information acquired in step S406. The display pattern is at least one of the light flashing cycle, light color, and light combination. Inside the traffic light determination unit 108, based on the information on the shape of the traffic light and the possible display patterns stored in the traffic light information storage unit 203, the in-image signal light interval estimation unit 205, the signal light flashing frequency estimation unit 206, the signal light frequency measurement unit 208 and the signal light / light-emitting object matching unit 209 perform the processing of steps S411 to S412.

[0028] As a result, if there is a pair of light-emitting elements that matches the expected light spacing (Yes in step S412), or if there is only one matching light-emitting element but not two or more (No in step S410), the signal aspect recognition unit 109 recognizes the traffic light aspect. That is, the signal aspect recognition unit 109 reads the traffic light aspect from the identified light. Then, the result output unit 110 outputs the aspect recognition result (step S413). Then, the process proceeds to step S415.

[0029] Also, if there is no matching light-emitting object in step S410 (No in step S410), and if there is no set of light-emitting objects that matches the expected light spacing in step S412 (No in step S412), the traffic light determination unit 108 determines that traffic light detection has failed (step S414). Furthermore, the image acquisition unit 101 determines whether or not the next image is input (step S415). If the next image is input (Yes in step S15), the process returns to step S401. On the other hand, if the next image is not input (No in step S415), the series of traffic light detection processes ends.

[0030] <Detailed explanation of step S409> Next, step S409 in FIG. 4 will be described in detail. FIG. 5 is a diagram showing a specific example of the process performed in step S409 in FIG. Here, a case will be described in which light emitting bodies H1 to H4 are photographed in an image and traffic light is extracted from among them. The traffic light determination unit 108 identifies the lights of the traffic light based on the movements of the light emitters H1 to H4 in the image. Specifically, the following methods can be used.

[0031] First, there is a method of focusing on the direction of movement of the light emitting bodies H1 to H4 in the image as the movement of the light emitting bodies H1 to H4. In this method, the traffic light determination unit 108 estimates the direction of movement, which is the direction in which the traffic light's light moves in the image, and identifies the light based on this direction of movement. In other words, as the vehicle travels, the relative position between the camera 14 and the traffic light changes. At this time, the traffic light moves in the image in accordance with the change in relative position. This direction of movement can be estimated from the change in relative position. In other words, the traffic light determination unit 108 estimates the direction of movement based on the change in the relative position between the camera 14 and the traffic light.

[0032] In FIG. 5, light-emitting objects H1 to H4 are captured in the image. In this case, the traffic light determination unit 108 determines the movement direction of the light-emitting objects H1 to H4 in the image based on the movement trajectories stored in the light-emitting object tracking result storage unit 207. In FIG. 5, this movement direction is indicated by a dotted arrow. Meanwhile, the traffic light determination unit 108 estimates the movement direction of the traffic light lights in the image based on the relative positions of the camera 14 and the traffic light. Then, if the movement directions of the two match, the traffic light determination unit 108 identifies the light-emitting objects H1 to H4 with the matching movement direction as the traffic light. That is, the traffic light determination unit 108 identifies the traffic light lights based on the estimated movement direction and the movement direction of the light-emitting objects H1 to H4 as the movement of the light-emitting objects H1 to H4.

[0033] As another method, the traffic light determination unit 108 can estimate the moving speed, which is the speed at which the light moves in the image, based on the relative position and the vehicle speed, and identify the light based on this moving speed. That is, as described above, the relative position between the camera 14 and the traffic light changes as the vehicle travels. At this time, the traffic light moves at a predetermined speed in the image in accordance with the change in relative position and the vehicle speed. This moving speed can be estimated from the change in relative position and the vehicle speed. The vehicle moving speed can be obtained by using GPS and trip meter information acquired by the self-position estimation unit 103.

[0034] In this case, the traffic light determination unit 108 determines the movement speeds of the light emitters H1 to H4 in the image of Fig. 5 based on the movement trajectories and vehicle speeds stored in the light emitter tracking result storage unit 207. Meanwhile, the traffic light determination unit 108 estimates the movement speeds of the traffic light lights in the image based on the relative positions of the camera 14 and the traffic light and the vehicle speed. Then, if the movement directions of the two match, the traffic light determination unit 108 identifies the light emitter with the matching movement speed as the traffic light. That is, the traffic light determination unit 108 identifies the traffic light light based on the estimated speed direction and the movement speeds of the light emitters as the movement of the light emitters H1 to H4.

[0035] Furthermore, the traffic light determination unit 108 may identify a traffic light by further taking into account at least one of the cycle of the flashing of the light, the interval between the lights, and the color of the light. In other words, the cycle of the flashing of the light is fixed for each traffic light. This allows the traffic light determination unit 108 to distinguish between the light within the traffic light and other light-emitting objects. Furthermore, the interval between the lights within a traffic light is fixed. Therefore, the traffic light determination unit 108 can distinguish between the light within the traffic light and other light-emitting objects based on the interval between the light-emitting objects in the image. Furthermore, the color of the light is fixed for each traffic light. This allows the traffic light determination unit 108 to distinguish between the light of the traffic light and other light-emitting objects. The traffic light determination unit 108 then uses these, so to speak, as an auxiliary factor in distinguishing between the light of the traffic light and other light-emitting objects. This improves the reliability of identifying the light of the traffic light. The traffic light determination unit 108 can also identify the lights of a traffic light by further taking into account the direction in which the camera 14 takes an image. In other words, by using information on the direction in which the camera 14 takes an image, the position of the traffic light in the image can be narrowed down.

[0036] <Detailed explanation of steps S411 to S412> Next, the processing in steps S411 and S412 in FIG. 4 will be specifically described. In step S409, there may be a plurality of light-emitting objects identified as traffic light. That is, the traffic light determination unit 108 may detect a plurality of traffic light lights. Furthermore, in a railway traffic light, a single traffic light may have a plurality of lights turned on or flashing. Therefore, in such a case, the traffic light determination unit 108 may identify a plurality of light-emitting objects as traffic light lights. In steps S411 and S412 in FIG. 4, the traffic light determination unit 108 determines whether or not a plurality of lights belong to one traffic light.

[0037] 6(a) and 6(b) are diagrams showing a specific example of the process performed in step S412. FIG. 6(a) shows a case where a traffic light Sg1 is photographed and lights L1 and L4 that are turned on at the traffic light Sg1 are identified. This traffic light Sg1 is a five-light traffic light with five lights that are turned on or flashing. FIG. 6(a) shows a case where the first light L1 from the top and the fourth light L4 from the top are turned on, and the other lights L2, L3, and L5 are turned off. It is also assumed here that lights L1 and L4 are photographed at a distance W apart in the image. In the figure, a circle indicates that a light is turned on, and a black circle indicates that a light is turned off.

[0038] The traffic light determination unit 108 identifies multiple lights as belonging to one traffic light based on the distance between the camera 14 and the traffic light. Specifically, the traffic light determination unit 108 estimates the spacing between the multiple lights in the image based on this distance. Then, based on the estimated spacing, it identifies the multiple lights as belonging to one traffic light. That is, the traffic light determination unit 108 estimates the spacing W in the image between lights L1 and L4 in FIG. 6(a). Then, if the spacing between the light-emitting objects identified as lights matches the estimated spacing W, the traffic light determination unit 108 identifies the multiple lights as belonging to one traffic light. FIG. 6(b) is a diagram illustrating this method. Here, the case where light-emitting elements H5 to H9 are photographed is shown. The distance between light-emitting elements H5 and H6 is W, and light-emitting elements H5 and H6 are identified as lights L1 and L4. The direction in which lights L1 and L4 are arranged may also be taken into consideration. In this case, lights L1 to L5 are arranged in the vertical direction. Therefore, light-emitting elements H5 and H6, which are arranged in the same direction and spaced apart by W, are identified as lights L1 and L4.

[0039] Alternatively, the traffic light determination unit 108 can estimate the size of the traffic light captured in the image based on the distance, and identify multiple lights as belonging to a single traffic light based on the estimated size. This method is also shown in Figure 6(b). Here, the image shows a case where light-emitting objects H5 to H9 are captured. The image also shows a case where light-emitting objects H5 and H6 are identified as traffic light lights L1 and L4. At this time, the traffic light determination unit 108 estimates the size of the traffic light captured in the image based on the distance. In FIG. 6(b), the estimated size of the traffic light in the image is shown by a dotted rectangle. In FIG. 6(b), the traffic light determination unit 108 selects lights L1 and L4 as a pair of lights that match the estimated size. This allows the traffic light determination unit 108 to identify that lights L1 and L4 belong to a single traffic light.

[0040] There are cases where two traffic lights are located close to each other and their lights are close to each other on the image. However, even in this case, the lights of each traffic light can be distinguished by the method described in Figure 5 for determining the direction and speed of movement of the light source. In other words, since the two traffic lights are located in different positions, the direction and speed of movement of their lights are different. Therefore, they can be distinguished from each other. Then, the combination of lights belonging to each traffic light can be determined by the method shown in Figure 6.

[0041] Furthermore, in the above example, map information obtained from a map information DB is used, but this is not necessarily required. In this case, the traffic light determination unit 108 can identify multiple lights as belonging to one traffic light based on the width of the track (e.g., rail) R (see FIG. 5). In this case, the traffic light determination unit 108 uses information such as the track width and the traffic light height to estimate the size at which the traffic light is captured in the image. Then, based on the estimated size, the unit identifies multiple lights as belonging to one traffic light. The track width is detected by the track detection unit 107.

[0042] <Detailed explanation of step S413> Next, a specific description will be given of the process of step S413 in Fig. 4. In step S413 in Fig. 4, the signal aspect recognition unit 109 recognizes the aspect of the traffic light. FIG. 7 is a diagram showing a flow in which the signal aspect recognition unit 109 recognizes the aspect of a traffic light. First, the traffic light determination unit 108 detects the light of a traffic light as described above (step S701). Then, the signal aspect recognition unit 109 detects the horizontal and vertical patterns of the lights (steps S702 and S703). Furthermore, the signal aspect recognition unit 109 determines the color of the light (step S704). Furthermore, the signal aspect recognition unit 109 determines the position of the light (step S705). In this way, the signal aspect recognition unit 109 reads the signal aspect of the traffic light from the combination of multiple lights.

[0043] FIG. 8(a) is a diagram showing the horizontal pattern of lights. In the diagram, "◯" indicates that the light is on, and "●" indicates that the light is off. A "horizontal pattern" is a horizontal pattern in which the lights are on or flashing. The traffic lights Sg2 to Sg4 shown in the diagram have a pattern in which two lights are on in the horizontal direction. In step S202, the signal aspect recognition unit 109 detects this lighting pattern. 8(b) is a diagram showing the vertical pattern of the lights. A "vertical pattern" is a vertical pattern in which the lights are lit or flashing. The traffic lights Sg5 to Sg9 shown in the figure have a pattern in which two lights are lit in the vertical direction.

[0044] In step S413, the signal aspect recognition unit 109 detects the horizontal and vertical patterns of the arrangement of lit or flashing lights as a combination of lit or flashing lights. That is, the signal aspect recognition unit 109 detects the horizontal and vertical patterns of the lit or flashing lights. The signal aspect recognition unit 109 then reads the aspect of the traffic light from these patterns. In this way, the signal aspect recognition unit 109 can interpret the number of lit or flashing lights and the combination of the light positions to read the aspect of the traffic light.

[0045] 8(a)-(b) show a pattern in which two lights are turned on, but the number of lights that are turned on may be one, or three or more. Also, not only turned-on lights but also blinking lights can be detected. The light position is determined, for example, as left / right in a horizontal pattern and top / middle / bottom in a vertical pattern. As a result, the signal aspect recognition unit 109 distinguishes between two lights located on the left and right in a horizontal pattern. Also, the signal aspect recognition unit 109 distinguishes between three lights located on the top / middle / bottom in a vertical pattern. When four or more lights are lit, the lighting pattern is determined using the light color or a combination of horizontal and vertical patterns.

[0046] 9(a) to 9(f) are diagrams showing examples of light illumination patterns that can be obtained in the above manner. In the diagrams, "◯" indicates that the light is on, "●" indicates that the light is off, and "☆" indicates that the light is flashing. The illustrated example shows the lighting patterns of the lights for traffic lights Sg11 to Sg15. Figures 9(a) to (c) show the lighting patterns of lights when passage is prohibited. The lighting pattern in Figure 9(a) shows a case where traffic lights Sg11 to Sg14 display a signal aspect by lighting red. In this case, traffic light Sg11 turns on lamps L112 and L113 and turns off the others. Traffic light Sg12 turns on lamps L121 and L124 and turns off the others. Traffic light Sg13 turns on lamps L132 and L135 and turns off the others. Traffic light Sg14 turns on lamps L142 and L145 and turns off the others.

[0047] The lighting pattern in Figure 9(b) shows a case where traffic lights Sg11 to Sg14 are illuminated in purple to indicate the signal aspect. In this case, traffic light Sg11 lights up lamp L113 and turns off the others. Traffic light Sg12 lights up lamp L121 and turns off the others. Traffic light Sg13 lights up lamp L131 and turns off the others. Traffic light Sg14 lights up lamp L141 and turns off the others.

[0048] In the lighting pattern of FIG. 9(c), for traffic light Sg15, lamp L152 is lit in yellow, lamp L156 is lit in red, and the others are turned off.

[0049] FIG. 9(d) shows the lighting pattern for when a train is forced to stop. In this case, the train may continue on after stopping temporarily. This lighting pattern has the purpose of, for example, maintaining a sufficient distance between trains. The lighting pattern in FIG. 9(d) shows a case where traffic lights Sg11 to Sg14 are illuminated in red to indicate a signal aspect. In this case, traffic light Sg11 turns on lamp L112 and the others turn off. Traffic light Sg12 turns on lamp L124 and the others turn off. Traffic light Sg13 turns on lamp L135 and the others turn off. Traffic light Sg14 turns on lamp L145 and the others turn off.

[0050] FIG. 9(e) shows the lighting pattern of the light in the case of a warning. The lighting pattern in Figure 9(e) shows a case where traffic lights Sg11 to Sg15 display their signal aspects by lighting their lights yellow. In this case, traffic light Sg11 lights lamp L113 and turns off the others. Traffic light Sg12 lights lamp L125 and turns off the others. Traffic light Sg13 lights lamp L136 and turns off the others. Traffic light Sg14 lights lamp L146 and turns off the others. Traffic light Sg15 lights lamp L154 and turns off the others.

[0051] FIG. 9(f) shows the lighting pattern of the light in the case of a preliminary warning. The lighting pattern in Figure 9(f) shows a case where traffic lights Sg11 to Sg15 display their signal aspects by flashing yellow. In this case, traffic light Sg11 flashes lamp L113 and turns off the others. Traffic light Sg12 flashes lamp L125 and turns off the others. Traffic light Sg13 flashes lamp L136 and turns off the others. Traffic light Sg14 flashes lamp L146 and turns off the others. Traffic light Sg15 flashes lamp L154 and turns off the others.

[0052] <Explanation of effect> According to the traffic light detection system 1 described above, the following effects can be obtained. To realize driverless railways, it is necessary to detect traffic signals and recognize their aspects. To achieve this, this embodiment uses image recognition technology. Image recognition technology has advantages such as low cost of sensors (in this case, for example, cameras 14) and no need to modify the detection target, making it inexpensive to implement. In such cases, it is difficult to recognize traffic signals in dark places such as tunnels or at night. However, this embodiment recognizes the lights of traffic signals, which are illuminants, and determines the aspect through image classification. Furthermore, this embodiment solves the problem of erroneous recognition due to the presence of illuminants such as headlights and streetlights in the railway environment. It identifies traffic signal lights from among the illuminants. Furthermore, even when multiple traffic signals are present in an image, it is possible to distinguish between them and select the signal whose aspect should be read.

[0053] Furthermore, the indications of railway traffic signals have different characteristics from those of automobile signals, as explained in (1) to (3) below. This makes it difficult to apply existing automobile signal recognition technology. In other words, while automobile signal recognition requires only distinguishing the color and shape of the light, this is not enough for railway traffic signals. (1) There is a possibility that multiple lights will come on. (2) The position of the light is significant. (3) There are multiple lights of the same color. In this embodiment, the traffic light aspect is recognized by interpreting not only the color of the lights but also the number and position of the lights. Therefore, even if the traffic light has the characteristics (1) to (3), the traffic light aspect can be recognized.

[0054] There are already methods for predicting the location of traffic lights using map information, but they have the following problems (4) and (5). (4) The influence of measurement errors on the position information is large, especially the influence of errors in the direction (orientation) of the camera 14. (5) Even if the location of the traffic light is predicted, it is difficult to see the outline of the traffic light at night, so it is necessary to recognize it using only the light. Street lights and the headlights of oncoming trains can cause false detections. Of these, problem (5) is similar to the above problem, and can be solved in this embodiment. Regarding (4), in this embodiment, the problem can be solved as follows.

[0055] The appearance of a traffic light is expressed as a composite of the light position in the image, the movement of the light due to time changes and train speed, the distance between the lights, the light color, and the flashing frequency of the lights. In this embodiment, the combination of detected lights that most closely resembles the estimated appearance is recognized as a traffic light, and the aspect is read. In other words, the movement of the light due to time changes and train speed, the distance between the lights, the light color, and the flashing frequency of the lights are robust features against measurement errors in position and direction, as well as against dark environments. These features are determined by the relative position between the camera 14 and the traffic light, and are less susceptible to measurement errors in vehicle position information or errors in the traffic light's position coordinates. Therefore, using these features can solve the problem (4) above. Furthermore, if information such as the shape, installation height, expected aspect, and zebra pattern of the back panel of the traffic light can be obtained from an HD map or an external database, that information can be used to modify the conditions.

[0056] <Explanation of signal recognition method> Here, the processing performed by the traffic light detection system 1 can be considered as a traffic light recognition method in which an image taken by the camera 14 installed in the vehicle is acquired, light-emitting objects are detected from the acquired image, the position of the camera 14 is estimated, and based on the estimated position of the camera 14, the traffic light is identified from the detected light-emitting objects, and the traffic light indication is read from the identified light.

[0057] <Program Description> The processing performed by the traffic light detection system 1 in the present embodiment described above is realized by the cooperation of software and hardware resources. That is, a processor such as a CPU provided in the traffic light detection system 1 executes a program that realizes each function of the traffic light detection system 1, thereby realizing each function.

[0058] Therefore, the processing performed by the traffic light detection system 1 in this embodiment can also be seen as a program that causes a computer to realize an image acquisition function that acquires images captured by the camera 14 installed in the vehicle, an illuminant detection function that detects illuminants from the acquired images, a position estimation function that estimates the position of the camera 14, an identification function that identifies traffic light lights from the detected illuminants based on the estimated position of the camera 14, and a reading function that reads the traffic light indication from the identified lights.

[0059] The program for realizing this embodiment can be provided not only by communication means but also by being stored on a recording medium such as a CD-ROM.

[0060] The traffic light detection system 1 described above is particularly suitable for use in recognizing railway traffic lights. However, the present invention is not limited to this application and can also be applied to recognizing automobile traffic lights. In other words, the present invention can be applied not only to railway vehicles but also to moving objects such as automobiles. The traffic light detection system 1 described above can also be installed as a wayside facility that can be mounted on a moving object such as a railway vehicle. In this case, the camera 14 in Fig. 1 needs to be mounted on the moving object, but the processor 11, the output unit 12, the input unit 13, the communication module 15, the internal memory 16, and the external memory 17 can be part of the wayside facility.

[0061] Although the present embodiment has been described above, the technical scope of the present invention is not limited to the scope of the above embodiment. It is clear from the claims that various modifications and improvements to the above embodiment are also included in the technical scope of the present invention. [Explanation of symbols]

[0062] 1...traffic light detection system, 101...image acquisition unit, 102...light-emitting object detection unit, 103...self-position estimation unit, 104...signal position acquisition unit, 105...light-emitting object tracking unit, 106...signal information acquisition unit, 107...trajectory detection unit, 108...traffic light judgment unit, 109...signal aspect recognition unit, 110...result output unit, Sg1 to Sg9, Sg11 to Sg15...traffic lights, H1 to H9...light-emitting objects, L1 to L5, L111 to L113, L121 to L125, L131 to L139, L141 to L147, L151 to L156...lights

Claims

1. an image acquisition unit that acquires an image captured by an image capture device disposed on the moving object; a light-emitting body detection unit that detects a light-emitting body from within the acquired image; a position estimation unit that estimates the position of the imaging device; an identification unit that identifies traffic light lights from among the detected light-emitting objects based on the estimated position of the image capture device, estimates the intervals between the multiple lights in the image based on the distance between the image capture device and the traffic light, and identifies the multiple lights as belonging to one traffic light when the intervals between the light-emitting objects identified as lights match the estimated intervals; a reading unit that reads the aspect of the traffic light from the combination of the plurality of lights; A signal recognition device comprising:

2. (delete)

3. (delete)

4. (delete)

5. The traffic light recognition device according to claim 1, wherein the identification unit estimates the size of the traffic light in the image based on the distance, and identifies the plurality of lights as belonging to one traffic light based on the estimated size.

6. an image acquisition unit that acquires an image captured by an image capture device disposed on the moving object; a light-emitting body detection unit that detects a light-emitting body from within the acquired image; a position estimation unit that estimates the position of the imaging device; an identification unit that identifies a traffic light from among the detected light-emitting objects based on the estimated position of the image capturing device, and identifies a plurality of lights as belonging to one traffic light; a reading unit that reads the aspect of the traffic light from the combination of the plurality of lights; Equipped with the moving body is a railway vehicle, The identification unit uses information about the width of the track to estimate the size at which the traffic light in the image is captured, and identifies multiple lights as belonging to one traffic light based on the estimated size.

7. The signal recognition device according to claim 1 , wherein the identification unit identifies the light based on a movement of the light-emitting body in the image.

8. The signal recognition device according to claim 7, wherein the identification unit estimates a movement direction, which is the direction in which the light moves in the image, and identifies the light based on the estimated movement direction and the movement direction of the light-emitting body as the movement of the light-emitting body.

9. The signal recognition device according to claim 7, wherein the identification unit estimates a moving speed, which is the speed at which the light moves in the image, and identifies the light based on the estimated moving speed and the moving speed of the light-emitting body as the movement of the light-emitting body.

10. The signal recognition device according to claim 8 or 9, wherein the identification unit identifies the light by further taking into account at least one of a cycle at which the light flashes, an interval between the lights, and a color of the lights.

11. The signal recognition device according to claim 8 , wherein the specifying unit specifies the light by further taking into account the direction in which the image is captured by the image capturing device.

12. (delete)

13. 2. The signal recognition device according to claim 1, wherein the reading unit reads the signal aspect of the traffic light from the combination of a horizontal pattern, which is a horizontal pattern in which the lights are turned on or flashing, and a vertical pattern, which is a vertical pattern in which the lights are turned on or flashing.

14. Acquire an image captured by an image capturing device disposed on the moving object; Detects light-emitting objects from the captured images, Estimating the position of the imaging device; identifying traffic light from among the detected light-emitting objects based on the estimated position of the image capturing device, and estimating the spacing between the multiple lights in the image based on the distance between the image capturing device and the traffic light; if the spacing between the light-emitting objects identified as lights matches the estimated spacing, identifying the multiple lights as belonging to a single traffic light; Reading the signal aspect of the traffic light from the combination of the plurality of lights; Signal recognition method.

15. On the computer, an image acquisition function for acquiring an image captured by an image capture device disposed on the moving object; A light-emitting object detection function that detects light-emitting objects from the captured images; a position estimation function for estimating the position of the imaging device; an identification function that identifies traffic light lights from among the detected light-emitting objects based on the estimated position of the image capture device, estimates the intervals between the multiple lights in the image based on the distance between the image capture device and the traffic light, and identifies the multiple lights as belonging to one traffic light if the intervals between the light-emitting objects identified as lights match the estimated intervals; a reading function for reading the aspect of the traffic light from the combination of the plurality of lights; A program to achieve this.

Images