Identification device, mobile body, identification method, and storage medium

Abstract

The present application provides a kind of to improve the reliability of the lighting condition of signal lamp identified using shooting part technology.Signal lamp lighting condition is identified by recognition device mounted on mobile body, the recognition device has: shooting part, periodically shooting the outside of the mobile body;Detection unit, for each image periodically obtained by the shooting part, the lighting mode of the signal lamp contained in image is detected in sequence;And decision part, in the same lighting mode of the signal lamp is detected for a predetermined time by the detection unit, the lighting mode is decided as the lighting condition of the signal lamp by the decision part, the predetermined time is twice more than the shooting cycle of the shooting part.

Description

Technical Field

[0001] This invention relates to a recognition device, a moving body, a recognition method, and a storage medium for identifying the lighting status of traffic lights. Background Technology

[0002] Patent document 1 discloses the following technology: a display screen inside the vehicle switches between the recognition results of signals in the camera and signals obtained through wireless communication based on the vehicle speed.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2009-61871 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] In devices that identify the lighting status of traffic lights based on images obtained by an imaging unit (camera), misidentification can sometimes occur due to differences between the imaging cycle of the imaging unit and the lighting cycle of the traffic lights. Therefore, it is desirable to improve the reliability of using an imaging unit to identify the lighting status of traffic lights.

[0008] The purpose of this invention is to provide a technology that can improve the reliability of the illumination status of signal lights identified by the camera.

[0009] Methods for solving problems

[0010] To achieve the above objectives, an identification device, as one aspect of the present invention, is mounted on a moving body and identifies the lighting status of a traffic light. The identification device comprises: an imaging unit that periodically captures images of the surrounding environment of the moving body; a detection unit that sequentially detects the lighting pattern of the traffic light contained in each image periodically obtained by the imaging unit; and a determination unit that, when the detection unit detects the same lighting pattern of the traffic light for a predetermined period of time, determines that lighting pattern as the lighting status of the traffic light, wherein the predetermined period is at least twice the imaging period of the imaging unit.

[0011] To achieve the above objective, an identification method for identifying the lighting status of a traffic light, as one aspect of the present invention, is characterized in that the identification method includes: a detection step, in which the lighting pattern of the traffic light contained in each image periodically obtained by a camera unit periodically photographing the outside of a moving object is sequentially detected; and a determination step, in which, if the same lighting pattern of the traffic light is detected for a predetermined time by the detection step, the lighting pattern is determined as the lighting status of the traffic light, wherein the predetermined time is more than twice the shooting period of the camera unit.

[0012] Invention Effects

[0013] According to the present invention, for example, a technique can be provided to improve the reliability of the illumination status of signal lights identified by the camera. Attached Figure Description

[0014] Figure 1 This is a block diagram representing the vehicle's control system.

[0015] Figure 2 This is a block diagram illustrating an example of the configuration of an identification device.

[0016] Figure 3 This is a schematic diagram used to illustrate the recognition process.

[0017] Figure 4 This diagram is used to illustrate false detections and undetected items in the testing department.

[0018] Figure 5 It is a diagram that schematically represents the road on which vehicles travel.

[0019] Figure 6 This is a diagram illustrating an example of external information provided by an external communication device.

[0020] Figure 7 This is a schematic diagram used to illustrate the recognition process.

[0021] Figure 8 This is a flowchart representing the identification process.

[0022] Explanation of reference numerals in the attached figures:

[0023] 100: Identification device; 110: Imaging unit (camera); 120: Processing unit; 121: Detection unit; 122: Decision unit; 123: Output unit; 130: Display unit; 140: Communication unit. Detailed Implementation

[0024] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments are not limited to the invention involved in the technical solution, and not all of the feature combinations described in the embodiments are necessary for the invention. Two or more features from the plurality of features described in the embodiments can be arbitrarily combined. Furthermore, identical or identical components are labeled with the same reference numerals, and repeated descriptions are omitted.

[0025] <First Implementation>

[0026] The identification device according to the first embodiment of the present invention will be described. The identification device according to the present invention is a device that is mounted on a moving body and uses a camera to capture images of the outside of the moving body to identify the illumination status (lighting state) of a traffic light. A vehicle traveling on a road can be cited as an example of a moving body. Hereinafter, an example of applying (mounting) the identification device according to the present invention to a four-wheeled vehicle will be described, but the identification device according to the present invention can also be applied to vehicles other than four-wheeled vehicles, such as straddle-type vehicles (motorized two-wheeled vehicles, three-wheeled vehicles).

[0027] First, refer to Figure 1 The overall control system of vehicle 1 will be explained. Figure 1 This is a block diagram representing the control device 2 of vehicle 1. Figure 1 In the diagram, the outline of vehicle 1 is illustrated by top view and side view. Vehicle 1 is exemplified as a four-wheeled passenger car of sedan type.

[0028] Control unit 2 controls various parts of vehicle 1. Control unit 2 includes multiple ECUs 20-29 that are communicatively connected via an in-vehicle network. Each ECU (Electronic Control Unit) includes a processor (such as a CPU), a storage device such as semiconductor memory, and an interface for external devices. The storage device stores the program executed by the processor, the data used by the processor during processing, etc. Each ECU may also have multiple processors, storage devices, and interfaces. For example, ECU 20 has a processor 20a and a memory 20b. The processor 20a executes the commands contained in the program stored in the memory 20b, thus performing the processing of ECU 20. Alternatively, ECU 20 may also have an application-specific integrated circuit (ASIC) or similar for performing the processing of ECU 20. The same applies to other ECUs.

[0029] The functions of each ECU 20 to 29 are explained below. It should be noted that the number of ECUs and their functions can be appropriately designed, and the implementation can be further subdivided or integrated than in this embodiment.

[0030] ECU20 performs vehicle control related to the autonomous driving of vehicle 1 (this vehicle) as described in this embodiment. In autonomous driving, at least one of the steering of vehicle 1 and vehicle speed (acceleration or deceleration) is automatically controlled. In this embodiment, an example of automatically controlling both the steering of vehicle 1 and vehicle speed is shown.

[0031] ECU 21 controls the electric power steering system 3. The electric power steering system 3 includes a mechanism that steers the front wheels based on the driver's driving operations (steering operations) on the steering wheel 31. Additionally, the electric power steering system 3 includes a motor that provides driving force for assisting steering operations or automatically steering the front wheels, a sensor that senses the steering angle, etc. When the vehicle 1 is in automatic driving mode, ECU 21 automatically controls the electric power steering system 3 in accordance with instructions from ECU 20, and controls the direction of travel of the vehicle 1.

[0032] ECU 22 and ECU 23 control the sensing units 41-43 that sense the surrounding conditions of the vehicle and process the information from the sensing results. Sensing unit 41 is a camera (hereinafter sometimes referred to as camera 41) that periodically captures images of the exterior of the vehicle 1. In this embodiment, camera 41 is mounted on the front of the vehicle 1 roof and inside the passenger compartment of the front window so that it can capture images of the area in front of the vehicle 1. By analyzing (image processing) the images captured by camera 41, it is possible to extract the information of landmarks such as traffic lights in front of the vehicle 1, the display of traffic lights, and lane markings (white lines, etc.) on the road.

[0033] The sensing unit 42 (optical radar sensing unit) is a Light Detection and Ranging (LIDAR) (hereinafter sometimes referred to as optical radar 42), which senses objects around the vehicle 1 using light or measures the distance to objects. In this embodiment, five optical radars 42 are provided: one at each corner of the front of the vehicle 1, one at the center of the rear, and one on each side of the rear. The sensing unit 43 (radar sensing unit) is a millimeter-wave radar (hereinafter sometimes referred to as radar 43), which senses objects around the vehicle 1 using radio waves or measures the distance to objects. In this embodiment, five radars 43 are provided: one at the center of the front of the vehicle 1, one at each corner of the front, and one at each corner of the rear.

[0034] ECU 22 controls one of the cameras 41 and each of the optical radars 42 and processes the information from the sensing results. ECU 23 controls the other camera 41 and each of the radars 43 and processes the information from the sensing results. By having two sets of devices for sensing the vehicle's surroundings, the reliability of the sensing results can be improved. Furthermore, by having different types of sensing units such as cameras, optical radars, and radars, the vehicle's surrounding environment can be analyzed from multiple perspectives. In this embodiment, ECU 22 and ECU 23 function as detection units, which sequentially detect the lighting pattern of the traffic lights included in each image periodically obtained by the camera 41. Here, the display of traffic lights may include the color display (illuminated color (red, green, yellow)) of the traffic lights indicating whether the vehicle can move, and the arrow display of the traffic lights indicating the direction in which the vehicle can move.

[0035] ECU 24 controls the gyroscope sensor 5, GPS sensor 24b, and communication device 24c, and processes the sensing or communication results. The gyroscope sensor 5 senses the rotational motion of vehicle 1. The direction of travel of vehicle 1 can be determined by the sensing results of the gyroscope sensor 5, wheel speed, etc. The GPS sensor 24b senses the current position of vehicle 1. The communication device 24c wirelessly communicates with a server providing map information and traffic information, and acquires this information. The communication device 24c wirelessly communicates with an information server providing a system that monitors road traffic conditions, such as VICS (Vehicle Information and Communication System) (registered trademark), and acquires traffic information indicating the road on which vehicle M is traveling and the traffic conditions of the planned route. The traffic information includes traffic obstacle information such as: traffic congestion information on the set route, time required to pass through traffic congestion points, information on disabled vehicles / construction, speed limits, and lane restrictions.

[0036] ECU24 can access a database 24a containing map information stored on a storage device, and ECU24 can perform route exploration from the current location to the destination. Database 24a can be configured on a network, and communication device 24c can access database 24a on the network and obtain information.

[0037] ECU 25 includes a communication device 25a capable of communicating with other vehicles, inter-vehicle communication, or information processing devices such as smartphones. For example, communication device 25a can wirelessly communicate with other vehicles in the vicinity and exchange information between vehicles, or it can exchange information wirelessly with external information processing devices. In this embodiment, communication device 25a and communication device 24c function as receiving units that receive information indicating the illumination status of traffic lights (color display, arrow display illumination information) via wireless communication.

[0038] ECU26 controls the power unit 6. The power unit 6 is a mechanism that outputs driving force to rotate the drive wheels of vehicle 1, and includes, for example, an engine and a transmission. It should be noted that the configuration of the power unit 6 is not limited to this example, and includes electric vehicles that use an electric motor as a power source, hybrid vehicles that combine an engine and an electric motor, etc. Electric vehicles are driven by electricity discharged from batteries such as rechargeable batteries, hydrogen fuel cells, metal fuel cells, and ethanol fuel cells.

[0039] For example, ECU 26 controls engine output in response to driver operations (accelerator or acceleration) sensed by the operation sensing sensor 7a located on the accelerator pedal 7A, or switches transmission gears based on information such as vehicle speed sensed by the vehicle speed sensor 7c. When the vehicle 1 is in autonomous driving mode, ECU 26 automatically controls the power unit 6 in response to instructions from ECU 20, and controls vehicle speed (vehicle 1 speed and acceleration / deceleration).

[0040] ECU27 controls lighting fixtures (headlights, taillights, etc.), including the turn indicator 8. Figure 1 In the case of this example, the direction indicator 8 is located at the front, rearview mirror and rear of the vehicle 1.

[0041] ECU 28 controls the input / output device 9. The input / output device 9 outputs information to passengers, including the driver, and receives information from passengers. Voice output device 91 notifies passengers of information via voice. Display device 92 notifies the driver of information via image display. Display device 92 may be located in front of the driver's seat and front passenger seat, forming a touch panel-type instrument panel that functions as a human-machine interface.

[0042] For example, ECU 28 performs the following display control: it displays road information, including the multiple lanes currently being traveled by vehicle 1, obtained from the location information and map information of vehicle 1 (this vehicle) along the path from the current location to the destination explored by ECU 24, on display device 92. Additionally, ECU 28 controls voice output device 91 and display device 92, providing the driver with voice, map display, and voice guidance information.

[0043] It should be noted that while voice and display are illustrated here, information can also be communicated via vibration or light. Furthermore, multiple methods such as voice, display, vibration, and light can be combined to communicate information. Moreover, the combination or notification method can vary depending on the level of the information to be communicated (e.g., urgency).

[0044] The input device 93 is located in a position that the driver can operate, including a switch group for inputting instructions for the vehicle 1 and a voice input device for inputting the voice of the passenger.

[0045] ECU 29 controls the braking device 10 and the parking brake (not shown). The braking device 10 is, for example, a disc brake, installed on each wheel of the vehicle 1, which applies resistance to the rotation of the wheels to decelerate or stop the vehicle 1. ECU 29 controls the operation of the braking device 10 in response to the driver's driving operation (braking operation) sensed by the operation sensing sensor 7b installed on the brake pedal 7B. When the vehicle 1 is in automatic driving mode, ECU 29 automatically controls the braking device 10 in response to instructions from ECU 20, and controls the deceleration and stopping of the vehicle 1. The braking device 10 and the parking brake can also operate to maintain the vehicle 1 in a stopped state. In addition, if the transmission of the power unit 6 has a parking lock mechanism, it can also be activated to maintain the vehicle 1 in a stopped state.

[0046] [Composition of the identification device]

[0047] Next, while referring to Figure 2 An example of the configuration of the identification device 100 of this embodiment will be described. Figure 2 This is a block diagram illustrating an example configuration of the identification device 100 according to this embodiment. The identification device 100 is a device that uses a camera to identify the lighting status (lighting state) of a traffic light, as described above, and may include a camera unit 110, a processing unit 120, a display unit 130, and a communication unit 140 that are communicatively connected to each other via a system bus 150.

[0048] For example, the shooting unit 110 is Figure 1The camera 41 periodically captures images of the exterior of the vehicle 1 within a predetermined shooting cycle (e.g., 60 ms). The capturing unit 110 can also be understood as an external sensor that acquires external information related to objects, including landmarks and signs present around the vehicle 1. It should be noted that, hereinafter, the capturing unit 110 is sometimes referred to as camera 110.

[0049] Processing unit 120 is, for example, Figure 1 The control device 2 (ECU 20-29) can be composed of a computer including a processor (such as a CPU), a storage device such as a semiconductor memory, and an interface with external devices. In this embodiment, the storage device stores a program (hereinafter sometimes referred to as a recognition program) for recognizing the lighting status of the traffic lights. By executing the commands contained in the recognition program through the processor, the processing unit 120 can perform the recognition processing of the lighting status of the traffic lights. In addition, the processing unit 120 in this embodiment may include a detection unit 121, a decision unit 122 (identification unit), and an output unit 123.

[0050] For example, the detection unit 121 is Figure 1 ECUs 23 and 24, for each image periodically acquired by camera 110, sequentially detect the lighting pattern of the traffic lights contained in the image. Specifically, by performing known image processing on the images acquired by camera 110, detection unit 121 can determine (extract) the traffic lights as objects contained in the image and can detect the lighting pattern of the determined traffic lights. The lighting pattern of the traffic lights refers to, for example, which color of the traffic light color display (green, yellow, red) is lit, or whether the arrow indicator of the traffic light is lit.

[0051] For example, decision section 122 is Figure 1 The ECU 20 determines (identifies, recognizes) the lighting status of the traffic lights based on the lighting pattern detected by the detection unit 121. In this embodiment, if the detection unit 121 detects the same lighting pattern of the traffic lights for a predetermined period of time, the determination unit 122 determines (identifies, recognizes) that lighting pattern detected for the predetermined period of time as the current (actual) lighting status of the traffic lights. The predetermined period can be set to more than twice the shooting cycle of the camera 110. In addition, for images periodically obtained by the camera 110, the predetermined period can also be defined by a predetermined number of times the detection unit 121 continuously detects the same lighting pattern of the traffic lights. In other words, the predetermined number of times is the number of images (i.e., the number of shots) that the detection unit 121 continuously detects as having the same lighting pattern, and can be set to more than twice. Hereinafter, an example in which the determination unit 122 uses a predetermined number of times to make a determination and the predetermined number of times is set to three will be described.

[0052] Output section 123 is, for example, Figure 1 The ECU 28 outputs information (hereinafter sometimes referred to as illumination information) indicating the illumination status of the signal lights determined by the decision unit 122 to the display unit 130, and displays the illumination information on the display unit 130. In addition, the output unit 123 may also output the illumination information obtained from the decision unit 122 to the unit in the ECU 20 responsible for autonomous driving control, so that the illumination information can be used in the autonomous driving (vehicle control) of the vehicle 1 executed by the ECU 20.

[0053] The display unit 130 is, for example, with Figure 1 The display device 92 corresponds to the display screen, which displays the lighting information output from the output unit 123. Additionally, the communication unit 140 is, for example, a... Figure 1 Communication devices 24c and 25a receive information indicating the lighting status of traffic lights from an external communication device (transmitting device) located on the road where vehicle 1 travels via wireless communication. The external communication device can be configured as a beacon-type device or a cellular device using a mobile communication network standardized according to communication specifications for mobile phones. Communication unit 140 can obtain information from the external communication device regarding the lighting status of multiple traffic lights present around vehicle 1 (for example, multiple traffic lights arranged in the direction of travel of vehicle 1). It should be noted that, hereinafter, information received from the external communication device will sometimes be referred to as external information.

[0054] [Processing Summary]

[0055] Next, an overview of the recognition process performed by the processing unit 120 will be described. As described above, based on the images periodically obtained by the camera 110, if the signal light detected by the detection unit 121 is in the same lighting pattern and this continues for a predetermined number of times (three times in this case), the processing unit 120 (determination unit 122) determines the continuously detected lighting state as the current (actual) lighting state of the signal light.

[0056] Figure 3 This is a schematic diagram illustrating the identification process for recognizing the lighting status of the signal light TS in the processing unit 120 (decision unit 122). Figure 3 (a) in the figure represents the information input to the decision unit 122 in a time series, that is, the information representing the lighting mode of the signal lamp TS detected by the detection unit 121. Specifically, Figure 3 In (a) of the detection unit 121, image processing is performed on the image obtained by the camera 110, and the result of detecting the lighting pattern of the signal light TS contained in the image is presented as a time series. Additionally, Figure 3(b) in the figure represents the information output from the decision unit 122 in a time sequence, that is, the information representing the lighting status of the signal light TS determined by the decision unit 122 (lighting information). Specifically, Figure 3 (b) in the diagram represents the lighting information of the signal lamp TS determined by the decision unit 122 based on the lighting mode of the signal lamp TS detected by the detection unit 121, in a time series format. Figure 3 In this context, "t" represents the time sequence number of the images periodically acquired by camera 110, that is, the time sequence number of the information on the lighting mode of the signal light TS detected by detection unit 121 based on each image. However, it can also be understood as the time corresponding to the shooting cycle of camera 110 (in other words, the detection cycle of detection unit 121). t = 1 to 4 are only used to represent, in time sequence, a portion of the recognition processing continuously performed by processing unit 120, and do not represent the content obtained from the first to fourth recognition processes. (The following will be discussed further.) Figure 4 as well as Figure 7 The "t" in it is the same.

[0057] If t = 1, then in the detection unit 121, based on the image obtained by the camera 110, it is detected that the lighting mode of the traffic light TS is red (R), and in the decision unit 122, it is also determined that the lighting state of the traffic light TS is red (R). Next, if t = 2, then... Figure 3 As shown in (a), the detection unit 121 detects that the illumination mode of the traffic light TS is green (B) based on the image subsequently obtained by the camera 110. However, at this stage, the number of times the detection unit 121 continuously detects the illumination mode (green B) of the traffic light TS is still only once, which does not reach the predetermined number (three times). Therefore, as shown in (a), the illumination mode of the traffic light TS is green (B) only once. Figure 3 As shown in (b), the decision unit 122 maintains the previously decided illumination status of the signal light TS (red light R).

[0058] If t = 3, then as follows Figure 3 As shown in (a), the detection unit 121 detects that the illumination mode of the traffic light TS is green (B) based on the image subsequently obtained by the camera 110. However, at this stage, the number of times the detection unit 121 continuously detects the illumination mode (green B) of the traffic light TS is still twice, which does not reach the predetermined number (three times), therefore... Figure 3 As shown in (b), the decision unit 122 maintains the previously decided illumination state of the signal light TS (red light R). Next, if t = 4, then... Figure 3As shown in (a), the detection unit 121 detects that the illumination pattern of the traffic light TS is green (B) based on the image subsequently obtained by the camera 110. During this stage, the number of times the detection unit 121 continuously detects the illumination pattern of the traffic light TS (green (B)) reaches a predetermined number (three times), therefore... Figure 3 As shown in (b), the decision unit 122 determines the lighting status of the signal light TS as green (B).

[0059] By performing the above-described identification process, even if false detection and / or non-detection occur in the detection unit 121, the determination unit 122 of this embodiment can accurately determine (identify) the lighting status of the signal light TS. That is, it can avoid or reduce situations where the lighting status of the signal light TS is determined based on false detection and / or non-detection in the detection unit 121. False detection by the detection unit 121 refers to, for example... Figure 4 As shown in (a) with t=2, although the signal light TS is actually lit in red, the detection unit 121 detects it as lit in green (B). The failure of the detection unit 121 to detect this means that, as in... Figure 4 As shown in (b) with t=2, although the signal light TS is actually lit (red), its illumination is not detected by the detection unit 121. Furthermore, according to the above recognition process, information indicating the lighting status of the signal light TS determined by the decision unit 122 (lighting information) can be provided by the output unit 123 to the display unit 130 (display) and displayed on the display unit 130. As described above, the lighting information output from the decision unit 122 can also be used for the autonomous driving (vehicle control) of the vehicle 1 executed by the ECU 20.

[0060] Here, when the communication unit 140 receives external information indicating the lighting status of the traffic light from an external communication device, the decision unit 122 of this embodiment can also determine the lighting status of the traffic light TS based on this external information. External information obtained from the external communication device tends to be highly reliable as information indicating the lighting status of the traffic light. Therefore, even if the detection unit 121 continuously detects that the same lighting pattern of the traffic light TS lasts for less than a predetermined time (predetermined number of times), the decision unit 122 can still determine the lighting status of the traffic light based on the external information received by the communication unit 140 from the external communication device.

[0061] Figure 5 This schematically represents the road on which vehicle 1 travels. Figure 5In the example shown, vehicle 1 travels in the direction of arrow A, and multiple traffic lights TS1 to TS3 are arranged in the direction of vehicle 1's travel. Traffic lights TS1 to TS3 respectively include color displays indicating whether travel is permitted (illuminated colors (red, green, yellow)), straight-ahead arrows indicating permission to go straight, and right-turn arrows indicating permission to turn right. Additionally, multiple external communication devices CA1 to CA2 are arranged along the road. External communication device CA1 is positioned near traffic light TS1, and external communication device CA2 is positioned near traffic light TS2 (between traffic lights TS1 and TS2). External communication devices CA1 to CA2 are devices that transmit (provide) external information to vehicle 1 via beacons, indicating the illumination status of the multiple traffic lights (including traffic lights TS1 to TS3) present around vehicle 1. As an example, ... Figure 6 As shown, the external communication device CA1, located near the traffic light TS1, provides (sends) external information to the vehicle 1 indicating the lighting status of traffic lights TS1 to TS3 respectively. Figure 6 This represents an example of external information provided by an external communication device CA1.

[0062] Figure 7 This is a schematic diagram illustrating the identification process in the processing unit 120 (decision unit 122). Figure 7 This paper presents an example of a recognition process that identifies the lighting status of a traffic light TS based not only on the lighting mode detected by the detection unit 121, but also on external information obtained from the external communication device CA. Figure 7 (a) shows the information input to the decision unit 122 in a time sequence, that is, the information indicating the lighting mode of the signal lamp TS detected by the detection unit 121. Figure 7 (b) shows, in time series, external information indicating the lighting pattern of the signal light TS provided from an external communication device (e.g., CA1). Figure 7 (c) shows the information output from the decision unit 122 in a time sequence, that is, the lighting information indicating the lighting status of the signal lamp TS determined by the decision unit 122.

[0063] If t = 1, then in the detection unit 121, based on the image obtained by the camera 110, it is detected that the lighting mode of the traffic light TS is red (R), and in the decision unit 122, it is also determined that the lighting status of the traffic light TS is red (R). At this stage, external information is not received. Next, if t = 2, then... Figure 7As shown in (a), based on the image subsequently obtained by camera 110, detection unit 121 detects that the illumination mode of signal light TS is green (B). At this stage, no external information is received. Furthermore, the number of times the illumination mode (green (B)) of signal light TS is continuously detected by detection unit 121 is still only one, which does not reach the predetermined number (three times). Therefore, as... Figure 7 As shown in (c), the decision unit 122 maintains the previously decided lighting status of the signal light TS (red light R).

[0064] If t = 3, then as follows Figure 7 As shown in (a), based on the image subsequently obtained by camera 110, detection unit 121 detects that the illumination mode of traffic light TS is green (B). On the other hand, communication unit 140 receives external information from external communication device CA1 that the illumination status of traffic light TS is green (B). At this stage, the number of times the illumination mode (green (B)) of traffic light TS is continuously detected by detection unit 121 is still twice, which does not reach the predetermined number (three times), but... Figure 7 As shown in (b), external information such as the green light B is received from an external communication device. Therefore, as Figure 7 As shown in (c), the decision unit 122 preferentially uses this external information to determine the lighting status of the traffic light TS as green (B). Next, if t = 4, then based on the image subsequently obtained by the camera 110, the detection unit 121 detects the lighting status of the traffic light TS as green (B) (see reference). Figure 7 In (a) of the text, the external information is also highlighted in green (see reference B). Figure 7 (b) in the text. Therefore, as in Figure 7 As shown in (c), the decision unit 122 determines the lighting status of the signal light TS as green (B). Thus, by further utilizing external information provided from an external communication device, the lighting status of the signal light TS can be determined (identified) with high accuracy and speed.

[0065] [Identification and processing flow]

[0066] Next, the process of identification processing performed by the processing unit 120 will be explained. Figure 8 This is a flowchart illustrating the identification process performed by the processing unit 120. Figure 8 The flowchart shown illustrates the repeated execution scenario. After step S16 is completed, the process can restart from step S11.

[0067] In step S11, the processing unit 120 acquires an image of the exterior of the vehicle 1 (front view in this embodiment) by having the camera 110 capture the exterior of the vehicle 1. In step S12, the processing unit 120 (detection unit 121) detects the lighting pattern of the traffic lights TS contained in the image obtained by the camera 110. For example, by performing known image processing on the image obtained by the camera 110, the detection unit 121 can detect the lighting color of the traffic lights TS in the image as the lighting pattern. Next, in step S13, the processing unit 120 (determination unit 122) determines whether the lighting pattern of the traffic lights TS detected by the detection unit 121 has been maintained for a predetermined number of times (three times in this embodiment). If the same lighting pattern has been maintained for a predetermined number of times, the process proceeds to step S16; otherwise, the process proceeds to step S14.

[0068] In step S14, the processing unit 120 (decision unit 122) determines whether external information has been obtained from the external communication device CA via the communication unit 140. External information refers to information indicating the lighting status of the traffic light TS, as described above, which can be obtained via beacon or cellular means, for example. If external information has been obtained, the process proceeds to step S15; otherwise, it returns to step S11. Next, in step S15, the processing unit 120 (decision unit 122) determines whether the elapsed time since the external information was obtained by the communication unit 140 in step S14 is within a predetermined period. Since the lighting status of the traffic light TS changes constantly, if the lighting status of the traffic light TS in the obtained external information is outdated, it is difficult to accurately determine the current lighting status of the traffic light TS in step S16 (described later). Therefore, in this step S15, a predetermined period (predetermined time) is set as the valid period for using external information, and it is determined whether the elapsed time since the external information was obtained is within the predetermined period. The specified period can be set based on information indicating the time during which the same illuminated color is maintained in the traffic light TS (e.g., the time for green to illuminate, the time for red to illuminate). The information indicating the time for maintaining the same illuminated color can, for example, be... Figure 6 The "Change Schedule" section displays the time information. Furthermore, the specified period can be set for each individual traffic light (TS) or multiple traffic light (TS) together.

[0069] Here, as described above, the external information received from the external communication device CA sometimes includes information indicating the illumination status of multiple traffic lights TS present around the vehicle 1. In this case, for traffic lights TS among the multiple traffic lights TS whose elapsed time after receiving the external information is within a predetermined period, the determination unit 122 can determine the current illumination status based on the external information. On the other hand, for traffic lights TS among the multiple traffic lights TS whose elapsed time after receiving the external information is outside the predetermined period, the current illumination status can be determined based on the detection results in the detection unit 121 without using the external information.

[0070] In step S16, the processing unit 120 (decision unit 122) determines (identifies, recognizes) the lighting status of the traffic light TS. For example, if the detection unit 121 detects the same lighting pattern of the traffic light TS three times in a predetermined number of times, the determination unit 122 determines the lighting pattern detected by measuring the predetermined number of times as the current (actual) lighting status of the traffic light TS. On the other hand, if external information is received from an external communication device CA and the elapsed time after receiving the external information is within a predetermined period, the determination unit 122 preferentially uses the external information and determines the lighting status of the traffic light TS contained in the external information as the current (actual) lighting status of the traffic light TS.

[0071] In the example above, as a result of confirming the specified period in step S15, the lighting state is determined in step S16 as long as the lighting state is within the specified period. However, as another method, the specified period processing can be omitted in step S15, and the lighting state of the signal light that matches the lighting state detected by the detection unit 121 in step S16 and the external information received from the external communication device CA can be determined as the lighting state. For example, if the detection unit 121 detects that the signal light is green, and the external information received from the external communication device CA also indicates that the signal light is green, the lighting state of the signal light can be determined as green without considering the specified period.

[0072] Furthermore, even when the signal light display information is received by the external communication device CA without reflecting the specified time, in addition to effectively utilizing the information sent by the external communication device CA regarding the illumination color and direction of travel of the signal lights configured in the direction of travel, it is also possible to effectively utilize the information sent by the external communication device CA regarding their shortest and longest illumination times. For example, if the illumination color is green, and communication is sometimes conducted with a minimum illumination time of 0 seconds and a maximum illumination time of 10 seconds, in this case, the accuracy of the following can be determined: although the signal light information was successfully acquired, the target signal light may change to yellow after 0 seconds, and the signal light illumination prediction information may be inaccurate, etc.

[0073] Thus, by comparing the external information (traffic light information) received by the communication unit 140 with the information obtained by the imaging unit 110, a high-precision display can be achieved. Furthermore, during the comparison, the shortest illumination time can be considered and a reference time can be set. That is, if the shortest reference time (shortest illumination time) is longer than the estimated time for a moving object to enter the intersection, the information (external information) received by the communication unit 140 is prioritized; conversely, if the shortest reference time (shortest illumination time) is shorter than the estimated time for a moving object to enter the intersection, the information received by the communication unit 140 and the information obtained by the detection unit 121 can also be compared.

[0074] As described above, the recognition device 100 of this embodiment detects the lighting pattern of the traffic light TS in each image periodically acquired by the camera 110. If the same lighting pattern is detected a predetermined number of times, the lighting pattern is determined as the current (actual) lighting state of the traffic light TS. Therefore, even if false detections or non-detections occur when detecting the lighting pattern of the traffic light TS based on the image, the situation where the current lighting state of the traffic light TS is determined (recognized) based on false detections or non-detections can be avoided or reduced. That is, the current lighting state of the traffic light TS can be identified with high precision and accuracy, and the reliability of the lighting state of the traffic light identified by the imaging unit is improved.

[0075] <Second Implementation>

[0076] In the first embodiment, an example of performing the above-described recognition processing on the color display of the traffic light TS was described, but the above-described recognition processing can also be performed on the arrow display of the traffic light TS. However, arrow displays are generally difficult to detect in known image processing methods, and false detections or non-detections are prone to occur in the detection unit 121. Therefore, when performing recognition processing on arrow displays, a second predetermined time set to be longer than the predetermined time used for color display can be used instead of the predetermined time used for color display. Similarly, a second predetermined number of times set to be more than the predetermined number of times used for color display can be used instead of the predetermined number of times used for color display. It should be noted that the configuration of the recognition device 100 and the processing content other than those described above are the same as in the first embodiment, so the description here is omitted.

[0077] <Third Implementation Method>

[0078] As explained in the first embodiment, the illumination information (determination result) indicating the illumination status of the signal light determined in the decision unit 122 is sometimes used in the automatic driving system of the ECU 20. Thus, when the illumination information is used for automatic driving, high accuracy is sought in this illumination information. Therefore, when the illumination information is used for automatic driving, a longer predetermined time (or a greater number of predetermined times) can be set compared to when the illumination information is not used for automatic driving. For example, if the predetermined number of times is set to three when only displaying the illumination information on the display unit 130, and the illumination information is used for automatic driving, this predetermined number of times can be set to four or more. This allows for more accurate illumination information, thereby improving the accuracy of automatic driving and reducing passenger discomfort. It should be noted that the configuration of the identification device 100 and the processing content other than those described above are the same as in the first embodiment, and therefore, the description is omitted here. Furthermore, the second embodiment can also be applied in this embodiment.

[0079] <Fourth Implementation>

[0080] The image obtained by camera 110 sometimes shows multiple traffic lights, including those indicating the direction of travel of vehicle 1 (TS), as well as those indicating directions different from the direction of travel of vehicle 1, and traffic lights at the intersection ahead. Thus, when the image obtained by camera 110 contains multiple traffic lights, the detection unit 121 can calculate the accuracy (determinacy, reliability) of the traffic light representing the direction of travel of vehicle 1 for each traffic light, and detect the lighting pattern of the traffic light with the highest accuracy among the multiple traffic lights. For example, the detection unit 121 can calculate the area of ​​the illuminated portion of each traffic light in the image and set the calculated area as the accuracy. In this way, by calculating the accuracy, even when the image contains multiple traffic lights, the traffic light TS indicating the direction of travel of vehicle 1 can be determined. Furthermore, regarding accuracy, after integrating information from multiple traffic lights, the accuracy can be improved by comparing it with other map information and information obtained through communication. It should be noted that the configuration of the identification device 100 and the processing other than those described above are the same as in the first embodiment, therefore, the description is omitted here. Furthermore, the second to third embodiments can also be applied to this embodiment.

[0081] <Summary of Implementation Methods>

[0082] 1. The identification device described in the above embodiment is an identification device (e.g., 100) mounted on a moving body (e.g., 1) and identifying the lighting status of a signal light (e.g., TS).

[0083] The identification device includes:

[0084] A camera unit (e.g., 110) periodically photographs the outside of the moving body;

[0085] A detection unit (e.g., 121) sequentially detects the lighting pattern of the signal lights contained in each image periodically obtained by the capturing unit; and

[0086] The decision unit (e.g., 122) determines that the same lighting pattern of the traffic light is the lighting status when the detection unit continuously detects the same lighting pattern of the traffic light for a predetermined period of time.

[0087] The scheduled time is more than twice the shooting cycle of the shooting unit.

[0088] According to this embodiment, even if false detection or non-detection occurs when detecting the lighting mode of a traffic light based on an image, the situation where the current lighting status of the traffic light is determined (identified) based on false detection or non-detection can be avoided or reduced. That is, the current lighting status of the traffic light can be identified with high precision and accuracy, and the reliability of the lighting status of the traffic light identified by the imaging unit is improved.

[0089] 2. In the above embodiments,

[0090] If the detection unit does not detect the same lighting pattern of the signal light for the predetermined time, the decision unit maintains the lighting status of the signal light as previously decided.

[0091] According to this embodiment, as long as the current lighting status of the traffic light cannot be reliably determined, the information indicating the lighting status of the traffic light will not be changed, thus improving accuracy.

[0092] 3. In the above embodiments,

[0093] The predetermined time is defined by the predetermined number of times the detection unit continuously detects the same lighting pattern of the signal light.

[0094] The predetermined number of times is two or more.

[0095] According to this embodiment, when a predetermined number of times is used instead of a predetermined time, the current lighting status of the traffic light can be identified with high precision and accuracy, just as when a predetermined time is used.

[0096] 4. In the above embodiments,

[0097] The identification device also includes an output unit (e.g., 123) that outputs information to a display unit (e.g., 130) indicating the lighting status of the signal light determined by the decision unit.

[0098] According to this embodiment, the status of traffic lights can be notified to users (e.g., drivers).

[0099] 5. In the above embodiments,

[0100] The decision unit determines the lighting status of the signal light for the predetermined time based on the color display of the signal light indicating whether the moving body can move.

[0101] According to this embodiment, the illumination status of the color display (lighting color (red, green, yellow)) of the traffic lights can be identified with high precision and accuracy.

[0102] 6. In the above embodiments,

[0103] When determining the lighting status based on the arrow display of the signal light indicating the direction in which the moving body can travel, the decision-making unit uses a second predetermined time that is longer than the predetermined time, instead of the predetermined time.

[0104] According to this embodiment, the lighting status of arrows for signal lights that are difficult to detect in known image processing and are prone to false detection and non-detection can also be identified with high precision and accuracy.

[0105] 7. In the above embodiments,

[0106] The identification device also includes a communication unit (e.g., 140) that receives information related to the illumination of the traffic light via wireless communication.

[0107] Even if the detection unit continuously detects that the same lighting pattern of the traffic light lasts for less than the predetermined time, the decision unit will still determine the lighting status of the traffic light based on the information received through the communication unit.

[0108] According to this implementation, information received via wireless communication tends to be more reliable than information detected by image detection. Therefore, by actively using such information, the lighting status of traffic lights can be identified with higher precision and accuracy.

[0109] 8. In the above embodiments,

[0110] For multiple traffic lights present around the mobile vehicle, the communication unit receives information from an external communication device (e.g., CA) located on the road on which the mobile vehicle travels.

[0111] For each of the plurality of traffic lights, if the elapsed time after receiving the information via the communication unit is within a predetermined period, the decision unit determines the lighting status based on the information.

[0112] For a traffic light whose elapsed time is outside the specified period, the decision unit determines the lighting status based on the detection results in the detection unit.

[0113] According to this embodiment, the lighting status of each signal light can be appropriately identified based on the information acquired by the identification device.

[0114] 9. In the above embodiments,

[0115] When the image obtained by the imaging unit contains multiple traffic lights, the detection unit calculates the accuracy of the traffic light representing the direction of travel of the moving body for each traffic light, and detects the lighting mode of the traffic light with the highest accuracy among the multiple traffic lights.

[0116] According to this embodiment, even when the image acquired by the imaging unit contains multiple traffic lights, it is possible to determine the traffic light indicating the vehicle's direction of travel and detect the illumination pattern of the determined traffic light. In other words, it is possible to identify the illumination status of the traffic light indicating the vehicle's direction of travel.

[0117] 10. In the above embodiments,

[0118] When the determination of the illumination status of the signal light is used for steering of the moving body and automatic speed control, the determination unit sets the predetermined time for a longer period compared to when the determination is not used for automatic control.

[0119] According to this embodiment, when the result of the signal light illumination status is used for autonomous driving, the result can be obtained more accurately, thereby improving the accuracy of autonomous driving and reducing passenger discomfort.

[0120] This invention is not limited to the above-described embodiments. Various changes and modifications can be made without departing from the spirit and scope of this invention.

Claims

1. An identification device mounted on a moving body and identifying a lighting condition of a signal light, characterized in that the identification device comprises: a photographing section that periodically photographs an outside of the moving body; a detection section that sequentially detects a lighting pattern of the signal light included in an image for each image periodically obtained by the photographing section; and a decision section that decides the lighting condition of the signal light as a lighting pattern of the signal light in a case where the same lighting pattern is detected by the detection section for a predetermined time, the predetermined time is twice or more the photographing period of the photographing section, in a case where the lighting condition is decided for a color display of the signal light indicating whether the moving body can travel, the decision section uses a first time as the predetermined time, and in a case where the lighting condition is decided for an arrow display of the signal light indicating a direction in which the moving body can travel, the decision section uses a second time longer than the first time as the predetermined time.

2. The identification device according to claim 1, characterized in that in a case where the same lighting pattern of the signal light is not detected by the detection section for the predetermined time, the decision section maintains the lighting condition of the signal light decided last time.

3. The identification device according to claim 1, characterized in that the predetermined time is defined by a predetermined number of times that the same lighting pattern of the signal light is continuously detected by the detection section, the predetermined number of times is two or more.

4. The identification device according to claim 1, characterized in that the identification device further comprises an output section that outputs information indicating the lighting condition of the signal light decided by the decision section to a display section.

5. The identification device according to claim 1, characterized in that the identification device further comprises a communication section that receives information about lighting of the signal light through wireless communication, even if a time during which the same lighting pattern of the signal light is continuously detected by the detection section does not satisfy the predetermined time, in a case where the information is received by the communication section, the decision section decides the lighting condition of the signal light based on the information.

6. The identification device according to claim 5, characterized in that the communication section receives the information from a communication device provided outside a road on which the moving body travels, for a plurality of signal lights existing in a periphery of the moving body, for a signal light of the plurality of signal lights for which an elapsed time after the information is received by the communication section is within a prescribed period, the decision section decides the lighting condition based on the information, for a signal light for which the elapsed time is outside the prescribed period, the decision section decides the lighting condition based on a detection result in the detection section.

7. The identification device according to claim 1, characterized in that In a case where a plurality of signal lights are included in the image obtained by the imaging section, the detection section calculates a degree of certainty of each signal light as a signal light in the traveling direction of the moving body, and detects the lighting pattern for the signal light having the highest degree of certainty among the plurality of signal lights.

8. The identification device according to claim 1, wherein In a case where the determination result of the lighting condition of the signal light is used for automatic control of steering and vehicle speed of the moving body, the determination section sets the predetermined time longer than in a case where the determination result is not used for the automatic control.

9. A moving body, comprising: The moving body includes the identification device according to any one of claims 1 to 8.

10. An identification method of identifying a lighting condition of a signal light, The identification method includes: a detection step of sequentially detecting a lighting pattern of the signal light included in each image periodically obtained by an imaging section that periodically images an outside of a moving body; and a determination step of determining the lighting pattern as a lighting condition of the signal light in a case where the same lighting pattern of the signal light is detected for a predetermined time by the detection step, The predetermined time is twice or more the imaging period of the imaging section, In the determination step, a first time is used as the predetermined time in a case where the lighting condition is determined for a color display of the signal light indicating whether the moving body can travel, and a second time longer than the first time is used as the predetermined time in a case where the lighting condition is determined for an arrow display of the signal light indicating a direction in which the moving body can travel.

11. A storage medium storing a program for causing a computer to execute the steps of the identification method according to claim 10. ​

Images