Control device, mobile object, control method and storage medium

Abstract

The present invention provides a control device, a mobile body, a control method and a storage medium, which determine the display of a signal light based on the comparison between the recognition result of a shooting unit and the display information received by a communication unit through communication. The control device is mounted on the mobile body, and displays the display of a signal light contained in an image captured by a shooting unit that captures the outside of the mobile body on a display unit, and the control device comprises: a recognition unit that recognizes the display of a signal light based on an image; a communication unit that receives the display information of the signal light through wireless communication; and a control unit that determines the display of the signal light based on the display of the signal light recognized by the recognition unit and the display information received by the communication unit, and displays the determined display of the signal light on the display unit.

Description

Technical Field

[0001] The present invention relates to a control device, a moving object, a control method and a storage medium. Background Art

[0002] Patent Document 1 discloses a technique in which the recognition result of either a signal recognition by a camera or a signal recognition acquired by wireless communication is switched according to the vehicle speed and displayed on a display inside the vehicle.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2009-61871 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, due to the surrounding environment of the road on which the vehicle is traveling, such as the influence of street trees on the side of the road that partially covers the traffic light, or weather conditions such as snowfall, the display of the traffic light may not be accurately recognized in the image captured by the camera (photographing unit). In addition, depending on the communication status, the recognition result may not be obtained via wireless communication.

[0008] In view of the above-mentioned problems, the present invention provides a technology capable of determining the display of a signal light based on a comparison between a recognition result of an imaging unit and display information received by a communication unit through communication.

[0009] Means for solving problems

[0010] A control device according to one embodiment of the present invention is mounted on a mobile body, and displays on a display unit a display of a signal light included in an image captured by a capturing unit capturing the outside of the mobile body, wherein:

[0011] The control device comprises:

[0012] a recognition unit for recognizing the display of the signal light according to the image;

[0013] a communication unit that receives display information of the signal light through wireless communication; and

[0014] a control unit that determines the display of the signal light based on the display of the signal light recognized by the recognition unit and the display information received by the communication unit, and displays the determined display of the signal light on the display unit,

[0015] The control unit comprises:

[0016] a lighting color determination unit that compares the lighting color of the signal light identified by the identification unit with the lighting color of the signal light acquired from the display information based on the wireless communication, and determines the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison; and

[0017] A traveling direction determination unit compares the traveling direction indication of the signal light identified by the identification unit with the traveling direction indication of the signal light obtained from the display information based on the wireless communication, and determines the traveling direction indication that is consistent with the comparison result as the traveling direction indication displayed on the display unit.

[0018] A mobile body according to another aspect of the present invention includes a control device that displays, on a display unit, a display of a signal light included in an image captured by a capturing unit that captures the outside world, wherein:

[0019] The control device comprises:

[0020] a recognition unit for recognizing the display of the signal light according to the image;

[0021] a communication unit that receives display information of the signal light through wireless communication; and

[0022] a control unit that determines the display of the signal light based on the display of the signal light recognized by the recognition unit and the display information received by the communication unit, and displays the determined display of the signal light on the display unit,

[0023] The control unit comprises:

[0024] a lighting color determination unit that compares the lighting color of the signal light identified by the identification unit with the lighting color of the signal light acquired from the display information based on the wireless communication, and determines the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison; and

[0025] A traveling direction determination unit compares the traveling direction indication of the signal light identified by the identification unit with the traveling direction indication of the signal light obtained from the display information based on the wireless communication, and determines the traveling direction indication that is consistent with the comparison result as the traveling direction indication displayed on the display unit.

[0026] A control method according to another aspect of the present invention is a control method for a control device mounted on a mobile body, which displays a display of a signal light included in an image captured by a capturing unit capturing the outside of the mobile body on a display unit, wherein:

[0027] The control method has the following features:

[0028] a recognition step, in which a recognition unit recognizes the display of the signal light based on the image;

[0029] a communication step, in which a communication unit receives display information of the signal light through wireless communication; and

[0030] a control step, in which a control unit determines the display of the signal light based on the display of the signal light recognized in the recognition step and the display information received in the communication step, and displays the determined display of the signal light on the display unit,

[0031] In the control step, there are:

[0032] a lighting color determining step of comparing the lighting color of the signal light identified in the identifying step with the lighting color of the signal light acquired from the display information based on the wireless communication, and determining the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison; and

[0033] A traveling direction determining step, in which the traveling direction indication of the signal light identified in the identifying step is compared with the traveling direction indication of the signal light obtained from the display information based on the wireless communication, and the traveling direction indication that is consistent with the comparison result is determined as the traveling direction indication displayed on the display unit.

[0034] A storage medium according to another aspect of the present invention stores a program for causing a computer to execute each step of a control method of a control device mounted on a mobile body, and displaying a display of a signal light included in an image captured by a capturing unit capturing the outside of the mobile body on a display unit, wherein:

[0035] The control method has the following features:

[0036] a recognition step, in which a recognition unit recognizes the display of the signal light based on the image;

[0037] a communication step, in which a communication unit receives display information of the signal light through wireless communication; and

[0038] a control step, in which a control unit determines the display of the signal light based on the display of the signal light recognized in the recognition step and the display information received in the communication step, and displays the determined display of the signal light on the display unit,

[0039] In the control step, there are:

[0040] a lighting color determining step of comparing the lighting color of the signal light identified in the identifying step with the lighting color of the signal light acquired from the display information based on the wireless communication, and determining the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison; and

[0041] A traveling direction determining step, in which the traveling direction indication of the signal light identified in the identifying step is compared with the traveling direction indication of the signal light obtained from the display information based on the wireless communication, and the traveling direction indication that is consistent with the comparison result is determined as the traveling direction indication displayed on the display unit.

[0042] Effects of the Invention

[0043] According to the present invention, the display of the signal light can be determined based on the comparison between the recognition result of the imaging unit and the display information received by the communication unit through communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a block diagram illustrating the basic structure of the control device.

[0045] Figure 2 This is a block diagram showing a configuration example in which the functional configuration of a control device is installed in a vehicle.

[0046] Figure 3A It is a diagram for explaining the flow of processing performed by the control device of the first embodiment.

[0047] Figure 3B This is a diagram for explaining the flow of processing performed by the control device related to the priority determination processing.

[0048] Figure 3C It is a diagram for explaining the flow of processing performed by the control device according to the second embodiment.

[0049] Figure 4 The figure schematically shows a road on which a vehicle travels.

[0050] Figure 5 The figure schematically shows the integration process of the display of a plurality of signal lights.

[0051] Figure 6 The diagram schematically shows a comparison process between a recognition result of a signal light display and display information and a determination process of a signal light display.

[0052] Figure 7 It is a diagram showing a setting example of a predetermined priority order related to the lighting colors of a traffic light.

[0053] Figure 8 This is a diagram for explaining an example of determining the indication of the traveling direction of a traffic light.

[0054] Fig. 9 FIG. 2 is a diagram schematically showing a comparison process of an integrated signal display and map information.

[0055] Fig.10 The diagram schematically shows a comparison process of the display of a traffic light and display information at an intermediate stage and a determination process of the display of the traffic light.

[0056] Description of Reference Numerals

[0057] 22, 23: ECU (identification unit);

[0058] 24c, 25a: communication device (communication unit);

[0059] 28: ECU (control unit). DETAILED DESCRIPTION

[0060] Hereinafter, the embodiments are described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention involved in the claims. In addition, the combination of features described in the embodiments may not all be necessary for the invention. It is also possible to arbitrarily combine two or more of the multiple features described in the embodiments. In addition, the same figure mark is marked on the same or similar structure, and repeated description is omitted.

[0061] [First embodiment]

[0062] In the first embodiment, the moving object includes a vehicle. In the following description, the vehicle is described as an example of the moving object, and a configuration for presenting (displaying) information to a vehicle occupant (driver) is described. Figure 1It is a block diagram illustrating the basic structure of the control device 2 that controls the vehicle 1. The control device 2 includes a processing unit C1 responsible for processing, a memory C2, a communication unit C3 that can communicate with a server and external devices on the network, etc. The sensor S includes, for example, multiple radars S1, and multiple optical radars S2 (Light Detection and Ranging (LIDAR: optical radar)), a gyro sensor S3, a GPS sensor S4, a vehicle speed sensor S5, etc. The sensor S and the camera CAM function as external sensors that obtain external information related to objects existing around the vehicle, marked targets, and vehicle position information. The sensor S and the camera CAM (external sensors) obtain information about the vehicle and various information around the vehicle, and input the obtained information into the control device 2.

[0063] The control device 2 performs image processing on the information input from the radar S1, optical radar S2 and camera CAM, extracts the target (object) existing around the vehicle, analyzes what kind of target is arranged around the vehicle, and monitors the target. For example, it can recognize the display of the traffic light in front of the road where the vehicle is traveling.

[0064] The control device 2 functions as a vehicle control device that performs automatic driving control of the vehicle based on the information of the extracted target object (subject) and the information input from the sensor S and the camera CAM, and as a display control device that displays the information of the target object (subject) such as the display of the traffic light on the display unit. The display control device is mounted on the mobile body (vehicle) and displays the display of the traffic light included in the image captured by the camera CAM (capturing unit) that captures the outside of the mobile body (vehicle) on the display unit. In addition, the control device 2 functions as a device having a navigation function such as mapping the current position of the vehicle on a map and providing route guidance to the destination. These devices and equipment are connected in a manner that can communicate data with each other via a communication medium such as CAN (Controller Area Network).

[0065] In addition, the gyro sensor S3 detects the rotational motion and posture of the vehicle, and the control device 2 can determine the direction of travel of the vehicle based on the detection results of the gyro sensor S3, the speed detected by the speed sensor S5, etc. In addition, the control device 2 can obtain road information such as the current position (position information) of the vehicle in the map information, the number of lanes in which the vehicle (this vehicle) is traveling, the shape of the road, the reduction or increase of lanes in the route, construction, the existence of a broken vehicle, etc., based on the detection results of the GPS sensor S4. The control device 2 can make multiple control states transition in stages based on the information of the surrounding environment of the vehicle, thereby controlling the automatic driving of the vehicle. That is, the control device 2 uses the information of the sensor S and the camera CAM to obtain the information of the surrounding environment of the vehicle, and based on the information of the surrounding environment, the control state of the vehicle is transitioned, thereby controlling the automatic driving of the vehicle.

[0066] On the vehicle Figure 1 In the case of the processing of the control device 2 shown, the control device 2 can be configured, for example, in an ECU of an identification processing system or an image processing system that processes information from a sensor S or a camera CAM, or in an ECU that controls a communication device, a display device, and an input / output device, or in an ECU that controls the drive of a vehicle or an ECU for automatic driving.

[0067] For example, as described below Figure 2 In that way, the functions of the control device 2 can also be dispersed among multiple ECUs set up to control the vehicle 1, such as ECU22 and ECU23 for radar, optical radar, and camera, ECU24 and ECU25 for communication control, ECU28 for input and output devices, and ECU20 for automatic driving.

[0068] Figure 2 is a control device 2 ( Figure 1 ) is a block diagram of a structural example provided in vehicle 1. Figure 2 , a top view and a side view are shown schematically of a vehicle 1. As an example, the vehicle 1 is a sedan-type four-wheeled passenger vehicle.

[0069] Figure 2The control device 2 controls various parts of the vehicle 1. The control device 2 includes a plurality of ECUs 20 to ECU 29 that are communicatively connected via an in-vehicle network. Each ECU (Electronic Control Unit) includes a processor represented by a CPU, a storage device such as a semiconductor memory, an interface with an external device, and the like. The storage device stores programs executed by the processor, data used by the processor in processing, and the like. Each ECU may also have a plurality of processors, storage devices, and interfaces, and the like. For example, ECU 20 includes a processor 20a and a memory 20b. The processor 20a executes the commands contained in the program stored in the memory 20b, thereby executing the processing of ECU 20. Alternatively, ECU 20 may also include a dedicated integrated circuit such as an ASIC for executing the processing of ECU 20. The same is true for other ECUs.

[0070] The following describes the functions and the like that are in charge of each of the ECUs 20 to 29. The number of ECUs and the functions that they are in charge of can be appropriately designed for the vehicle 1, and can be subdivided or integrated compared to the present embodiment.

[0071] ECU 20 performs vehicle control related to the automatic driving of vehicle 1 (host vehicle) of this embodiment. In the automatic driving, steering, lane change, acceleration and deceleration of vehicle 1 are automatically controlled. Specific control-related processing related to the automatic driving will be described in detail later.

[0072] ECU21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism for steering the front wheels according to the driver's driving operation (steering operation) of the steering wheel 31. In addition, the electric power steering device 3 includes a motor that exerts a driving force for assisting the steering operation or automatically steering the front wheels, a sensor for detecting the steering angle, etc. When the driving state of the vehicle 1 is automatic driving, the ECU21 automatically controls the electric power steering device 3 in accordance with the instruction from the ECU20 to control the travel direction of the vehicle 1.

[0073] ECU22 and ECU23 control the detection units 41 to 43 for detecting the surrounding conditions of the vehicle and process the information of the detection results. Figure 1 The structure corresponding to the camera CAM is a photographing device (hereinafter sometimes referred to as camera 41) that detects objects in front of the vehicle 1 by photographing. In the case of this embodiment, the camera 41 is installed on the inner side of the front window at the front part of the roof of the vehicle 1 in a manner that can photograph the front of the vehicle 1. By analyzing (image processing) the image captured by the camera 41, the outline of the target object such as the traffic light located in front of the vehicle 1, the display of the traffic light, and the lane dividing line (white line, etc.) on the road can be extracted.

[0074] The detection unit 42 (optical radar detection unit) is a Light Detection and Ranging (LIDAR: optical radar) (hereinafter sometimes referred to as optical radar 42), which detects a target object around the vehicle 1 using light or measures the distance to the target object. Figure 1 In the present embodiment, five optical radars 42 are provided, one at each corner of the front of the vehicle 1, one at the center of the rear, and one at each side of the rear.

[0075] The detection unit 43 (radar detection unit) is a millimeter wave radar (hereinafter sometimes referred to as radar 43), which detects a target object around the vehicle 1 or measures the distance to the target object by radio waves. Figure 1 In the present embodiment, five radars 43 are provided, one at the center of the front portion of the vehicle 1, one at each of the front corners, and one at each of the rear corners.

[0076] ECU22 controls the camera 41 and each optical radar 42 on one side and processes the information of the detection results. ECU23 controls the camera 41 and each radar 43 on the other side and processes the information of the detection results. By having two sets of devices for detecting the surrounding conditions of the vehicle, the reliability of the detection results can be improved. In addition, by having different types of detection units such as cameras, optical radars, and radars, the surrounding environment of the vehicle can be analyzed in many aspects. In this embodiment, ECU22 and ECU23 function as recognition units to recognize the display of the traffic light based on the image captured by the camera 41 (shooting unit). Here, the display of the traffic light includes the lighting color of the traffic light (red, green, yellow) and the indication of the direction of travel of the traffic light shown by the arrow display.

[0077] ECU24 controls the gyro sensor 5, GPS sensor 24b, and communication device 24c, and processes information of detection results or communication results. Gyro sensor 5 detects the rotational motion of vehicle 1. The traveling direction of vehicle 1 can be determined based on the detection result of gyro sensor 5, wheel speed, etc. GPS sensor 24b detects the current position of vehicle 1. Communication device 24c wirelessly communicates with a server that provides map information and traffic information to obtain such information. Communication device 24c wirelessly communicates with an information providing server of a system that monitors the traffic conditions of roads, such as VICS (Vehicle Information and Communication System) (registered trademark), to obtain traffic information indicating the traffic conditions of the road on which vehicle M is traveling and the road on which it is scheduled to travel. The traffic information includes traffic obstacle information such as congestion information on the set route, time information required for passing through congested locations, faulty vehicle / construction information, speed limit / lane restriction information, etc. In addition, communication device 24c obtains map information that maintains display information of signal lights (lighting color, travel mode of signal lights) through wireless communication. The communication device 24c acquires map information that at least holds display information of the traveling direction of a traffic light indicating the direction of an arrow signal as display information of the traffic light through wireless communication, and stores the information in a storage device.

[0078] ECU24 can access the database 24a of map information built in the storage device, and ECU24 can search for routes from the current location to the destination. Database 24a can be configured on the network, and communication device 24c can access database 24a on the network to obtain information. Gyro sensor 5 and GPS sensor 24b are respectively Figure 1 The structure corresponding to the gyro sensor S3 and the GPS sensor S4.

[0079] ECU25 has a communication device 25a that can communicate with other vehicles, roads, or information processing devices such as smart phones. For example, the communication device 25a can communicate with other vehicles in the vicinity to exchange information between vehicles, or can communicate with external information processing devices through wireless communication. The communication device 25a and the communication device 24c are Figure 1 In the present embodiment, the communication device 25a and the communication device 24c function as communication units and receive the display information of the signal light through wireless communication.

[0080] ECU26 controls the power device 6. The power device 6 is a mechanism that outputs a driving force for rotating the driving wheels of the vehicle 1, and includes, for example, an engine and a transmission. In addition, the structure of the power device 6 is not limited to this example, and includes electric vehicles using an electric motor as a power source, hybrid vehicles having both an engine and an electric motor, and the like. Among them, the electric vehicle is driven by, for example, electricity discharged by a battery such as a rechargeable battery, a hydrogen fuel cell, a metal fuel cell, and an alcohol fuel cell.

[0081] The ECU 26 controls the engine output in accordance with the driver's driving operation (accelerator operation or accelerator operation) detected by the operation detection sensor 7a provided on the accelerator pedal 7A, or controls the engine output based on the vehicle speed sensor 7c ( Figure 1 The transmission is switched according to the vehicle speed and other information detected by the vehicle speed sensor S5. When the driving state of the vehicle 1 is automatic driving, the ECU 26 automatically controls the power unit 6 in accordance with the instruction from the ECU 20 to control the acceleration and deceleration of the vehicle 1.

[0082] The ECU 27 controls the lighting devices (headlights, taillights, etc.) including the direction indicator 8 (turn indicator). Figure 1 In the example of FIG. 1 , the direction indicators 8 are provided at the front portion, the door mirrors, and the rear portion of the vehicle 1 .

[0083] ECU28 controls the input-output device 9. The input-output device 9 outputs information to passengers including the driver and accepts input of information from the passengers. The sound output device 91 reports information to the passengers by sound. The display device 92 reports information to the driver by displaying images. The display device 92 is, for example, arranged in front of the driver's seat and the front passenger seat, and constitutes a touch panel-type instrument panel that functions as a human-machine interface. In the present embodiment, ECU28 functions as a control unit, determines the display of the signal light based on the display of the signal light recognized by the recognition unit (ECU22, ECU23) and the display information received by the communication unit (communication device 24c, 25a), and displays the determined display of the signal light on the display device 92 (display unit).

[0084] On the route from the current location to the destination searched by ECU 24, ECU 28 performs display control to display road information including multiple lanes currently traveled by vehicle 1 obtained from the position information and map information of vehicle 1 (host vehicle) on display device 92. In addition, ECU 28 controls the sound output device 91 and display device 92 to provide the driver with sound, map display, and guidance information based on sound.

[0085] In addition, although sound and display are exemplified here, information may also be reported by vibration or light. In addition, multiple of sound, display, vibration or light may be combined to report information. Furthermore, the combination may be different or the notification method may be different depending on the level of the information to be notified (e.g., urgency).

[0086] The input device 93 is disposed at a position where the driver can operate it, and includes a switch group for inputting instructions to the vehicle 1 and a voice input device for inputting voices of passengers.

[0087] ECU29 controls the braking device 10 and the parking brake (not shown). The braking device 10 is, for example, a disc braking device, which is provided on each wheel of the vehicle 1 and decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheel. ECU29 controls the operation of the braking device 10 in accordance with the driver's driving operation (brake operation) detected by the operation detection sensor 7b provided on the brake pedal 7B. When the driving state of the vehicle 1 is automatic driving, ECU29 automatically controls the braking device 10 in accordance with the instructions from ECU20 to control the deceleration and stopping of the vehicle 1. The braking device 10 and the parking brake can also be operated to maintain the stopped state of the vehicle 1. In addition, when the transmission of the power unit 6 is equipped with a parking lock mechanism, it can also be operated to maintain the stopped state of the vehicle 1.

[0088] (Overview of the process)

[0089] Next, an overview of the processing executed by the control device 2 will be described. Figure 4 Schematically shows a road on which the vehicle 1 is traveling. Figure 4 As shown, vehicle 1 is traveling in a lane LN1 in which it can go straight or turn left, in the direction of arrow 440. Lane LN2 adjacent to lane LN1 is a right-turn lane, and lane LN3 is an oncoming lane. Other vehicles 431 and 432 can travel in the opposite direction to arrow 440.

[0090] The first intersection 410 and the second intersection 420 are located in front of the vehicle 1. The first intersection 410 is the intersection closest to the traveling position of the vehicle 1, and the second intersection 420 is the intersection separated from the first intersection 410 by a distance L1.

[0091] A first signal light 411 is provided at the first intersection 410. The first signal light 411 having a lighted display portion of "red", "yellow" and "green" is provided with arrow signal portions 412 and 413 for indicating the direction of travel by means of arrow signals. Figure 4In the display example shown, the lighting color of the first signal light 411 is "red" (shaded display), and the arrow signal of the arrow signal portion 412 indicating the instruction to turn right is in a lit state. In addition, the arrow signal of the arrow signal portion 413 indicating the instruction to go straight is in an extinguished state (non-lit state). By the lighting display of the arrow signal portion 412 indicating the instruction to turn right, the other vehicle 415 is in a state of turning right from the lane LN2 at the first intersection 410.

[0092] A second signal light 421 is provided at the second intersection 420. The second signal light 421 having a lighted display portion of "red", "yellow" and "green" is provided with arrow signal portions 422 and 423 for indicating the direction of travel by means of arrow signals. Figure 4 In the display example shown, the lighting color of the second signal lamp 421 is "red" (shaded display), and the arrow signal of the arrow signal portion 422 indicating the instruction of right turn is in an off state (non-lit state). In addition, the arrow signal of the arrow signal portion 423 indicating the instruction of straight travel is in a lit state. By the lighting display of the arrow signal portion 423 indicating the instruction of straight travel, the other vehicle 425 is in a state of going straight from the lane LN1 to the second intersection 420.

[0093] In addition, in the present embodiment, the case of "red" is illustrated as the lighting color of each signal light, but in the processing of the control device 2 in the present embodiment, the lighting color is not limited to "red", and the same applies to the lighting colors of "yellow" and "green". In addition, as the display example of the arrow signal, the display examples of right turn and straight are shown, but even if it is the display of the arrow signal of left turn, the control device 2 can also process it in the same way.

[0094] An external communication device 414 is arranged at a distance L2 before the first intersection 410, and an external communication device 424 is arranged at a distance L3 before the second intersection 420. The external communication devices 414 and 424 are configured as external communication devices of a cellular type using a beacon system or a mobile communication network standardized by a communication standard for mobile phones, for example. The external communication devices 414 and 424 have a communication function based on V2X (Vehicle to Everything), and the external communication devices 414 and 424 communicate with a signal control system that controls the display of the signal lights (the first signal light 411 and the second signal light 421), and publish the display information of the signal lights received from the signal control system to the mobile body. The external communication devices 414 and 424 are arranged above the lanes (LN1 and LN2). When the moving body travels below the external communication devices 414 and 424, the communication unit (communication devices 24c and 25a) of the moving body can receive the display information of the traffic lights sent from the external communication devices 414 and 424.

[0095] The communication unit (communication devices 24c, 25a) of the mobile body receives the display information of the traffic light when traveling under the external communication devices 414, 424. For example, the display information of the traffic light includes at least the following information.

[0096] First signal light 411: current lighting color "red"

[0097] Arrow signal display status (lit for right turn, not lit for straight going)

[0098] Next light signal "green", change the scheduled time (T1 second later)

[0099] Second signal light 421: current lighting color "red"

[0100] Arrow signal display status (right turn not lit, straight ahead lit)

[0101] The next light turns green and changes the scheduled time (T2 seconds later).

[0102] A signal control system that controls the display of a traffic light sometimes changes the control timing of the display of a traffic light (lighting color, indication of the direction of travel) according to the congestion of an intersection in order to achieve smooth traffic. Even when the display information of a traffic light is changed, the cellular external communication device can communicate the changed display information of the traffic light in real time.

[0103] On the other hand, in the case of a beacon-type external communication device, for example, the display information received by the communication unit (communication device 24c, 25a) of the mobile body from the external communication device 414 is not updated until the display information of the changed signal light is received from the next external communication device 424. In the present embodiment, the predetermined time (reference time) from the time of traveling under the external communication device is set as the time during which the display information of the signal light received from the beacon-type or cellular-type external communication device 414, 424 can maintain accurate information.

[0104] (Processing Flow)

[0105] Next, the flow of processing executed by the control device 2 will be described. Figure 3A is a diagram illustrating the flow of processing by the control device 2. Figure 5-6 1 is a diagram schematically showing the internal processing of the ECU 28 for determining the display of the signal lamp.

[0106] exist Figure 3A In S301, the camera 41 (CAM) captures the outside of the moving object (vehicle). In S302, the recognition unit (ECU22, ECU23) recognizes the display of the signal light contained in the image (captured image) captured by the camera 41 (CAM). The recognition unit (ECU22, ECU23) can extract the signal light contained in the captured image and recognize the display of the signal light by performing image processing on the captured image.

[0107] In S303, the recognition unit (ECU22, ECU23) determines whether a plurality of displays are acquired as the display of the signal light. If a plurality of displays are not acquired (S303-No), the recognition unit (ECU22, ECU23) advances the process to S305. On the other hand, if it is determined by the determination process of S303 that a plurality of displays are acquired (S303-"Yes"), the recognition unit (ECU22, ECU23) advances the process to S304.

[0108] In S304, the recognition unit (ECU22, ECU23) integrates the displays of the plurality of signal lights. The displays of the plurality of signal lights are, for example, Figure 4 In a road structure where the first intersection 410 and the second intersection 420 are connected as shown, it is possible that the display of the first signal light 411 and the display of the second signal light 421 are captured by the camera 41 . Figure 5 Schematically showing the integration process of the display of a plurality of signal lights. The display 510 of the first signal light is based on the display of the first signal light 411 ( Figure 4 ) and the display of the second signal light 520 is based on the display of the second signal light 421 ( Figure 4 ) and the display identified.

[0109] The integration processing of the display of the signal light is the addition of the display parts, and the recognition unit (ECU22, ECU23) obtains the integrated signal display 530 obtained by adding the display of the right turn arrow signal in the display 510 of the first signal light and the display of the straight arrow signal in the display 520 of the second signal light. The lighting color is "red" which is common to the display 510 of the first signal light and the display 520 of the second signal light, and the recognition unit (ECU22, ECU23) obtains the integrated signal display 530 with the lighting color "red".

[0110] exist Figure 5 In the example shown, the lighting colors are consistent, but the present invention is not limited to this display example. As for the case where the lighting colors are different, for example, when the lighting color in the display 510 of the first signal light is "green" and the lighting color in the display 520 of the second signal light is "red", the recognition unit (ECU22, ECU23) can obtain the integrated signal display 530 of the lighting colors ("red", "green") obtained by adding these lighting colors.

[0111] Description Return to Figure 3A In S305, the communication unit (communication device 24c, 25a) receives the display information of the signal light from the external communication device 414, 424 through wireless communication.

[0112] In S306, ECU 28 (control unit) compares the recognition result (S302, S304) of the display of the signal lamp with the display information (S305).

[0113] Then, in S307, ECU28 (control unit) determines the display of the traffic light based on the comparison result of S306. ECU28 (control unit) determines the driving position of the moving body based on the position information of the moving body acquired by the position information acquisition unit (ECU24, gyro sensor 5, GPS sensor 24b). ECU28 (control unit) determines the display of the traffic light related to the intersection closest to the driving position among the indications of the traveling direction of the traffic light recognized by the recognition unit (ECU22, ECU23).

[0114] Here, Figure 6 The diagram schematically shows a comparison process ( S306 ) of the recognition result of the display of the traffic light and the display information 610 , 620 received by communication, and a determination process ( S307 ) of the display of the traffic light. Figure 6 ST61 schematically shows the comparison process and the decision process of the display of the traffic light based on the display information 610. Here, the display information 610 shows the display state in which the lighting color is red and the indication of the traveling direction is a right turn.

[0115] in addition, Figure 6 ST62 schematically shows the comparison process based on the display information 620 and the decision process of the display of the traffic light. The lighting color of the display information 620 is red, and the indication of the traveling direction shows the display state of going straight.

[0116] In this embodiment, the ECU 28 (control unit) can function as a lighting color determination unit and a traveling direction determination unit as a functional structure. Here, the lighting color determination unit compares the lighting color of the signal light identified by the identification unit (ECU 22, ECU 23) with the lighting color of the signal light obtained from the display information based on the wireless communication of the communication device 24c, and determines the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on the comparison result.

[0117] In addition, the travel direction determination unit compares the travel direction indication of the signal light identified by the identification unit (ECU22, ECU23) with the travel direction indication of the signal light obtained from the display information based on wireless communication of the communication device 24c, and determines the travel direction indication that is consistent with the comparison result as the travel direction indication displayed on the display unit.

[0118] (Light up color decision)

[0119] Figure 7 1 is a diagram showing an example of setting a predetermined priority order related to the lighting color of a signal light, wherein the lighting color determination unit determines the lighting color according to the predetermined priority order ( Figure 7 ) to determine the lighting color of the signal light displayed on the display. Figure 7 The predetermined priority order shown is an example, and although “green” and “red” are described as examples of lighting colors, it is also possible to set a predetermined priority order including three colors without excluding “yellow”.

[0120] exist Figure 7 In the example of setting the priority order shown, when the display color 701 recognized by the recognition unit (ECU 22, ECU 23) and the display information 702 received by the communication unit (communication device 24c) are consistent (710, 750), the lighting color determination unit determines the lighting color that is consistent with the two as the lighting color of the signal light displayed on the display unit. In this case, the lighting color determination unit determines the remaining time until the current lighting color is changed together with the lighting color based on the received display information.

[0121] In the priority setting example, when the display color 701 recognized by the recognition unit (ECU22, ECU23) is inconsistent with the display information 702 received by the communication unit (720, 740), the lighting color determination unit determines the lighting color "red" as the lighting color of the signal light displayed on the display unit in order to prevent entry into the intersection when the light is red.

[0122] In the case where the information to be compared cannot be obtained, or the state of the camera 41 cannot be matched with the state of the control device 2 (730, 760, 770), "unknown" indicating undetermined information is set for the recognized display color 701 and the display information 702. In the case where undetermined information ("unknown") is set for at least one of the recognized display color 701 and the display information 702, the lighting color determination unit determines the lighting color of the signal light displayed on the display unit based on the information that can be obtained according to a predetermined priority order. For example, in the case where the display information 702 cannot be received from the communication unit and undetermined information ("unknown") is set in the display information 702 (730, 760), the lighting color determination unit determines the lighting color of the signal light displayed on the display unit based on the display color recognized by the recognition unit (ECU22, ECU23).

[0123] Regarding the lighting color of the signal light, Figure 6 In the example of the decision processing (ST61, ST62), the lighting color (red) of the integrated signal display 530 and the display information 610 is consistent, and the lighting color decision unit determines the lighting color according to the predetermined priority order ( Figure 7 750), a consistent lighting color (red) is determined as the lighting color of the signal light displayed on the display unit.

[0124] (Determination of the direction of travel)

[0125] The traveling direction determination unit excludes the traveling direction indications of the signal lights recognized by the recognition unit (ECU22, ECU23) that are inconsistent with the traveling direction indications of the signal lights acquired from the display information based on wireless communication. That is, the traveling direction determination unit determines the traveling direction indications of the signal lights that are consistent with the traveling direction indications of the signal lights acquired from the display information based on wireless communication, among the traveling direction indications of the signal lights recognized by the recognition unit (ECU22, ECU23), as the traveling direction indications displayed on the display unit.

[0126] When the recognition unit (ECU 22, ECU 23) acquires the displays of a plurality of signal lights included in the image captured by the camera 41, it recognizes the integrated signal display 530 that integrates the displays of the plurality of signal lights as the display of the signal lights ( Figure 5The traveling direction determination unit determines the traveling direction indication that matches the comparison result between the integrated signal display 530 and the display information (610, 620) based on the wireless communication as the traveling direction indication to be displayed on the display unit.

[0127] Figure 8 This is a diagram showing an example of determining the indication of the traveling direction of a traffic light, in which the traveling direction determination unit compares the arrow signal 801 (indication of the traveling direction of the traffic light) identified by the identification unit (ECU22, ECU23) with the indication of the traveling direction of the traffic light obtained from the display information 802 based on the wireless communication of the communication device 24c, and determines the indication of the traveling direction that is consistent with the comparison result as the indication of the traveling direction displayed on the display unit.

[0128] like Figure 8 As shown, when the arrow signal 801 recognized by the recognition unit (ECU22, ECU23) is consistent with the display information 802 received by the communication unit (communication device 24c) (810, 840), the traveling direction determination unit determines the indication of the traveling direction that is consistent with the comparison result as the indication of the traveling direction displayed on the display unit (803). In this case, the traveling direction determination unit determines the remaining time for changing the display of the current arrow signal together with the indication of the traveling direction (the direction of the arrow signal) based on the received display information 802. In addition, when the comparison results are inconsistent (820, 830), the traveling direction determination unit determines that there is no indication of the traveling direction (803).

[0129] Regarding arrow signals, Figure 6 In the example of decision processing (ST61), by comparing the identified arrow signal (straight ahead, right turn) with the display information (right turn) received by the communication unit (communication devices 24c, 25a), the travel direction determination unit determines the travel direction indication (right turn) that is consistent with the comparison result as the travel direction indication displayed on the display unit.

[0130] In addition, Figure 6 In the example of decision processing (ST62), by comparing the identified arrow signal (straight ahead, right turn) with the display information (straight ahead) received by the communication unit (communication devices 24c, 25a), the travel direction determination unit determines the travel direction indication (straight ahead) that is consistent with the comparison result as the travel direction indication displayed on the display unit.

[0131] When the communication unit (communication device 24c, 25a) receives the display information of the traffic light through wireless communication with the external communication device 414, 424 arranged above the lane where the moving body is traveling, in addition to sending the lighting color of the traffic light arranged in the direction of travel and the direction of travel indication, the minimum lighting time and the maximum lighting time of the traffic light are also sent from the external communication device 414, 424. For example, there is a case where the lighting color is green, the minimum lighting time is 0 seconds, and the maximum lighting time is 10 seconds. In this case, although the acquisition of the traffic light information based on the communication is successful, there is a possibility that the traffic light of the displayed object will change to yellow lighting after 0 seconds, and the lighting prediction information of the traffic light may become unreliable.

[0132] Therefore, as in the present embodiment, by comparing the signal light information acquired by the communication unit with the information displayed by the signal light identified by the imaging unit, the display of the signal light to be displayed is determined, and a high-precision display can be performed. In addition, when comparing, the reference time may be set in consideration of the minimum lighting time. That is, when the minimum reference time at which the display of the signal light changes is longer than the time estimated for the moving body to enter the intersection, the signal light information acquired by the communication unit is prioritized, and when the minimum reference time is shorter than the estimated time, the information of the communication unit and the imaging unit are compared.

[0133] (Variation Example)

[0134] In the first embodiment, the basic processing of the lighting color determination unit and the travel direction determination unit is described. In this example, the processing (priority determination processing) in which ECU28 (control unit) determines the display of the traffic light (lighting color, indication of the travel direction) by giving priority to the information received from the external communication devices 414, 424 within a reference time from the moment when the display information is received from the external communication devices 414, 424 is described. Then, after the reference time has passed, ECU28 (control unit) changes the setting in such a way as to cancel the setting of the priority determination processing and return to the basic processing described in the first embodiment. The priority determination processing described in this example is, for example, Figure 3A The processing executed in S305. Figure 3B It is a diagram for explaining the flow of processing of the control device 2 related to the priority determination processing.

[0135] exist Figure 3B In S310, the communication unit (communication device 24c, 25a) receives the display information of the signal light from the external communication device 414, 424 through wireless communication.

[0136] In S311, ECU 28 (control unit) determines the display of the traffic light related to the intersection closest to the traveling position based on the received display information.

[0137] In this step, the lighting color determination unit determines the lighting color according to the predetermined priority order ( Figure 7 ), the lighting color of the signal light displayed on the display unit is determined based on the lighting color of the signal light obtained from the display information (priority determination processing of the lighting color). In addition, the traveling direction determination unit determines the indication of the traveling direction displayed on the display unit based on the indication of the traveling direction of the signal light obtained from the display information, regardless of the result of the comparison, within a reference time from the reception of the display information (priority determination processing of the indication of the traveling direction).

[0138] In S312, ECU 28 (control unit) starts measuring time based on the internal clock upon receipt of the display information.

[0139] In S313, ECU28 (control unit) determines whether the measured time has passed the reference time. If the reference time has not passed (S313-No), the process returns to S311 and the same process is repeatedly performed. On the other hand, in the determination of S313, if the reference time has passed (S313-Yes), ECU28 (control unit) causes the process to enter S314.

[0140] In S314, ECU28 (control unit) cancels the setting of the priority determination process performed in S311. After the reference time has passed, the lighting color determination unit determines the lighting color in accordance with a predetermined priority order ( Figure 7 ) to determine the lighting color of the signal light displayed on the display unit. In addition, after a reference time has passed, the traveling direction determination unit determines the traveling direction indication that matches the comparison result as the traveling direction indication displayed on the display unit, regardless of the priority determination processing of the traveling direction indication.

[0141] [Second embodiment]

[0142] In the first embodiment, a structure for determining the display of a signal light based on the display of the signal light recognized by the recognition unit (ECU 22, ECU 23) and the display information received by the communication unit (communication device 24c, 25a) is described. In the second embodiment, a structure for determining the display of the signal light by combining map information that holds the display information of the traveling direction of the signal light is described. The structure of the control device 2 is similar to that described in the first embodiment. Figure 1 , Figure 2 same.

[0143] Figure 3C FIG. 2 is a diagram for explaining the flow of processing performed by the control device according to the second embodiment. Figure 3C In the embodiment, the processes of S301 to S305, S306, and S307 are the same as those described in the first embodiment, and the repeated description is omitted. The processes of the different steps are described below.

[0144] After the integration process in S304, Figure 3C In S308, the communication unit (communication device 24c, 25a) acquires map information that maintains display information of the traffic light (lighting color, direction of travel of the traffic light) through wireless communication. The map information at least maintains display information indicating the direction of the arrow signal as display information of the traffic light. In addition, the acquisition of map information is not limited to wireless communication, and the control device 2 may also pre-have map information that at least maintains display information of the direction of travel of the traffic light indicating the direction of the arrow signal in a storage device or the like as display information of the traffic light.

[0145] In S309 , the ECU 28 (control unit) compares the integrated signal display 530 acquired in S304 with the map information ( S308 ).

[0146] Fig. 9 Schematically shows the comparison process of the integrated signal display 530 (S304) and the map information 910 (S308). Among the various information stored in the map information, Fig. 9 The map information 910 shown shows the display information of the traffic light related to the intersection at the position closest to the traveling position of the moving body. The map information 910 shows, as the display information of the traffic light, the presence or absence of a display part having three colors ("red", "yellow" and "green"), and the arrow signal part indicating the right turn as the direction of travel. The information of the lighted display part does not include the color limitation information related to the currently lit lighted color.

[0147] Based on the comparison process, ECU28 (control unit) obtains the intermediate stage signal light display 920. In the comparison process between the integrated signal display 530 and the map information 910, the lighting color cannot be limited, so ECU28 (control unit) sets the lighting color display (red) in the integrated signal display 530 as the intermediate stage signal light display 920.

[0148] In addition, regarding the arrow signal, the travel direction determination unit determines (sets) the travel direction indication of the signal light identified by the identification unit (ECU22, ECU23) that is consistent with the display information (right turn) maintained by the map information 910 at the intersection closest to the driving position (for example, straight ahead, right turn) as the travel direction indication displayed on the display unit.

[0149] That is, the traveling direction determination unit determines (sets) the traveling direction indications (right turn) of the remaining traffic lights as the traveling direction indications displayed on the display unit, except for the traveling direction indications (straight) of the traffic lights that are inconsistent with the display information stored in the map information 910. The determination in this step is an intermediate stage determination, and the final display of the traffic lights is performed based on the following processing of S310 and S307.

[0150] In S310, ECU 28 (control unit) compares the display 920 of the intermediate-stage signal lamp (S309) with the display information (S305).

[0151] Then, in S307, ECU28 (control unit) determines the display of the traffic light based on the comparison result of S306. ECU28 (control unit) determines the driving position of the moving body based on the position information of the moving body acquired by the position information acquisition unit (ECU24, gyro sensor 5, GPS sensor 24b), and ECU28 (control unit) determines the display of the traffic light related to the intersection closest to the driving position based on the comparison of the display 920 (S309) of the traffic light in the middle stage with the display information (S305).

[0152] Here, Fig.10 The diagram schematically shows a comparison process ( S310 ) of the intermediate-stage traffic light display 920 and the display information 1010 , 1020 received by communication, and a decision process ( S307 ) of the traffic light display. Fig.10 ST101 schematically shows the comparison process and the decision process of the display of the traffic light based on the display information 1010. Here, the display information 1010 shows the display state in which the lighting color is red and the indication of the traveling direction is a right turn.

[0153] in addition, Fig.10 ST102 schematically shows the comparison process based on the display information 1020 and the decision process of the display of the traffic light. The lighting color of the display information 1020 is red, and the indication of the traveling direction indicates the display state of going straight.

[0154] Regarding the lighting color of the signal light, Fig.10In the example of the decision processing (ST101, ST102), the display 920 of the intermediate stage signal light is consistent with the lighting color (red) of the display information 1010, and the lighting color decision unit determines the lighting color according to the predetermined priority order ( Figure 7 750), a consistent lighting color (red) is determined as the lighting color of the signal light displayed on the display unit.

[0155] Regarding arrow signals, Fig.10 In the example of decision processing (ST101), by comparing the arrow signal (right turn) displayed by the traffic light 920 in the intermediate stage with the display information 1010 (right turn) received by the communication unit (communication devices 24c, 25a), the travel direction determination unit determines the travel direction indication (right turn) that is consistent with the comparison result as the travel direction indication displayed on the display unit.

[0156] In addition, Fig.10 In the example of decision processing (ST102), by comparing the arrow signal (right turn) 920 of the traffic light display in the intermediate stage with the display information 1020 (straight ahead) received by the communication unit (communication devices 24c, 25a), the indicated directions of the arrow signals are inconsistent, and the travel direction determination unit determines that there is no indication of the travel direction.

[0157] (Variation Example)

[0158] When the communication unit (communication device 24c, 25a) cannot obtain display information and map information based on wireless communication, the recognition unit (ECU22, ECU23) does not integrate the displays of the plurality of signal lights even when the displays of the plurality of signal lights included in the image captured by the camera 41 are obtained, but determines the display of the signal light with higher recognition accuracy among the displays of the plurality of signal lights recognized by image processing of the image. The recognition unit can determine the display of the signal light with higher recognition accuracy based on a comparison of the results of image processing such as the resolution and pixel value of the image of the signal light extracted from the image. At this time, the lighting color determination unit of the control unit (ECU28) determines the lighting color of the signal light based on the display of the signal light determined by the recognition unit, and the travel direction determination unit determines the indication of the travel direction displayed on the display unit based on the display of the signal light determined by the recognition unit.

[0159] In addition, when the recognition unit (ECU22, ECU23) cannot recognize the traffic light based on the image captured by the camera 41 and only obtains the display information through wireless communication, the lighting color determination unit and the travel direction determination unit determine the display of the traffic light based on the display information obtained through wireless communication, and the control unit (ECU28) displays the determined display of the traffic light on the display unit.

[0160] (Summary of Implementation and Modifications)

[0161] The above-mentioned embodiments disclose at least the following control device, a moving object (vehicle) having the control device, a control method, and a program.

[0162] Configuration 1. The control device of the above embodiment is mounted on a mobile body (1), and displays a display of a signal light included in an image captured by a camera (41) capturing the outside of the mobile body on a display unit (92), wherein:

[0163] The control device (2) comprises:

[0164] a recognition unit (22, 23) for recognizing the display of the signal light based on the image;

[0165] a communication unit (24c, 25a) for receiving display information of the signal light through wireless communication; and

[0166] a control unit (28) that determines the display of the signal light based on the display of the signal light recognized by the recognition unit and the display information received by the communication unit, and displays the determined display of the signal light on the display unit,

[0167] The control unit (28) comprises:

[0168] a lighting color determination unit that compares the lighting color of the signal light identified by the identification unit with the lighting color of the signal light acquired from the display information based on the wireless communication, and determines the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison; and

[0169] A traveling direction determination unit compares the traveling direction indication of the signal light identified by the identification unit with the traveling direction indication of the signal light obtained from the display information based on the wireless communication, and determines the traveling direction indication that is consistent with the comparison result as the traveling direction indication displayed on the display unit.

[0170] According to the control device of Configuration 1, the display of the signal light can be determined based on the comparison between the recognition result of the imaging unit and the display information received by the communication unit through communication.

[0171] Configuration 2. In the control device of the above embodiment, the control unit (28) determines the travel position of the mobile body based on the position information of the mobile body acquired by the position information acquisition unit,

[0172] The control device includes map information as display information of the traffic light, and the map information at least holds display information of the traveling direction of the traffic light indicating the direction of the arrow signal.

[0173] The traveling direction determination unit will also determine, among the traveling direction indications of the traffic light identified by the identification unit, the traveling direction indication of the traffic light at the intersection closest to the driving position and consistent with the display information maintained by the map information, as the traveling direction indication displayed on the display unit.

[0174] Configuration 3. In the control device of the above embodiment, the traveling direction determining unit is configured as follows:

[0175] The method further comprises excluding, from among the indications of the travel direction of the signal light recognized by the recognition unit, indications of the travel direction that are inconsistent with the indications of the travel direction of the signal light acquired from the display information based on the wireless communication, and

[0176] excluding the indication of the traveling direction of the traffic light which is inconsistent with the display information held in the map information,

[0177] The remaining traveling direction indications of the signal lights are determined as the traveling direction indications displayed on the display unit.

[0178] Configuration 4. In the control device of the above embodiment, the traveling direction determining unit is configured as follows:

[0179] Among the indications of the travel direction of the signal light identified by the identification unit, the indication of the travel direction of the signal light that is consistent with the indication of the travel direction of the signal light obtained from the display information based on the wireless communication and consistent with the display information maintained by the map information is determined as the indication of the travel direction displayed on the display unit.

[0180] According to the control device of configurations 2 to 4, the display of the traffic light can be determined based on the recognition result of the imaging unit and the comparison between the display information received by the communication unit and the map information.

[0181] Configuration 5. In the control device of the above embodiment, when the display of a plurality of signal lights included in the image is acquired, the recognition unit (22, 23) recognizes an integrated signal display obtained by integrating the displays of the plurality of signal lights as the display of the signal light,

[0182] The traveling direction determination unit determines, as the traveling direction indication to be displayed on the display unit, an indication of the traveling direction that matches a result of a comparison between the integrated signal display and the display information based on the wireless communication.

[0183] According to the control device of structure 5, when the displays of multiple signal lights included in the captured image are obtained, the display (integrated signal display) obtained by integrating the displays of the multiple signal lights can be obtained as a comparison object, and the display of the signal lights can be determined based on the comparison results of the comparison processing using the integrated signal display.

[0184] Configuration 6. In the control device of the above embodiment, the communication unit (24c, 25a) receives display information of the traffic light arranged at the intersection closest to the driving position through wireless communication with an external communication device (414, 424) arranged above the lane in which the moving body is traveling,

[0185] The external communication device communicates with a signal control system that controls the display of the signal light, and distributes the display information of the signal light received from the signal control system.

[0186] According to the control device of structure 6, the communication unit of the mobile body can receive the display information of the signal light by communicating with the external communication device, and by using the received display information, it can determine the display of the signal light based on the recognition result of the shooting unit and the comparison of the display information received by the communication unit through communication.

[0187] Configuration 7. In the control device of the above embodiment, when the communication unit (24c, 25a) receives the display information of the signal light from the external communication device (414, 424),

[0188] The lighting color determination unit determines the lighting color of the signal light to be displayed on the display unit based on the lighting color of the signal light acquired from the display information, regardless of a predetermined priority order based on a result of the comparison, within a reference time from when the display information is received,

[0189] The traveling direction determination unit determines the traveling direction indication to be displayed on the display unit based on the traveling direction indication of the traffic light acquired from the display information, regardless of a result of the comparison, within a reference time from receiving the display information.

[0190] Configuration 8. In the control device of the above embodiment, the lighting color determination unit determines the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on the result of the comparison after the reference time has passed,

[0191] The traveling direction determination unit determines, after the reference time has elapsed, the traveling direction indication that matches the comparison result as the traveling direction indication to be displayed on the display unit.

[0192] According to the control device of Structure 7 and Structure 8, even when the signal control system changes the control timing of the display of the traffic light according to the congestion condition of the intersection, it is possible to reduce the impact of the change in the control timing by switching the comparison processing before and after a reference time that can maintain accurate information for the displayed information, thereby determining the display of the traffic light with high precision.

[0193] Configuration 9. In the control device of the above embodiment, when the communication unit (24c, 25a) cannot obtain the display information and the map information based on the wireless communication,

[0194] The recognition unit (22, 23) does not integrate the displays of the plurality of traffic lights even when the displays of the plurality of traffic lights included in the image are acquired.

[0195] The recognition unit (22, 23) determines the display of a signal light having a higher recognition accuracy among the displays of the plurality of signal lights recognized by image processing of the image,

[0196] The lighting color determination unit determines the lighting color of the signal light based on the display of the signal light determined by the recognition unit,

[0197] The traveling direction determination unit determines an indication of the traveling direction to be displayed on the display unit based on the display of the signal lamp determined by the recognition unit.

[0198] According to the control device of configuration 9, when the display information and the map information used in the comparison process cannot be acquired, the display of the traffic light can be determined based on the display of the traffic light with higher recognition accuracy determined by the recognition unit.

[0199] Configuration 10. The moving object of the above embodiment has the control device (2) described in any one of Configurations 1 to 9.

[0200] According to the moving object of Configuration 10, it is possible to provide a moving object including the control device described in any one of Configurations 1 to 9.

[0201] Configuration 11. The control method of the above embodiment is a control method of a control device (2) mounted on a mobile body (1), which displays a display of a signal light included in an image captured by a camera (41) capturing the outside of the mobile body on a display unit, wherein:

[0202] The control method has the following features:

[0203] a recognition step (S302), in which the recognition unit (22, 23) recognizes the display of the signal light based on the image;

[0204] a communication step (S305), in which the communication unit (24c, 25a) receives display information of the signal light through wireless communication; and

[0205] a control step (S306, S307), in which the control unit determines the display of the signal light based on the display of the signal light identified in the identification step and the display information received in the communication step, and displays the determined display of the signal light on the display unit,

[0206] In the control step, there are:

[0207] a lighting color determining step of comparing the lighting color of the signal light identified in the identifying step with the lighting color of the signal light acquired from the display information based on the wireless communication, and determining the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison; and

[0208] A traveling direction determining step, in which the traveling direction indication of the signal light identified in the identifying step is compared with the traveling direction indication of the signal light obtained from the display information based on the wireless communication, and the traveling direction indication that is consistent with the comparison result is determined as the traveling direction indication displayed on the display unit.

[0209] According to the control method of configuration 11, the display of the signal light can be determined based on the comparison between the recognition result of the imaging unit and the display information received by the communication unit through communication.

[0210] Configuration 12. In the program of the above-mentioned embodiment, each step of the control method described in Configuration 11 is executed on a computer.

[0211] According to this configuration, it is possible to provide a program that causes a computer to execute each step of the above-described control method.

[0212] (Other embodiments)

[0213] The present invention can also provide a program that realizes the functions of the above-mentioned embodiments to a system or control device for controlling a mobile body via a network or a storage medium, and one or more processors in a computer of the system or control device read the program and execute the processing of the control device.

[0214] The present invention is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the gist of the invention.

Claims

1. A control device, the control device being mounted on a mobile body, and causing a display unit to display a signal light included in an image captured by a capturing unit capturing an external environment of the mobile body, It is characterized in that The control device comprises: a recognition unit for recognizing the display of the signal light according to the image; a communication unit that receives display information of the signal light through wireless communication; and a control unit that determines the display of the signal light based on the display of the signal light recognized by the recognition unit and the display information received by the communication unit, and displays the determined display of the signal light on the display unit, The control unit comprises: a lighting color determination unit that compares the lighting color of the signal light identified by the identification unit with the lighting color of the signal light acquired from the display information based on the wireless communication, and determines the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison; as well as A traveling direction determination unit compares the traveling direction indication of the signal light identified by the identification unit with the traveling direction indication of the signal light obtained from the display information based on the wireless communication, and determines the traveling direction indication that is consistent with the comparison result as the traveling direction indication displayed on the display unit.

2. The control device according to claim 1, characterized in that: The control unit determines the traveling position of the moving body based on the position information of the moving body acquired by the position information acquisition unit, The control device includes map information as display information of the traffic light, and the map information at least holds display information of the traveling direction of the traffic light indicating the direction of the arrow signal. The traveling direction determination unit will also determine, among the traveling direction indications of the traffic light identified by the identification unit, the traveling direction indication of the traffic light at the intersection closest to the driving position and consistent with the display information maintained by the map information, as the traveling direction indication displayed on the display unit.

3. The control device according to claim 2, characterized in that: The traveling direction determining unit is configured as follows: The method further comprises excluding, from among the indications of the travel direction of the signal light recognized by the recognition unit, indications of the travel direction that are inconsistent with the indications of the travel direction of the signal light acquired from the display information based on the wireless communication, and excluding the indication of the traveling direction of the traffic light which is inconsistent with the display information held in the map information, The remaining traveling direction indications of the signal lights are determined as the traveling direction indications displayed on the display unit.

4. The control device according to claim 2, characterized in that: The traveling direction determining unit is configured as follows: Among the indications of the travel direction of the signal light identified by the identification unit, the indication of the travel direction of the signal light that is consistent with the indication of the travel direction of the signal light obtained from the display information based on the wireless communication and consistent with the display information maintained by the map information is determined as the indication of the travel direction displayed on the display unit.

5. The control device according to any one of claims 1 to 4, characterized in that: When the displays of a plurality of signal lights included in the image are acquired, the recognition unit recognizes an integrated signal display obtained by integrating the displays of the plurality of signal lights as the display of the signal light, The traveling direction determination unit determines, as the traveling direction indication to be displayed on the display unit, an indication of the traveling direction that matches a result of a comparison between the integrated signal display and the display information based on the wireless communication.

6. The control device according to claim 2, characterized in that: The communication unit receives display information of a traffic light disposed at an intersection closest to the driving position through wireless communication with an external communication device disposed above a lane in which the moving body is traveling. The external communication device communicates with a signal control system that controls the display of the signal light, and distributes the display information of the signal light received from the signal control system.

7. The control device according to claim 6, characterized in that: When the communication unit receives the display information of the signal light from the external communication device, The lighting color determination unit determines the lighting color of the signal light to be displayed on the display unit based on the lighting color of the signal light acquired from the display information, regardless of a predetermined priority order based on a result of the comparison, within a reference time from when the display information is received, The traveling direction determination unit determines the traveling direction indication to be displayed on the display unit based on the traveling direction indication of the traffic light acquired from the display information, regardless of a result of the comparison, within a reference time from receiving the display information.

8. The control device according to claim 7, characterized in that: The lighting color determination unit determines the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison after the reference time has elapsed, The traveling direction determination unit determines, after the reference time has elapsed, the traveling direction indication that matches the comparison result as the traveling direction indication to be displayed on the display unit.

9. The control device according to claim 2, characterized in that: When the communication unit cannot acquire the display information and the map information based on the wireless communication, Even when the recognition unit acquires the displays of the plurality of traffic lights included in the image, the recognition unit does not integrate the displays of the plurality of traffic lights. The recognition unit determines, among the plurality of signal light displays recognized by the image processing of the image, a signal light display having a higher recognition accuracy of the signal light display; The lighting color determination unit determines the lighting color of the signal light based on the display of the signal light determined by the recognition unit, The traveling direction determination unit determines an indication of the traveling direction to be displayed on the display unit based on the display of the signal lamp determined by the recognition unit.

10. A mobile body comprising a control device for displaying, on a display unit, a display of a signal light included in an image captured by a capturing unit capturing the outside world, It is characterized in that The control device comprises: a recognition unit for recognizing the display of the signal light according to the image; a communication unit that receives display information of the signal light through wireless communication; and a control unit that determines the display of the signal light based on the display of the signal light recognized by the recognition unit and the display information received by the communication unit, and displays the determined display of the signal light on the display unit, The control unit comprises: a lighting color determination unit that compares the lighting color of the signal light identified by the identification unit with the lighting color of the signal light acquired from the display information based on the wireless communication, and determines the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison; as well as A traveling direction determination unit compares the traveling direction indication of the signal light identified by the identification unit with the traveling direction indication of the signal light obtained from the display information based on the wireless communication, and determines the traveling direction indication that is consistent with the comparison result as the traveling direction indication displayed on the display unit.

11. A control method, comprising: a control method for controlling a control device mounted on a moving body, wherein the control device displays on a display unit a display of a signal light included in an image captured by a capturing unit capturing an external environment of the moving body, It is characterized in that The control method has the following features: a recognition step, in which a recognition unit recognizes the display of the signal light based on the image; a communication step, in which a communication unit receives display information of the signal light through wireless communication; as well as a control step, in which a control unit determines the display of the signal light based on the display of the signal light recognized in the recognition step and the display information received in the communication step, and displays the determined display of the signal light on the display unit, In the control step, there are: a lighting color determination step of comparing the lighting color of the signal light identified in the identification step with the lighting color of the signal light acquired from the display information based on the wireless communication, and determining the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison; as well as A traveling direction determining step, in which the traveling direction indication of the signal light identified in the identifying step is compared with the traveling direction indication of the signal light obtained from the display information based on the wireless communication, and the traveling direction indication that is consistent with the comparison result is determined as the traveling direction indication displayed on the display unit.

12. A storage medium storing a program for causing a computer to execute each step of a control method of a control device, the control device being mounted on a mobile body and displaying on a display unit a display of a signal light included in an image captured by a capturing unit capturing the outside of the mobile body, It is characterized in that The control method has the following features: a recognition step, in which a recognition unit recognizes the display of the signal light based on the image; a communication step, in which a communication unit receives display information of the signal light through wireless communication; as well as a control step, in which a control unit determines the display of the signal light based on the display of the signal light recognized in the recognition step and the display information received in the communication step, and displays the determined display of the signal light on the display unit, In the control step, there are: a lighting color determination step of comparing the lighting color of the signal light identified in the identification step with the lighting color of the signal light acquired from the display information based on the wireless communication, and determining the lighting color of the signal light displayed on the display unit according to a predetermined priority order based on a result of the comparison; as well as A traveling direction determining step, in which the traveling direction indication of the signal light identified in the identifying step is compared with the traveling direction indication of the signal light obtained from the display information based on the wireless communication, and the traveling direction indication that is consistent with the comparison result is determined as the traveling direction indication displayed on the display unit.

Images