Traffic signal light recognition method and device, and intelligent driving equipment

Abstract

This application relates to a traffic light recognition method, apparatus, and intelligent driving device. The method includes: acquiring a first image captured by a first camera; recognizing a traffic light based on the first image to obtain a first recognition result; if the first recognition result indicates an abnormality in the recognition of the traffic light, acquiring a second image captured by a second camera, wherein the field of view (FOV) ranges of the first camera and the second camera do not overlap or the FOV ranges partially overlap; and recognizing the traffic light based on the second image. The technical solution provided by the embodiments of this application can effectively assist intelligent driving devices in observing traffic lights, improve the accuracy and speed of recognition under limited computing power, reduce the data processing burden, reduce energy consumption, and help the intelligent driving system operate efficiently, ensuring driving safety.

Description

Traffic signal lamp recognition method and device and intelligent driving equipment TECHNICAL FIELD

[0001] The present application relates to the field of intelligent driving, and more particularly, to a traffic signal lamp recognition method, device and intelligent driving equipment. BACKGROUND

[0002] Traffic signal lamp recognition is crucial to an intelligent driving system, which mainly collects and analyzes the state of a traffic signal lamp through a vehicle-mounted camera to guide the driving decision of an intelligent driving equipment. However, the recognition of the traffic signal lamp may be inaccurate, leading to the intelligent driving equipment taking wrong driving behaviors.

[0003] To solve this problem, it is urgent to find a method to assist the intelligent driving system in making correct recognition of the traffic signal lamp, so as to normally control the vehicle and ensure the safety of the vehicle.

[0004] SUMMARY

[0005] The present application provides a traffic signal lamp recognition method, device and intelligent driving equipment, which can effectively assist the intelligent driving equipment in observing the traffic signal lamp.

[0006] In a first aspect, a traffic signal lamp recognition method is provided, comprising: acquiring a first image collected by a first camera; recognizing a traffic signal lamp according to the first image to obtain a first recognition result; if the first recognition result indicates that the recognition of the traffic signal lamp is abnormal, acquiring a second image collected by a second camera, the field of view (FOV) range of the first camera and the second camera being non-overlapping or partially overlapping; and recognizing the traffic signal lamp according to the second image.

[0007] Based on the above technical solution, the situation that the intelligent driving system cannot observe the traffic signal lamp can be effectively avoided, and the accuracy and speed of recognizing the traffic signal lamp can be improved in the case of limited computing resources at the vehicle end, which helps to reduce the burden of data processing and vehicle energy consumption, and is also conducive to the more efficient operation of the intelligent driving system, providing protection for driving safety.

[0008] In some possible implementation manners, the horizontal FOV range of the first camera and the second camera is non-overlapping, or the horizontal FOV range of the first camera and the second camera is partially overlapping.

[0009] In some possible implementation manners, the vertical FOV range of the first camera and the second camera is non-overlapping, or the vertical FOV range of the first camera and the second camera is partially overlapping.

[0010] With reference to the first aspect, in some possible implementations of the first aspect, the method further includes: identifying the first traffic signal light according to the first image; and obtaining a second image captured by a second camera, including: obtaining the second image captured by the second camera when the first identification result indicates that the first traffic signal light switches from being within the FOV range of the first camera to being beyond the FOV range of the first camera; or obtaining the second image captured by the second camera when the first identification result indicates that the first traffic signal light is a malfunctioning light.

[0011] In some possible implementations, the first identification result can indicate that the first image includes a plurality of first traffic signal lights, and all the plurality of first traffic signal lights are traffic signal lights for indicating driving behaviors of the intelligent driving device at the first intersection.

[0012] In some possible implementations, when the vehicle is at the first driving position, the first traffic signal light is identified according to the first image, and when the vehicle continues to drive forward to a second driving position, the first traffic signal light switches from being within the FOV range of the first camera to being beyond the FOV range. At this time, the vehicle cannot identify the traffic signal light according to the first image.

[0013] In some possible implementations, the traffic signal light can be affected by extreme weather, accident damage, or disrepair, and can have circuit damage, aging, or other hardware problems. When the vehicle identifies the first traffic signal light according to the first image, the first identification result indicates that the first traffic signal light is a malfunctioning light, and at this time, the vehicle cannot make decisions on driving actions according to the malfunctioning light.

[0014] With reference to the first aspect, in some possible implementations of the first aspect, before obtaining the second image captured by the second camera, the method further includes: determining a position of the first traffic signal light according to the first image; and determining the second camera according to the position of the first traffic signal light.

[0015] Determining the second camera according to the position of the first traffic signal light includes: selecting the second camera from a plurality of cameras outside the cabin according to the position of the first traffic signal light.

[0016] Determining the second camera according to the position of the first traffic signal light includes: determining a position of the first traffic signal light relative to the intelligent driving device according to the first image, and determining the second camera to be turned on at a corresponding position.

[0017] In some possible implementations, determining the second camera includes: determining a plurality of second cameras.

[0018] Based on the above technical solution, when the first recognition result indicates that the recognition of the traffic light is abnormal, the approximate location of the traffic light can be estimated based on the first image, and the camera that is more likely to recognize the traffic light can be used first, which helps to improve the recognition efficiency of the traffic light.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, if the first recognition result indicates that the recognition of the traffic light is abnormal, acquiring the second image captured by the second camera includes: acquiring the second image captured by the second camera when about to pass through the first intersection and the first recognition result indicates that the first image does not include the traffic light.

[0020] In some possible implementations, when a vehicle is far from the first intersection, the temporary traffic light is too small to be recognized by the first image. When a vehicle is close to the first intersection, the temporary traffic light is outside the field of view (FOV) of the first camera, so the vehicle cannot recognize the temporary traffic light by the first image.

[0021] Based on the above technical solution, when the intelligent driving device is about to pass through an intersection and the first recognition result indicates that the first image does not include traffic lights, a second image captured by a second camera can be acquired. In this way, by acquiring images of the first intersection using cameras with different FOV ranges, the success rate of the intelligent driving device in acquiring traffic light indicators at the first intersection can be improved, thereby helping to enhance the user's driving experience and driving safety.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, when about to pass through the first intersection and the first recognition result indicates that the first image does not include traffic lights, acquiring the second image captured by the second camera includes: when about to pass through the first intersection, map information indicates that there are traffic lights at the first intersection, and the first recognition result indicates that the first image does not include traffic lights, acquiring the second image captured by the second camera.

[0023] Based on the above technical solution, combined with the real-time perception data and map information from the first camera, intelligent driving devices can more accurately understand road conditions, which is conducive to providing more comprehensive support for the decision to activate the second camera.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, before acquiring the second image captured by the second camera, the method further includes: when the first recognition result indicates that the first image does not include traffic lights, determining the second camera according to the driving rules of the local area, wherein the driving rules of the local area include: left-hand drive or right-hand drive.

[0025] Based on the above technical solutions, intelligent driving equipment can better adapt to traffic environments in different regions, improve the accuracy and efficiency of traffic light recognition, and thus provide effective protection for driving safety.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, the first camera is a front-viewing camera and the second camera is a side-viewing camera in the first direction.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, acquiring the second image captured by the second camera includes: acquiring the second image captured by the second camera and acquiring the third image captured by the third camera, wherein the third camera is a side-view camera in the second direction, and the first direction and the second direction are opposite directions.

[0028] In some possible implementations, the second camera is the left-side camera, and the third camera is the right-side camera.

[0029] Based on the above technical solution, by simultaneously employing two side-view cameras from different directions for traffic light recognition, more comprehensive monitoring of the surrounding environment can be provided in a short time, reducing blind spots and improving the recognition rate of traffic lights, thereby enhancing the reliability of the intelligent driving system. On the one hand, it can effectively reduce the risk of intelligent driving equipment failing to recognize traffic lights; on the other hand, since the two side-view cameras from different directions are activated according to the scenario requirements, it can also effectively save the computing resources and power consumption of the intelligent driving equipment.

[0030] In conjunction with the first aspect, in some implementations of the first aspect, before acquiring the second image captured by the second camera, the method further includes: acquiring a third image captured by a third camera, wherein the third camera is a side-view camera in a second direction, and the first direction and the second direction are opposite directions; wherein acquiring the second image captured by the second camera includes: acquiring the second image captured by the second camera when the third image does not include traffic lights.

[0031] Based on the above technical solutions, enabling the third and second cameras as needed helps to efficiently utilize vehicle-side resources and reduce the pressure on the intelligent driving system in data processing.

[0032] In conjunction with the first aspect, in some implementations of the first aspect, before acquiring the second image captured by the second camera, the method further includes: after the first recognition result indicates that the recognition of the traffic light has been abnormal for a first duration, acquiring the second image captured by the second camera, wherein the first duration can be determined based on at least one of the following: vehicle speed, weather conditions, road conditions, and vehicle braking performance.

[0033] Based on the above technical solution, when an anomaly occurs in the first recognition result, the second camera is not immediately activated. Instead, a certain amount of time is reserved to review the first recognition result. This allows for continuous monitoring of the traffic light status and effectively avoids resource waste caused by activating the second camera due to recognition errors.

[0034] In conjunction with the first aspect, in some implementations of the first aspect, after recognizing traffic lights based on the second image, the method further includes: controlling the intelligent driving device based on the second recognition result of the second image.

[0035] Secondly, embodiments of this application provide a traffic light recognition device, which includes: an acquisition unit for acquiring a first image captured by a first camera; a recognition unit for recognizing a traffic light based on the first image to obtain a first recognition result; the acquisition unit is further configured to acquire a second image captured by a second camera when the first recognition result indicates that an abnormal situation has occurred in the recognition of the traffic light, wherein the field of view (FOV) ranges of the first camera and the second camera do not overlap or the FOV ranges of the first camera and the second camera partially overlap; the recognition unit is further configured to recognize the traffic light based on the second image.

[0036] In conjunction with the second aspect, in some implementations of the second aspect, the identification unit is further configured to: identify the first traffic light based on the first image; the acquisition unit is specifically configured to: acquire the second image captured by the second camera when the first identification result indicates that the first traffic light has switched from being within the FOV range of the first camera to being outside the FOV range of the first camera; or, acquire the second image captured by the second camera when the first identification result indicates that the first traffic light is a fault light.

[0037] In conjunction with the second aspect, in some implementations of the second aspect, the device further includes a determining unit, which is configured to: determine the position of a first traffic light based on the first image; the determining unit is also configured to: determine a second camera based on the position of the first traffic light.

[0038] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is specifically used to: acquire the second image captured by the second camera when the first recognition result indicates that the first image does not include traffic lights when the first intersection is about to be passed.

[0039] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is specifically used to: when approaching the first intersection, if map information indicates that there is a traffic light at the first intersection, and the first recognition result indicates that the first image does not include the traffic light, acquire the second image captured by the second camera.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the device further includes a determining unit, which is configured to: if the first recognition result indicates that the first image does not include traffic lights, determine the second camera according to the driving rules of the area, wherein the driving rules of the area include: left-hand drive or right-hand drive.

[0041] In conjunction with the second aspect, in some implementations of the second aspect, the first camera is a front-viewing camera and the second camera is a side-viewing camera in the first direction.

[0042] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is specifically used to: acquire a second image captured by a second camera and acquire a third image captured by a third camera, wherein the third camera is a side-view camera in the second direction, and the first direction and the second direction are opposite directions.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is further configured to: acquire a third image acquired by a third camera before the acquisition unit acquires the second image acquired by the second camera, wherein the third camera is a side-view camera in the second direction, and the first direction and the second direction are opposite directions; wherein, the acquisition unit is specifically configured to: acquire the second image acquired by the second camera when the third image does not include traffic lights.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is further configured to: acquire a second image captured by a second camera after the first recognition result indicates that the recognition of the traffic signal light has been abnormal for a first duration, wherein the first duration can be determined based on at least one of the following: vehicle speed, weather conditions, road conditions, and vehicle braking performance.

[0045] In conjunction with the second aspect, in some implementations of the second aspect, the device further includes a control unit, which is configured to: control the intelligent driving device based on a second recognition result of the second image after the recognition unit recognizes the traffic light based on the second image.

[0046] Thirdly, embodiments of this application provide a traffic light recognition device, which includes: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, so that the device performs any of the possible methods described in the first aspect above.

[0047] Fourthly, embodiments of this application provide an intelligent driving device including any of the traffic light recognition devices described in the second or third aspect above.

[0048] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the intelligent driving device is a vehicle.

[0049] Fifthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to implement any of the possible methods described in the first aspect above.

[0050] In a sixth aspect, embodiments of this application provide a computer program product comprising computer program code that, when executed on a computer, enables the computer to implement any of the possible methods described in the first aspect above.

[0051] In a seventh aspect, embodiments of this application provide a chip that includes circuitry for performing any of the possible methods described in the first aspect above. Attached Figure Description

[0052] Figure 1 is a functional block diagram of the vehicle provided in an embodiment of this application.

[0053] Figure 2 is a schematic block diagram of the intelligent driving system provided in an embodiment of this application.

[0054] Figure 3 is a diagram of an intelligent driving system architecture provided in an embodiment of this application.

[0055] Figure 4 is a schematic diagram of a traffic light outside the FOV range of a forward-looking camera provided in an embodiment of this application.

[0056] Figure 5 is a schematic diagram of a traffic light outside the FOV range of a forward-looking camera provided in an embodiment of this application.

[0057] Figure 6 is a schematic diagram of a traffic light recognition method 600 provided in an embodiment of this application.

[0058] Figure 7 is a schematic diagram of a vehicle driving process provided in an embodiment of this application.

[0059] Figure 8 is a comparative schematic diagram of a normal traffic signal light and a faulty traffic signal light provided in an embodiment of this application.

[0060] Figure 9 is a schematic diagram of a temporary traffic light setup provided in an embodiment of this application.

[0061] Figure 10 is a schematic diagram of traffic light recognition using cameras on both the left and right sides, provided in an embodiment of this application.

[0062] Figure 11 is a schematic diagram of a temporary traffic light setup provided in an embodiment of this application.

[0063] Figure 12 is a scenario diagram of right-hand drive driving provided in an embodiment of this application.

[0064] Figure 13 is a scenario diagram of right-hand drive driving provided in an embodiment of this application.

[0065] Figure 14 illustrates a traffic light recognition method 1400 provided in an embodiment of this application.

[0066] Figure 15 is a schematic block diagram of a traffic signal light recognition device 1500 provided in an embodiment of this application. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. "At least one" refers to one or more. For example, "at least one of A and B," similar to "A and / or B," describes the association relationship between related objects, indicating that three relationships can exist. For example, at least one of A and B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0068] The prefixes such as "first" and "second" used in this application embodiment are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this application embodiment does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not constitute unnecessary restrictions due to the use of such prefixes. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0069] Figure 1 is a functional block diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 may include a sensing system 110, a computing platform 120, and a display device 130. The sensing system 110 may include one or more sensors for sensing information about the environment surrounding the vehicle 100. For example, the sensing system 110 may include a positioning system, which may be a Global Positioning System (GPS), a BeiDou Navigation Satellite System, or another positioning system. As another example, the sensing system 110 may include one or more of the following: an inertial measurement unit (IMU), an accelerometer, a lidar, a millimeter-wave radar, an ultrasonic radar, and a camera device.

[0070] Some or all of the functions of vehicle 100 can be controlled by computing platform 120. Computing platform 120 may include one or more processors, such as processors 121 to 12n (n being a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement some or all of the functions of the aforementioned units. Furthermore, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. In addition, the computing platform 120 may also include a memory for storing instructions. Some or all of the processors 121 to 12n can call the instructions in the memory to implement the corresponding functions.

[0071] The in-cabin display devices 130 are mainly divided into two categories: the first is the in-vehicle display screen; the second is the projection display screen, such as the head-up display (HUD). An in-vehicle display screen is a physical display screen and an important component of the in-vehicle infotainment system. Multiple displays can be installed in the cabin, such as the digital instrument cluster display, the central control screen, the display screen in front of the front passenger (also known as the front-seat passenger), the display screen in front of the left rear passenger, the display screen in front of the right rear passenger, and even the car window can be used as a display screen. A head-up display, also known as a head-up display system, is mainly used to display driving information such as speed and navigation on a display device in front of the driver (such as the windshield). This reduces the driver's eye-shift time, avoids pupil changes caused by eye-shifting, and improves driving safety and comfort. Examples of HUDs include combiner-HUD (C-HUD) systems, windshield-HUD (W-HUD) systems, and augmented reality HUD (AR-HUD) systems. It should be understood that HUDs can also evolve into other types of systems as technology progresses, and this application does not limit them.

[0072] The above description of the display device 130 uses an in-vehicle display screen and a projection display screen as examples, but the embodiments of this application are not limited thereto. For example, the display device 130 can also be a light display screen or a projection screen.

[0073] Optionally, the structure of the vehicle 100 described above is merely illustrative. In actual applications, various components of the vehicle 100 may be added or removed as needed.

[0074] Vehicle 100 may include an intelligent driving system, which may include an advanced driving assistant system (ADAS) and an autonomous driving system (ADS). The intelligent driving system uses various sensors on the vehicle (including but not limited to: lidar, millimeter-wave radar, camera devices, ultrasonic sensors, global positioning system, inertial measurement unit) to acquire information from the vehicle's surroundings, and analyzes and processes the acquired information to achieve functions such as obstacle perception, target recognition, vehicle positioning, path planning, and driver monitoring / alerts, thereby improving the safety, automation, and comfort of driving the vehicle.

[0075] For example, Figure 2 shows a schematic block diagram of an intelligent driving system 200 provided in an embodiment of this application. The intelligent driving system 200 may include three functional modules: a perception module 210, a planning module 220, and a control module 230. The perception module 210 perceives the environment surrounding the vehicle through sensors and outputs corresponding perception data to the planning module 220. The planning module 220 obtains information such as road topology and target objects based on the information acquired by the perception module 210. The planning module 220 can determine a planned trajectory over a period of time based on the road topology and target object information. The planning module 220 can send this planned trajectory to the control module 230. After receiving the planned trajectory from the planning module 220, the control module 230 can output control signals to control the actuators to take corresponding actions, such as steering, acceleration, and deceleration.

[0076] The above-mentioned sensing module 210 can be the above-mentioned sensing system 110, and the planning module 220 and the control module 230 can be located in the above-mentioned computing platform 120.

[0077] Vehicle-based driving automation systems are classified into five levels (or L0-L5) based on the degree to which they can perform dynamic driving tasks, according to the role allocation in performing these tasks and the presence or absence of an operational design domain (ODD), such as the external conditions (road, traffic, weather, lighting, etc.) defined during the system's design. Levels 0-2 represent driver assistance, where the system assists humans in performing dynamic driving tasks, but the driver remains the primary driver. Levels 3-5 represent autonomous driving, where the system performs dynamic driving tasks in place of the human under the designed operating conditions; when activated, the system becomes the primary driver. The names and definitions of each level are as follows:

[0078] Level 0 Driving Automation (also known as Emergency Assistance): The system cannot continuously perform lateral or longitudinal motion control of the vehicle in dynamic driving tasks, but it has the ability to continuously perform partial target and event detection and response in dynamic driving tasks. Level 1 Driving Automation (also known as Partial Driver Assistance): The system continuously performs lateral or longitudinal motion control of the vehicle in dynamic driving tasks under its design operating conditions, and has the ability to perform partial target and event detection and response adapted to the performed lateral or longitudinal motion control. Level 2 Driving Automation (also known as Combined Driver Assistance): The system continuously performs lateral and longitudinal motion control of the vehicle in dynamic driving tasks under its design operating conditions, and has the ability to perform partial target and event detection and response adapted to the performed lateral and longitudinal motion control. Level 3 Driving Automation (also known as Conditionally Automated Driving): The system continuously performs all dynamic driving tasks under its design operating conditions. Level 4 Driving Automation (also known as Highly Automated Driving): The system continuously performs all dynamic driving tasks under its design operating conditions and automatically executes the minimum risk strategy. Level 5 driving automation (also known as fully automated driving): The system continuously performs all dynamic driving tasks under any drivable conditions and automatically executes the least risk strategy.

[0079] Traffic light recognition is a crucial function in the field of intelligent driving equipment, which uses onboard sensing devices to observe the status of traffic lights.

[0080] For example, traffic light recognition may include the following tasks: traffic light detection, traffic light classification, and traffic light tracking.

[0081] Traffic light detection can be performed based on single-frame image data in a video stream. The goal is to locate one or more traffic lights in the video stream and determine their positions within the video stream.

[0082] Traffic light classification can be based on the results of traffic light detection, with the aim of determining the category and color of the detected traffic lights. The categories of traffic lights include, but are not limited to: motor vehicle traffic lights, non-motor vehicle traffic lights, pedestrian crossing lights, turn signals, and hazard warning lights.

[0083] Traffic light tracking can be performed based on multiple consecutive frames in a video stream. The goal is to continuously monitor changes in the state of detected traffic lights as the vehicle moves. These changes include, but are not limited to, changes in the position and color of the traffic lights within the video stream. For example, traffic lights detected by a forward-looking camera are tracked temporally based on the intersection-over-union (IoU) ratio, and the same traffic light is assigned the same ID, which can be used for tracking in subsequent frames.

[0084] Figure 3 illustrates an architecture diagram of an intelligent driving system provided in an embodiment of this application. As shown in Figure 3, the process of traffic light recognition by the intelligent driving device is as follows: the camera collects data, and the collected data is converted into a video stream by an image signal processor (ISP). The traffic light recognition module receives the video stream provided by the ISP and, in conjunction with navigation map data and vehicle positioning information, outputs the color, type, and location of the detected traffic light. The decision module determines the traffic light corresponding to the lane where the vehicle is currently located based on the color, type, and location information of the traffic light and decides whether the vehicle needs to proceed, prepare, or stop. Simultaneously, the decision module transmits the selected traffic light to the human-machine interface (HMI) on the cockpit / instrument panel for visual display, indicating the status of the traffic lights at the intersection ahead to the driver. The planning and control module plans and controls the vehicle's behavior based on the status of the traffic lights and surrounding environmental information, and finally transmits the control commands to the chassis for execution.

[0085] The traffic signal recognition module can be located in the perception module 210. Some functions of the decision-making module and planning control can be implemented by the planning module 220, and other functions can be implemented by the control module 230.

[0086] For example, the camera may be part of the perception system 110 of the vehicle 100; the ISP may be one of the processors in the computing platform 120; the traffic light recognition module may be part of the perception module 210; the decision module may be part of the planning module 220; some functions of the planning and control module may be implemented by the planning module 220, and other functions may be implemented by the control module 230; the human-machine interface on the cockpit / instrument may be a display device 130, which can display the results of traffic light recognition during the process. For example, the display device may show that the current intersection requires braking, or it may show the remaining time of the red light at the current intersection. It should be understood that the intelligent driving system architecture diagram shown in Figure 3 is only an example. In actual applications, the modules in the above intelligent driving system may be added or deleted according to actual needs.

[0087] In intelligent driving systems, cameras serve as the "eyes" of the intelligent driving device. They can collect video streams to help the intelligent driving device observe the surrounding environment and provide driving assistance to the driver. They are important sensors in intelligent driving devices.

[0088] In the field of traffic light recognition, traffic lights at intersections ahead can be used to indicate the current movement of vehicles. Therefore, in intelligent driving systems, the perception and detection of traffic lights are mainly performed using forward-facing cameras. Forward-facing cameras can monitor the road conditions ahead of the vehicle through their forward field of view, meeting the basic requirements for safe driving. However, the installation location and physical characteristics of forward-facing cameras may limit their field of view. If only forward-facing cameras are used for traffic light recognition, there may be situations where the position of the traffic lights exceeds the field of view (FOV) of the forward-facing camera.

[0089] For example, Figure 4 illustrates a schematic diagram of a traffic light outside the FOV range of a forward-looking camera according to an embodiment of this application. As shown in Figure 4(A), when the intelligent driving device approaches the intersection from a distance, the forward-looking camera can identify the traffic light at the intersection. However, as the intelligent driving device continues to approach the intersection, as shown in Figure 4(B), the position of the traffic light exceeds the horizontal field of view (HFOV) range of the forward-looking camera, that is, the traffic light is located in the blind spot to the side and front of the intelligent driving device.

[0090] For example, Figure 5 illustrates a schematic diagram of a traffic light outside the field of view (FOV) of a forward-looking camera according to an embodiment of this application. As shown in Figure 5(A), when the intelligent driving device approaches the intersection from a distance, the forward-looking camera can identify the traffic light at the intersection. However, as the intelligent driving device continues to approach the intersection, as shown in Figure 5(B), the position of the traffic light exceeds the vertical field of view (VFOV) of the forward-looking camera, meaning the traffic light is located in the blind spot above the intelligent driving device.

[0091] When the location of a traffic light is outside the field of view (FOV) of the forward-facing camera, the camera will be unable to observe the status of the traffic light, causing the intelligent driving device to make an incorrect decision.

[0092] For example, if a traffic light is located on the side of an intersection, and the autonomous driving device approaches the intersection, but the traffic light is outside the HFOV (Head-of-Voice) range of the forward-facing camera, the camera cannot observe the traffic light's status. Simultaneously, the navigation map does not mark the traffic light at this intersection. Therefore, the autonomous driving device will treat this intersection as one without traffic lights and proceed directly through it. However, if the actual traffic light at this intersection is red, the autonomous driving device may be running a red light, or it may collide with other objects.

[0093] For example, if a traffic light is too close to the stop line at an intersection, when the autonomous driving device approaches the intersection, the traffic light's position is outside the HFOV (Head-of-Voice) range of the forward-facing camera, making it impossible for the camera to observe the traffic light's status. However, the map marks this intersection as having a traffic light. The autonomous driving device will then treat this intersection as one with a traffic light. For safety reasons, the device will take a conservative approach, slowing down and stopping before the stop line to obey traffic rules and avoid running a red light. However, if the traffic light at this intersection is actually green, this results in unnecessary stopping and could even lead to a rear-end collision. Furthermore, because the forward-facing camera cannot observe the traffic light's status, the autonomous driving device cannot automatically start based on the traffic light's indication. Therefore, the autonomous driving device will be temporarily unable to drive autonomously, requiring the driver to take over the vehicle.

[0094] For example, if a traffic light is located above the stop line at an intersection, and the autonomous driving device approaches the intersection, but the traffic light is outside the VFOV (Vehicle-of-Flight) range of the forward-facing camera, the camera cannot observe the traffic light's status. Furthermore, the map does not mark the traffic light at this intersection. Therefore, the intersection would be considered an intersection without traffic lights, and the autonomous driving device would proceed directly through it. However, if the actual traffic light at this intersection is red, the autonomous driving device might run a red light, or it might collide with other objects.

[0095] As can be seen from the above, in order to address the problem that intelligent driving devices cannot observe traffic lights in the blind spot of the forward-facing camera, solutions need to be sought to reduce the number of incorrect decisions made by intelligent driving devices and violations of traffic rules.

[0096] Currently, it's common practice for vehicle manufacturers to equip vehicles with multiple cameras for reversing assistance or driving recording. These multiple cameras can include, but are not limited to, front-view cameras, side-view cameras, surround-view cameras, rear-view cameras, and interior cameras. The placement of these cameras can be determined based on the specific design of the vehicle and its perception requirements. Typically, the primary purpose of cameras used for reversing assistance or driving recording is to provide the recorded video for human viewing; therefore, a resolution of only a few hundred thousand pixels is generally sufficient. However, with further technological advancements and the continuous expansion of intelligent driving scenarios, the focus of in-vehicle camera applications has shifted to how to automate environmental perception during driving. Intelligent driving devices demand increasingly higher resolution from in-vehicle cameras; for example, some vehicle manufacturers are already using 8-megapixel cameras. Such high-resolution cameras generate far more data than low-resolution cameras. Therefore, if an intelligent driving system uses multiple in-vehicle cameras for traffic light recognition, it will generate a large amount of video streams. Given the limited computing resources and double data rate (DDR) bandwidth of intelligent driving devices, this increases the difficulty of data processing and the computational burden on the intelligent driving system.

[0097] As can be seen from the above, the problem of traffic lights not being able to be observed in the blind spot of the forward-facing camera of intelligent driving equipment cannot be solved simply by connecting cameras from more angles.

[0098] This application provides a traffic light recognition method that can save resources of intelligent driving systems. By using different cameras that start and stop at different times to recognize traffic lights, it solves the problem of traffic lights exceeding the field of view (FOV) of the intelligent driving device's cameras, thus helping the intelligent driving device make correct decisions.

[0099] Figure 6 is a schematic diagram of a traffic light recognition method 600 provided in an embodiment of this application. The method 600 can be executed by a vehicle 100; or, the method 600 can be executed by a processor in a computing platform 120; or, the method 600 can be executed by an intelligent driving system 200; or, the method 600 can be executed by a perception module 210 in the intelligent driving system 200.

[0100] The method 600 may include the following steps:

[0101] S601: Acquire the first image captured by the first camera.

[0102] For example, the first camera can be a forward-facing camera. As one of the core sensors in intelligent driving devices, the forward-facing camera has multiple functions, meeting the basic needs of intelligent driving devices in most driving scenarios, including: forward collision warning (FCW), pedestrian collision warning (PCW), traffic sign recognition (TSR), lane departure warning (LDW), and lane keeping assist (LKA). Therefore, the forward-facing camera can remain constantly on to ensure safe driving.

[0103] The embodiments of this application do not limit the installation position of the first camera. For example, the first camera can be a front-view camera, which can be installed in at least one of the following positions: above the front grille, above the windshield, above the bumper, above the hood, etc.

[0104] Due to differences in vehicle configuration requirements, traffic light size, and other factors, the recognition distance of the first camera varies. This application embodiment does not limit the recognition distance of the first camera.

[0105] S602: Identify the traffic light based on the first image and obtain the first identification result.

[0106] Optionally, the method further includes: identifying a first traffic light based on the first image.

[0107] S603: If the first recognition result indicates that the recognition of the traffic light is abnormal, acquire the second image captured by the second camera, wherein the field of view (FOV) ranges of the first camera and the second camera do not overlap or the FOV ranges of the first camera and the second camera partially overlap.

[0108] In one embodiment of step S603, if the first recognition result indicates that the recognition of the traffic light is abnormal, the second image captured by the second camera is acquired, including: when the first recognition result indicates that the first traffic light switches from being within the FOV range of the first camera to being outside the FOV range of the first camera, the second image captured by the second camera is acquired; or, when the first recognition result indicates that the first traffic light is a faulty light, the second image captured by the second camera is acquired.

[0109] Figure 7 illustrates a vehicle driving process according to an embodiment of this application. As shown in Figure 7(A), the vehicle travels from a first driving position to a second driving position. For example, the distance between the first driving position and the stop line at the first intersection can be 150 meters. The second driving position can be a position where the intelligent driving device continues to travel from the first driving position, closer to the first intersection, and the distance between the second driving position and the stop line at the first intersection can be 20 meters. As shown in Figure 7(B), when the vehicle is at the first driving position, it identifies the first traffic light based on the first image, meaning the first traffic light at the first intersection is within the HFOV range of the first camera. As shown in Figure 7(C), when the vehicle continues to travel forward to the second driving position, the first traffic light switches from being within the HFOV range of the first camera to being outside the HFOV range of the first camera. At this time, the vehicle cannot identify the traffic light based on the first image, therefore, it is necessary to activate the second camera to capture a second image.

[0110] In real-world road environments, traffic lights may malfunction due to extreme weather, accidents, or years of neglect, resulting in circuit damage, aging, or other hardware issues. For example, as the intelligent driving device approaches the first intersection, it identifies the first traffic light based on a first image and obtains a first identification result. This result indicates that the first traffic light at the first intersection is faulty. Unable to make decisions about the vehicle's driving actions based on this faulty light, the intelligent driving device activates a second camera to capture a second image.

[0111] For example, a traffic light malfunction may manifest as one or more traffic lights not lighting up. For instance, if the first image identifies that all the traffic lights at the first intersection are not working, it could be due to a power outage, a short circuit causing the main fuse to blow, or a lightning strike causing the mainboard to burn out, among other reasons.

[0112] For example, a traffic light malfunction may manifest as follows: the type of traffic light displayed is unknown, i.e., the traffic light is not a standard arrow-type traffic light or a full-screen traffic light. Figure 8 shows a comparative schematic diagram of a normal traffic light and a malfunctioning traffic light provided in an embodiment of this application. Figure 8(A) shows a normally functioning arrow-type traffic light, which clearly indicates the direction of travel, including: left turn, straight ahead, and right turn. Figure 8(B) shows a malfunctioning traffic light, in which the intelligent driving device cannot make a correct driving decision based on the traffic light's indication.

[0113] For example, a malfunctioning traffic light may manifest as the traffic light displaying an unknown color, meaning the color displayed is not red, yellow, or green.

[0114] The lack of traffic lights can cause traffic jams or accidents, and repairing traffic lights usually takes time. Therefore, traffic management departments sometimes set up temporary traffic lights at intersections or implement manual traffic control to ensure that traffic order is maintained during the repair process.

[0115] However, temporary traffic lights are designed to be relatively small for rapid deployment and flexible movement. When the intelligent driving device is far from the first intersection, it cannot recognize the temporary traffic lights based on the first image. Conversely, when the intelligent driving device is close to the first intersection, the position of the temporary traffic lights may be limited by site conditions and exceed the FOV (field of view) of the first camera. For example, Figure 9 shows a schematic diagram of setting up temporary traffic lights according to an embodiment of this application. As shown in Figure 9, the original traffic lights on the first road are malfunctioning and unusable, and temporary traffic lights are set up on the side of the first intersection. When a vehicle is about to pass through the first intersection, it recognizes the first traffic light based on the first image and obtains a first recognition result. The first recognition result indicates that the first traffic light at the first intersection is faulty, but the temporary traffic light, because it is set on the side of the first intersection, is outside the FOV of the first camera. This causes the intelligent driving device to be unable to correctly plan and control its behavior based on the first image, thus requiring the activation of a second camera to capture a second image.

[0116] Optionally, before acquiring the second image captured by the second camera, the method further includes: determining the position of the first traffic light based on the first image; and determining the second camera based on the position of the first traffic light.

[0117] For example, the position of the first traffic light relative to the intelligent driving device can be determined based on the first image, and the second camera at the corresponding position can be activated to acquire the second image captured by the second camera. For instance, the intelligent driving device activates the first camera at the first driving position to recognize the traffic light at the first intersection, acquires the first image captured by the first camera, and identifies the first traffic light based on the first image, which is located on the left side of the first image. When the intelligent driving device continues to drive forward to the second driving position, the first traffic light changes from being within the FOV (Field of View) of the first camera to being outside the FOV of the first camera. At this time, the intelligent driving device cannot recognize the traffic light based on the first image, but based on the first image acquired by the intelligent device at the first driving position, it can determine that the first traffic light is located to the left front of the intelligent driving device, thus determining that the second camera that can be activated is the left-side camera.

[0118] For example, the first recognition result may indicate that the first image includes multiple first traffic lights, and these multiple first traffic lights are all traffic lights used to instruct the intelligent driving device on its driving behavior at the first intersection. Therefore, the number of second cameras determined above can be multiple. For example, the intelligent driving device activates the first camera at the first driving position to recognize the traffic lights at the first intersection, acquires the first image captured by the first camera, and identifies two first traffic lights based on the first image, one of which is located on the left side of the first image and the other on the right side. When the intelligent driving device continues to drive forward to the second driving position, the two first traffic lights switch from being within the FOV range of the first camera to being outside the FOV range of the first camera. At this time, the intelligent driving device cannot recognize the traffic lights based on the first image, but based on the first image acquired by the intelligent device at the first driving position, it can determine that the two first traffic lights are located to the left and right front of the intelligent driving device, respectively. Therefore, it is determined that the second cameras that can be activated are the left camera and the right camera, that is, traffic light recognition is performed simultaneously through the left camera and the right camera.

[0119] In one implementation, as the intelligent driving device approaches a first intersection, it activates a first camera to identify the traffic lights at the intersection, acquiring a first image captured by the camera. Based on the first image, it identifies the first traffic light and obtains a first identification result. The first identification result indicates that the first traffic light at the first intersection is a fault light, located on the right side of the first image. The intelligent driving device cannot make decisions about the vehicle's driving actions based on the fault light. Since the first image indicates that the fault light is located to the right front of the intelligent driving device, it is estimated that a temporary traffic light may also be located to the right front of the intelligent driving device. Therefore, the right-side camera can be activated to identify the traffic light, with the aim of locating the temporary traffic light.

[0120] In another implementation, two first traffic lights are identified based on the first image. The first identification result indicates that both first traffic lights at the first intersection are faulty, with one first traffic light located on the left side of the first image and the other on the right side. Since the relative positions of the temporary traffic lights and vehicles are uncertain, traffic lights can be identified simultaneously using both the left and right cameras.

[0121] As can be seen, based on the method provided in some embodiments of this application, when the first recognition result indicates that the recognition of the traffic light is abnormal, the approximate location of the traffic light can be estimated based on the first image, and the camera that is more likely to recognize the traffic light can be used preferentially, which helps to improve the recognition efficiency of the traffic light.

[0122] Optionally, the first camera is a front-view camera, and the second camera is a side-view camera in the first direction.

[0123] As mentioned above, the field of view (FOV) ranges of the first camera and the second camera do not overlap or the FOV ranges partially overlap. When the first camera is a front-view camera, the second camera can be a side-view camera. For example, the second camera can be a side-view camera in the first direction.

[0124] The FOV (Field of View) of a side-view camera primarily covers the area on the side of the vehicle and can be used for blind spot detection (BSD), lane change warning (LCW), and door open warning (DOW). This application does not limit the installation location of the side-view camera. For example, the side-view camera can be installed below the side mirrors on both sides of the intelligent driving device, and can be a left-side camera and a right-side camera. The side-view camera in the first direction can be either a left-side camera or a right-side camera.

[0125] Optionally, acquiring the second image captured by the second camera includes: acquiring the second image captured by the second camera and acquiring the third image captured by the third camera, wherein the third camera is a side-view camera in the second direction, and the first direction and the second direction are opposite directions.

[0126] In some embodiments provided in this application, the first camera may be a front-view camera, and the second and third cameras may be left-side cameras or right-side cameras, respectively.

[0127] Figure 10 illustrates a schematic diagram of traffic light recognition using both left and right cameras according to an embodiment of this application. As shown in Figure 10, the light-colored fan-shaped area 1 represents the FOV range of the front-view camera, and the dark-colored fan-shaped areas 2 and 3 represent the FOV ranges of the left and right cameras, respectively. The dark and light-colored fan-shaped areas overlap, indicating an overlap between the FOV ranges of the front-view and side-view cameras. For example, if the first recognition result indicates an abnormality in traffic light recognition, and if the first camera is a front-view camera, both the left and right cameras can be activated simultaneously for traffic light recognition.

[0128] As can be seen, based on the methods provided in some embodiments of this application, by simultaneously employing two side-view cameras in different directions for traffic light recognition, more comprehensive monitoring of the surrounding environment can be provided in a short time, reducing blind spots and improving the recognition rate of traffic lights, thereby enhancing the reliability of the intelligent driving system. On the one hand, it can effectively reduce the risk that the intelligent driving device cannot recognize traffic lights; on the other hand, the two side-view cameras in different directions are activated according to the scenario requirements, thus effectively saving vehicle-side computing resources.

[0129] Optionally, before acquiring the second image captured by the second camera, the method further includes: acquiring a third image captured by a third camera, wherein the third camera is a side-view camera in a second direction, and the first direction and the second direction are opposite directions; wherein acquiring the second image captured by the second camera includes: acquiring the second image captured by the second camera when the third image does not include traffic lights.

[0130] In some embodiments, if the first recognition result indicates an abnormality in traffic light recognition, the third camera can be activated to capture a third image, and a third recognition result can be obtained based on the third image. If the third recognition result indicates that the third image does not include the traffic light, the second camera is activated to capture a second image. For example, the third camera can be kept on while the second camera is activated, meaning that traffic light recognition can be performed simultaneously using both the second and third cameras.

[0131] For example, the first camera can be a front-view camera, the second camera can be a right-side camera, and the third camera can be a left-side camera. In this example, the intelligent driving device is about to pass through the first intersection. It activates the first camera to recognize the traffic lights at the first intersection, acquires a first image captured by the first camera, identifies the first traffic light based on the first image, and obtains a first recognition result. The first recognition result indicates that the first traffic light at the first intersection is a fault light. The intelligent driving device cannot make decisions about the vehicle's driving actions based on the fault light and needs to activate other cameras to recognize the traffic lights. The intelligent driving device first activates the left-side camera to acquire a third image and obtains a third recognition result based on the third image. The third recognition result indicates that the third image does not include the traffic light. Therefore, the intelligent driving device activates the right-side camera to acquire a second image. For example, when the right-side camera is activated, the left-side camera can remain on, that is, traffic light recognition is performed simultaneously using both the left and right cameras.

[0132] As can be seen from the above, enabling the third and second cameras as needed helps to efficiently utilize vehicle resources and reduce the pressure on the intelligent driving system in data processing.

[0133] In one embodiment of step S603, if the first recognition result indicates that the recognition of the traffic light is abnormal, the second image captured by the second camera is acquired, including: when the first intersection is about to be passed and the first recognition result indicates that the first image does not include the traffic light, the second image captured by the second camera is acquired.

[0134] For example, as the intelligent driving device approaches the first intersection, it identifies the traffic lights based on the first image and obtains a first identification result. The first identification result indicates that the first image does not include the traffic lights. In other words, the intelligent driving device acquires a second image captured by a second camera because the first image does not identify the traffic lights at the first intersection.

[0135] In real-world road environments, intersections that originally had traffic lights may need to have those lights temporarily removed. For example, road construction or urban development might be underway near the first intersection, requiring the temporary removal of the existing traffic lights. Traffic management departments typically use temporary traffic lights to maintain traffic order and ensure road safety at intersections.

[0136] However, temporary traffic lights are designed to be relatively small for rapid deployment and flexible movement. When the autonomous driving device is far from the first intersection, although the temporary traffic lights are within the field of view (FOV) of the first camera, they are not visible due to their inconspicuousness. Furthermore, because construction sites and fences occupy part of the road surface, temporary traffic lights often have to be placed close to the edge of the construction area. This means that when the autonomous driving device is close to the first intersection, the temporary traffic lights may be located to the side of the device, meaning their position may be limited by site conditions and exceed the FOV of the first camera.

[0137] For example, Figure 11 shows a schematic diagram of setting up temporary traffic lights according to an embodiment of this application. As shown in Figure 11, the traffic lights at the first intersection are temporarily removed due to construction, and temporary traffic lights are set up on the side of the first intersection. As shown in Figure 11(A), when the intelligent driving device is far from the first intersection, although the temporary traffic lights are within the FOV of the first camera, they cannot be identified by the first image because they are too small. As shown in Figure 11(B), when the intelligent driving device is close to the first intersection, the temporary traffic lights are outside the FOV of the first camera. Therefore, in both the scenarios of Figure 11(A) and Figure 11(B), the traffic lights are identified based on the first image, and a first identification result is obtained. The first identification result indicates that the first image never includes the traffic lights. This results in the intelligent driving device being unable to obtain any traffic light indication information based on the first image, thus making it impossible to correctly plan and control the behavior of the intelligent driving device. Therefore, the second camera is activated to collect a second image.

[0138] Based on the traffic light recognition method provided in some embodiments of this application, when the intelligent driving device is about to pass through an intersection and the first recognition result indicates that the first image does not include traffic lights, a second image captured by a second camera can be acquired. In this way, by acquiring images of the first intersection using cameras with different FOV ranges, the success rate of the intelligent driving device in acquiring traffic light indicators at the first intersection can be improved, thereby helping to enhance the user's driving experience and driving safety.

[0139] Optionally, when about to pass the first intersection and the first recognition result indicates that the first image does not include traffic lights, the second image captured by the second camera is acquired, including: when about to pass the first intersection, map information indicates that there are traffic lights at the first intersection, and the first recognition result indicates that the first image does not include traffic lights, the second image captured by the second camera is acquired.

[0140] Currently, in-vehicle maps can integrate cutting-edge technologies such as satellite remote sensing, real-time sensing, big data analysis, and artificial intelligence to provide users with accurate and real-time updated map information. In some embodiments, traffic light recognition can be assisted based on map information. For example, as the intelligent driving device is about to pass through a first intersection, the map information indicates that there is a traffic light at the first intersection, and the intelligent driving system expects to recognize the traffic light at the first intersection. However, when recognizing the traffic light based on a first image, the first recognition result indicates that the first image does not include the traffic light, meaning that the actual perceived result is inconsistent with the expected information. The intelligent device cannot obtain the traffic light information of the first intersection based on the first image, therefore, a second camera is activated to capture a second image.

[0141] In some embodiments of this application, combining real-time perception data from the first camera with map information can help intelligent driving devices understand road conditions more accurately, which is beneficial for providing more comprehensive support for the decision to activate the second camera.

[0142] Optionally, before acquiring the second image captured by the second camera, the method further includes: when the first recognition result indicates that the first image does not include traffic lights, determining the second camera according to the driving rules of the area, wherein the driving rules of the area include: left-hand drive or right-hand drive.

[0143] Countries that use right-hand drive driving rules include China, the United States, and Canada; countries that use left-hand drive driving rules include Japan, the United Kingdom, and New Zealand.

[0144] In some embodiments, determining the second camera based on the driving rules of the region includes: if the driving rules of the region are right-hand drive, the left-hand camera can be enabled for traffic light recognition; if the driving rules of the region are left-hand drive, the right-hand camera can be enabled for traffic light recognition.

[0145] Figure 12 illustrates a scenario of right-hand drive driving according to an embodiment of this application. As shown in Figure 12(A), the intelligent driving device is about to pass through the first intersection. It identifies the traffic lights based on the first image and obtains a first identification result, indicating that the first image does not include the traffic lights. It can be seen that traffic light a at the first intersection is used to indicate the movement of vehicles in the north-south lane, and traffic light a is located to the left front of the intelligent driving device. Traffic light b at the first intersection is used to indicate the movement of vehicles in the east-west lane, and traffic light b is located to the right front of the intelligent driving device. Since the intelligent driving device is driving in the north-south lane, it needs to identify traffic light a, which indicates the movement of vehicles in the north-south lane, thus determining to activate the left-side camera. As shown in Figure 12(B), traffic light a is within the FOV range of the left-side camera.

[0146] For example, if the intelligent driving device is in a driving scenario in mainland China, and the first recognition result indicates that the first image does not include traffic lights, the left-side camera can be turned on by default.

[0147] Figure 13 illustrates a left-hand drive driving scenario provided by an embodiment of this application. As shown in Figure 13(A), the intelligent driving device is about to pass through the first intersection. It identifies the traffic lights based on the first image and obtains a first identification result, which indicates that the first image does not include the traffic lights. It can be seen that traffic light c at the first intersection is used to indicate the movement of vehicles in the north-south lane, and traffic light c is located to the right front of the intelligent driving device. Traffic light d at the first intersection is used to indicate the movement of vehicles in the east-west lane, and traffic light d is located to the left front of the intelligent driving device. Since the intelligent driving device is driving in the north-south lane, it needs to identify the traffic light c indicating the movement of vehicles in the north-south lane, i.e., determine to activate the right-side camera. As shown in Figure 13(B), traffic light c is within the FOV range of the right-side camera.

[0148] For example, if the intelligent driving device is in a driving scenario in the UK, and the first recognition result indicates that the first image does not include traffic lights, the right-side camera can be turned on by default.

[0149] For example, the intelligent driving device identifies two first traffic lights based on the first image. One first traffic light is located on the left side of the first image, and the other is located on the right side. Both traffic lights are outside the field of view (FOV) of the forward-facing camera after being identified. If the intelligent driving device is in a driving scenario in mainland China, as shown in Figure 12, it is estimated that the traffic light in front of the vehicle on the left is used to indicate the movement of vehicles in the current lane. Therefore, the left-side camera is activated by default to detect the traffic light.

[0150] Based on the solutions provided in some embodiments of this application, the second camera is determined according to the driving rules of the local area, which helps intelligent driving equipment to better adapt to the traffic environment of different regions, improve the accuracy and efficiency of traffic light recognition, and thus provide effective protection for driving safety.

[0151] In some embodiments, while the first recognition result indicates that the recognition of the traffic light has malfunctioned, a second image is acquired by a second camera.

[0152] For example, the intelligent driving device acquires a first image from a first camera at a first driving position, identifies a first traffic light based on the first image, and obtains a first identification result. The distance between the first driving position and the stop line at the first intersection can be 200 meters. Simultaneously with the first identification result indicating that the first traffic light at the first intersection is a fault light, a second image from a second camera is immediately acquired.

[0153] Based on some embodiments of this application, the rapid activation of the second camera helps ensure the safe driving of intelligent driving devices and facilitates faster collection of environmental information from intelligent driving devices for traffic light recognition.

[0154] Optionally, before acquiring the second image captured by the second camera, the method further includes: after the first recognition result indicates that the recognition of the traffic light has been abnormal for a first duration, acquiring the second image captured by the second camera, wherein the first duration can be determined based on at least one of the following: the distance of the vehicle to the first intersection, the vehicle speed, the weather conditions, the road surface conditions, and the vehicle braking performance.

[0155] For example, the intelligent driving device acquires a first image from a first camera at a first driving position, identifies a first traffic light based on the first image, and obtains a first identification result. The distance between the first driving position and the stop line at the first intersection can be 200 meters. The first identification result indicates that the first traffic light at the first intersection is faulty. However, if the first identification result indicates an abnormality in the identification of the traffic light, a second image from a second camera is not immediately acquired. If, before the vehicle continues to drive forward to a second driving position, the first identification result continues to indicate that the first traffic light at the first intersection is faulty, then a second image from a second camera is acquired at the second driving position. The distance between the second driving position and the stop line at the first intersection can be 80 meters. The aforementioned first duration can be the time taken from the first driving position to the second driving position. In some embodiments, if the first identification result indicates that the first traffic light at the first intersection is not faulty before the vehicle reaches the second driving position, then it is not necessary to acquire a second image from the second camera. That is, the second camera does not need to be activated, saving resources.

[0156] For example, when the intelligent driving device is at the first driving position, it identifies the first traffic light based on the first image, meaning the first traffic light at the first intersection is within the field of view (FOV) of the first camera. As the intelligent driving device continues to move forward to the second driving position, the first traffic light switches from being within the FOV of the first camera to being outside the FOV of the first camera. However, if the first identification result indicates an abnormality in the identification of the traffic light, a second image captured by the second camera is not immediately acquired. Until the vehicle continues to move forward to the third driving position, if the first traffic light remains outside the FOV of the first camera, a second image captured by the second camera is acquired at the third driving position. The first duration can be the time taken from the second driving position to the third driving position.

[0157] For example, when the weather conditions are severe or the road conditions are poor, such as when it is raining or snowing and the road is slippery, the first time can be shortened accordingly so that the vehicle has sufficient preparation and reaction time.

[0158] For example, if the vehicle speed is high or the vehicle braking performance is poor, the first duration can also be shortened accordingly.

[0159] Based on the solutions provided in some embodiments of this application, when an abnormal situation occurs in the first recognition result, the second camera is not immediately activated, but a certain amount of time is reserved to review the first recognition result. This allows for continuous monitoring of the traffic light status and effectively avoids resource waste caused by activating the second camera due to recognition errors.

[0160] Optionally, after enabling the second camera to identify the traffic lights at the first intersection, the method further includes: fusing and deduplicating duplicate lights identified by multiple cameras.

[0161] For example, if the left and right cameras identify the same traffic light at the first intersection, the intelligent driving system can determine the same traffic light based on built-in rules. For instance, based on the location, color, and type of the traffic light, it can determine that the two traffic lights are actually the same traffic light. In this way, the two traffic lights can be treated as one traffic light to achieve the purpose of deduplication, which can optimize the traffic light recognition results and build a real-time traffic light perception map for intelligent driving devices.

[0162] S604: Identify traffic lights based on the second image.

[0163] Optionally, after recognizing the traffic light based on the second image, the method further includes: controlling the intelligent driving device based on a second recognition result of the second image.

[0164] For example, when the vehicle is in manual driving mode, the display device in the cabin can be controlled to show the recognition results of traffic lights, and the driver can control the vehicle based on the recognition results. As another example, when the intelligent driving device is in intelligent driving mode, the intelligent driving system can control the intelligent driving device based on the recognition results of traffic lights.

[0165] Figure 14 illustrates a traffic light recognition method 1400 provided in an embodiment of this application. As shown in Figure 14, the method 1400 includes:

[0166] S1401: Traffic light recognition based on the first image captured by the forward-looking camera.

[0167] S1402: Based on the first image, determine whether there are no traffic lights within the FOV range of the forward-looking camera.

[0168] As the vehicle travels from a distance toward the first intersection, traffic light identification is performed based on the first image captured by the forward-looking camera. If there is no traffic light in the first image, confirming that there is no traffic light within the FOV of the forward-looking camera, then S1406 is executed.

[0169] If the first traffic light is identified based on the first image, then step S1403 is executed.

[0170] S1403: Based on the first image, the first traffic light is identified, and it is determined whether the first traffic light is outside the FOV range of the forward-looking camera, or whether the first traffic light is a faulty traffic light.

[0171] If a first traffic light is detected based on the first image at the first driving position, and the first traffic light is outside the FOV range of the forward-looking camera at the second driving position, then S1404 is executed.

[0172] If a first traffic light is identified based on the first image, and the first traffic light is a faulty traffic light, then S1404 is executed.

[0173] If a first traffic light is identified based on the first image, and the first traffic light never exceeds the FOV of the forward-looking camera and the first traffic light is not a faulty light, then no other steps are performed.

[0174] S1404: Based on the first image, determine that the first traffic light is located on the left, right, or both sides of the vehicle.

[0175] If the first traffic light is on the right side of the vehicle, execute S1405; if the first traffic light is on the left side of the vehicle, execute S1406; if the traffic light is on both the left and right sides of the vehicle, execute S1406.

[0176] S1405: Control the right-side camera to turn on and perform traffic light recognition based on the second image captured by the right-side camera.

[0177] S1406: Control the left-side camera to turn on and perform traffic light recognition based on the second image captured by the left-side camera.

[0178] If the traffic lights are located on the left and right sides of the vehicle, the left and right cameras can be turned on simultaneously, and traffic light recognition can be performed based on the second images captured by the left and right cameras.

[0179] S1407: Completes traffic light recognition and deduplication of multiple cameras.

[0180] Figure 15 shows a schematic block diagram of a traffic light recognition device 1500 provided in an embodiment of this application. The traffic light recognition device 1500 includes: an acquisition unit 1510 for acquiring a first image captured by a first camera; a recognition unit 1520 for recognizing a traffic light based on the first image to obtain a first recognition result; the acquisition unit 1510 is further configured to acquire a second image captured by a second camera when the first recognition result indicates an abnormality in the recognition of the traffic light, wherein the field of view (FOV) ranges of the first and second cameras do not overlap or the FOV ranges of the first and second cameras partially overlap; the recognition unit 1520 is further configured to recognize the traffic light based on the second image.

[0181] Optionally, the identification unit 1520 is further configured to: identify the first traffic light based on the first image; the acquisition unit 1510 is specifically configured to: acquire the second image captured by the second camera when the first identification result indicates that the first traffic light has switched from being within the FOV range of the first camera to being outside the FOV range of the first camera; or, acquire the second image captured by the second camera when the first identification result indicates that the first traffic light is a fault light.

[0182] Optionally, the device further includes a determining unit 1540, which is configured to: determine the position of the first traffic light based on the first image; and the determining unit 1540 is also configured to: determine the position of the second camera based on the position of the first traffic light.

[0183] Optionally, the acquisition unit 1510 is specifically used to: acquire a second image captured by a second camera when the first image is about to pass through the first intersection and the first recognition result indicates that the first image does not include traffic lights.

[0184] Optionally, the acquisition unit 1510 is specifically used to: when the map information indicates that there is a traffic light at the first intersection and the first recognition result indicates that the first image does not include the traffic light, acquire the second image captured by the second camera.

[0185] Optionally, the device further includes a determining unit 1540, which is configured to: if the first recognition result indicates that the first image does not include traffic lights, determine the second camera according to the driving rules of the area, wherein the driving rules of the area include: left-hand drive or right-hand drive.

[0186] Optionally, the first camera is a front-view camera, and the second camera is a side-view camera in the first direction.

[0187] Optionally, the acquisition unit 1510 is specifically used to: acquire a second image captured by a second camera and acquire a third image captured by a third camera, wherein the third camera is a side-view camera in the second direction, and the first direction and the second direction are opposite directions.

[0188] Optionally, the acquisition unit 1510 is further configured to: acquire a third image acquired by a third camera before the acquisition unit acquires the second image acquired by the second camera, wherein the third camera is a side-view camera in the second direction, and the first direction and the second direction are opposite directions; wherein, the acquisition unit 1510 is specifically configured to: acquire the second image acquired by the second camera when the third image does not include traffic lights.

[0189] Optionally, before acquiring the second image captured by the second camera, the acquisition unit 1510 is further configured to: acquire the second image captured by the second camera after the first recognition result indicates that the recognition of the traffic light has been abnormal for a first duration, wherein the first duration can be determined based on at least one of the following: vehicle speed, weather conditions, road conditions, and vehicle braking performance.

[0190] Optionally, the device further includes a control unit 1530, which is configured to control the intelligent driving device based on a second recognition result of the second image after the recognition unit recognizes the traffic light based on the second image.

[0191] It should be understood that the division of units in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the units in the device can be implemented by a processor calling software; for example, the device includes a processor connected to memory, which stores instructions. The processor calls the instructions stored in memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be, for example, a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. The functions of some or all units can be implemented through the design of the hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all units are implemented through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby implementing the functions of some or all units. All units of the above devices can be implemented entirely through processor calling software, or entirely through hardware circuits, or partially through processor calling software with the remaining parts implemented through hardware circuits.

[0192] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0193] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0194] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together as a System-on-a-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and AI processor, CPU and GPU, etc.

[0195] This application also provides a traffic light recognition device, which includes: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, so that the device performs the methods or steps described in the above embodiments.

[0196] This application also provides an intelligent driving device, including the traffic light recognition device 1500 described above.

[0197] Alternatively, the intelligent driving device can be a vehicle.

[0198] This application also provides a computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the methods or steps described in the above embodiments.

[0199] This application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to implement the methods or steps performed in the above embodiments.

[0200] This application also provides a chip, which includes a circuit for performing the methods or steps described in the above embodiments.

[0201] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, power-on erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0202] It should be understood that in the embodiments of this application, the memory may include read-only memory and random access memory, and provides instructions and data to the processor.

[0203] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0204] Those skilled in the art will recognize that the units of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0205] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0206] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0207] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0208] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0209] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0210] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be covered. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A traffic light recognition method, characterized in that, include: Acquire the first image captured by the first camera; Based on the first image, traffic lights are identified, and a first identification result is obtained; If the first recognition result indicates that the recognition of the traffic light is abnormal, the second image captured by the second camera is obtained, and the field of view (FOV) ranges of the first camera and the second camera do not overlap or the FOV ranges of the first camera and the second camera partially overlap. The traffic light is identified based on the second image.

2. The method according to claim 1, characterized in that, The method further includes: The first traffic light was identified based on the first image; If the first recognition result indicates that the traffic light recognition is abnormal, the second image captured by the second camera is acquired, including: When the first recognition result indicates that the first traffic light switches from being within the FOV of the first camera to being outside the FOV of the first camera, the second image captured by the second camera is acquired; or, When the first recognition result indicates that the first traffic light is a faulty light, the second image captured by the second camera is acquired.

3. The method according to claim 2, characterized in that, Before acquiring the second image from the second camera, the method also includes: Based on the first image, determine the position of the first traffic light; The second camera is located based on the position of the first traffic light.

4. The method according to claim 1, characterized in that, If the first recognition result indicates that the traffic light recognition is abnormal, the second image captured by the second camera is acquired, including: When the first intersection is about to be reached and the first recognition result indicates that the first image does not include the traffic light, the second image captured by the second camera is acquired.

5. The method according to claim 4, characterized in that, The step of acquiring a second image from a second camera when the first intersection is about to be reached and the first recognition result indicates that the first image does not include the traffic light includes: When approaching the first intersection, if the map information indicates that the traffic light exists at the first intersection, and the first recognition result indicates that the first image does not include the traffic light, then the second image captured by the second camera is obtained.

6. The method according to claim 5, characterized in that, Before acquiring the second image captured by the second camera, the method further includes: When the first recognition result indicates that the first image does not include the traffic light, the second camera is determined according to the driving rules of the area, which include: left-hand drive or right-hand drive.

7. The method according to any one of claims 1-6, characterized in that, The first camera is a front-view camera, and the second camera is a side-view camera in the first direction.

8. The method according to claim 7, characterized in that, The acquisition of the second image captured by the second camera includes: The system acquires the second image captured by the second camera and the third image captured by the third camera, wherein the third camera is a side-view camera in the second direction, and the first direction and the second direction are opposite directions.

9. The method according to claim 7, characterized in that, Before acquiring the second image captured by the second camera, the method further includes: Acquire a third image captured by a third camera, wherein the third camera is a side-view camera in the second direction, and the first direction and the second direction are opposite directions; The acquisition of the second image captured by the second camera includes: When the traffic light is not included in the third image, the second image captured by the second camera is obtained.

10. The method according to any one of claims 1-9, characterized in that, Before acquiring the second image captured by the second camera, the method further includes: After the first recognition result indicates that the traffic light recognition has malfunctioned for a first duration, the second camera image is acquired. The second image acquired by the head, the first duration can be determined based on at least one of the following: vehicle speed, weather conditions, road conditions, and vehicle braking performance.

11. The method according to any one of claims 1-10, characterized in that, After identifying the traffic light based on the second image, the method further includes: The intelligent driving device is controlled based on the second recognition result of the second image.

12. A traffic signal light recognition device, characterized in that, include: The acquisition unit is used to acquire the first image captured by the first camera; The recognition unit is used to recognize traffic lights based on the first image and obtain a first recognition result; The acquisition unit is further configured to acquire a second image captured by the second camera when the first recognition result indicates that the recognition of the traffic light is abnormal, wherein the field of view (FOV) ranges of the first camera and the second camera do not overlap or the FOV ranges of the first camera and the second camera partially overlap. The identification unit is also used to identify the traffic light based on the second image.

13. The apparatus according to claim 12, characterized in that, The identification unit is further configured to: The first traffic light was identified based on the first image; The acquisition unit is specifically used for: When the first recognition result indicates that the first traffic light has switched from being within the FOV of the first camera to being outside the FOV of the first camera, the second image captured by the second camera is acquired; or, When the first recognition result indicates that the first traffic light is a faulty light, the second image captured by the second camera is acquired.

14. The apparatus according to claim 13, characterized in that, The device also includes a determining unit. The determining unit is configured to: determine the position of the first traffic light based on the first image before the acquiring unit acquires the second image captured by the second camera; The determining unit is further configured to: determine the second camera based on the position of the first traffic light.

15. The apparatus according to claim 12, characterized in that, The acquisition unit is specifically used for: When the first intersection is about to be reached and the first recognition result indicates that the first image does not include the traffic light, the second image captured by the second camera is acquired.

16. The apparatus according to claim 15, characterized in that, The acquisition unit is specifically used for: When approaching the first intersection, if the map information indicates that the traffic light exists at the first intersection, and the first recognition result indicates that the first image does not include the traffic light, then the second image captured by the second camera is obtained.

17. The apparatus according to claim 16, characterized in that, The apparatus further includes a determining unit, the determining unit being configured to: If the first recognition result indicates that the first image does not include the traffic light, the second camera is determined according to the driving rules of the area, which include: left-hand drive or right-hand drive.

18. The apparatus according to any one of claims 12-17, characterized in that, The first camera is a front-view camera, and the second camera is a side-view camera in the first direction.

19. The apparatus according to claim 18, characterized in that, The acquisition unit is specifically used for: The system acquires the second image captured by the second camera and the third image captured by the third camera, wherein the third camera is a side-view camera in the second direction, and the first direction and the second direction are opposite directions.

20. The apparatus according to claim 18, characterized in that, The acquisition unit is further configured to: Before the acquisition unit acquires the second image captured by the second camera, it acquires a third image captured by a third camera, wherein the third camera is a side-view camera in the second direction, and the first direction and the second direction are opposite directions; The acquisition unit is specifically used for: When the traffic light is not included in the third image, the second image captured by the second camera is obtained.

21. The apparatus according to any one of claims 12-20, characterized in that, The acquisition unit is further configured to: After the first recognition result indicates that the traffic light recognition has been abnormal for a first duration, the second image captured by the second camera is acquired. The first duration can be determined based on at least one of the following: vehicle speed, weather conditions, road conditions, and vehicle braking performance.

22. The apparatus according to any one of claims 12-21, characterized in that, The device further includes: a control unit, The control unit is used for: After the recognition unit recognizes the traffic light based on the second image, it controls the intelligent driving device based on the second recognition result of the second image.

23. A traffic signal light recognition device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to cause the apparatus to perform the method as described in any one of claims 1 to 11.

24. An intelligent driving device, characterized in that, The traffic signal recognition device includes any one of claims 12-23.

25. The intelligent driving device according to claim 24, characterized in that, The intelligent driving device is a vehicle.

26. A computer-readable storage medium, characterized in that, It stores instructions that, when executed by a processor, cause the processor to implement the method as described in any one of claims 1 to 11.

27. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 11.

28. A chip, characterized in that, The chip includes circuitry for performing the method as described in any one of claims 1 to 11.

Images