System and method for providing traffic signal light roi parameters to autonomous vehicles

Abstract

An image sequence and a vehicle location associated with each image are received at a traffic light ROI management system. At least one traffic light is detected in each image. An ECS traffic light ROI is defined for each image. The ECS traffic light ROI encompasses the detected traffic light. A visual feature template is generated for each image. The visual feature template is based on the ECS traffic light ROI of the image. Each visual feature template is mapped to a high definition (HD) map at the vehicle location associated with the image. The HD map is sent to an autonomous vehicle to enable the autonomous vehicle to identify a real-time traffic light ROI in a real-time image based on a match between a first visual feature template and a real-time visual feature of the real-time image at a vehicle location associated with the first visual feature template.

Description

Technical Field

[0001] This technical field generally relates to autonomous vehicles, and more specifically to systems and methods for generating regions of interest for traffic light detection. Background Technology

[0002] Automated driving systems (ADS) in autonomous vehicles typically rely on images captured by the vehicle's cameras to detect traffic lights on road sections. These images can be high-resolution, such as 8-megapixel images. Using onboard computing resources to implement object detection algorithms to detect traffic lights in images on autonomous vehicles can lead to increased computing resource utilization and traffic light detection latency.

[0003] In some cases, the route to traffic lights may include complex road terrain structures, such as bumps, curves, or hills. Compensating for these complex terrain structures to achieve accurate traffic light detection can place additional strain on the computing resources of autonomous vehicles. Summary of the Invention

[0004] In one embodiment, an edge computing system (ECS) for providing traffic light region of interest (ROI) parameters to an autonomous vehicle includes a processor and a memory. The memory includes instructions that, when executed by the processor, cause the processor to receive an image sequence and vehicle positions associated with each image from a first autonomous vehicle; detect at least one traffic light in each image of the image sequence; define an ECS traffic light ROI for each image of the image sequence, the ECS traffic light ROI of each image surrounding at least one traffic light in the image; generate a visual feature template for each image of the image sequence, the visual feature template of each image being based on the ECS traffic light ROI of the image; map each visual feature template of each image of the image sequence to a vehicle position associated with the image on a high-definition (HD) map; and send the HD map to a second autonomous vehicle so that the second autonomous vehicle can identify a real-time traffic light ROI in the real-time image based on a match between a first visual feature template on the HD map and a real-time visual feature associated with the real-time image at the vehicle position associated with the first visual feature template.

[0005] In this embodiment, each image in the image sequence is in forward chronological order, and the memory also includes instructions that, when executed by the processor, cause the processor to define the ECS traffic light ROI in each image of the image sequence in reverse chronological order.

[0006] In an embodiment, the memory also includes instructions that, when executed by the processor, cause the processor to discard the first image of the image sequence when it determines that at least one traffic light detected in the first image of the image sequence has dropped below a traffic light size threshold.

[0007] In an embodiment, the memory also includes instructions that, when executed by the processor, cause the processor to generate a visual feature template of the image in the frequency domain by applying a fast Fourier transform to the ECS traffic light ROI of each image in the image sequence.

[0008] In an embodiment, the memory further includes instructions that, when executed by a processor, cause the processor to select a second visual feature template based on a first image in an image sequence, the first image being associated with a first vehicle location; determine whether the second visual feature template can be used to identify a first ECS traffic light ROI in a second image in an image sequence, the second image being associated with a second vehicle location, based on a match between the second visual feature template and visual features associated with the second image; and map the second visual feature template onto the first and second vehicle locations on an HD map based on the determination.

[0009] In an embodiment, the memory further includes instructions that, when executed by a processor, cause the processor to determine the ROI offset of the ECS traffic light ROI in consecutive pairs of images based on the movement of the ECS traffic light ROI in consecutive pairs of images in an image sequence; generate an ROI scaling ratio for each vehicle position associated with each image in the image sequence based on the relationship between each ROI offset and the vehicle position associated with the ROI offset; and map each ROI scaling ratio to a vehicle position associated with the ROI scaling ratio on an HD map.

[0010] In one embodiment, the memory also includes instructions that, when executed by the processor, cause the processor to send an HD map, including a mapped ROI scaling ratio, to a second autonomous vehicle, enabling the second autonomous vehicle to adjust the size of the real-time traffic light ROI in the real-time image according to the ROI scaling ratio associated with the vehicle's location.

[0011] In one embodiment, a computer-readable medium for providing traffic light region of interest (ROI) parameters to an autonomous vehicle includes instructions stored thereon that, when executed by a processor, cause the processor to receive an image sequence and vehicle positions associated with each image from a first autonomous vehicle; detect at least one traffic light in each image of the image sequence; define an ECS traffic light ROI for each image of the image sequence, the ECS traffic light ROI of each image surrounding at least one traffic light in the image; generate a visual feature template for each image of the image sequence, the visual feature template of each image being based on the ECS traffic light ROI of the image; map each visual feature template of each image of the image sequence to a vehicle position associated with the image on a high-definition (HD) map; and send the HD map to a second autonomous vehicle so that the second autonomous vehicle can identify a real-time traffic light ROI in a real-time image based on a match between a first visual feature template on the HD map and a real-time visual feature associated with a real-time image at a vehicle position associated with the first visual feature template.

[0012] In an embodiment, the computer-readable medium further includes instructions to cause the processor to receive the image sequence in forward chronological order; and to define the ECS traffic light ROI in each image of the image sequence in reverse chronological order.

[0013] In an embodiment, the computer-readable medium further includes instructions to cause the processor to discard the first image of the image sequence when it determines that at least one detected traffic light in the first image has fallen below a traffic light size threshold.

[0014] In an embodiment, the computer-readable medium further includes instructions to cause the processor to generate a visual feature template of the image in the frequency domain by applying a fast Fourier transform to the ECS traffic light ROI of each image in the image sequence.

[0015] In an embodiment, the computer-readable medium further includes instructions to cause the processor to select a second visual feature template based on a first image in an image sequence, the first image being associated with a first vehicle location; to determine whether the second visual feature template can be used to identify a first ECS traffic light ROI in a second image in an image sequence, the second image being associated with a second vehicle location, based on a match between the second visual feature template and visual features associated with the second image; and to map the second visual feature template onto the first and second vehicle locations on an HD map based on the determination.

[0016] In an embodiment, the computer-readable medium further includes instructions to cause the processor to determine the ROI offset of the ECS traffic light ROI in consecutive pairs of images based on the movement of the ECS traffic light ROI in consecutive pairs of images in an image sequence; to generate an ROI scaling ratio for each vehicle position associated with each image in the image sequence based on a relationship between each ROI offset and the vehicle position associated with the ROI offset; and to map each ROI scaling ratio to a vehicle position associated with the ROI scaling ratio on an HD map.

[0017] In an embodiment, the computer-readable medium further includes instructions to cause the processor to send an HD map, including a mapped ROI scaling ratio, to a second autonomous vehicle, enabling the second autonomous vehicle to adjust the size of a real-time traffic light ROI in a real-time image based on the ROI scaling ratio associated with the vehicle's location.

[0018] In one embodiment, a method for providing traffic light Region of Interest (ROI) parameters to an autonomous vehicle includes receiving an image sequence and vehicle positions associated with each image from a first autonomous vehicle at a traffic light ROI management system; detecting at least one traffic light in each image of the image sequence at the traffic light ROI management system; defining an ECS (Electronic Control System) traffic light ROI for each image of the image sequence at the traffic light ROI management system, wherein the ECS traffic light ROI of each image surrounds at least one traffic light in the image; and generating each image of the image sequence at the traffic light ROI management system. The visual feature templates for each image are based on the ECS traffic light ROI of the image; at the traffic light ROI management system, each visual feature template of each image in the image sequence is mapped to the vehicle location associated with the image on a high-definition (HD) map; and the HD map is sent from the traffic light ROI management system to a second autonomous vehicle so that the second autonomous vehicle can identify the real-time traffic light ROI in the real-time image based on the matching between the first visual feature template of the HD map and the real-time visual features associated with the real-time image at the vehicle location associated with the first visual feature template.

[0019] In an embodiment, the method further includes receiving an image sequence in forward chronological order at a traffic light ROI management system; and defining the ECS traffic light ROI for each image in the image sequence in reverse chronological order at the traffic light ROI management system.

[0020] In an embodiment, the method further includes discarding the first image of the image sequence when the traffic light ROI management system determines that at least one detected traffic light in the first image has fallen below a traffic light size threshold.

[0021] In an embodiment, the method further includes generating a visual feature template of the image in the frequency domain by applying a fast Fourier transform to the ECS traffic light ROI of each image in the image sequence at the traffic light ROI management system.

[0022] In an embodiment, the method further includes selecting a second visual feature template at a traffic light ROI management system based on a first image in an image sequence, the first image being associated with a first vehicle location; determining at the traffic light ROI management system whether the second visual feature template can be used to identify a first ECS traffic light ROI in a second image in an image sequence, the second image being associated with a second vehicle location, based on a match between the second visual feature template and visual features associated with a second image; and determining at the traffic light ROI management system to map the second visual feature template to the first and second vehicle locations on an HD map.

[0023] In an embodiment, the method further includes determining, at the traffic light ROI management system, the ROI offset of the ECS traffic light ROI in consecutive pairs of images in the image sequence based on the movement of the ECS traffic light ROI in consecutive pairs of images; generating, at the traffic light ROI management system, an ROI scaling ratio associated with each vehicle position in each image sequence based on the relationship between each ROI offset and the vehicle position associated with the ROI offset; and mapping each ROI scaling ratio to the vehicle position associated with the ROI scaling ratio on the HD map at the traffic light ROI management system. Attached Figure Description

[0024] Exemplary embodiments will be described below with reference to the accompanying drawings, wherein the same numbers denote the same elements.

[0025] Figure 1 This is a functional block diagram of an autonomous vehicle coupled to an edge computing system (ECS), which includes an embodiment of a traffic light region of interest (ROI) management system;

[0026] Figure 2 This is a schematic diagram of an autonomous vehicle that captures a sequence of images, including traffic lights, to be sent to an ECS (Electronic Control System) including a traffic light ROI (Region of Interest) management system.

[0027] Figure 3This is a schematic diagram of multiple autonomous vehicles communicatively coupled to an embodiment of an ECS including a traffic signal ROI management system;

[0028] Figure 4 This is a functional block diagram of an embodiment of an ECS including a traffic signal ROI management system;

[0029] Figure 5 This is a flowchart illustrating an example of a method for generating multiple ECS traffic light ROIs based on an image sequence received from an autonomous vehicle, as an embodiment of a traffic light ROI management system.

[0030] Figure 6a and Figure 6b This is a block diagram illustrating an example of the ECS traffic light ROI location in a pair of consecutive images within an image sequence.

[0031] Figure 7 This is a schematic diagram illustrating an example of the association between each of multiple visual feature templates and multiple vehicle locations on a road segment including traffic lights;

[0032] Figure 8 This is a schematic diagram illustrating the vehicle location of an autonomous vehicle and an example of the vehicle location associated with multiple visual feature templates on a road segment including traffic lights.

[0033] Figure 9 A flowchart illustrating a method for providing traffic light ROI parameters to autonomous vehicles; and

[0034] Figure 10 It is a graph showing distance as a function of vehicle position. Detailed Implementation

[0035] The following detailed descriptions are merely exemplary in nature and are not intended to limit application and use. Furthermore, they are not intended to be construed as being bound by any express or implied theory presented in the foregoing description, summary of the invention, or the following detailed descriptions. As used herein, the term "module" refers to any hardware, software, firmware, electronic control components, processing logic devices, and / or processor devices, individually or in any combination, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped), and memory executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components providing the described functionality.

[0036] The embodiments of this disclosure can be described herein in terms of functional and / or logical block components and various processing steps. It should be understood that such block components can be implemented by any number of hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of this disclosure can employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices. Furthermore, those skilled in the art will understand that embodiments of this disclosure can be combined with any number of system practices, and the systems described herein are merely exemplary embodiments of this disclosure.

[0037] For the sake of brevity, conventional techniques related to signal processing, data transmission, signaling, control, and other functional aspects of the system (and its individual operating components) are not described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent exemplary functional relationships and / or physical couplings between various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in the embodiments of this disclosure.

[0038] refer to Figure 1 This diagram illustrates a functional block diagram of an autonomous vehicle communicatively coupled to an edge computing system (ECS) including an embodiment of a traffic light region of interest (ROI) management system. The autonomous vehicle 100 typically includes a chassis 112, a body 114, front wheels 116, and rear wheels 118. The body 114 is disposed on the chassis 112 and substantially surrounds the components of the autonomous vehicle 100. The body 114 and chassis 112 may together form a frame. The front wheels 116 and rear wheels 118 are each rotatably coupled to the chassis 112 near a corresponding corner of the body 114.

[0039] Automated vehicle 100 is, for example, an autonomously controlled vehicle that transports passengers from one location to another. While the automated vehicle 100 is described as a passenger car in the illustrated embodiment, other examples of automated vehicles include, but are not limited to, motorcycles, trucks, SUVs, RVs, ships, and aircraft. In one embodiment, the automated vehicle 100 is a so-called Level 4 or Level 5 automation system. A Level 4 system signifies “high automation,” referring to the driving mode-specific performance of the Automated Driving System (ADS) in all aspects of a dynamic driving task, even if the human driver does not respond appropriately to intervention requests. A Level 5 system signifies “full automation,” referring to the full-time performance of the ADS in all road and environmental conditions that a human driver can manage in all aspects of a dynamic driving task.

[0040] As shown in the figure, an autonomous vehicle 100 typically includes a propulsion system 120, a transmission system 122, a steering system 124, a braking system 126, a vehicle sensor system 128, an actuator system 130, at least one data storage device 132, at least one controller 134, and a vehicle communication system 136. In various embodiments, the propulsion system 120 may include an internal combustion engine, an electric motor (such as a traction motor), and / or a fuel cell propulsion system. The transmission system 122 is configured to transmit power from the propulsion system 120 to the front wheels 116 and the rear wheels 118 according to a selectable speed ratio. According to various embodiments, the transmission system 122 may include a step-ratio automatic transmission, a continuously variable transmission (CVT), or other suitable transmission. The braking system 126 is configured to provide braking torque to the front wheels 116 and the rear wheels 118. In various embodiments, the braking system 126 may include friction brakes, brake-by-wire brakes, regenerative braking systems (e.g., electric motors), and / or other suitable braking systems. The steering system 124 influences the position of the front wheels 116 and the rear wheels 118. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of this disclosure, the steering system 124 may not include a steering wheel.

[0041] The vehicle sensor system 128 includes one or more vehicle sensing devices 140a-140n that sense observable conditions of the external and / or internal environments of the autonomous vehicle 100. Examples of vehicle sensing devices 140a-140n include, but are not limited to, radar, lidar, GPS, optical cameras, thermal imagers, ultrasonic sensors, and / or other sensors. The actuator system 130 includes one or more actuator devices 142a-142n that control one or more vehicle features, such as, but not limited to, the propulsion system 120, the drivetrain 122, the steering system 124, and the braking system 126. In various embodiments, vehicle features may also include internal and / or external vehicle features, such as, but not limited to, doors, the trunk, and cabin features (e.g., air, music, and lighting).

[0042] Vehicle communication system 136 is configured to communicate wirelessly with other entities (“vehicle-to-everything (V2X)” communication). For example, vehicle communication system 136 is configured to communicate wirelessly with other vehicles 148 (“vehicle-to-vehicle (V2V)” communication), drive system infrastructure (“vehicle-to-infrastructure (V2I)” communication), remote systems, ECS 150, and / or personal devices. In embodiments, vehicle communication system 136 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communication. However, additional or alternative communication methods, such as dedicated short-range communication (DSRC) channels, are also considered within the scope of this disclosure. A DSRC channel refers to a one-way or two-way short-to-medium-range wireless communication channel specifically designed for automotive use, along with a set of corresponding protocols and standards.

[0043] Data storage device 132 stores data used for automatically controlling the autonomous vehicle 100. Data storage device 132 may be part of controller 134, separate from controller 134, or part of controller 134 and a separate system.

[0044] The controller 134 includes at least one processor 144 and a computer-readable storage device 146. The computer-readable storage device 146 may also be referred to as a computer-readable medium 146. The processor 144 may be any custom or commercially available processor, central processing unit (CPU), graphics processing unit (GPU), auxiliary processor among several processors associated with the controller 134, semiconductor-based microprocessor (in the form of a microchip or chipset), macroprocessor, any combination thereof, or any device typically used for executing instructions. For example, the computer-readable storage device 146 may include volatile and non-volatile memory in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operational variables when the processor 144 is powered off. The computer-readable storage device 146 may be implemented using any of a number of known storage devices, such as a programmable read-only memory (PROM), an electrical PROM (EPROM), an electrically erasable PROM (EEPROM), flash memory, or any other electrical, magnetic, optical, or combined storage device capable of storing data, some of which represent executable instructions used by the controller 134 to control the autonomous vehicle 100.

[0045] The instructions may include one or more separate programs, each comprising an ordered list of executable instructions for implementing logical functions. When executed by processor 144, these instructions receive and process signals from vehicle sensor system 128, execute logic, calculations, methods, and / or algorithms for automatically controlling components of autonomous vehicle 100, and generate control signals to actuator system 130 based on the logic, calculations, methods, and / or algorithms to automatically control one or more components of autonomous vehicle 100. Although Figure 1 Only one controller 134 is shown, but alternative embodiments of the autonomous vehicle 100 may include any number of controllers 134 that communicate via any suitable communication medium or combination of communication media and cooperate in processing sensor signals, executing logic, calculations, methods and / or algorithms, and generating control signals to automatically control the features of the autonomous vehicle 100.

[0046] In various embodiments, one or more instructions of controller 134 are embodied to provide ADS functionality as described with reference to one or more embodiments herein. Controller 134 or one of its functional modules is configured to implement the functionality based on traffic light ROI parameters received at ECS 150 from embodiments of traffic light ROI management system 152.

[0047] refer to Figure 2 A schematic diagram of an autonomous vehicle 100 is shown, which captures image sequences including traffic lights 200a and 200b for transmission to an ECS 150 comprising an embodiment of a traffic light ROI management system 152. The autonomous vehicle 100 includes a vehicle sensor system 128 and a vehicle communication system 136. The vehicle sensor system 128 includes one or more vehicle sensors 140a-140n. Examples of vehicle sensors 140a-140n include, but are not limited to, radar, lidar, global positioning system (GPS), optical cameras, thermal imagers, ultrasonic sensors, and / or other sensors.

[0048] Vehicle sensor system 128 is configured to capture a sequence of images as the autonomous vehicle 100 approaches traffic lights 200a and 200b on road segment 202. Each image in the sequence includes traffic lights 200a and 200b. Vehicle sensor system 128 is configured to identify the vehicle position of the autonomous vehicle 100 and associate the vehicle position with the image as each image is captured. The images in the sequence are arranged in forward chronological order. The images are arranged in the order in which they were captured by vehicle sensor system 128. For example, the first image in the sequence is the oldest image in the sequence, while the last image in the sequence is a newer image. Vehicle communication system 136 is configured to send the image sequence and the vehicle position associated with each image to traffic light ROI management system 152 at ECS 150. Although two traffic lights 200a and 200b are displayed on road segment 202, road segment 202 may include fewer or more traffic lights.

[0049] refer to Figure 3A schematic diagram of multiple autonomous vehicles 100 communicatively coupled to an ECS 150 including an embodiment of a traffic light ROI management system 152 is shown. Each of the multiple autonomous vehicles 100 in the group 300 is configured to capture an image sequence as the autonomous vehicle 100 approaches traffic lights 200a, 200b on road segment 202, for transmission to the ECS 150 including the traffic light ROI management system 152. Each autonomous vehicle 100 is configured to transmit the image sequence and the vehicle location associated with each image to the traffic light ROI management system 152 at the ECS 150. Although the number of autonomous vehicles 100 in the group 300 is shown as five, the group 300 may include more or fewer autonomous vehicles 100.

[0050] The traffic light ROI management system 152 is configured to receive image sequences from each of the autonomous vehicles 100 in the group 300. The traffic light ROI management system 152 is configured to process each image sequence individually. The traffic light ROI management system 152 is configured to perform reverse object (traffic light) detection and tracking on each image in the image sequence to generate an ECS traffic light ROI for each image in the image sequence (including images of traffic lights exceeding a traffic light size threshold).

[0051] In this embodiment, the traffic light ROI management system 152 is configured to generate an ROI scaling ratio associated with each vehicle position in the image based on the movement of the ECS traffic light ROIs in the image sequence. The traffic light ROI management system 152 is configured to determine the ECS ROI offsets in consecutive pairs of images in the image sequence. Each ROI offset is based on the movement of the ECS traffic light ROI in the consecutive pairs of images. Each ROI offset represents a movement of the ECS traffic light ROI from a first position in a pair of consecutive images to a second position in a pair of consecutive images. The movement of the ECS traffic light ROI, as a function of vehicle positions on road segment 202, defines the terrain structure of road segment 202. The traffic light ROI management system 152 is configured to generate an ROI scaling ratio associated with each vehicle position in each image in the image sequence based on the relationship between each ROI offset and the vehicle position associated with the ROI offset.

[0052] The traffic light ROI management system 152 is configured to map each ROI scaling ratio to a vehicle location on a high-definition (HD) map including road segment 202. The vehicle location on the HD map is associated with the ROI scaling ratio. The HD map is stored in an ECS database 304. The traffic light ROI management system 152 is configured to repeat this process for each image in the image sequence received from the autonomous vehicle 100 in group 300 to define, refine, and / or update the ROI scaling ratio for each of the multiple vehicle locations on road segment 202. In an embodiment, the traffic light ROI management system 152 is configured to generate ROI scaling ratios only for vehicle locations associated with images in the image sequence that include traffic lights exceeding a traffic light size threshold.

[0053] In one embodiment, the traffic light ROI management system 152 is configured to receive ECS traffic light ROIs and vehicle locations associated with each image in an image sequence including the ECS traffic light ROIs. The traffic light ROI management system 152 is configured to generate a visual feature template for each ECS traffic light ROI. In another embodiment, the traffic light ROI management system 152 is configured to generate a visual feature template for each ECS traffic light ROI in the frequency domain by applying a Fast Fourier Transform to the ECS traffic light ROI.

[0054] The traffic light ROI management system 152 is configured to map each visual feature template to a vehicle location on a high-definition (HD) map including road segment 202. The vehicle locations on the HD map are associated with images in an image sequence that include ECS traffic light ROIs associated with the visual feature templates. The HD map is stored in an ECS database 304. The traffic light ROI management system 152 is configured to repeat this process for each image in the image sequence received from the autonomous vehicle 100 in group 300 to define, refine, and / or update the visual feature template for each of the multiple vehicle locations on road segment 202. In an embodiment, the traffic light ROI management system 152 is configured to generate visual feature templates only for vehicle locations in the image sequence associated with images including traffic lights exceeding a traffic light size threshold.

[0055] In one embodiment, the traffic light ROI management system 152 is configured to identify ECS traffic light ROIs that include robust visual features. Each ECS traffic light ROI identified as having robust visual features is based on an image associated with a vehicle location. In one embodiment, the identified ECS traffic light ROI is associated with a vehicle location on road segment 202, which is associated with the image on which the ECS traffic light ROI is based. In one embodiment, the identified ECS traffic light ROI may be associated with multiple vehicle locations on road segment 202.

[0056] A traffic light ROI management system 152 is configured to send an HD map to autonomous vehicles 100 and 302. The HD map includes an ROI scaling ratio and visual feature templates mapped to different vehicle positions on road segment 202. The ROI scaling ratio and visual feature templates may be referred to as traffic light ROI parameters. In one embodiment, the traffic light ROI management system 152 is configured to send an HD map including an ROI scaling ratio and visual feature templates mapped to vehicle positions on road segment 202, which includes traffic lights 200a and 200b within a predetermined vicinity of the location of the autonomous vehicles 100 and 302. In another embodiment, the traffic light ROI management system 152 is configured to send an HD map including an ROI scaling ratio and visual feature templates mapped to vehicle positions on road segment 202, which includes traffic lights 200a and 200b on a route that the autonomous vehicles 100 and 302 are expected to take (e.g., during a highway journey).

[0057] Automated vehicle 100,302 includes a vehicle sensor system 128. The vehicle sensor system 128 includes one or more vehicle sensing devices 140a-140n. Examples of vehicle sensing devices 140a-140n include, but are not limited to, radar, lidar, global positioning system (GPS), optical cameras, thermal imagers, ultrasonic sensors, and / or other sensors.

[0058] Vehicle sensor system 128 is configured to capture real-time images as autonomous vehicles 100, 302 approach traffic lights 200a, 200b on road segment 202, and generate a real-time traffic light ROI including traffic lights 200a, 200b for each captured real-time image. Uneven or bumpy terrain on road segment 202 may cause traffic lights 200a, 200b to "jump" outside the real-time traffic light ROI. In an embodiment, autonomous vehicles 100, 302 are configured to identify the ROI scaling ratio in an HD map of the vehicle positions of autonomous vehicles 100, 302 mapped to road segment 202, and apply the ROI scaling ratio to the real-time images captured at the vehicle positions.

[0059] Autonomous vehicles 100 and 302 are configured to adjust the size of the real-time traffic light ROI according to the ROI scaling ratio. Adjusting the size of the real-time traffic light ROI in each real-time image captured by the autonomous vehicles 100 and 302 using the ROI scaling ratio ensures that traffic lights 200a and 200b remain within the real-time traffic light ROI during terrain changes on road segments 202 leading to traffic lights 200a and 200b, and enables the ADS of the autonomous vehicles 100 and 302 to take appropriate actions based on the state of traffic lights 200a and 200b on road segments 202. An onboard traffic light object detection algorithm at the autonomous vehicles 100 and 302 is used to process the traffic light ROI to detect traffic lights within the traffic light ROI.

[0060] In this embodiment, the autonomous vehicles 100, 302 are configured to identify visual feature templates in an HD map mapped to the vehicle positions of the autonomous vehicles 100, 302 on road segment 202. The autonomous vehicles 100, 302 are configured to identify real-time traffic light ROIs in real-time images captured by the vehicle sensor system 128 at the vehicle positions based on a match between the identified visual feature templates and real-time visual features associated with real-time images. An onboard traffic light object detection algorithm at the autonomous vehicles 100, 302 is used to process the traffic light ROIs to detect traffic lights within the traffic light ROIs. Although a group 302 receiving an HD map including at least one of the ROI scaling factor and visual feature templates is shown as a single autonomous vehicle 100, the group 302 may include a greater number of autonomous vehicles 100, 302.

[0061] refer to Figure 4A functional block diagram of an ECS 150 is shown, which includes an embodiment of a traffic light ROI management system 152. The ECS 150 is configured to communicatively couple to an autonomous vehicle 100, represented by group 300, to receive image sequences, and to communicatively couple to the autonomous vehicle 100, represented by group 302, to transmit an HD map including at least one of an ROI scaling ratio and a visual feature template, the ROI scaling ratio and the visual feature template being associated with different vehicle positions on road segments 202 including traffic lights 200a and 200b. The ROI scaling ratio and the visual feature template may be referred to as traffic light ROI parameters.

[0062] In one embodiment, ECS 150 includes an ECS traffic light ROI management system 152 and an ECS database 304. In another embodiment, the ECS traffic light ROI management system 152 includes one or more processors 402, a memory 404, and an ECS database 304. In another embodiment, the memory 404 includes a traffic light ROI module 406 and an ROI scaling module 408. In another embodiment, the memory 404 includes a traffic light ROI module 406 and a visual feature template module 410. In another embodiment, the memory 404 includes the traffic light ROI module 406, the ROI scaling module 408, and the visual feature template module 410. In another embodiment, the memory 404 includes a template selection module 412. ECS 150 may include other components that facilitate the operation of ECS 150.

[0063] refer to Figure 5 The diagram illustrates an embodiment of a traffic light ROI management system 152, showing an example flowchart of a method 500 for generating multiple ECS traffic light ROIs based on image sequences received from autonomous vehicles 100, 300. This method 500 can be executed via hardware circuitry, firmware, software, and / or a combination thereof.

[0064] At point 502, when the autonomous vehicles 100 and 300 approach one or more traffic lights 200a and 200b on road segment 202, a sequence of images is captured by the vehicle sensor system 128 of the autonomous vehicles 100 and 300. The images in the sequence are arranged in forward chronological order. The images are arranged according to the order in which the vehicle sensor system 128 captured them. For example, the first image in the sequence is the oldest image in the sequence, and the last image in the sequence is a newer image in the sequence.

[0065] At point 504, the autonomous vehicles 100 and 300 associate their vehicle positions with each image in the image sequence. The vehicle sensor system 128 is configured to capture the vehicle positions of the autonomous vehicles 100 and 300 as they travel on road segment 202 toward traffic lights 200a and 200b. The vehicle position at the time the vehicle sensor system 128 captures each image in the image sequence is associated with that image. At point 506, the autonomous vehicles 100 and 300 send the image sequence and the vehicle positions associated with each image to the traffic light ROI management system 152 at ECS 150.

[0066] The traffic light ROI management system 152 applies a reverse object detection and tracking algorithm to each image in the image sequence in reverse chronological order. In the reverse chronological order of the images, the first image is the newest image in the sequence, and the last image is the oldest image in the sequence. At 508, the traffic light ROI module 406 receives the images in the reverse chronologically ordered image sequence.

[0067] At 510, the traffic light ROI module 406 uses an object tracking algorithm to detect traffic lights 200a and 200b in the image. In this embodiment, each detected traffic light 200a and 200b in the image is represented as a bounding box. At 512, the traffic light ROI module 406 determines whether the size of each detected traffic light 200a and 200b represented as a bounding box in the image is greater than a traffic light size threshold.

[0068] If the traffic light ROI module 406 determines that the size of each detected traffic light 200a, 200b represented as a bounding box in the image is not greater than a traffic light size threshold, then at 514, the traffic light ROI module 406 discards the image. At 516, the traffic light ROI module 406 determines whether there are any unprocessed images in the image sequence. If the ECS traffic light ROI module 406 determines that there are still unprocessed images in the image sequence, then method 500 returns to 508, and the traffic light ROI module 406 receives the next image in the reverse time-ordered image sequence for processing. If the traffic light ROI module 406 determines that there are no unprocessed images in the image sequence, then at 518, method 500 ends.

[0069] If the traffic light ROI module 406 determines that the size of each detected traffic light 200a, 200b represented as a bounding box in the image is greater than a traffic light size threshold, then at 520, the traffic light ROI module 406 generates an ECS traffic light ROI that encloses the detected traffic lights 200a, 200b represented as bounding boxes in the image. At 522, the traffic light ROI module 406 associates the vehicle positions associated with the image with the ECS traffic light ROI. Method 500 proceeds to 516.

[0070] At 516, the traffic light ROI module 406 determines whether there are any unprocessed images in the image sequence. If the traffic light ROI module 406 determines that there are unprocessed images in the image sequence, then method 500 returns to 508, and the traffic light ROI module 406 receives the next image in the reverse time-ordered image sequence for processing. If the traffic light ROI module 406 determines that there are no unprocessed images in the image sequence, then at 518, method 500 ends.

[0071] Method 500 generates multiple traffic light ROIs as output based on image sequences received from autonomous vehicles 100 and 300. Traffic light ROI management system 152 uses the multiple traffic light ROIs to generate traffic light ROI parameters. In one embodiment, the traffic light ROI parameter is an ROI scaling ratio. In another embodiment, the traffic light ROI parameter is a visual feature template. In yet another embodiment, the traffic light ROI parameter includes both the ROI scaling ratio and the visual feature template.

[0072] refer to Figure 6a and Figure 6b A block diagram illustrating an example of the ECS traffic light ROI location in a pair of consecutive images 600a, 600b of an image sequence is shown. In this embodiment, an ROI scaling module 408 receives the image sequence and multiple ECS traffic light ROIs based on the image sequence. The ROI scaling module 408 is configured to generate an ROI scaling ratio based on the movement of the ECS traffic light ROIs in the image sequence.

[0073] The first image 600a includes a first ECS traffic light ROI 602a, and the second image 600b includes a second ECS traffic light ROI 602b. The first and second images 600a and 600b are of the same size. The first and second images 600a and 600b can be defined by an x-y coordinate system, where the x-axis extends along the width of each image in the first and second images 600a and 600b, and the y-axis extends along the height of each image in the first and second images 600a and 600b. The first ECS traffic light ROI 602a and the second ECS traffic light ROI 602b can be of different sizes.

[0074] In this example, the position of the center point 604a of the first ECS traffic light ROI 602a is determined by the coordinates (x... i ,y i The location of the center point 604b of the second ECS traffic light ROI 602b is indicated by coordinates (x...). i+1 ,y i+1 () represents the distance di. Distance di indicates the movement of the ECS traffic light ROI within a continuous image. In this example, distance di represents the distance between the center point 604a and the center point (x). i y i The position of ) to the center point 604b is in (x i+1 y i+1 The movement of the position. Distance d i This can be referred to as ROI offset or ROI movement. As shown below, the distance d... i It is defined as a function of the movement of the center points 604a and 604b of the first and second ECS traffic signal ROIs 602a and 602b in the first and second images 600a and 600b.

[0075] d i =distance((x i ,y i ),(x i+1 ,y i+1 ))

[0076] In this example, the first image 600a is at the first vehicle position p. i The image was captured at location p, and the second image 600b was captured at the second vehicle location p. i+1 Captured at location d. i With position p i Related. Distance d i＝1 With position p i＝1 Correlation. For each pair of consecutive images in the image sequence, calculate the distance d corresponding to position p. Figure 10An example of a graph is shown that can represent distance d (also known as ROI offset or movement of the ECS traffic light ROI) as a function of vehicle position p.

[0077] As a function of vehicle positions on road segment 202, the ROI offset defines the terrain structure of road segment 202. For example, as shown in the graph, a relatively large ROI offset value indicates the presence of bumpy locations in road segment 202. The ROI scaling ratio module 408 is configured to generate an ROI scaling ratio for each vehicle position associated with each image in an image sequence that includes the ECS traffic light ROI, based on the ROI offset associated with the vehicle position. The ROI scaling ratio is a traffic light ROI parameter. The ROI scaling ratio module 408 is configured to map the ROI scaling ratios associated with different vehicle positions on road segment 202 onto an HD map including road segment 202. The HD map including the mapped ROI scaling ratios is stored in the ECS database 304.

[0078] In one embodiment, the visual feature template module 410 is configured to receive the ECS traffic light ROI and the vehicle location associated with each image in the image sequence that includes the ECS traffic light ROI. The visual feature template module 410 is configured to generate a visual feature template for each ECS traffic light ROI. In another embodiment, the visual feature template module 410 is configured to generate a visual feature template for each ECS traffic light ROI in the frequency domain by applying a Fast Fourier Transform to the ECS traffic light ROI.

[0079] The visual feature template module 410 is configured to map each visual feature template to a vehicle location on a high-definition (HD) map including road segment 202. The vehicle locations on the HD map are associated with images in the image sequence that include ECS traffic light ROIs associated with the visual feature templates. The HD map is stored in the ECS database 304. The visual feature template module 410 is configured to repeat this process for each image in the image sequence received from the autonomous vehicle 100 in group 300 to define, refine, and / or update a visual feature template for each of the multiple vehicle locations on road segment 202. In an embodiment, the visual feature template module 410 is configured to generate visual feature templates only for vehicle locations associated with images in the image sequence that include traffic lights exceeding a traffic light size threshold.

[0080] In this embodiment, the template selection module 412 is configured to receive a set of visual feature templates and a sequence of images previously received from different autonomous vehicles 100, 300 traveling on road segment 202 toward traffic lights 200a, 200b. The template selection module 412 is configured to generate a reduced set of visual feature templates, which can be used by the autonomous vehicles 100, 302 to identify real-time traffic light ROIs in real-time images captured by the autonomous vehicles 100, 302 traveling on road segment 202 toward traffic lights 200a, 200b. Using the reduced set of visual feature templates reduces the amount of space required for the visual feature templates associated with road segment 202 and reduces the memory needed to store the visual feature templates.

[0081] refer to Figure 7 This diagram illustrates an example of the association between each of a plurality of visual feature templates and multiple vehicle locations on road segment 202, including traffic lights 200a and 200b. Vehicle locations are represented as circles on road segment 202. Template selection module 412 is configured to select a subset of visual feature templates generated by visual feature template module 410 for mapping onto an HD map.

[0082] In this embodiment, each visual feature template is a frequency domain representation of an ECS traffic light ROI associated with an image previously captured by the autonomous vehicles 100, 300 at a vehicle location on road segment 202. The template selection module 412 is configured to use the frequency domain representation of each image in the image sequence received from the autonomous vehicles 100, 300 on road segment 202 to predict the ECS traffic light ROI of images captured at nearby vehicle locations and identify visual feature templates that can be used at multiple vehicle locations on road segment 202.

[0083] For example, Figure 7 In the middle, visual feature template V i Visual feature template V can be used to accurately determine the real-time traffic light ROI at each vehicle location represented by black dots. j It can be used to accurately determine the real-time traffic light ROI at each vehicle location represented by gray dots, and V k It can be used to accurately determine the real-time ROI of traffic lights in each vehicle location represented by white dots.

[0084] The visual feature template set generated by the visual feature template module 410 for road segment 202 may, for example, include thirteen different visual feature templates. The template selection module 412 identifies the visual feature template V. i V j V kA reduced set of visual features is used to determine the real-time traffic light ROI in real-time images captured by future autonomous vehicles 100, 302 at potential vehicle locations represented by circles on road segment 202. Each visual feature template in the reduced set is mapped to a vehicle location on road segment 202 where the visual feature template can be used to determine the real-time traffic light ROI.

[0085] refer to Figure 8 The image shows the vehicle position 800 of the autonomous vehicle 100 and multiple visual feature templates V on a road segment 202 including traffic lights 200a and 200b. i V j V k A schematic diagram illustrating an example of associated vehicle locations. Vehicle locations are represented as circles on road segment 202. In this example, the visual feature template V... i Visual feature template V can be used to accurately determine the real-time traffic light ROI at the location of vehicles represented by black dots. j It can be used to accurately determine the real-time traffic light ROI at the vehicle location represented by gray dots, and V k This can be used to correctly identify the real-time traffic light ROI at the vehicle location, represented by white dots. The autonomous vehicle 100 is positioned at vehicle location 800, represented by a square. The HD map does not include a visual feature template associated with vehicle location 800.

[0086] The autonomous vehicle 100 is configured to run a visual feature template scaling algorithm in real time. The visual feature template scaling algorithm uses the size of the traffic light ROI (ROI) at the autonomous vehicle 100 (ROISize). vehicle And visual feature template V associated with the location of another vehicle on road segment 202 within the area. j ROISize Vj The scaling ratio of the visual feature template is calculated. This region is represented by the viewing angle 802 associated with the vehicle position 800 of the autonomous vehicle 100. Visual feature template V j In the frequency domain, the scaling ratio of the visual feature template is calculated using the following equation.

[0087]

[0088] The visual feature template scaling algorithm applies a scaling ratio to the visual feature template V. j To generate visual feature template V j A scaled-down version. Autonomous vehicles 100 use visual feature template V. jA scaled-down version that identifies real-time traffic light ROIs in real-time images captured by the autonomous vehicle 100 at vehicle location 800.

[0089] refer to Figure 9 This document illustrates an example of a method 900 for providing traffic light ROI parameters to an autonomous vehicle using an embodiment of a traffic light ROI management system 152 located at an ECS 150. Method 900 is executed by the traffic light ROI management system 152. Method 900 can be executed by the traffic light ROI management system 152 in conjunction with other components of the ECS 150. Method 900 can be executed via hardware circuitry, firmware, software, and / or a combination thereof.

[0090] At 902, the traffic light ROI management system 152 receives image sequences and vehicle locations associated with each image from the first automated driving vehicles 100 and 300. At 904, the traffic light ROI management system 152 detects at least one traffic light 200a and 200b in each image of the image sequence. At 906, the traffic light ROI management system 152 defines an ECS traffic light ROI for each image of the image sequence. The ECS traffic light ROI of each image surrounds at least one traffic light 202a and 202b in the image. At 908, the traffic light ROI management system 152 generates a visual feature template for each image of the image sequence. The visual feature template for each image is based on the ECS traffic light ROI of the image. At 910, the traffic light ROI management system 152 maps each visual feature template of each image in the image sequence to the vehicle location associated with that image on a high-resolution (HD) map. At point 912, the HD map is sent from the traffic light ROI management system 152 to the second autonomous vehicle 100, 302, enabling the second autonomous vehicle 100, 302 to identify real-time traffic light ROIs in the real-time image based on a match between a first visual feature template of the HD map and real-time visual features associated with the real-time image at the vehicle location associated with the first visual feature template. An onboard traffic light object detection algorithm at the autonomous vehicle 100 is used to process the traffic light ROI to detect traffic lights within the traffic light ROI.

[0091] Using a traffic light ROI management system 152 at the edge computing system 150 can reduce the computer resource usage at the autonomous vehicle 100. Creating intersection models and traffic light models using crowdsourced vehicle sensor data can further reduce the computer resource usage at the autonomous vehicle 100. Generating traffic light ROI parameters at the edge computing system 150 based on the crowdsourced vehicle sensor data used by the autonomous vehicle 100 in real time allows the autonomous vehicle 100 to more accurately identify real-time traffic light ROIs in real-time images and compensate for complex road terrain.

[0092] Processing full-resolution images in traffic light detection algorithms for autonomous vehicles can make the algorithm run very slowly. This problem can be solved by extracting smaller sub-images (regions of interest) that include the traffic lights from the full-resolution image containing the traffic lights before running the traffic light object detection algorithm. Extracting smaller sub-images (ROIs) before running the full, complex traffic light detection algorithm can effectively reduce the response time of real-time traffic light detection algorithms.

[0093] While at least one exemplary embodiment has been described in the foregoing detailed description, it should be understood that numerous variations exist. It should also be understood that the exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of this disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient roadmap for implementing one or more exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. An edge computing system (ECS) for providing traffic signal region of interest (ROI) parameters to autonomous vehicles, comprising: a processor; and a memory including instructions that, when executed by the processor, cause the processor to: receive, from a first autonomous vehicle, a sequence of images and a vehicle location associated with each image; detect at least one traffic signal in each image in the sequence of images; define an ECS traffic signal ROI for each image in the sequence of images, the ECS traffic signal ROI for each image encompassing the at least one traffic signal in the image; generate a visual feature template for each image in the sequence of images, the visual feature template for each image being based on the ECS traffic signal ROI for the image; map each visual feature template for each image in the sequence of images to a high definition (HD) map at the vehicle location associated with the image; and send the HD map to a second autonomous vehicle to enable the second autonomous vehicle to identify a real-time traffic signal ROI in a real-time image based on a match between a first visual feature template of the HD map and a real-time visual feature associated with the real-time image at a vehicle location associated with the first visual feature template. each image in the sequence of images is ordered in a forward temporal order, and the memory further includes instructions that, when executed by the processor, cause the processor to define the ECS traffic signal ROI in each image in the sequence of images in a reverse temporal order.

2. The system of claim 1, wherein, the memory further includes instructions that, when executed by the processor, cause the processor to discard a first image of the sequence of images upon determining that a size of the at least one detected traffic signal in the first image falls below a traffic signal size threshold.

3. The system of claim 2, wherein, the memory further includes instructions that, when executed by the processor, cause the processor to generate the visual feature template for each image in the sequence of images in a frequency domain by applying a fast Fourier transform to the ECS traffic signal ROI for the image.

4. The system of claim 1, wherein, the memory further includes instructions that, when executed by the processor, cause the processor to:

5. The system of claim 1, wherein, select a second visual feature template from a first image in the sequence of images, the first image being associated with a first vehicle location; determine whether the second visual feature template is usable for identifying a first ECS traffic signal ROI in a second image of the sequence of images based on a match between the second visual feature template and a visual feature associated with the second image, the second image being associated with a second vehicle location; and in accordance with the determination, map the second visual feature template to the first vehicle location and the second vehicle location on the HD map. the memory further includes instructions that, when executed by the processor, cause the processor to:

6. The system of claim 1, wherein, determine a ROI shift of the ECS traffic signal ROI in successive pairs of images in the sequence of images based on movement of the ECS traffic signal ROI in the successive pairs of images. ​ generating, according to a relationship between each ROI offset and a vehicle location associated with the ROI offset, a ROI scale ratio for each vehicle location associated with each image in the sequence of images; and mapping each ROI scale ratio to a vehicle location on the HD map associated with the ROI scale ratio.

7. The system of claim 6, wherein, The memory further includes instructions that, when executed by the processor, cause the processor to send the HD map including the mapped ROI scale ratios to the second autonomous vehicle to enable the second autonomous vehicle to adjust a size of a real-time traffic light ROI in a real-time image according to the ROI scale ratio associated with a vehicle location of the second autonomous vehicle.

8. A computer-readable medium including instructions stored thereon for providing traffic light region of interest (ROI) parameters to an autonomous vehicle, the instructions, when executed by a processor, cause the processor to: receive, from a first autonomous vehicle, a sequence of images and a vehicle location associated with each image; detect at least one traffic light in each image in the sequence of images; define an ECS traffic light ROI for each image in the sequence of images, the ECS traffic light ROI for each image encompassing the at least one traffic light in the image; generate a visual feature template for each image in the sequence of images, the visual feature template for each image based on the ECS traffic light ROI for the image; map each visual feature template for each image in the sequence of images to a HD map at a vehicle location associated with the image; and send the HD map to a second autonomous vehicle to enable the second autonomous vehicle to identify a real-time traffic light ROI in a real-time image according to a match between a first visual feature template of the HD map and a real-time visual feature associated with the real-time image at a vehicle location associated with the first visual feature template.

9. The computer-readable medium of claim 8, further including instructions to cause the processor to: receive the sequence of images in a forward temporal order; and define the ECS traffic light ROI in each image of the sequence of images in a reverse temporal order.

10. The computer-readable medium of claim 9, further including instructions to cause the processor to discard a first image of the sequence of images upon determining that a size of at least one detected traffic light in the first image falls below a traffic light size threshold.

Images