Method for detecting grouped traffic sign objects for crowdsourcing

Abstract

A method for detecting grouped traffic sign objects for crowdsourcing. A method for detecting grouped traffic sign objects includes receiving image data from multiple vehicles, the image data capturing multiple traffic signs, and refining the image data by filtering the image data to remove inappropriate image frames and assigning each traffic sign captured in the image data to one or more sign groups. For each corresponding sign group in the one or more sign groups, the method further includes processing the corresponding sign group to extract metadata associated with each of the traffic signs assigned to the corresponding sign group, inferring the intent of the corresponding sign group based on the metadata, inferring the context of each of the traffic signs assigned to the corresponding sign group based on the metadata, and storing the corresponding sign group in a data store.

Description

[0001] introduce

[0002] The information provided in this section is for the purpose of presenting the general context of this disclosure. The work of the currently named inventors is neither expressly nor implicitly acknowledged as prior art to this disclosure, to the extent that it is described in this section and in any other way that it may not be considered prior art at the time of filing.

[0003] This disclosure generally relates to the detection of traffic sign objects used in crowdsourcing. In particular, in the field of autonomous vehicle technology, the detection and storage of traffic objects is fundamental to ensuring safe and efficient navigation. Current systems utilize advanced object recognition algorithms to identify traffic signs and other relevant objects from sensor data, including camera, LiDAR, and radar data. Once detected, these objects are categorized and stored in a database as individual traffic objects. This information is then used by the vehicle's autonomous system to make informed decisions regarding navigation and maneuvering. By storing each detected traffic sign as an isolated entity, the system can anticipate, manage, and respond to various traffic scenarios as the vehicle approaches the traffic sign.

[0004] It is worth noting that co-located traffic objects can lead to ambiguity, such as when a co-located traffic object is applied to different types of vehicles (e.g., light motorized vehicles, trucks). Furthermore, some co-located traffic objects may be designed to support and / or enhance other co-located traffic objects (e.g., speed recommendations based on road conditions) rather than directly conflict with them. This underscores the importance of developing approaches that help resolve potential conflicts when autonomous vehicles encounter multiple potentially conflicting or complementary traffic signs simultaneously. Summary of the Invention

[0005] One aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the hardware to perform operations including receiving image data from a plurality of vehicles, the image data capturing a plurality of traffic signs, and refining the image data by filtering the image data to remove unfit image frames, and assigning each traffic sign captured in the image data to one or more sign groups. For each corresponding sign group in the one or more sign groups, the operations further include processing the corresponding sign group to extract metadata associated with each of the traffic signs assigned to the corresponding sign group, inferring the intent of the corresponding sign group based on the metadata, inferring the context of each of the traffic signs assigned to the corresponding sign group based on the metadata, and storing the corresponding sign group in a data storage.

[0006] Implementations of this disclosure may include one or more of the following optional features. In some implementations, each of the traffic signs assigned to a corresponding sign group is positioned close to each other. In some examples, the received image data is captured over two or more days. In some implementations, filtering the image data further includes identifying multiple image frames in the image data capturing duplicate traffic signs, and fusing the duplicate traffic signs captured in the identified image frames.

[0007] In some examples, metadata includes one or more of the following: semantic data, color data, location, position, size, elevation, and shape. In some implementations, data storage includes lookup tables. In some examples, processing each of the corresponding tag groups further includes identifying dependencies of the corresponding tag group based on metadata. In these examples, dependencies can include either independent or supplemental. In some implementations, the intent of the corresponding tag group includes one of the following: informational, enforcement, or warning. In some examples, the context of the corresponding tag group includes one or more of the following: vehicle type, time of day, vehicle location, and environment.

[0008] Another aspect of this disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving image data from a plurality of vehicles, the image data capturing a plurality of traffic signs, and refining the image data by: filtering the image data to remove inappropriate image frames, and assigning each traffic sign captured in the image data to one or more sign groups. For each corresponding sign group in the one or more sign groups, the operations further include processing the corresponding sign group to extract metadata associated with each of the traffic signs assigned to the corresponding sign group, inferring the intent of the corresponding sign group based on the metadata, inferring the context of each of the traffic signs assigned to the corresponding sign group based on the metadata, and storing the corresponding sign group in a data storage.

[0009] This aspect may include one or more of the following optional features. In some implementations, each of the traffic signs assigned to a corresponding sign group is positioned close to each other. In some examples, the received image data is captured over two or more days. In some implementations, filtering the image data further includes identifying multiple image frames in the image data capturing duplicate traffic signs, and fusing the duplicate traffic signs captured in the identified image frames.

[0010] In some examples, the metadata includes one or more of the following: semantic data, color data, location, position, size, altitude, and shape. In some implementations, the data storage includes lookup tables. In some examples, processing each of the corresponding flag groups further includes identifying dependencies of the corresponding flag group based on metadata. In some implementations, the intent of the corresponding flag group includes one of the following: informational, mandatory, or warning. In some examples, the context of the corresponding flag group includes one or more of the following: vehicle type, time of day, vehicle location, and environment.

[0011] Another aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the hardware to perform operations including identifying an approaching group of signs, the group comprising a plurality of traffic signs, and receiving an intent to receive the group of signs and a context of the group of signs. The operations also include receiving a vehicle context of a vehicle, using the intent and context of the group of signs to disambiguate the group of signs to identify a corresponding traffic sign among the plurality of traffic signs corresponding to the vehicle's vehicle context, and transmitting the corresponding traffic sign among the plurality of traffic signs to vehicle control of the vehicle.

[0012] A computer-implemented method executed on data processing hardware, the method causing the data processing hardware to perform operations including: receiving image data from multiple vehicles, the image data capturing multiple traffic signs; refining the image data by: filtering the image data to remove unsuitable image frames; and assigning each of the multiple traffic signs captured in the image data to one or more sign groups; and for each corresponding sign group in the one or more sign groups, processing the corresponding sign group to: extract metadata associated with each of the traffic signs assigned to the corresponding sign group from the corresponding sign group; inferring the intent of the corresponding sign group based on the metadata; inferring the context of each of the traffic signs assigned to the corresponding sign group based on the metadata; and storing the corresponding sign group in a data storage. Each of the traffic signs assigned to the corresponding sign group is located close to each other. The received image data was captured over two or more days. Filtering the image data further includes: identifying multiple image frames in the image data capturing duplicate traffic signs; and fusing duplicate traffic signs captured in the identified image frames. The metadata includes one or more of the following: semantic data; color data; location; position; size; altitude; and shape. The data storage includes a lookup table. The processing of each corresponding flag group further includes identifying the dependencies of the corresponding flag group based on metadata. Dependencies can be either independent or complementary. The intent of the corresponding flag group includes one of the following: informational; mandatory; or warning. The context of the corresponding flag group includes one or more of the following: vehicle type; time of day; vehicle location; and environment.

[0013] A system includes: data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, said operations including: receiving image data from a plurality of vehicles, the image data capturing a plurality of traffic signs; refining the image data by: filtering the image data to remove inappropriate image frames; and assigning each of the plurality of traffic signs captured in the image data to one or more sign groups; and for each corresponding sign group in the one or more sign groups, processing the corresponding sign group to: extract metadata associated with each of the traffic signs assigned to the corresponding sign group from the corresponding sign group; infer the intent of the corresponding sign group based on the metadata; infer the context of each of the traffic signs assigned to the corresponding sign group based on the metadata; and storing the corresponding sign group in a data storage. Each of the traffic signs assigned to the corresponding sign group is located close to each other. The received image data was captured over two or more days. Filtering the image data further includes: identifying a plurality of image frames in the image data capturing duplicate traffic signs; and fusing the duplicate traffic signs captured in the identified image frames. The metadata includes one or more of the following: semantic data; color data; location; position; size; altitude; and shape. The data storage includes lookup tables. Processing each of the corresponding tag groups further includes identifying dependencies of the corresponding tag group based on the metadata. The intent of the corresponding tag group includes one of the following: informational; mandatory; or warning. The context of the corresponding tag group includes one or more of the following: vehicle type; time of day; vehicle location; and environment.

[0014] A computer-implemented method executed on data processing hardware, the method causing the data processing hardware to perform operations including: identifying an approaching group of signs, the group of signs comprising a plurality of traffic signs; receiving an intent of the group of signs and a context of the group of signs; receiving a vehicle context of a vehicle; using the intent of the group of signs and the context of the group of signs to disambiguate the group of signs to identify a corresponding traffic sign among the plurality of traffic signs corresponding to the vehicle context of the vehicle; and transmitting the corresponding traffic sign among the plurality of traffic signs to a vehicle control of the vehicle.

[0015] Details of one or more implementations of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the specification, the drawings, and the claims. Attached Figure Description

[0016] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0017] Figure 1 This is a schematic diagram of an example system for detecting traffic sign objects used in crowdsourcing groups.

[0018] Figure 2 yes Figure 1 A schematic diagram of an example component of the system.

[0019] Figure 3 yes Figure 1 A schematic diagram of an example component of the system.

[0020] Figure 4 yes Figure 1 Example data storage for example systems.

[0021] Figures 5A-5C yes Figure 4 Example data storage example flag group.

[0022] Figure 6 This is a flowchart of an example operation layout for a method used to detect traffic sign objects grouped for crowdsourcing.

[0023] Figure 7 This is a flowchart of an example operation layout for a method used to detect traffic sign objects grouped for crowdsourcing.

[0024] Throughout the accompanying drawings, corresponding reference numerals indicate the relevant parts. Detailed Implementation

[0025] The example configuration will now be described more fully with reference to the accompanying drawings. The example configuration is provided so that this disclosure will be thorough and will fully communicate the scope of this disclosure to those skilled in the art. Specific details, such as examples of specific components, devices, and methods, are set forth to provide a thorough understanding of the configuration of this disclosure. It will be apparent to those skilled in the art that specific details are not required, the example configuration may be embodied in many different forms, and the specific details and example configuration should not be construed as limiting the scope of this disclosure.

[0026] The terminology used herein is for the purpose of describing specific exemplary configurations only and is not intended to be limiting. As used herein, the singular articles “a,” “an,” and “the” may also be intended to include plural forms unless the context clearly indicates otherwise. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore specify the presence of features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as requiring them to be performed in the specific order discussed or illustrated, unless specifically identified as such. Additional or alternative steps may be employed.

[0027] When an element or layer is described as being “on”, “joined to”, “connected to”, “attached to”, or “coupled to” another element or layer, it can be directly on, joined to, connected to, attached to, or coupled to another element or layer, or there may be intermediate elements or layers present. Conversely, when an element is described as being “directly on”, “directly joined to”, “directly connected to”, “directly attached to”, or “directly coupled to” another element or layer, there may be no intermediate elements or layers present. Other terms used to describe relationships between elements should be interpreted in the same way (e.g., “between” versus “directly between”, “adjacent” versus “directly adjacent”, etc.). As used herein, the term “and / or” includes any and all combinations of one or more associated listed items.

[0028] The terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers, and / or sections. These elements, components, regions, layers, and / or sections should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or section from another. Terms such as “first,” “second,” and other numerical terms do not imply order or sequence unless the context clearly indicates otherwise. Therefore, without departing from the teachings of the example configuration, the first element, component, region, layer, or section discussed below may be referred to as the second element, component, region, layer, or section.

[0029] In this application, including the following definitions, the term "module" may be replaced by the term "circuit". The term "module" may refer to, be a part of, or include the following: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor (shared, dedicated, or grouped) for executing code; memory (shared, dedicated, or grouped) for storing code executed by the processor; other suitable hardware components that provide the aforementioned functionality; or combinations of some or all of the above, such as in a system-on-a-chip.

[0030] The term "code" as used above can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared processor" includes a single processor that executes some or all of the code from multiple modules. The term "group processor" includes processors that, in combination with additional processors, execute some or all of the code from one or more modules. The term "shared memory" includes a single memory that stores some or all of the code from multiple modules. The term "group memory" includes memory that, in combination with additional memory, stores some or all of the code from one or more modules. The term "memory" can be a subset of the term "computer-readable medium." The term "computer-readable medium" does not include transient electrical and electromagnetic signals propagating through the medium, and therefore can be considered tangible and non-transitory memory. Non-limiting examples of non-transitory memory include tangible computer-readable media, including non-volatile memory, magnetic storage devices, and optical storage devices.

[0031] The apparatus and methods described in this application can be implemented, in whole or in part, by one or more computer programs executed by one or more processors. The computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. The computer program may also include and / or depend on stored data.

[0032] A software application (i.e., a software resource) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0033] Non-transitory memory can be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disks or tapes.

[0034] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0035] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, which includes at least one programmable processor, at least one input device, and at least one output device. The at least one programmable processor can be dedicated or general-purpose and is coupled to receive data and instructions from and transfer data and instructions to the storage system.

[0036] The processes and logic described in this specification can be executed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic can also be executed by a special-purpose logic circuit system, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, or operatively coupled to receive data from or transfer data to one or more mass storage devices, such as a magnetic disk, magneto-optical disk, or optical disk. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented or incorporated therein by a dedicated logic circuit system.

[0037] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, and optional keyboard and pointing devices, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including audible, voice, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0038] refer to Figure 1In some implementations, system 100 includes a vehicle 10 communicating with a remote system 60 via a network 40. The network 40 may include a wireless local area network (WLAN), which facilitates communication and interoperability between the vehicle 10 and the remote system 60 within the vehicle 10's environment. Therefore, network 40 may include wireless fidelity. (e.g., IEEE 802.11), low-rate wireless personal area networks (e.g., IEEE 802.15.4), global interoperability for microwave access (WiMAX), 3G, 4G, LTE, 5G, and digital subscriber lines (DSL). Near Field Communication (NFC) or any other wireless standard or Ethernet (e.g., IEEE 802.3). System 100 may additionally include one or more access points (APs) (not shown) configured to facilitate wireless communication between vehicle 10 and remote system 60.

[0039] As shown, vehicle 10 and / or remote system 60 execute flag grouping system 110, which executes group generator model 200. Figure 2 ) and signage model 300 ( Figure 3 The group generator model 200 is configured to receive image data 202 from multiple vehicles 10 (including vehicles 10), identifying co-located traffic signs 510. Figures 5A-5C Traffic signs 510 are grouped into sign groups 232, which are stored in a centralized data storage 250. Figure 2 It is worth noting, and as will be described in further detail below, that by classifying co-located traffic signs 510 into corresponding sign groups 232, when a vehicle 10 encounters a sign group 232 of traffic signs 510 in the future, the metadata 234 extracted from the co-located traffic signs 510 in sign group 232 can be used to meaningfully resolve ambiguities between conflicting or complementary traffic signs 510. For example, when a vehicle 10 approaches sign group 232, the sign identifier model 300 can determine, based on the metadata 234 of sign group 232, whether one or more co-located traffic signs 510 are applicable to the vehicle 10, and generate instructions for the vehicle 10 to proceed according to the applicable traffic sign 510.

[0040] In the example shown, the sign grouping system 110 is implemented within vehicle 10. However, the sign grouping system 110 can be implemented in any other propulsion system, such as, but not limited to, motorcycles, trucks, off-road vehicles, bicycles, agricultural equipment, trains, aircraft, and the like. Vehicle 10 includes data processing hardware 12 and memory hardware 14 storing instructions that, when executed on the data processing hardware 12, cause the data processing hardware 12 to perform operations. Additionally, although the sign grouping system 110 is described as being implemented by the data processing hardware 12 and memory hardware 12 of vehicle 10, it can be implemented on other computing devices (e.g., computing devices communicating with vehicle 10), such as, but not limited to, smartphones, tablets, smart displays, desktop / laptop computers, smartwatches, smart appliances, or smart glasses / headsets.

[0041] As in Figure 1 and Figure 2 As shown, vehicle 10 is configured to receive sensor data 18 detected / captured by sensor system 16. Sensor system 16 may include one or more cameras, a forward collision mitigation system, radio detection and ranging (RADAR), light detection and ranging (LiDAR) capable of capturing image data, and other external sensors of vehicle 10. Although Figure 1 The sensor system 16 shown is positioned at the front of the vehicle 10, but it should be understood that the sensor system 16 may include sensors located throughout the vehicle 10. For example, the sensor system 16 can provide 360-degree surround sensing of the environment 102 of the vehicle 10.

[0042] System 100 may further include an image subsystem 20 for extracting image data (e.g., pixels) from images of the environment 102 of the vehicle 10 to generate image data 202 of the environment 100. For example, image subsystem 20 may continuously receive sensor data 18 including images captured by sensor system 16 and convert the received sensor data 18 into image data 202 including one or more image frames 204. Here, each image frame 204 may capture one or more traffic signs 510 within the environment 102 and in the vicinity of the vehicle 10. In some instances, image subsystem 20 additionally converts the image frames 204 of image data 202 into semantic data.

[0043] The remote system 60 (e.g., a server, a cloud computing environment) also includes data processing hardware 62 and memory hardware 64 for storing instructions that, when executed on the data processing hardware 62, cause the data processing hardware 62 to perform operations. In some examples, the execution of the flag grouping system 110 is shared across the vehicle 10 and the remote system 60. See reference... Figure 1-5C In more detail, the sign grouping system 110, executed on vehicle 10 and / or remote system 60, implements group generator model 200 and sign identifier model 300. As will become clear, group generator model 200 and sign identifier model 300 cooperate to utilize crowdsourced vehicle data to detect co-located traffic signs 510, create sign groups 232 that associate co-located traffic signs 510 together, and use these sign groups to meaningfully disambiguate the co-located traffic signs 510 when vehicle 10 encounters sign groups 232 in the future. To utilize crowdsourcing, sign grouping system 110 receives image data 202 from one or more vehicles 10 in system 100 and / or one or more passes of the same vehicle 10 in system 100 to detect co-located traffic signs 510. In other words, image data 202 can be captured by one or more vehicles 10 over one or more days. Optionally, the same vehicle 10 may capture image data 202 of the same co-located sign 510 in a single day or by passing through it multiple times over several days.

[0044] Continue to refer to Figure 1 and Figure 2 The group generator model 200 includes an object detector 210, a flag processor 220, a flag grouper 230, and an inference module 240. The group generator model 200 further has access to a data storage 240 stored on memory hardware 14, 64. The data storage 250 stores flag groups 232 generated by the group generator model 200. When a vehicle 10 in system 100 reports / transmits image data 202 to the group generator model 200, the group generator model 200 processes the image data 202. Specifically, as in... Figure 2 As shown, object detector 210 receives image data 202, including one or more image frames 204 capturing traffic signs 510, as input, and performs object detection on the one or more image frames 204 to extract information and / or identify one or more traffic signs 510. For example, object detector 210 performs one or more object recognition techniques (e.g., object classification, object recognition, etc.) to identify one or more traffic signs 510 in the image frames 204 of image data 202. As noted above, object detector 210 can aggregate image data 202 because it is collected from multiple vehicles 10 over multiple days and / or multiple passes.

[0045] Subsequently, the sign processor 220 receives image data 202 including one or more image frames 204 as input and refines the image data 202 to generate refined image data 202 as output. For example, the sign processor 220 filters the image data 202 to remove unsuitable image frames 204 from the image data 202. As used herein, an unsuitable image frame 204 can generally refer to an image frame 204 in which the corresponding Global Positioning System (GPS) data indicates that the image frame 204 suffers from position errors and / or in which the image quality of the image frame 204 makes it unsuitable for further processing by the group generator model 200. The sign processor 220 can further refine the image data 202 by performing data clustering and de-duplication. Here, the sign processor 220 can identify one or more image frames 204 in the image data 202 reported by one or more vehicles 10 and perform clustering to group two or more image frames 204 of the image data 202 containing the same traffic sign 510. The sign processor 220 can then detect duplicate clusters in the grouped image data 202 and fuse the image data 202 together. In other words, the sign processor 220 can identify multiple image frames 204 in the aggregated image data 202 that capture duplicate (i.e., identical) traffic signs 510 and fuse the image data 202 for the duplicate traffic signs 510. Additionally or alternatively, the sign processor 220 identifies false detections (e.g., where traffic sign 510 is incorrectly identified as present in the image data 202) and filters out any image frames 204 of the incorrectly detected traffic sign 510.

[0046] The sign grouper 230 receives refined image data 202 from the sign processor 220 and assigns each traffic sign 510 detected in the image data 202 to one or more sign groups 232. The sign grouper 230 further processes the image data 202 to extract metadata 234 associated with each traffic sign 510. For example, the metadata 234 may include one or more of the following: semantic data of the traffic sign 510 (i.e., text and / or characters), color data of the traffic sign 510 (e.g., red, orange, yellow, black, and white), location (i.e., heading) of the traffic sign 510, position of the traffic sign 510, dimensions of the traffic sign 510 (e.g., height, width), altitude at which the traffic sign 510 is installed, and shape of the traffic sign 510. When the metadata 234 of each of the traffic signs 510 indicates that the traffic signs 510 are close to each other in location, the sign grouper 230 may assign one or more traffic signs 510 to a specific sign group 232. Additionally or alternatively, when the corresponding metadata 234 of traffic sign 510 indicates that traffic sign 510 is co-located and shares the same location and / or altitude, traffic sign 510 may be assigned to a specific sign group 232.

[0047] For each assignment of traffic sign 510, sign grouper 230 can calculate the confidence that sign 510 belongs to a specific sign group 232. For example, sign grouper 230 can identify a confidence factor based on metadata 234, which includes one or more of the following: the number of vehicles 10 that observed traffic sign 510, the number of times traffic sign 510 was observed to pass by, the number of image frames 204 from the corresponding sensor system 16 of the vehicles 10 in system 100, the amount of time between detections of traffic sign 510, the standard deviation of the position of traffic sign 510, the standard deviation of the heading of traffic sign 510, and the standard deviation of the altitude of traffic sign 510. Then, when the confidence factor exceeds a confidence threshold, sign grouper 230 can assign traffic sign 510 to a specific sign group 232, which indicates that traffic sign 510 should be grouped with other traffic signs 510 in the specific sign group 232. In some instances, sign grouper 230 can assign one or more traffic signs 510 to a specific sign group 232 based on semantic data derived from the corresponding metadata 234 of each traffic sign 510 (e.g., color, relative location, relative altitude, etc.).

[0048] In addition to grouping co-located traffic signs 510 into one or more sign groups 232, the sign grouper 230 also infers the intent 242 of a corresponding sign group 232 based on the metadata 234 of each of the traffic signs 510 in that sign group 232. The intent 242 of a sign group 232 may include one or more of the following: information (e.g., parking area, high-occupancy vehicle (HOV) lane, etc.), mandates (e.g., speed limit, disabled parking, no parking, stop sign, etc.), and warnings (e.g., slippery road, sharp bend, steep slope, etc.). In other words, the intent 242 of a sign group 232 may refer to the type of sign assigned to the sign group 232. In some cases, the sign grouper 230 may additionally infer the intent 242 of a particular sign group 232 based on guidance and traffic rules on traffic signs corresponding to a specific area in which the vehicle 10 is traveling.

[0049] Subsequently, sign grouper 230 infers the corresponding context 244 for each traffic sign 510 in the corresponding sign group 232 based on metadata 234. That is, sign grouper 230 infers the individual intent 242 of the sign group 232 as a whole, and the individual context 244 of each traffic sign 510 in the sign group 232. Context 242 may include one or more of the following: vehicle type (e.g., light motor vehicle, truck, bicycle, etc.), time of day (e.g., school hours, after 11:00 PM, etc.), vehicle location (e.g., lane position), vehicle direction, and environment (e.g., weather conditions, road conditions, etc.). In other words, context 242 may refer to what each traffic sign 510 in the corresponding sign group 232 intends to convey to the vehicle 10 and / or how to determine which traffic sign 510 in the sign group 232 is applicable to a particular vehicle 10, whether based on the time of day, the type of vehicle 10, or the specific environmental conditions present when the vehicle 10 encounters the sign group 232.

[0050] In some instances, the sign grouper 230 additionally identifies the dependency 246 of each traffic sign 510 based on the metadata 234 of each traffic sign 510 in the sign group 232. As used herein, the dependency 246 of a traffic sign 510 refers to whether a particular traffic sign 510 is treated independently of other traffic signs 510 in its corresponding sign group 232 (i.e., conflicting with other traffic signs 510) or complements (i.e., complementary to other traffic signs 510) other traffic signs 510 in its corresponding sign group 232. For example, a traffic sign 510 may be identified as having an independent dependency 246 when it is unrelated to any other traffic sign 510 in the sign group 232. Conversely, a traffic sign 510 may be identified as having a complementary dependency 246 when it enhances and / or further elaborates on another traffic sign 510 in the sign group 232. For each sign group 232, the sign grouper 230 may further associate the intent 242, the corresponding context 244 of each traffic sign 510, and, where applicable, the corresponding dependency 246 of each traffic sign 510 with the sign group 232, and store the corresponding sign group 232 in the data storage 250. In some instances, the data storage 250 includes a lookup table 400. Figure 4 It stores each flag group 232 and its corresponding intent 242, context 244 and dependency 246 as a record in lookup table 400.

[0051] refer to Figure 4-5C Example sign groups 232a-232c generated by sign grouper 230 are shown, each example sign group 232a-232c including two or more traffic signs 510a-510g. Figure 4 The corresponding lookup table 400 is also shown, which stores each record generated by the group generator model 200 for each flag group 232a-232c. See details. Figure 4 and Figure 5A The sign grouper 230 can assign co-located traffic signs 510a-510c to sign group 232a (i.e., group ID 1). Here, the sign grouper 230 can assign each of the traffic signs 510a-510c to sign group 232a based on the corresponding metadata 234a-234c of the co-located traffic signs 510a-510c. The metadata 234a-234c may include the corresponding height H of each of the traffic signs 510a-510c. 510a -H 510c The corresponding width W for each of the traffic signs 510a-510c 510a -W 510cAnd the graphic elements 512a-512c displayed on each of the traffic signs 510a-510c. The sign grouper 230 can identify that, based on metadata 234a-234c, each of the traffic signs 510a-510c has the same corresponding width W. 510a -W 510c and the same corresponding height H 510a -H 510c Based on the common dimensions identified in metadata 234a-234c, sign grouper 230 can determine that signs 510a-510c belong to the same sign group 232a. Furthermore, sign grouper 230 can identify that graphic element 512a of traffic sign 510a includes "speed limit 75", graphic element 512b of traffic sign 510b includes "truck speed 65", and graphic element 512c of traffic sign 510c includes "minimum speed 55". Based on this metadata 234a-234c, as in... Figure 4 As shown, sign grouper 230 infers that sign group 232a includes the corresponding mandatory intent 242a (i.e., speed limit). Furthermore, sign grouper 230 can store location metadata 234a and graphic elements 512a-512c of traffic signs 510a-510c in lookup table 400.

[0052] The sign grouper 230 can further determine that traffic signs 510a and 510b have independent dependencies 246a, 246b, while traffic sign 510c has a supplementary dependency 246c, which enhances / applies to sign group 232a as a whole. These dependencies 246a-246c are also stored in lookup table 400 along with the records for sign group 232a. Finally, the sign grouper 230 infers the specific context 244a-244c associated with each traffic sign 510a-510c based on metadata 234a-234c (i.e., via graphical elements 512a-512c). As shown, the traffic sign context 244a-244c is applied based on the type of vehicle 10 approaching sign group 232a. Specifically, when vehicle 10 is a light motor vehicle, context 244a associated with traffic sign 510a is applied, and it (via graphic element 512a) conveys that light motor vehicle 10 encountering sign group 232a should comply with a speed limit of 75 miles per hour (mph). Similarly, when vehicle 10 is a truck, context 244b associated with traffic sign 510b is applied, and it (via graphic element 512b) conveys that truck-type vehicle 10 encountering sign group 232b should comply with a speed limit of 65 mph. It is worth noting that because context 244c associated with traffic sign 510c is applied to all vehicles 10 (i.e., with supplementary dependency 246c), it (via graphic element 512c) conveys that all vehicles 10 encountering sign group 232a should comply with a minimum speed limit of 55 mph, regardless of their type.

[0053] Now for reference Figure 4 and Figure 5B The sign grouper 230 can assign co-located traffic signs 510d and 510e to sign group 232b (i.e., group ID 2). Here, the sign grouper 230 can assign each of the traffic signs 510d and 510e to sign group 232b based on the corresponding metadata 234d and 244e of the co-located traffic signs 510d and 510e. The metadata 234d and 234e may include the corresponding height H of each of the traffic signs 510d and 510e. 510d H 510e The corresponding width W for each of traffic signs 510d and 510e 510d W 510e And the graphic elements 512d and 512e displayed on each of the traffic signs 510d and 510e. When the sign grouper 230 can identify that, based on metadata 234d and 234e, each of the traffic signs 510d and 510e has a different shape, and based on different widths W... 510d W510e and different heights H 510d H 510e When traffic signs 510d and 510e have different sizes, sign grouper 230 can further identify that they are installed on the same post and assign them to the same sign group 232b accordingly. Furthermore, sign grouper 230 can identify that graphic element 512d of traffic sign 510d includes a right-curved arrow, while graphic element 512e of traffic sign 510e includes "40mph". Based on this metadata 234d and 234e, as in... Figure 4 As shown, sign grouper 230 infers that sign group 232b includes the corresponding warning intent 242b (i.e., slow down on the right curve). Furthermore, sign grouper 230 can store location metadata 234d and graphic elements 512d, 512e of traffic signs 510d, 510e in lookup table 400.

[0054] The sign grouper 230 can further determine that traffic sign 510d has an independent dependency 246d, while traffic sign 510e has a supplementary dependency 246e intended to complement traffic sign 510d. For example, the sign grouper 230 interprets the semantic metadata 234e of traffic sign 510d and determines that traffic sign 510d is providing additional guidance to traffic sign 510e. Here, the sign grouper 230 can identify that traffic signs 510d and 510e share the same color scheme (e.g., black text / symbols on a yellow background), indicating that both traffic signs 510d and 510e indicate a warning and should be interpreted together. These dependencies 246d and 246e are also stored in lookup table 400 along with the records of sign group 232b. Finally, the sign grouper 230 infers the specific context 244d and 244e associated with each traffic sign 510d and 510e based on the metadata 234d and 234e (i.e., via graphic elements 512d and 512e). As shown, traffic sign contexts 244d and 244e are applied based on the position of the vehicle 10 approaching sign group 232b. Specifically, when the vehicle 10 is in the exit ramp lane, contexts 244d and 244e associated with traffic signs 510d and 510e are applied, and (via graphic elements 512d and 512e) any vehicle 10 encountering sign group 232b is warned to reduce its speed to the recommended 40 mph while driving along the right-hand exit ramp.

[0055] Now for reference Figure 4 and 5CThe sign grouper 230 can assign co-located traffic signs 510f and 510g to sign group 232c (i.e., group ID 3). Here, the sign grouper 230 can assign each of the traffic signs 510f and 510g to sign group 232c based on the corresponding metadata 234f and 244g of the co-located traffic signs 510f and 510g. The metadata 234f and 234g may include graphic elements 512f and 512g displayed on each of the traffic signs 510f and 510g. When processing the metadata 234f and 234g of the traffic signs 510f and 510g, the sign grouper 230 identifies that the graphic element 512f of the traffic sign 510f includes "School speed limit 20 when flashing", and the graphic element 512g of the traffic sign 510g includes "Speed ​​limit 35". Thereafter, as in Figure 4 As shown, the sign grouper 230 can perform semantic interpretation and infer that the sign group 232c includes the corresponding mandatory intent 242c (i.e., speed limit). Furthermore, the sign grouper 230 can store location metadata 234c and graphic elements 512f, 512g of traffic signs 510g, 510f in a lookup table 400.

[0056] The sign grouper 230 can further determine that traffic signs 510f and 510g have independent dependencies 246f and 246g, such that traffic signs 510f and 510g are not applicable simultaneously. These dependencies 246f and 246g are also stored in lookup table 400 along with the records of sign group 232c. Finally, the sign grouper 230 infers the specific context 244f and 244g associated with each traffic sign 510f and 510g based on the metadata 234f and 234g (i.e., via graphical elements 512f and 512g). As shown, the traffic sign context 244f and 244g are applied based on the time of day when vehicle 10 is approaching sign group 232c. Specifically, when vehicle 10 approaches sign group 232c during school hours (e.g., when traffic sign 510f includes flashing beacons from 7:00–8:20 am and 3:00–4:20 pm), context 244f associated with traffic sign 510f is applied, and it (via graphic element 512f) conveys that the approaching vehicle 10 should comply with a speed limit of 20 mph. Similarly, when vehicle 10 approaches sign group 232c outside of school hours and / or when the beacons are not flashing, context 244g associated with traffic sign 510g is applied, and it (via graphic element 512g) conveys that the approaching vehicle 10 may comply with a speed limit of 35 mph.

[0057] Now for reference Figure 1 and Figure 3Furthermore, as noted above, the sign grouping system 110 further includes a sign identifier 300 that communicates with a data storage 250 that stores records of sign groups 232 and their corresponding associated metadata 234, intent 242, context 244, and dependencies 246. The sign identifier 300 may continuously receive image data 202 capturing the environment 102 of the vehicle 10, as well as the vehicle context 24 of the vehicle 10. For example, the vehicle context 24 may include one or more of the following: the type of vehicle 10 (i.e., light motor vehicle, truck, etc.), the speed of the vehicle 10, the trajectory of the vehicle 10, the location of the vehicle 10, the time of day, the weather conditions of the environment 102 of the vehicle 10, the distance between the vehicle 10 and a specific sign group 232, the altitude of the vehicle, the heading of the vehicle 10, and the lane (i.e., location) of the vehicle 10. As vehicle 10 drives along the road, it receives image data 202, and vehicle context 24 identifies whether vehicle 10 is approaching a sign group 232 of co-located sign 510. Here, sign identifier model 300 can determine, based on vehicle context 24, whether the approaching sign group 232 will be applied to vehicle 10, and whether the sign group 232 includes multiple traffic signs 510 that may be ambiguous.

[0058] The sign identifier model 300 can query the data storage 250 to determine if it already contains the sign group 232 of the approaching sign. When the data storage 250 contains the corresponding information for the approaching sign group 232, it returns the metadata 234, intent 242, context 244, and dependencies 246 of the sign group 232 to the sign identifier model 300. The sign identifier model 300 then uses the metadata 234, including the intent 242 and context 244 of the sign group 232, to resolve ambiguities in the sign group 232 and identify which of the multiple traffic signs 510 in the sign group 232 corresponds to the vehicle context 24 of the vehicle 10. Thereafter, the sign identifier can transmit the corresponding traffic sign 510 corresponding to the vehicle context 24 to the vehicle controls of the vehicle 10 (e.g., steering, speed, braking).

[0059] In an implementation where data storage 250 does not include records of approaching sign groups 232 identified by vehicle 10, sign grouping system 110 can generate on-the-fly records of approaching sign groups 232. For example, sign grouping system 110 can infer / extrapolate the intent 242 of approaching sign groups 232 based on similar sign groups 232 stored in data storage 250. Sign grouping system 110 can then flag on-the-fly records of approaching sign groups 232 to request verification of intent 242 of sign group 232 when the number of observations (e.g., via other vehicles 10 and / or multiple passes) reaches a verification threshold. Once the verification threshold is met, on-the-fly records of sign groups 232 can be added to lookup table 400 stored in data storage 250 for further passage of vehicle 10. Optionally, on-the-fly records of sign groups 232 can be manually reviewed before being added to lookup table 400 to confirm compliance with traffic control policies.

[0060] Figure 6 This includes a flowchart of an example operation layout for a method 600 for detecting traffic sign objects 510 used for crowdsourcing groups. See also... Figure 1-5C To describe method 600. Data processing hardware (e.g., Figure 1 Data processing hardware 12, 62) can execute data stored in memory hardware (e.g., Figure 1 Instructions on the memory hardware (14, 64) are used to perform an example operational arrangement for method 600. At operation 602, method 600 includes receiving image data 202 from a plurality of vehicles 10, the image data 202 capturing a plurality of traffic signs 510. At operation 604, method 600 further includes refining the image data 202 to remove unsuitable image frames 204 from the image data 202, and at operation 606, assigning each of the plurality of traffic signs 510 captured in the image data 202 to one or more sign groups 232.

[0061] For each corresponding sign group 232 in one or more sign groups 232, method 600 further includes processing the corresponding sign group 232 for operations 608-614. Specifically, at operation 608, method 600 includes processing the corresponding sign group 232 to extract metadata 234 associated with each of the traffic signs 510 assigned to the corresponding sign group 232. Method 600 further includes, at operation 610, processing the corresponding sign group 232 to infer the intent 242 of the corresponding sign group 232 based on the metadata 234. At operation 612, method 600 further includes processing the corresponding sign group 232 to infer the context 244 of each of the traffic signs 510 assigned to the corresponding sign group 232 based on the metadata 232. At operation 614, method 600 further includes processing the corresponding sign group 232 to store the corresponding sign group 232 in data storage 250.

[0062] Figure 7 This includes a flowchart of an example operation layout for a method 700 for detecting traffic sign objects 510 used for crowdsourcing groups. See also... Figure 1-5C To describe method 700. Data processing hardware (e.g., Figure 1 Data processing hardware 12, 62) can execute data stored in memory hardware (e.g., Figure 1 Instructions on the memory hardware (14, 64) are executed to perform an example operational arrangement for method 700. At operation 702, method 700 includes identifying an approaching group of signs 232. Here, group of signs 232 includes a plurality of traffic signs 510. Method 700 also includes, at operation 704, receiving an intent 242 of group of signs 232 and a context 244 of group of signs 232.

[0063] At operation 706, method 700 further includes receiving the vehicle context 24 of the vehicle 10. Method 700 also includes, at operation 708, using the intent 242 of the sign group 232 and the context 244 of the sign group 232 to disambiguate the sign group 232, thereby identifying the corresponding traffic sign 510 among the plurality of traffic signs 510 corresponding to the vehicle context 24 of the vehicle 10. Method 700 further includes, at operation 710, transmitting the corresponding traffic sign 510 among the plurality of traffic signs 510 to the vehicle control of the vehicle 10.

[0064] Several implementations have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are within the scope of the following claims.

[0065] The foregoing description has been provided for illustrative and descriptive purposes. It is not intended to be exhaustive or limiting of this disclosure. Individual elements or features of a particular configuration are generally not limited to that particular configuration, but where applicable, they are interchangeable and can be used in the chosen configuration, even if not specifically shown or described. This can also be varied in many ways. Such variations should not be considered as departing from this disclosure, and all such modifications are intended to be included within the scope of this disclosure.

Claims

1. A computer-implemented method executed on data processing hardware, the method causing the data processing hardware to perform operations, the operations including: Image data is received from multiple vehicles, and the image data captures multiple traffic signs; The image data is refined using the following methods: Filter image data to remove inappropriate image frames; and Each traffic sign among multiple traffic signs captured in the image data will be assigned to one or more sign groups; and For each corresponding flag group in one or more flag groups, process the corresponding flag group as follows: Extract the associated metadata of each traffic sign in the corresponding sign group and assign it to the corresponding sign group; Inferring the intent of the corresponding tag group based on metadata; The context assigned to each traffic sign in the corresponding sign group is inferred based on metadata; and Store the corresponding flag group in the data store.

2. The method according to claim 1, wherein, Each traffic sign assigned to a corresponding sign group is positioned close to the others.

3. The method according to claim 1, wherein, The received image data was captured over two or more days.

4. The method according to claim 1, wherein, Filtered image data further includes: Identify multiple image frames in image data that capture repeating traffic signs; and Repeating traffic signs captured in the identified image frames.

5. The method according to claim 1, wherein, Metadata includes one or more of the following: Semantic data; Color data; position; Location; size; Altitude; and shape.

6. The method according to claim 1, wherein, Data storage includes lookup tables.

7. The method according to claim 1, wherein, Processing each of the corresponding flag groups further includes identifying the dependencies of the corresponding flag group based on metadata.

8. The method according to claim 7, wherein, Dependency can be either independent or complementary.

9. The method according to claim 1, wherein, The intent of the corresponding group of symbols includes one of the following: information; Forced; or warn.

10. The method according to claim 1, wherein, The context of the corresponding flag group includes one or more of the following: Types of transportation; Time of day; Location of vehicles; and environment.

Images