Traffic sign quality evaluation method

Abstract

A method includes receiving image data captured by a traffic feature detection system. The image data represents traffic characteristics within the environment. The method includes identifying a type of traffic feature. The method includes isolating a portion of the image data containing a traffic feature. Based on a portion of the image data containing a traffic feature, the method includes determining a pixel color value of the traffic feature. Based on the type of traffic feature, the method includes determining an expected pixel color value for the traffic feature. Based on a comparison of the pixel color value determined for the traffic feature and an expected pixel color value, the method includes determining a state of health of the traffic feature.

Description

Technical Field

[0001] The information provided in this section is for the purpose of presenting the general context of this disclosure. The work of the currently named inventors, to the extent described in this section, and in aspects that may not qualify as prior art at the time of filing, is neither expressly nor implicitly acknowledged as prior art to this disclosure.

[0002] This disclosure generally relates to systems and methods for determining the health status or quality assessment of traffic features. More specifically, this disclosure relates to determining the health status of traffic features based on image data captured by a traffic feature detection system for vehicles, and processing the captured image data to compare the color and contrast of the traffic features with expected values. Background Technology

[0003] Over time, traffic features such as stop signs, speed limit signs, and street signs become damaged or degraded. Traditionally, municipal and road authorities responsible for repairing or replacing traffic features rely on manual inspections or individual complaints to identify features requiring repair or replacement. Therefore, problems with traffic features often go unresolved until their degraded quality substantially impacts road safety. Summary of the Invention

[0004] One aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations. The operations include receiving image data captured by a traffic feature detection system. The image data represents traffic features within an environment. The operations include identifying the type of traffic feature. The operations include isolating a portion of the image data containing the traffic feature. Based on the portion of image data containing the traffic feature, the operations include determining pixel color values ​​for the traffic feature. Based on the type of traffic feature, the operations include determining expected pixel color values ​​for the traffic feature. Based on a comparison of the determined pixel color values ​​for the traffic feature with the expected pixel color values, the operations include determining a health status for the traffic feature.

[0005] Implementations of this disclosure may include one or more of the following optional features. In some implementations, determining the pixel color value of a traffic feature includes determining one or more clusters of pixel color values ​​via k-means clustering. Alternatively, determining the pixel color value of a traffic feature includes determining the pixel color value of the traffic feature based on the average of the principal clusters in one or more clusters of pixel color values.

[0006] In some examples, the comparison between the expected pixel color value and the determined pixel color value is based on the Euclidean distance between them. The expected pixel color value and the determined pixel color value are represented by one selected from a group consisting of (i) RGB coordinate values, (ii) HSV coordinate values, (iii) HSL coordinate values, and (iv) YUV coordinate values. In some aspects, the health status of traffic features is also based on the contrast of the determined traffic features.

[0007] In some implementations, the operation further includes isolating a second portion of the image data surrounding the portion of the image data containing traffic features. Furthermore, the operation includes determining pixel color values ​​of the environment based on the second portion of the image data. In a further implementation, the health status of the traffic features is also based on a comparison of the determined pixel color values ​​of the traffic features with the determined pixel color values ​​of the environment.

[0008] In some examples, image data is aggregated from a traffic feature detection system deployed at multiple vehicles. In some aspects, the comparison of expected pixel color values ​​with determined pixel color values ​​is based on the level of ambient light present in the image data. In some implementations, the operation also includes adjusting the operation of the traffic feature detection system based on the determined health status of the traffic features. Furthermore, the operation may also include generating an alert to repair the traffic features based on a degradation indication of the determined health status of the traffic features.

[0009] Another aspect of this disclosure provides a system. The system includes memory hardware storing instructions that, when executed on data processing hardware in communication with the memory hardware, cause the data processing hardware to perform operations. The operations include receiving image data captured by a traffic feature detection system. The image data represents traffic features within an environment. The operations include identifying the type of traffic feature. The operations include isolating a portion of the image data containing the traffic feature. Based on the portion of the image data containing the traffic feature, the operations include determining pixel color values ​​for the traffic feature. Based on the type of traffic feature, the operations include determining expected pixel color values ​​for the traffic feature. Based on a comparison of the determined pixel color values ​​for the traffic feature with the expected pixel color values, the operations include determining a health status for the traffic feature. This aspect may include one or more of the following optional features.

[0010] In some implementations, determining the pixel color value of a traffic feature includes determining one or more clusters of pixel color values ​​via k-means clustering. Alternatively, determining the pixel color value of a traffic feature includes determining the pixel color value of the traffic feature based on the average of the principal clusters among one or more clusters of pixel color values.

[0011] In some examples, the comparison between the expected pixel color value and the determined pixel color value is based on the Euclidean distance between them. The expected pixel color value and the determined pixel color value are represented by one selected from a group consisting of (i) RGB coordinate values, (ii) HSV coordinate values, (iii) HSL coordinate values, and (iv) YUV coordinate values. In some aspects, the health status of traffic features is also based on the contrast of the determined traffic features.

[0012] In some aspects, the operation further includes isolating a second portion of the image data, the second portion surrounding the portion of the image data containing the traffic feature. Furthermore, the operation includes determining pixel color values ​​of the environment based on the second portion of the image data. Determining the health status of the traffic feature is also based on a comparison of the determined pixel color values ​​of the traffic feature with the determined pixel color values ​​of the environment. In some embodiments, image data is aggregated from a traffic feature detection system located at multiple vehicles.

[0013] Another aspect of this disclosure provides a vehicle. The vehicle includes memory hardware storing instructions that, when executed on data processing hardware in communication with the memory hardware, cause the data processing hardware to perform operations. The operations include receiving image data captured by a traffic feature detection system. The image data represents traffic features within an environment. The operations include identifying the type of traffic feature. The operations include isolating a portion of the image data containing the traffic feature. Based on the portion of the image data containing the traffic feature, the operations include determining pixel color values ​​for the traffic feature. Based on the type of traffic feature, the operations include determining expected pixel color values ​​for the traffic feature. Based on a comparison of the expected pixel color values ​​with pixel color values ​​determined for the traffic feature, the operations include determining a health status for the traffic feature. This aspect may include one or more of the following optional features.

[0014] In some implementations, determining the pixel color value of a traffic feature includes determining one or more clusters of pixel color values ​​via k-means clustering. Alternatively, determining the pixel color value of a traffic feature includes determining the pixel color value of the traffic feature based on the average of the principal clusters among one or more clusters of pixel color values.

[0015] In some examples, the comparison between the expected pixel color value and the determined pixel color value is based on the Euclidean distance between them. The expected pixel color value and the determined pixel color value are represented by one selected from a group consisting of (i) RGB coordinate values, (ii) HSV coordinate values, (iii) HSL coordinate values, and (iv) YUV coordinate values. In some aspects, the health status of traffic features is also based on the contrast of the determined traffic features.

[0016] In some aspects, the operation further includes isolating a second portion of the image data, the second portion surrounding the portion of the image data containing the traffic feature. Furthermore, the operation includes determining pixel color values ​​of the environment based on the second portion of the image data. Determining the health status of the traffic feature is also based on a comparison of the determined pixel color values ​​of the traffic feature with the determined pixel color values ​​of the environment. In some embodiments, image data is aggregated from a traffic feature detection system located at multiple vehicles.

[0017] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the specification, drawings, and claims. Attached Figure Description

[0018] The accompanying drawings described herein are for illustrative purposes only for the selected configurations and are not intended to limit the scope of this disclosure.

[0019] Figure 1 It is an environmental view of vehicles traveling along the road.

[0020] Figure 2 This is a system diagram of the vehicle traffic feature detection system and the back-end system that communicates with the traffic feature detection system.

[0021] Figure 3 This is a flowchart of an example method for determining the health status of traffic features in a vehicle environment.

[0022] Figure 4 It is a schematic diagram of image data being processed to determine the health status of traffic features represented by image data.

[0023] Figure 5 It is a graph depicting sample data generated by a traffic feature detection system to determine health status.

[0024] Figures 6 to 8 It is a graph depicting the Euclidean distance and contrast value of a stop sign as its color decays over time.

[0025] Figure 9 It is a chart that compares the environmental contrast of the determined traffic characteristics with the health status values ​​of the determined traffic characteristics.

[0026] Figure 10 It is a chart that compares the environmental contrast of the determined traffic features with the contrast of the determined traffic features.

[0027] Figure 11 It is a chart that compares the environmental contrast of the determined traffic characteristics with the health status values ​​of the determined traffic characteristics.

[0028] Figure 12 It is a chart that compares the determined chromaticity values ​​for a traffic feature with the expected federally recommended ideal chromaticity values ​​for that type of traffic feature.

[0029] In all the accompanying drawings, the corresponding reference numerals denote the corresponding parts. Detailed Implementation

[0030] The example configuration will now be described more fully with reference to the accompanying drawings. The example configuration is provided so that this disclosure will be thorough and will fully communicate the scope of this disclosure to those skilled in the art. Specific details, such as examples of specific components, apparatus, and methods, are set forth to provide a thorough understanding of the configuration of this disclosure. It will be apparent to those skilled in the art that specific details are not required, the example configuration may be embodied in many different forms, and the specific details and example configuration should not be construed as limiting the scope of this disclosure.

[0031] The terminology used herein is for the purpose of describing a particular exemplary configuration only and is not intended to be limiting. As used herein, the singular articles “a,” “an,” and “the” may also be intended to include plural forms unless the context clearly indicates otherwise. The terms “comprises,” “comprising,” “including,” and “having” are inclusive and therefore specify the presence of features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein should not be construed as requiring them to be performed in the specific order discussed or shown, unless specifically identified as such. Additional or alternative steps may be employed.

[0032] When an element or layer is referred to as “on another element or layer,” “joined to,” “connected to,” “attached to,” or “linked to” another element or layer, it may be directly on, joined to, attached to, or linked to the other element or layer, or there may be intermediate elements or layers present. Conversely, when an element is referred to as “directly on another element or layer,” “directly joined to,” “directly connected to,” “directly attached to,” or “directly linked to” another element or layer, there may be no intermediate elements or layers present. Other terms used to describe relationships between elements should be interpreted in a similar manner (e.g., “between” vs. “directly between,” “adjacent” vs. “directly adjacent,” etc.). As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0033] The terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers, and / or parts. These elements, components, regions, layers, and / or parts should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or part from another. Unless the context clearly indicates otherwise, terms such as “first,” “second,” and other numerical terms do not imply order or sequence. Therefore, without departing from the teachings of the example configuration, the first element, component, region, layer, or part discussed below may be referred to as the second element, component, region, layer, or part.

[0034] In this application, including the following definitions, the term "module" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include the following: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor (shared, dedicated, or grouped) for executing code; memory (shared, dedicated, or grouped) for storing code executed by the processor; other suitable hardware components that provide the described functionality; or some or all of the above, such as in a system-on-a-chip.

[0035] The term "code" as used above can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared processor" includes a single processor that executes some or all of the code from multiple modules. The term "group processor" includes processors that, in combination with additional processors, execute some or all of the code from one or more modules. The term "shared memory" covers a single memory that stores some or all of the code from multiple modules. The term "group memory" includes memory that, in combination with additional memory, stores some or all of the code from one or more modules. The term "memory" can be a subset of the term "computer-readable medium." The term "computer-readable medium" does not include transient electrical and electromagnetic signals propagating through the medium and can therefore be considered tangible and non-transitory memory. Non-limiting examples of non-transitory memory include tangible computer-readable media, which include non-volatile memory, magnetic memory, and optical memory.

[0036] The apparatus and methods described in this application can be implemented, partially or entirely, by one or more computer programs executed by one or more processors. The computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. The computer program may also include and / or depend on stored data.

[0037] A software application (i.e., a software resource) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "program," or "application." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0038] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0039] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0040] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0041] The processes and logic flows described in this specification can be executed by one or more programmable processors (also known as data processing hardware) that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). As an example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0042] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) for displaying information to the user and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0043] Referring now to the accompanying drawings and the configuration shown therein, vehicle 100 is equipped with a traffic feature detection system 102 for identifying traffic features 12 in the environment 10 of vehicle 10. Figure 1 and Figure 2For example, the traffic feature detection system 102 is enabled by the electronic control unit (ECU) or control module 104 of the vehicle 100, which has data processing hardware that executes instructions stored on the memory hardware 106 of the vehicle 100. As the vehicle 100 travels along road 14 of environment 10, the traffic feature detection system 102 processes image data captured by camera 108 of the vehicle 100 (e.g., a camera mounted on the windshield viewing from in front of the vehicle 100) to detect and identify traffic features 12 within environment 10, such as stop signs, speed limit signs, yield signs, traffic lights, pedestrian crossings, lane merging and lane markings, and railway crossings. Based on the identification of traffic features 12, the traffic feature detection system 102 can send notifications 110 to the driver of the vehicle 100, such as messages generated on the vehicle 100's display screen, and / or adjust the operation of the advanced driver assistance system (ADAS) 112 of the vehicle 100, since the ADAS at least partially controls the operation of the vehicle 100 along road 14.

[0044] As discussed below, image data captured by camera 108 of vehicle 100 and representing traffic features 12 in environment 10 can be processed for determining the health status or quality assessment 400 of traffic features 12. Although the determination of the health status 400 of traffic features 12 is discussed at a back-end system 200 communicating with telematics system 114 of vehicle 100, it should be understood that at least part of the methods and techniques described herein can be performed at vehicle 100. Furthermore, the health status 400 of traffic features 12 can be determined based on a single image data frame, a series of image data frames captured during a driving session of vehicle 100 (e.g., video images captured by camera 108), and / or an aggregation or collection of image data collected by cameras at multiple vehicles (e.g., a convoy).

[0045] Regarding Figure 3 300 Methods for Discussion Figure 1 , Figure 2 and Figure 4 The configuration shown. Figure 3 A flowchart is provided showing an exemplary arrangement of the operation of a method 300 for determining the health status 400 of a traffic feature 12 based on image data 206 received from a traffic feature detection system 102 of at least one vehicle 100 communicating with a backend system 200. For example, the backend system 200 may include a remote server or cloud computing system 202 that executes instructions stored on a memory storage device 204.

[0046] At operation 302 of method 300, backend system 200 receives training image data 208 representing one or more traffic features 12 within environment 10. For example, training data 208 may include the LISA traffic sign dataset or other image data captured during testing. Training data 208 may be stored in memory 204. At operation 304 of method 300, a training engine or traffic sign detection training model 210 for the backend system, such as You Only LookOnce version 4 (YOLOv4) or YOLOv7, is trained based on the training data 208. In some examples, backend system 200 trains the traffic sign detection training model 210 at least in part based on image data 206 captured by vehicle 100 while vehicle 100 is traveling within environment 10.

[0047] At operation 306 of method 300, backend system 200 receives captured image data 206 representing traffic features 12 in environment 10. For example... Figure 4 As shown, at operation 308 of method 300, the backend system 200 detects traffic features 12 in image data 206 based on a trained traffic sign detection training model 210. For example, the traffic sign detection model 210 is configured to recognize traffic features 12, including stop signs, speed limit signs, yield signs, traffic lights, pedestrian crossings, lane merging and lane lines, and railway crossings. The traffic sign detection model 210 is capable of recognizing traffic features 12 in both color and grayscale image data 206.

[0048] Based on the detected traffic feature 12 in image data 206, backend system 200 can crop or isolate a portion 206a of image data 206, wherein the cropped portion 206a represents traffic feature 12. In other words, at operation 310 of method 300, backend system 200 isolates portion 206a of image data 206 containing traffic feature 12. As discussed further below, backend system 200 can also isolate or crop a second portion 206b of image data 206, which represents the environment 10 in image data 206 immediately surrounding or behind traffic feature 12.

[0049] As discussed further below, the quality determination module 212 of the backend system identifies the type or characterization 12a of traffic feature 12, and based on type 12a, the quality determination module 212 determines one or more qualities 400a-d of traffic feature 12 representing a health state 400 of traffic feature 12. The health state 400 can then be associated with image data 206 and the associated traffic feature 12 for further determination. The data aggregation module 214 of the backend system 200 can receive the identified and / or cropped image data 206 for sorting the image data 206 based on the identified traffic feature 12. For example, semantic data 216 associated with image data 206 (such as the physical distance 12b of traffic feature 12 from vehicle 100 in image data 206, a unique identifier associated with traffic feature 12, a timestamp 12d when image data 206 was captured, and GPS coordinates of vehicle 100, etc.) can be applied by the aggregation module 214 to feature map 218 and associated with the identification of image data 206 and traffic feature 12 for storage in traffic feature database 220. Therefore, image data 206 representing the same traffic feature 12 at different instances, collected by cameras at multiple vehicle locations, can be aggregated or categorized together and stored in the traffic feature database 220 of the memory storage device 204. In other words, image data 206 collected at different instances can be associated with the same traffic feature 12, such that the aggregation or collection of image data 206 can be used to determine the health status 400 of traffic feature 12.

[0050] At operation 312 of method 300, quality determination module 212 determines the dominant color or pixel color value 402 in image data 206 and associated with traffic feature 12. For example, quality determination module 212 may first filter the captured image data 206 based on criteria such as the maximum distance 12b of traffic feature 12 from vehicle 100 when the image data is captured, the minimum size of traffic feature 12 in image data 206, and the confidence level 12c associated with identifying traffic feature 12. When image data 206 meets the threshold criteria, quality determination module 212 may determine only the health status 400 of traffic feature 12. The value or color of each pixel or portion of image data 206 may then be clustered, such as using K-means clustering, to generate a histogram 406 representing K-clusters 404, and the pixel color value 402 of traffic feature 12 may be determined based on the dominant K-cluster 404. In other words, the pixel value, color, or hue 402 of traffic feature 12 is determined by determining the pixel value through one or more K-clusters 404 via K-means clustering and determining the pixel color value 402 of traffic feature 12 based on the mean or median of the primary or dominant K-clusters 404. The pixel color value 402 can be expressed as red, green, and blue (RGB) coordinate values ​​408, expressed as a grayscale pixel value, or using any suitable color coordinate system.

[0051] For example, when traffic feature 12 is a stop sign, the pixel values ​​of image data 206 can typically be clustered into a red K-cluster 404 representing the red portion of traffic feature 12 in image data 206 and a white K-cluster 404 representing the white portion of traffic feature 12 in image data 206. Other K-clusters 404 may represent a portion of the environment 10 within the cropped portion 206a of image data 206, foreign objects or substances at traffic feature 12 (e.g., decals or paint), etc. In the example of a stop sign, the pixel color value 402 of traffic feature 12 can be determined as the average of the red K-cluster values ​​404.

[0052] At operation 314 of method 300, quality determination module 212 determines the health status 400 of traffic feature 12 based on the determined pixel color value 402 for traffic feature 12. For example, based on type 12a of traffic feature 12, quality determination module 212 determines the expected pixel color value 410 of traffic feature 12. In the example of a stop sign, the expected pixel color value 410 could be a red hue defined by the Federal Highway Administration (FHWA). For example, the expected pixel color value 410 is derived from the ideal chromaticity values ​​recommended by FHWA based on type 12a of traffic feature 12.

[0053] Based on a comparison of the expected pixel color value 410 and the determined pixel color value 402 of traffic feature 12, the quality determination module 212 determines the health state 400 of traffic feature 12. Since the expected pixel color value 410 and the determined pixel color value 402 can be represented as RGB coordinates 408, the comparison between the expected pixel color value 410 and the determined pixel color value 402 can be based on the Euclidean distance 400, 400a between their RGB coordinates 408. That is, the health state 400 can be at least partially based on the Euclidean distance 400a between the determined pixel color value 402 and the expected pixel color value 410 of traffic feature 12. The larger the Euclidean distance 400a, the more likely traffic feature 12 is to degrade from its original state.

[0054] In some examples, the Euclidean distance 400a can be based on the expected pixel color value 410 and the determined pixel color value 402 in any suitable coordinate system, such as hue, saturation, and value (HSV) coordinates, hue, saturation, and lightness (HSL) coordinates, and lightness, blue projection, and red projection (YUV) coordinates. Therefore, the Euclidean distance 400a (ED) can be represented in RGB coordinates as follows:

[0055]

[0056] Where LM represents the pixel value 402 of the determined traffic feature 12, and I represents the expected pixel color value 410 of the traffic feature 12.

[0057] Similarly, the Euclidean distance 400a can be represented in HSV / HSL coordinates as follows:

[0058]

[0059] The Euclidean distance 400a can be expressed in YUV coordinates by the following formula:

[0060]

[0061] Therefore, the Euclidean distance 400a can compare the current color scale 402 of traffic feature 12 with the recommended or expected color scale 410 of type 12a of traffic feature 12 to measure how the color 402 of traffic feature 12a deteriorates from its initial or ideal value 410.

[0062] Furthermore, the health status 400 of traffic feature 12 can be based on the determined contrast 400, 400b of traffic feature 12. Contrast 400b represents the ratio of the difference in intensity / brightness between two or more target regions of traffic feature 12. In the example of a stop sign, contrast 400b represents the difference between red K-cluster 404 representing the red portion of traffic feature 12 in image data 206 and white K-cluster 404 representing the white portion of traffic feature 12 in image data 206, and thus measures the visibility of one color compared to another. Contrast 400b (μ) can be expressed by the following formula:

[0063]

[0064] Where, μ l Indicates the first K-cluster 404 (e.g., white), μ p This represents the second K-cluster 404 (e.g., red). Therefore, the health status 400 considers contrast 400b as a representation of the visibility of traffic feature 12 to the driver. The higher the contrast 400b, the more visible traffic feature 12 is to the driver.

[0065] like Figure 3 As shown, the determined Euclidean distance 400a and the determined contrast 400b for traffic feature 12 can be combined to determine a uniform health quality 412 for traffic feature 12. In some examples, weights are assigned to the Euclidean distance 400a and the contrast 400b to determine an average weighted score for the uniform health quality 412. For example, the two measures can be normalized to a common range, such as [0,1] or other suitable ranges. That is, the uniform health quality 412 can be based on the Euclidean distance 400a and the contrast 400b as values ​​between zero and one (e.g., ...). Figure 9 ) or along any suitable scale (e.g., Figure 10 and Figure 11 To determine.

[0066] Furthermore, when assigning weights to the Euclidean distance 400a and contrast 400b, the perceived brightness or level of ambient light present in image data 206 can be considered. That is, at least a portion of image data 206 can be converted to the L*a*b* color space, where L* represents perceived brightness, a* represents the green-red contrasting color, and b* represents the blue-yellow contrasting color. L* can define black as 0.0 and white as 1.0. In some examples, an entire frame of image data 206 is converted to the L*a*b* color space to determine the perceived brightness of image data 206. Alternatively, only a portion of image data 206 (such as the pixel color value 402 of the determined traffic feature 12, or alternatively, the corresponding pixel color value of any portion of image data 206, such as a portion representing the environment) is converted to the L*a*b* color space to determine the perceived brightness. When image data 206 is captured in a sunny environment, the perceived brightness of image data 206 can be greater than 0.5, and the Euclidean distance 400a can be given more weight as a more important factor in determining the uniform health quality 412, because image capture is considered to occur under more ideal conditions and is less likely to be affected by weather conditions. When image data 206 is captured in a dark or cloudy environment, the perceived brightness of image data 206 can be less than 0.5, and the contrast ratio 400b can be given more weight as a more important factor, because the color of traffic feature 12 may appear darker and the contrast ratio 400b is less affected by dark conditions. Therefore, the uniform health state 412(H) can be expressed as:

[0067]

[0068] Where α represents the weight applied to a contrast ratio of 400b, CR represents the contrast ratio of 400b, and β represents the weight applied to a Euclidean distance of 400a. When the perceived brightness is less than or equal to 0.5, α is greater than β. When the perceived brightness is greater than 0.5, α is less than β. The sum of α and β equals 1.

[0069] In some examples, the health status 400 of traffic feature 12 can also be based on environmental contrast 400, 400c, which represents the visibility of traffic feature 12 compared to the background or environment surrounding traffic feature 12. Therefore, the quality determination module 212 can isolate a second portion 206b of image data 206 surrounding the first portion 206a, where the second portion 206b represents the environment 10 at or near traffic feature 12. The quality determination module 212 determines, for example, a pixel color value 414 representing the environment 10 in the second portion 206b via K-means clustering, and determines the environmental contrast 400c based on the determined pixel color value 402 of traffic feature 12 and the determined pixel color value 414 of environment 10. Figure 4 As shown, the second part 206b may only represent the environment 10 immediately surrounding the traffic feature 12 to provide a more accurate representation of visibility to the driver viewing the traffic feature 12.

[0070] Furthermore, the quality determination module 212 can determine the chromaticity 400, 400d associated with traffic feature 12, such as to confirm or verify the health status 400 determined for traffic feature 12. For example, the quality determination module 212 can convert the pixel color value 402 determined for traffic feature 12 into the corresponding chromaticity value 400d, and draw the chromaticity along the chromaticity map 416 to compare the chromaticity value 400d for traffic feature 12 with the expected value 410 of the chromaticity for type 12a of traffic feature 12. Figure 12 The expected value 410 of traffic feature 12 can be based on guidance from FHWA.

[0071] refer to Figure 2 The determined health status 400 and / or uniform health quality 412 for traffic feature 12 can be stored in traffic feature database 220. Each unique traffic feature 12 can have one or more determined health status values ​​400 assigned to it. Vehicle 100 can communicate with traffic feature database 220, such as via vehicle 100's telematics system 114, to adjust the operation of traffic feature detection system 102 based on the determined health status 400 of traffic feature 12.

[0072] For example, based on the location of vehicle 100 determined by navigation or GPS module 116 of vehicle 100, traffic feature detection system 102 can predict one or more traffic features 12 along the route of vehicle 100. Navigation module 116 can adjust the route of vehicle 100, such as to avoid traffic features 12 with a determined health state 400 below a threshold level. Based on the health state 400 of traffic feature 12, system 102 can generate notification 110 to the driver, such as to warn the driver of the degraded traffic feature indicating health state 400, and therefore traffic feature 12 may be invisible to the driver. For example, based on the determined health state 400 of traffic feature 12, traffic feature detection system 102 can adjust its operation, such as to highlight traffic features 12 with poor health state 400 in a head-up display or windshield display. Furthermore, ADAS 112 can adjust its operation, such as using health state 400 as a weighting factor when controlling vehicle 100 using image data captured by camera 108 or a map from navigation module 116.

[0073] Furthermore, the traffic feature database 220 can communicate with the event processing module 222 of the backend system 200, which can identify and cluster traffic features 12 with health status 400 indicating deterioration. That is, the event processing module 222 can determine traffic features 12 with poor health status 400, and classify or group them based on factors such as the location of the traffic feature 12, the type 12a of the traffic feature 12, and / or the municipality responsible for the traffic feature 12. Based on the determined health status 400 indication of deterioration for one or more traffic features 12, the publisher module 224 of the backend system 200 can generate or publish alerts or reports to repair the traffic feature 12. For example, the publisher module 224 can populate a publicly available database so that the municipality 20 responsible for the traffic feature 12 can be informed of the health status 400.

[0074] For example, based on a uniform health status 412 (i.e., Euclidean distance 400a and contrast 400b) indicating poor quality of traffic feature 12, system 200 can recommend repairing or replacing traffic feature 12. Based on an environmental contrast 400c indicating that traffic feature 12 blends into its surrounding environment 10 or is invisible compared to its surrounding environment 10, system 200 can recommend adjustments to traffic feature 12 or environment 10, such as a recommendation to prune trees behind traffic feature 12 or to adjust the height of traffic feature 12 to make it more visible. The chromaticity 400d of traffic feature 12 can be analyzed to determine whether traffic feature 12 complies with the FHWA guidelines for type 12a of traffic feature 12.

[0075] refer to Figure 5-8 The determined health status values ​​400, associated with traffic feature 12 and stored in traffic feature database 220, were plotted to illustrate the impact of changes in color and contrast on the visibility of traffic feature 12 for the driver and traffic feature detection system 102. Figure 6 As shown in Figure 600, as the Euclidean distance 400a of the determined traffic feature 12 increases over time (i.e., as the color of the traffic feature travels further from its original color), the contrast 400b of the traffic feature 12 decreases. In other words, as the fading of the traffic feature 12 increases, the Euclidean distance 400a increases and the contrast 400b decreases, indicating a deterioration in the health status 400 of the traffic feature 12. That is, as the traffic feature 12 fades or changes color, the traffic feature 12 becomes less noticeable and more difficult to identify. In the example of the stop sign, and as... Figure 7 As shown in Figure 700, when traffic feature 12 has essentially not faded, the determined contrast 400b between the red and white portions of traffic feature 12 is highest, thus the stop sign retains a higher percentage of its original red 410. Similarly, and as... Figure 8 As shown in Figure 800, when traffic feature 12 has essentially not faded, the Euclidean distance 400a between the determined pixel color value 402 and the expected pixel color value 410 of traffic feature 12 can be minimized, and therefore the stop sign retains a higher percentage of its original red 410. Therefore, the Euclidean distance 400a and contrast 400b are strongly correlated with the good or bad health 400 of traffic feature 12.

[0076] Figure 9 Figure 900 shows experimental data that compares the determined environmental contrast 400c of traffic feature 12 with the uniform health quality 412 of traffic feature 12. As shown, the traffic sign detection system 102 can detect traffic features 12 with low scores on the uniform health quality 412 (i.e., traffic feature 12 may be damaged or deteriorated and therefore difficult to identify) and / or low scores on the environmental contrast 400c (i.e., traffic feature 12 may appear to blend into the surrounding environment 10 or not stand out compared to the surrounding environment 10). Furthermore, Figure 10 Figure 1000 illustrates experimental data that compares a determined environmental contrast 400c with a determined contrast 400b of traffic feature 12, where human perception values ​​representing the visibility of traffic feature 12 to a human driver can be used to validate the data. As shown, traffic feature 12 that meets the minimum contrast 400b and has a high environmental contrast value 400c appears most visible. These are rated highest by human perception values, thus validating the determined contrast 400b as an effective method for determining the health status 400 of traffic feature 12. Similarly, Figure 11 Figure 1100 shows a comparison of environmental contrast 400c with uniform health quality 412.

[0077] Therefore, the traffic feature detection system 102, which collaborates with the backend system 200, is configured to use a deep learning model 210 to detect traffic features 12 (such as stop signs, speed limit signs, yield signs, etc.) and measure one or more health status values ​​400 associated with the traffic feature 12 (such as Euclidean distance 400a, contrast 400b, ambient contrast 400c, and / or chromaticity 400d). This provides an automated, quantitative health quality assessment for traffic features 12 that lack human perception. By tracking the health status 400 in the traffic sign database 220, the system 200 can track the current health quality 400 and the rate of deterioration of the traffic feature 12. This allows for prediction of when the traffic feature 12 will need to be replaced, or provides recommendations on when the traffic feature 12 is ready to be replaced. Therefore, the system 200 ensures the timely replacement of traffic features 12, reduces monitoring costs for the municipal authorities responsible for traffic features 12, and improves road safety.

[0078] In some examples, the traffic feature detection system 102 and the backend system 200 can be configured to identify other roadside features, such as storefront signs and billboard advertisements. Thus, the backend system 200 determines a health status value 400 for these other roadside features, which can help business entities determine whether to repair or replace their storefront signs or develop different, more prominent billboard advertisements.

[0079] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

[0080] The foregoing description is provided for illustrative and descriptive purposes. It is not intended to be exhaustive or limiting of this disclosure. Elements or features of a particular configuration are generally not limited to that particular configuration, but are interchangeable where applicable and can be used in selected configurations, even if not specifically shown or described. They can also be varied in many ways. Such variations should not be considered as departing from this disclosure, and all such modifications are intended to be included within the scope of this disclosure.

Claims

1. A computer-implemented method, when executed on data processing hardware, causing the data processing hardware to perform operations, the operations including: Receive image data captured by a traffic feature detection system, the image data representing traffic features in the environment; Identify the type of the traffic feature; Isolate a portion of the image data containing the traffic features; Based on the portion of the image data containing the traffic feature, the pixel color value of the traffic feature is determined; Based on the type of the traffic feature, determine the expected pixel color value of the traffic feature; as well as The health status of the traffic feature is determined by comparing the determined pixel color value with the expected pixel color value.

2. The method according to claim 1, wherein, Determining the pixel color values ​​of the traffic feature includes: One or more clusters are used to determine pixel color values ​​via k-means clustering; and The pixel color value of the traffic feature is determined based on the average value of the main clusters in the one or more pixel color value clusters.

3. The method according to claim 1, wherein, The comparison between the expected pixel color value and the determined pixel color value is based on the Euclidean distance between the expected pixel color value and the determined pixel color value, which are represented by one of the group consisting of (i) RGB coordinate values, (ii) HSV coordinate values, (iii) HSL coordinate values ​​and (iv) YUV coordinate values.

4. The method according to claim 1, wherein, The health status of the traffic feature is also based on the determined contrast of the traffic feature.

5. The method according to claim 1, wherein, The operation also includes: A second portion of the image data is isolated, the second portion surrounding the portion of the image data containing the traffic features; and The second part of the image data determines the pixel color values ​​of the environment.

6. The method according to claim 5, wherein, The health status of the traffic feature is also based on a comparison between the determined pixel color value of the traffic feature and the determined pixel color value of the environment.

7. The method according to claim 1, wherein, The image data is aggregated from a traffic feature detection system installed at multiple vehicles.

8. The method according to claim 1, wherein, The comparison between the expected pixel color value and the determined pixel color value is based on the level of ambient light present in the image data.

9. The method according to claim 1, wherein, The operation also includes adjusting the operation of the traffic feature detection system based on the determined health status of the traffic feature.

10. The method according to claim 1, wherein, The operation also includes generating an alert to repair the traffic feature based on the determined indication of health deterioration of the traffic feature.

Images