METHOD FOR EVALUATING THE QUALITY OF TRAFFIC SIGNS

Abstract

A computer-implemented procedure that, when executed in data processing hardware, causes the data processing hardware to perform operations that include: Receiving image data (206) acquired by a traffic facility detection system (102), wherein the image data (206) represent a traffic facility (12) in an environment (10); Identifying the type of traffic facility (12); Isolating a part of the image data (206a) that contains the traffic control device (12); Determining a pixel color value for the traffic control device (12) based on the part of the image data (206a) that contains the traffic control device (12); Determining an expected pixel color value for the traffic control device (12) based on the type of traffic control device; Isolating a second part of the image data (206b) that surrounds the part of the image data (206a) that contains the traffic control device (12); Determining a pixel color value for the environment (10) based on the second part of the image data (206b); and Determining a health status (400) for the traffic facility (12) based on a comparison of the expected pixel color value and the determined pixel color value for the traffic facility (12); wherein the health status (400) for the traffic facility (12) is further based on a comparison of the determined pixel color value for the traffic facility (12) and the determined pixel color value for the environment (10).

Description

INTRODUCTION

[0001] The information in this section serves to present the general context of the disclosure. Works of the inventors mentioned herein, insofar as they are described in this section, as well as aspects of the description that may not have been prior art at the time of filing, are neither expressly nor implicitly admitted as prior art against the present disclosure.

[0002] This disclosure relates generally to systems and methods for determining the health status or quality assessment of traffic control devices. In particular, this disclosure relates to determining the health status of traffic control devices based on image data acquired by a vehicle's traffic control device detection system and to processing the acquired image data to compare the colors and contrast ratios of the traffic control device with expected values.

[0003] Over time, traffic control devices such as stop signs, speed limit signs, road signs, and the like become damaged or destroyed. Traditionally, municipalities and road authorities responsible for repairing or replacing traffic control devices rely on manual inspections or individual complaints to identify devices that need repair or replacement. Consequently, problems with traffic control devices are often not addressed until their deterioration significantly impacts road safety.

[0004] DE 102013216892 A1 discloses a method and a system for determining the location of an object, in particular a moving vehicle such as a railway track inspection train. DE 102019116962 A1 discloses a system in which an infrastructure node receives data from a vehicle to determine one or more physical properties of an infrastructure element, such as a pothole in a road. DE 102022112256 A1 describes a system for classifying a road that predicts a future functional condition of the road based on sensor data from a vehicle and using a deterioration model. SUMMARY

[0005] One aspect of the disclosure provides for a computer-implemented method which, when executed on data processing hardware, causes the data processing hardware to perform operations. These operations include receiving image data acquired by a traffic control device detection system. The image data is representative of a traffic control device in an environment. The operations include identifying the type of traffic control device. The operations include isolating a portion of the image data containing the traffic control device. The operations include determining a pixel color value for the traffic control device based on the portion of the image data containing the traffic control device. The operations include determining an expected pixel color value for the traffic control device based on the type of traffic control device.The operations also include isolating a second part of the image data that surrounds the part of the image data containing the traffic control device. Furthermore, the operations include determining a pixel color value for the environment based on this second part of the image data. The operations include determining the health status of the traffic control device based on a comparison of the expected pixel color value and the determined pixel color value for the traffic control device. The health status of the traffic control device is further based on a comparison of the determined pixel color value for the traffic control device and the determined pixel color value for the environment.

[0006] Implementations of the disclosure may include one or more of the following optional features. In some implementations, determining the pixel color value for the traffic control device involves determining one or more clusters of pixel color values ​​by k-means clustering. Determining the pixel color value for the traffic control device also involves determining the pixel color value for the traffic control device based on an average value from a primary cluster of the one or more clusters of pixel color values.

[0007] In some examples, the comparison of the expected pixel color value and the determined pixel color value is based on the Euclidean distance between the expected and determined pixel color values. The expected and determined pixel color values ​​are represented by values ​​selected from the group consisting of (i) RGB coordinate values, (ii) HSV coordinate values, (iii) HSL coordinate values, and (iv) YUV coordinate values. In some aspects, the health of the traffic control device also depends on a specific contrast ratio of the traffic control device.

[0008] In some examples, image data from traffic control device detection systems mounted on numerous vehicles is aggregated. In some aspects, the comparison between the expected pixel color value and the detected pixel color value is based on the level of ambient light present in the image data. In some implementations, the operations further include adjusting the operation of the traffic control device detection system based on the detected health status of the traffic control device. Additionally, the operations may also include generating a warning to repair the traffic control device based on the detected health status indicating deterioration.

[0009] Another aspect of the revelation envisages a system. The system includes memory hardware that stores instructions which, when executed on the data processing hardware in communication with the memory hardware, cause the data processing hardware to perform operations. These operations include receiving image data captured by a traffic control device detection system. The image data is representative of a traffic control device in an environment. The operations include identifying the type of traffic control device. The operations include isolating a portion of the image data containing the traffic control device. The operations include determining a pixel color value for the traffic control device based on the portion of the image data containing the traffic control device. The operations include determining an expected pixel color value for the traffic control device based on the type of traffic control device.The operations involve determining the health status of the traffic control device based on a comparison of the expected pixel color value and the determined pixel color value. This aspect may include one or more of the following optional features.

[0010] In some implementations, determining the pixel color value for the traffic control device involves determining one or more clusters of pixel color values ​​using k-means clustering. Furthermore, determining the pixel color value for the traffic control device also includes determining the pixel color value based on the average value of a primary cluster of the one or more clusters of pixel color values.

[0011] In some examples, the comparison of the expected pixel color value and the determined pixel color value is based on the Euclidean distance between the expected and determined pixel color values. The expected and determined pixel color values ​​are represented by values ​​selected from the group consisting of (i) RGB coordinate values, (ii) HSV coordinate values, (iii) HSL coordinate values, and (iv) YUV coordinate values. In some aspects, the health status of the traffic control device is also based on a specific contrast ratio of the traffic control device.

[0012] In some aspects, the operations also include isolating a second part of the image data that surrounds the portion containing the traffic control device. Furthermore, the operations include determining a pixel color value for the environment based on this second part of the image data. Determining the health status of the traffic control device is also based on comparing the determined pixel color value for the traffic control device with the determined pixel color value for the environment. In some implementations, image data from traffic control device detection systems mounted on a large number of vehicles are aggregated.

[0013] Another aspect of the disclosure involves a vehicle. The vehicle includes storage hardware that holds instructions which, when executed on data processing hardware communicating with the storage hardware, cause the data processing hardware to perform operations. These operations include receiving image data captured by a traffic control device detection system. The image data is representative of a traffic control device in an environment. The operations include identifying the type of traffic control device. The operations include isolating a portion of the image data containing the traffic control device. The operations include determining a pixel color value for the traffic control device based on the portion of the image data containing the traffic control device.The operations include determining an expected pixel color value for the traffic control device based on the device type. The operations also include determining the health status of the traffic control device based on a comparison of the expected pixel color value and the actual pixel color value. This aspect may include one or more of the following optional features.

[0014] In some implementations, determining the pixel color value for the traffic control device involves determining one or more clusters of pixel color values ​​using k-means clustering. Determining the pixel color value for the traffic control device also involves determining the pixel color value based on the average value of a primary cluster of the one or more clusters of pixel color values.

[0015] In some examples, the comparison of the expected pixel color value and the determined pixel color value is based on the Euclidean distance between the expected and determined pixel color values. The expected and determined pixel color values ​​are represented by values ​​selected from the group consisting of (i) RGB coordinate values, (ii) HSV coordinate values, (iii) HSL coordinate values, and (iv) YUV coordinate values. In some aspects, the health status of the traffic control device is also based on a specific contrast ratio of the traffic control device.

[0016] In some aspects, the operations also include isolating a second part of the image data that surrounds the part of the image data containing the traffic control device. Furthermore, the operations include determining a pixel color value for the environment based on this second part of the image data. Determining the health status of the traffic control device is also based on comparing the determined pixel color value for the traffic control device with the determined pixel color value for the environment. In some implementations, the image data from traffic control device detection systems mounted on a large number of vehicles are aggregated.

[0017] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and in the description below. Further aspects, features, and advantages will become apparent from the description and the drawings, as well as from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described here serve only to illustrate selected configurations and are not intended to limit the scope of this disclosure. Fig. 1 is a view of the surroundings of a vehicle driving on a road. Fig. Figure 2 is a system diagram of a vehicle traffic control device detection system and a backend system that communicates with the traffic control device detection system. Fig. Figure 3 is a flowchart of an example of a procedure for determining the health status of a traffic facility in the vicinity of the vehicle. Fig. Figure 4 is a schematic representation of the processing of image data to determine the health status of the traffic facility represented by the image data. Fig. Figure 5 is a diagram that shows an example of data generated by the traffic facility data collection system for determining health status. Fig. Figures 6-8 are diagrams showing the determined values ​​for the Euclidean distance and the contrast ratio for a stop sign, while the colors of the stop sign fade over time. Fig. Figure 9 is a diagram in which determined environmental contrast ratios for traffic facilities are compared with determined health status values ​​for the traffic facilities. Fig. Figure 10 is a diagram comparing determined environmental contrast ratios for traffic facilities with determined contrast ratios for the traffic facilities. Fig. Figure 11 is a diagram in which determined environmental contrast ratios for traffic facilities are compared with determined health status values ​​for the traffic facilities. Fig. Figure 12 is a diagram that compares determined chromaticity values ​​for traffic facilities with an expected, federally recommended ideal chromaticity value for the type of traffic facility.

[0019] The corresponding reference numbers denote the corresponding parts in the drawings. DETAILED DESCRIPTION

[0020] Exemplary configurations are now described in more detail with reference to the accompanying drawings. Exemplary configurations are provided so that this disclosure is thorough and conveys the full scope of the disclosure to those skilled in the art. Specific details are listed, such as examples of specific components, devices, and processes, to provide a thorough understanding of the configurations of this disclosure. It is clear to those skilled in the art that specific details need not be used, that exemplary configurations can be implemented in many different forms, and that the specific details and exemplary configurations should not be interpreted in such a way as to limit the scope of the disclosure.

[0021] The terminology used here serves only to describe certain exemplary configurations and is not intended to be restrictive. As used here, the singular articles "a," "an," and "the" can also include the plural forms unless the context clearly indicates otherwise. The terms "comprises," "comprehensive," "containing," and "exhibiting" are inclusive and therefore specify the presence of features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The procedural steps, processes, and operations described here are not to be interpreted as necessarily being carried out in the order discussed or presented, unless they are explicitly identified as such.Additional or alternative steps can be applied.

[0022] When an element or layer is described as "on," "engaging," "connected," "attached to," or "coupled" with another element or layer, it may be directly on, engaged, connected, attached, or coupled to that other element or layer, or there may be intervening elements or layers. Conversely, when an element is described as "directly on," "directly engaged with," "directly connected to," "directly attached to," or "directly coupled to" another element or layer, there must be no intervening elements or layers. Other words used to describe the relationship between elements should be interpreted similarly (e.g., "between" versus "directly between," "next to" versus "directly beside," etc.).As used here, the term “and / or” includes all combinations of one or more of the related listed elements.

[0023] The terms "first," "second," "third," etc., may be used here to describe different elements, components, areas, layers, and / or sections. These elements, components, areas, layers, and / or sections should not be restricted by these terms. These terms may only be used to distinguish one element, component, area, layer, or section from another. Terms such as "first," "second," and other numerical terms do not imply any sequence or order unless clearly indicated by the context. Thus, a first element, component, area, layer, or section discussed below could be referred to as a second element, component, area, layer, or section without deviating from the lessons of the exemplary configurations.

[0024] In this application, including the definitions below, the term "module" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include: an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; memory (shared, dedicated, or group) that stores the code executed by a processor; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, e.g., in a system-on-a-chip.

[0025] The term "code," as used above, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared processor" refers to a single processor that executes some or all of the code from multiple modules. The term "group processor" refers to a processor that, in combination with other processors, executes some or all of the code from one or more modules. The term "shared memory" refers to a single memory that stores some or all of the code from multiple modules. The term "group memory" refers to a memory that, in combination with other memories, stores some or all of the code from one or more modules. The term "memory" can be a subset of the term "computer-readable medium."The term "computer-readable medium" excludes transitory electrical and electromagnetic signals propagating through a medium and can therefore be considered tangible and non-transient storage. Non-restrictive examples of non-transient storage include tangible, computer-readable media, including non-volatile memory, magnetic storage, and optical storage.

[0026] The devices and methods described in this application can be implemented in whole or in part by one or more computer programs executed by one or more processors. The computer programs contain processor-executable instructions stored on at least one non-transitory, tangible, computer-readable medium. The computer programs may also contain and / or access stored data.

[0027] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform a task. In some examples, a software application may be called an "application," "app," or "program." Examples of applications include system diagnostics applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0028] Non-transitory memory can be physical devices used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program status information) for use by a computer. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and floppy disks or tapes.

[0029] These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a procedural and / or object-oriented high-level language and / or in assembly / machine language. The terms "machine-readable medium" and "computer-readable medium" used here refer to any computer program product, non-transient computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) that serves to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.The term "machine-readable signal" refers to any signal that serves to provide machine instructions and / or data to a programmable processor.

[0030] Various implementations of the systems and techniques described herein can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system comprising at least one programmable processor, which can be used for special or general purposes and is coupled such that it receives data and instructions from and transmits data and instructions to a storage system, as well as at least one input device and at least one output device.

[0031] The processes and logical sequences described in this description can be executed by one or more programmable processors, also known as data processing hardware, which run one or more computer programs to perform functions by responding to input data and producing outputs. The processes and logical sequences can also be executed by specialized logic circuits, such as an FPGA (Field Programmable Gate Array) or an ASIC (application-specific integrated circuit). Processors suitable for executing a computer program include, for example, both general-purpose and specialized microprocessors, as well as one or more processors from any type of digital computer. Generally, a processor receives instructions and data from read-only memory, random-access memory, or both.The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is operationally connected to them to receive data from or transmit data to them. However, a computer does not necessarily have to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable media; magneto-optical disks; and CD-ROM and DVD-ROM disks.The processor and memory can be supplemented by special logic circuits or integrated into them.

[0032] To enable interaction with a user, one or more aspects of the revelation can be implemented on a computer that has a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display), or touchscreen monitor to show information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, with which the user can input information into the computer. Other types of devices can also be used for user interaction; feedback to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and user input can be received in any form, including auditory, verbal, or tactile input.Furthermore, a computer can interact with a user by sending and receiving documents to and from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0033] A vehicle 100 is equipped with a traffic control device detection system 102, with which traffic control devices 12 in a vicinity 10 of the vehicle 100 can be detected ( Fig. 1 and Fig. 2) For example, the traffic equipment detection system 102 is activated by an electronic control unit (ECU) or control module 104 of the vehicle 100, which has data processing hardware that executes instructions stored in the memory hardware 106 of the vehicle 100. As the vehicle 100 travels along a road 14 of the environment 10, the traffic equipment detection system 102 processes image data captured by a camera 108 of the vehicle 100 (e.g., a windshield-mounted camera facing forward in front of the vehicle 100) to detect and identify the traffic equipment 12 within the environment 10, such as stop signs, speed limit signs, yield signs, traffic lights, pedestrian crossings, lane merges and lane markings, pedestrian crossings, railroad crossings, and the like.Based on the identification of the traffic equipment 12, the traffic equipment detection system 102 can transmit a notification 110 to a driver of the vehicle 100, such as a message generated on a screen of the vehicle 100, and / or adjust the operation of an advanced driver assistance system (ADAS) 112 of the vehicle 100, since the ADAS at least partially controls the operation of the vehicle 100 along the road 14.

[0034] As explained below, the image data captured by the camera 108 of the vehicle 100, depicting the traffic control devices 12 in the environment 10, can be processed to determine a health status or quality rating 400 of the traffic control device 12. Although the determination of the health status 400 of the traffic control device 12 is described in a backend system 200 in communication with a telematics system 114 of the vehicle 100, it should be clear that at least parts of the procedures and techniques described here can be carried out in the vehicle 100. Furthermore, the health status 400 of the traffic control device 12 can be determined based on a single frame of image data, a series of frames of image data captured during a journey of the vehicle 100 (e.g.,video images taken by camera 108), and / or a collection or set of image data collected by cameras on a large number of vehicles (e.g., a fleet of vehicles).

[0035] The in the Fig. 1, Fig. 2 and Fig. The 4 configurations shown are used in connection with procedure 300 of the Fig. 3 discussed. Fig. Figure 3 shows a flowchart of an exemplary sequence of operations for a procedure 300 for determining the health status 400 of the traffic control device 12 based on image data 206 received by the traffic control device detection system 102 of at least one vehicle 100 in communication with the backend system 200. The backend system 200 may, for example, comprise a remote server or a cloud computing system 202 that executes instructions stored in memory 204.

[0036] In operation 302 of procedure 300, the backend system 200 receives training image data 208 depicting one or more traffic control devices 12 within the environment 10. The training data 208 includes, for example, the LISA traffic sign dataset or other image data acquired during the test procedures. The training data 208 can be stored in a memory 204. In operation 304 of procedure 300, a training engine or traffic sign recognition training model 210 of the backend system, such as You Only Look Once Version 4 (YOLOv4) or YOLOv7, is trained based on the training data 208. In some examples, the backend system 200 trains the traffic sign recognition training model 210 at least partially based on the image data 206 acquired by the vehicle 100 while driving in the environment 10.

[0037] In operation 306 of procedure 300, the backend system 200 receives the captured image data 206, which are representative of a traffic facility 12 in the vicinity 10. As in Fig. As shown in Figure 4, the backend system 200, in operation 308 of procedure 300, detects the traffic control device 12 in the image data 206 based on the trained traffic sign recognition training model 210. The traffic sign recognition model 210 is configured, for example, to detect traffic control devices 12 such as stop signs, speed limit signs, priority signs, traffic lights, pedestrian crossings, lane merges and lane markings, pedestrian crossings, level crossings, and similar features. The traffic sign recognition model 210 is capable of detecting the traffic control device 12 in the color and grayscale image data 206.

[0038] Based on the detection of the traffic control device 12 in the image data 206, the backend system 200 can crop or isolate a portion 206a of the image data 206, where the cropped portion 206a is representative of the traffic control device 12. In other words, in operation 310 of procedure 300, the backend system 200 isolates the portion 206a of the image data 206 that contains the traffic control device 12. As explained further below, the backend system 200 can also isolate or crop a second portion 206b of the image data 206, which represents the environment 10 immediately around or behind the traffic control device 12 in the image data 206.

[0039] As explained below, a quality determination module 212 of the backend system identifies a type or characterization 12a of the traffic equipment 12, and based on the type 12a, the quality determination module 212 determines one or more qualities 400a-d of the traffic equipment 12 that are representative of the health status 400 of the traffic equipment 12. The health status 400 can then be linked to the image data 206 and the associated traffic equipment 12 for further determinations. A data aggregation module 214 of the backend system 200 can receive the identified and / or cropped image data 206 in order to sort the image data 206 based on the identified traffic equipment 12.For example, semantic data 216, linked to the image data 206, such as a physical distance 12b of the traffic control device 12 from the vehicle 100 in the image data 206, a unique identifier associated with the traffic control device 12, GPS coordinates of the vehicle 100, and a timestamp 12d indicating when the image data 206 was acquired, and similar information, can be applied to a traffic control device map 218 and linked to the image data 206 and the identity of the traffic control device 12 by the aggregation module 214 for storage in a traffic control device database 220. Accordingly, image data 206, captured by cameras on a large number of vehicles and representative of the same traffic control device 12 at different times, can be aggregated or sorted and stored in the traffic control device database 220 of the storage device 204.In other words, image data 206 collected at different instances can be associated with the same traffic facility 12, so that the aggregation or collection of image data 206 can be used to determine the health status 400 for the traffic facility 12.

[0040] In operation 312 of procedure 300, the quality determination module 212 determines a dominant color or pixel color value 402 in the image data 206 that is associated with the traffic control device 12. For example, the quality determination module 212 may first filter the acquired image data 206 based on criteria such as a maximum distance 12b of the traffic control device 12 from the vehicle 100 when the image data was acquired, a minimum size of the traffic control device 12 in the image data 206, a confidence level 12c associated with the identification of the traffic control device 12, and the like. The quality determination module 212 may only determine the health status 400 of the traffic control device 12 if the image data 206 meets the threshold criteria. The values ​​or colors for each pixel or part of the image data 206 may then be clustered, e.g.Using K-means clustering to generate a histogram 406 representing the K-clusters 404, the pixel color value 402 for the traffic control device 12 can be determined based on the most dominant K-cluster 404. That is, the pixel value, color, or hue 402 for the traffic control device 12 is determined by identifying one or more K-clusters 404 of pixel values ​​using K-means clustering and determining the pixel color value 402 for the traffic control device 12 based on an average or median value of a primary or dominant K-cluster 404. The pixel color value 402 can be expressed as a red, green, and blue (RGB) coordinate value 408, as a grayscale pixel value, or using any suitable color coordinate system.

[0041] If, for example, the traffic device 12 is a stop sign, the pixel values ​​for the image data 206 can generally be clustered into a red K-cluster 404, representing the red parts of the traffic device 12 in the image data 206, and a white K-cluster 404, representing the white parts of the traffic device 12 in the image data 206. Other K-clusters 404 may be representative of parts of the environment 10 within the section 206a of the image data 206, foreign objects or substances on the traffic device 12 (e.g., a sticker or paint), and the like. In the example of the stop sign, the pixel color value 402 for the traffic device 12 can be determined as the average of the red K-cluster values ​​404.

[0042] In Operation 314 of Procedure 300, the Quality Determination Module 212 determines the health status 400 for the traffic device 12 based on the determined pixel color value 402 for the traffic device 12. For example, based on the type 12a of the traffic device 12, the Quality Determination Module 212 determines an expected pixel color value 410 for the traffic device 12. In the example of the stop sign, the expected pixel color value 410 could be a shade of red defined by the Federal Highway Administration (FHWA). The expected pixel color value 410 is derived, for example, from ideal chromaticity values ​​recommended by the FHWA based on the type 12a of the traffic device 12.

[0043] Based on a comparison of the expected pixel color value 410 and the determined pixel color value 402 for traffic equipment 12, the quality determination module 212 determines the health status 400 for traffic equipment 12. Since the expected pixel color value 410 and the determined pixel color value 402 can be represented as RGB coordinates 408, the comparison of the expected pixel color value 410 and the determined pixel color value 402 can be based on the Euclidean distance 400, 400a between the RGB coordinates 408 of the expected pixel color value 410 and the determined pixel color value 402. That is, the health status 400 can be based, at least partially, on the Euclidean distance 400a between the determined pixel color value 402 and the expected pixel color value 410 for traffic equipment 12. The larger the Euclidean distance 400a, the more likely it is that the traffic facility 12 has deteriorated compared to its original state.

[0044] In some examples, the Euclidean distance 400a can be based on the expected pixel color value 410 and the determined pixel color value 402 in any suitable coordinate system, such as hue, saturation, and value (HSV), hue, saturation, and lightness (HSL), and luma, blue projection, and red projection (YUV). Thus, the Euclidean distance 400a (ED) can be represented in RGB coordinates as follows: EDRGB=(LMR−IR)2+(LMG−IG)2+(LMB−IB)2 where LM is representative of the determined pixel value 402 of the traffic facility 12 and I is representative of the expected pixel color value 410 of the traffic facility 12.

[0045] Similarly, the Euclidean distance 400a can be represented in HSV / HSL coordinates as follows: EDHSV / HSL=(LMH−IH)2+(LMS−IS)2+(LMV / L−IV / L)2

[0046] The Euclidean distance 400a can be represented in YUV coordinates as follows: EDYUV=(LMY−IY)2+(LMU−IU)2+(LMV−IV)2

[0047] Thus, the Euclidean distance 400a can compare the current color scale 402 of the traffic device 12 with the recommended or expected color scale 410 of type 12a of the traffic device 12 in order to measure how the color 402 of the traffic device 12a has deteriorated from its original or ideal value 410.

[0048] Furthermore, the health status 400 for traffic device 12 can be based on a specific contrast ratio 400, 400b of the traffic device 12. The contrast ratio 400b is representative of the ratio of the difference between the intensity / luminance between two or more target areas of the traffic device 12. In the example of the stop sign, the contrast ratio 400b is representative of the difference between the red K-cluster 404, which represents the red parts of the traffic device 12 in the image data 206, and the white K-cluster 404, which represents the white parts of the traffic device 12 in the image data 206, and thus measures how visible one color is compared to the other. The contrast ratio 400b (µ) can be represented as follows: μ=μl−μpμp where µ I for a first K-cluster 404 (e.g. white) and µ pThis represents a second K-cluster 404 (e.g., red). Thus, the health status 400 takes the contrast ratio 400b into account as representative of the visibility of the traffic control device 12 for drivers. The higher the contrast ratio 400b, the better the traffic control device 12 is visible to drivers.

[0049] As in Fig. As shown in Figure 3, the determined Euclidean distance 400a and the determined contrast ratio 400b for the traffic device 12 can be combined to determine a uniform health quality 412 for the traffic device 12. In some examples, weights are assigned to the Euclidean distance 400a and the contrast ratio 400b to determine an average weighted score for the uniform health quality 412. For example, both metrics can be normalized to a common range, such as [0, 1] or another suitable range. That is, the uniform health quality 412 can be determined based on the Euclidean distance 400a and the contrast ratio 400b as a value between zero and one (e.g., Fig. 9) or along a suitable scale (e.g. Fig. 10 and Fig. 11) be determined.

[0050] Furthermore, the perceived brightness or the intensity / level of the ambient light in the image data 206 can be taken into account when assigning weights to the Euclidean distance 400a and the contrast ratio 400b. That is, at least a part of the image data 206 can be converted into the L*a*b* color space, where L* represents the perceived brightness, a* the green-red opponent colors, and b* the blue-yellow opponent colors. L* can define black at 0.0 and white at 1.0. In some examples, the entire image of the image data 206 is converted into the L*a*b* color space to determine the perceived brightness of the image data 206. Optionally, only a part of the image data 206 is converted (e.g., the determined pixel color value 402 for the traffic control device 12, or optionally corresponding pixel color values ​​for any part of the image data 206, e.g.,Representative portions of the image data 206 for the environment are converted into the L*a*b* color space to determine the perceived brightness. If the image data 206 are taken in a sunny environment, the perceived brightness of the image data 206 may be greater than 0.5, and the Euclidean distance 400a may be given more weight as the more significant factor in determining the uniform health quality 412, since it is assumed that the image acquisition takes place under more ideal conditions and is less likely to be affected by weather conditions.If the image data 206 is acquired in a dark or cloudy environment, the perceived brightness of the image data 206 may be less than 0.5, and the contrast ratio 400b may be given more weight as the more significant factor, since the colors of the traffic control device 12 may appear darker and the contrast ratio 400b is less susceptible to dark conditions. The uniform health condition 412 (H) can therefore be represented as follows: H=α(CR)+β(ED) where α represents the weight of the contrast ratio 400b, CR represents the contrast ratio 400b, and β represents the weight of the Euclidean distance 400a. If the perceived brightness is less than or equal to 0.5, then α is greater than β. If the perceived brightness is greater than 0.5, then α is less than β. The sum of α and β is equal to one.

[0051] In some examples, the health status 400 of the traffic device 12 can also be based on an ambient contrast ratio 400, 400c, which represents the visibility of the traffic device 12 compared to the background or the environment of the traffic device 12. Thus, the quality determination module 212 can isolate the second part 206b of the image data 206, which surrounds the first part 206a, where the second part 206b is representative of the environment 10 at or near the traffic device 12. The quality determination module 212 determines a pixel color value 414 that is representative of the environment 10 in the second part 206b, e.g., via K-means clustering, and determines the ambient contrast ratio 400c based on the determined pixel color value 402 of the traffic device 12 and the determined pixel color value 414 for the environment 10. As in Fig. As shown in Figure 4, the second part 206b can only represent the environment 10 that immediately surrounds the traffic device 12 in order to provide drivers viewing the traffic device 12 with a more accurate representation of its visibility.

[0052] Furthermore, the quality determination module 212 can determine a chromaticity 400, 400d, which is assigned to the traffic device 12, for example, to check or confirm the health status 400 determined for the traffic device 12. For example, the quality determination module 212 can translate the determined pixel color value 402 for the traffic device 12 into a corresponding chromaticity value 400d and record the chromaticity along a chromaticity diagram 416 in order to compare the chromaticity value 400d for the traffic device 12 with the expected chromaticity value 410 for type 12a of the traffic device 12 ( Fig. 12). The expected value of 410 for traffic facility 12 can be based on the FHWA guidelines.

[0053] As in Fig. As shown in Figure 2, the determined health status 400 and / or the uniform health quality 412 for the traffic equipment 12 can be stored in the traffic equipment database 220. One or more specific health status values ​​400 can be assigned to each unique traffic equipment 12. The vehicle 100 can communicate with the traffic equipment database 220, for example via the vehicle 100's telematics system 114, to adjust the operation of the traffic equipment detection system 102 based on the determined health status 400 for the traffic equipment 12.

[0054] For example, the traffic equipment detection system 102 can predict one or more traffic equipment installations 12 along a route of the vehicle 100 based on the vehicle 100's position determined by a navigation or GPS module 116 of the vehicle 100. The navigation module 116 can adjust the route of the vehicle 100 to, for example, avoid traffic equipment installations 12 whose determined health status 400 is below a threshold. Based on the health status 400 of the traffic equipment installation 12, the system 102 can generate a notification 110 to the driver, for example, to inform them that the health status 400 indicates a malfunctioning traffic equipment installation and that the traffic equipment installation 12 may therefore not be visible to the driver.Based on the determined health status 400 for the traffic device 12, the traffic device detection system 102 can, for example, adjust its operation to highlight traffic devices 12 with poor health status 400 in a head-up display or a display on the windshield. Furthermore, the ADAS 112 can adjust its operation by, for example, using the health status 400 as a weighting factor when using the image data captured by the camera 108 or the map from the navigation module 116 to control the vehicle 100.

[0055] Furthermore, the traffic equipment database 220 can communicate with an event processing module 222 of the backend system 200, which can identify and group or cluster traffic equipment 12 with a health status 400 indicating deterioration. That is, the event processing module 222 can identify traffic equipment 12 with a poor health status 400 and sort or group the traffic equipment 12, for example, based on the location of the traffic equipment 12, its type 12a, and / or the municipality responsible for the traffic equipment 12. Based on the identified health status 400 for one or more traffic equipment 12 indicating deterioration, a publishing module 224 of the backend system 200 can generate or publish a warning or a report for the repair of the traffic equipment 12.For example, the publication module 224 can populate a publicly accessible database so that a municipality 20, which is responsible for the transport facility 12, can be informed about the health status 400.

[0056] Based on the uniform health condition 412 (i.e., the Euclidean distance 400a and the contrast ratio 400b), which indicates poor quality of the traffic device 12, the system 200 may, for example, recommend repairing or replacing the traffic device 12. Based on the environment contrast ratio 400c of the traffic device 12, which indicates that the traffic device 12 is indistinguishable or invisible compared to its environment 10, the system 200 may recommend an adjustment to the traffic device 12 or the environment 10, e.g., recommending that trees behind the traffic device 12 be trimmed, or recommending that the height of the traffic device 12 be adjusted to make it more visible.The chromaticity 400d of the traffic device 12 can be analyzed to determine the conformity of the traffic device 12 with the FHWA guidelines for type 12a of the traffic device 12.

[0057] In Fig. Figures 5-8 graphically represent the determined health status values ​​400, which are assigned to traffic device 12 and stored in the traffic device database 220, to illustrate the effects of a change in color and contrast ratio on the visibility of traffic device 12 for drivers and the traffic device detection system 102. As shown in diagram 600 of Fig. As shown in Figure 6, the contrast ratio 400b of the traffic control device 12 decreases over time with increasing determined Euclidean distance 400a of the traffic control device 12 (i.e., as the color of the traffic control device deviates further from its original color). In other words, as the traffic control device 12 fades, the Euclidean distance 400a increases and the contrast ratio 400b decreases, indicating a deterioration in the health 400 of the traffic control device 12. That is, the more the traffic control device 12 fades or discolors, the more difficult it is to detect and identify. In the example of the stop sign, and as shown in diagram 700 of Figure 6, the contrast ratio 400b decreases with increasing Euclidean distance 400a of the traffic control device 12. Fig. As shown in Figure 7, the determined contrast ratio 400b between the red and white parts of the traffic control device 12 is highest when the traffic control device 12 has not faded significantly and thus the stop sign has retained a higher percentage of its original red color 410. Similarly, and as shown in diagram 800 of Fig. As shown in Figure 8, the Euclidean distance 400a between the determined pixel color value 402 and the expected pixel color value 410 of the traffic device 12 can be smallest when the traffic device 12 has not faded significantly and thus the stop sign has retained a higher percentage of its original red color 410. Therefore, the Euclidean distance 400a and the contrast ratio 400b are strongly correlated with the good or bad health status 400 of the traffic device 12.

[0058] Fig. Figure 9 shows a diagram 900 with experimental data comparing the determined environmental contrast ratio 400c of the traffic facility 12 with the uniform health quality 412 of the traffic facility 12. As shown, the traffic sign detection system 102 may be able to detect traffic facilities 12 that have low scores for uniform health quality 412 (i.e., the traffic facility 12 may be damaged or deteriorated and therefore difficult to detect) and / or low scores for the environmental contrast ratio 400c (i.e., the traffic facility 12 may appear inconspicuous and non-prominent compared with the environment 10). Furthermore, Figure 9 shows Fig. Figure 10 presents a diagram 1000 with experimental data comparing the determined ambient contrast ratio 400c with the determined contrast ratio 400b for the traffic control devices 12, the data being validated with human perception values ​​representative of the visibility of the traffic control devices 12 to human drivers. As shown, the traffic control devices 12 that meet the minimum contrast ratio 400b and have higher values ​​for the ambient contrast ratio 400c are the most visible. These were rated highest by human perception values, thus confirming the determined contrast ratio 400b as an efficient method for determining the health 400 of the traffic control device 12. Similarly, Figure 10 shows that the traffic control devices 12 meet the minimum contrast ratio 400b and have higher values ​​for the ambient contrast ratio 400c. Fig. 11 a diagram 1100, in which the environmental contrast ratio 400c is compared with the uniform health quality 412.

[0059] The traffic equipment detection system 102, in conjunction with the backend system 200, is configured to use deep learning models 210 to detect traffic equipment 12 (e.g., stop signs, speed limit signs, yield signs, etc.) and to measure one or more health status values ​​400 associated with the traffic equipment 12 (e.g., Euclidean distance 400a, contrast ratio 400b, ambient contrast ratio 400c, and / or chromaticity 400d). This enables an automated, quantitative assessment of the health of traffic equipment 12 that is independent of human perception. By tracking the health status 400 in the traffic sign database 220, the system 200 can monitor the current health status 400 as well as the deterioration rates of the traffic equipment 12.In this way, it is possible to predict when the traffic control devices 12 will need to be replaced, or to make a recommendation as to when the traffic control device 12 should be replaced. System 200 thus ensures the timely replacement of the traffic control devices 12, reduces monitoring costs for the municipalities responsible for the traffic control devices 12, and increases road safety.

[0060] In some examples, the traffic equipment detection system 102 and the backend system 200 can be configured to also detect other street-side features, such as commercial signs on shop windows and billboards. Accordingly, the backend system 200 determines health status values ​​400 for the other street-side equipment, which can help businesses decide whether to repair or replace their shop sign or develop other, more eye-catching advertising displays.

[0061] Several implementations have been described. However, it goes without saying that various modifications can be made without deviating from the spirit and scope of the disclosure. Accordingly, other embodiments also fall within the scope of protection of the following claims.

[0062] The foregoing description serves for illustration and description purposes. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular configuration are generally not restricted to that particular configuration but are optionally interchangeable and may be used in a selected configuration even if they are not specifically shown or described. The same may also be varied in many ways. Such variations are not to be considered outside the scope of disclosure, and all such modifications are to be included within the scope of protection of the disclosure.

Claims

[1] Computer-implemented method which, when executed in data processing hardware, causes the data processing hardware to perform operations which include: Receiving image data (206) acquired by a traffic facility detection system (102), wherein the image data (206) represent a traffic facility (12) in an environment (10); Identifying the type of traffic facility (12); Isolating a part of the image data (206a) that contains the traffic control device (12); Determining a pixel color value for the traffic control device (12) based on the part of the image data (206a) that contains the traffic control device (12); Determining an expected pixel color value for the traffic control device (12) based on the type of traffic control device; Isolating a second part of the image data (206b) that surrounds the part of the image data (206a) that contains the traffic control device (12); Determining a pixel color value for the environment (10) based on the second part of the image data (206b); and Determining a health status (400) for the traffic facility (12) based on a comparison of the expected pixel color value and the determined pixel color value for the traffic facility (12); wherein the health status (400) for the traffic facility (12) is further based on a comparison of the determined pixel color value for the traffic facility (12) and the determined pixel color value for the environment (10). [2] Method according to claim 1, wherein determining the pixel color value for the traffic control device (12) comprises: Determining one or more clusters of pixel color values ​​by k-means clustering; and Determining the pixel color value for the traffic facility (12) based on an average value of a primary cluster of one or more clusters of pixel color values. [3] Method according to claim 1, wherein the comparison of the expected pixel color value and the determined pixel color value is based on the Euclidean distance between the expected pixel color value and the determined pixel color value, wherein the expected pixel color value and the determined pixel color value are represented by values ​​selected from the group consisting of (i) RGB coordinate values, (ii) HSV coordinate values, (iii) HSL coordinate values ​​and (iv) YUV coordinate values. [4] Method according to claim 1, wherein the health status (400) for the traffic device (12) is further based on a specific contrast ratio of the traffic device (12). [5] Method according to claim 1, wherein the image data (206) from traffic control system detection systems (102) attached to a plurality of vehicles are aggregated. [6] Method according to claim 1, wherein the comparison of the expected pixel color value and the determined pixel color value is based on a level of ambient light present in the image data (206). [7] Method according to claim 1, wherein the operations further include adjusting the operation of the traffic equipment detection system (102) on the basis of the determined health status (400) of the traffic equipment (12). [8] Method according to claim 1, wherein the operations further comprise generating a warning to repair the traffic device (12) based on the determined health status (400) of the traffic device indicating deterioration.

Images