Full-range image sensor profiling system

Abstract

Techniques are disclosed for performing an image sensor profiling process and follower architecture that facilitate performing image quality measurements within different ranges of parameters, such as light level, exposure value, contrast target, confidence interval, color ratio, temperature, etc. An image sensor profile containing an image sensor performance profile dataset may then be generated from the measurements. This data set provides contrast detection probability (CDP) measurement data over a full range of operating parameters of the image sensor. The image sensor performance profile dataset may quantify image quality and performance over a full range of operations. The image sensor performance profile dataset may be implemented to generate an image sensor model that may be deployed in a vehicle and used to modify a function of the vehicle during operation.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63,646,156, filed May 13, 2024; U.S. Provisional Application No. 63 / 723,195, filed November 21, 2024; and U.S. Provisional Application No. 63 / 787,102, filed April 11, 2025, the contents of each of which are incorporated herein by reference in their entirety. Technical Field

[0003] The aspects described herein generally relate to performing an image sensor profiling process, and more specifically, to performing image sensor profiling to measure image sensor performance across different parameter ranges, which can be implemented to generate an image sensor model deployed in a vehicle and used to modify the vehicle's functionality during operation. Background Technology

[0004] Conventional image sensor testing for image quality verification typically requires collecting large amounts of data by analyzing images acquired over time and under various conditions. Therefore, for vehicle-based applications, this testing may require collecting images and / or videos over several days or weeks while driving during daytime and nighttime driving, traversing tunnels, etc. Once a sufficiently large set of image data is acquired in this manner, a manual process is usually implemented where users review the images to identify quality-related issues. This process is not only laborious and time-consuming, but also often fails to identify quality problems across all operating parameters because the image data collection process essentially samples the image set under random (but not all possible) operating conditions. Therefore, current methods for evaluating image sensor quality have various drawbacks and fail to provide a robust and comprehensive evaluation of the image sensor. Attached Figure Description

[0005] The accompanying drawings, which are incorporated herein and form a part of this specification, illustrate aspects of this disclosure and, together with embodiments, further serve to explain the principles of the aspects and enable those skilled in the art to manufacture and use the aspects.

[0006] Figure 1 Example vehicles are described according to one or more aspects of this disclosure.

[0007] Figure 2 This disclosure describes various examples of electronic components used in vehicle safety systems according to one or more aspects of this disclosure;

[0008] Figure 3 This describes an instance architecture for generating a full-range image sensor performance profiling dataset, based on one or more aspects of this disclosure;

[0009] Figure 4 This describes example test patterns for generating a full-range image sensor performance profiling dataset, according to one or more aspects of this disclosure.

[0010] Figure 5 Describe an example process flow based on one or more aspects of this disclosure;

[0011] Figures 6A to 6B This describes instance performance plots derived from an image sensor performance profiling dataset according to one or more aspects of this disclosure;

[0012] Figures 7A to 7C This illustration shows example plots of contrast detection probability (CDP) versus light range under different contrast targets according to one or more aspects of this disclosure;

[0013] Figure 8 The illustration describes an example of CDP versus light level plot and 2D heatmap according to one or more aspects of this disclosure, the 2D heatmap being mapped to CDP values ​​represented in an image sensor performance profiling dataset at a set of different light levels and exposure values;

[0014] Figures 9A to 9B This describes an example 2DCDP thermal map of two different image sensors according to one or more aspects of this disclosure, which plots the probability of a specified contrast detection within a specified confidence interval of exposure value and light level.

[0015] Figure 10A and Figure 10B The illustration is based on one or more aspects of this disclosure, and is generated from an image sensor performance profiling dataset of a sensor to determine low-light performance.

[0016] Figure 11A and Figure 11B Description of one or more aspects corresponding to this disclosure Figure 9A and Figure 9B Instance plots of two different image sensors for 2DCDP thermal images;

[0017] Figures 12A to 12B This illustrates an example 2D thermal image representing a color ratio value mapping within a range of exposure values ​​and light levels, according to one or more aspects of this disclosure; and

[0018] Figures 13A to 13B This describes a set of example 2D heatmaps illustrating different color ratio mappings within a range of exposure values ​​and light levels, according to one or more aspects of this disclosure, wherein the color ratio mappings correspond to... Figure 9A and Figure 9B Two different image sensors for 2DCDP thermal images.

[0019] Exemplary aspects of this disclosure will be described with reference to the accompanying drawings. The first appearance of an element is typically indicated by one or more leftmost numerals in the corresponding reference numerals. Detailed Implementation

[0020] In the following description, numerous specific details are set forth to provide a thorough understanding of the various aspects of this disclosure. However, those skilled in the art will understand that aspects including structures, systems, and methods can be practiced without these specific details. The descriptions and representations herein are common means used by those skilled in the art to most effectively communicate the substance of their work to others skilled in the art. In other instances, well-known methods, procedures, components, and circuit systems have not been described in detail to avoid unnecessarily obscuring aspects of this disclosure.

[0021] I. Example Vehicle Architecture

[0022] Figure 1 Demonstrates various aspects of the security system 200 according to this disclosure (see also) Figure 2 Vehicle 100. Vehicle 100 and safety system 200 are exemplary in nature and are therefore simplified for illustrative purposes. The positions and relational distances of elements (as discussed herein, the diagrams are not drawn to scale) are provided by way of example rather than limitation. Safety system 200 may include various components depending on the requirements of a particular implementation and / or application, and may facilitate navigation and / or control of vehicle 100. Vehicle 100 may be an autonomous vehicle (AV) which may include any level of automation (e.g., levels 0 to 5), including no automation or full automation (level 5). Vehicle 100 may implement safety system 200 as part of any suitable type of autonomous or driver assistance control system (including, for example, AV and / or advanced driver assistance systems (ADAS)). Safety system 200 may include one or more components that are integrated during manufacturing as part of vehicle 100, part of an add-on or aftermarket device, or a combination thereof. Thus, as Figure 2 The various components of the safety system 200 shown can be integrated as part of the vehicle system and / or as part of the aftermarket system installed in the vehicle 100.

[0023] One or more processors 102 may be integrated with or separate from the electronic control unit (ECU) or engine control unit of the vehicle 100, which may be considered herein as a dedicated type of ECU. Safety system 200 may generate data for controlling or assisting in controlling the ECU and / or other components of the vehicle 100 to directly or indirectly control the driving of the vehicle 100. However, the aspects described herein are not limited to implementations within autonomous or semi-autonomous vehicles, as these are provided by way of example. The aspects described herein can be implemented as part of any suitable type of vehicle capable of driving in specific driving environments with or without any suitable level of human assistance. Thus, in various aspects, various vehicle components (such as those referenced herein) Figure 2 One or more of the vehicle components discussed may be implemented as parts of a standard vehicle (i.e., a vehicle without autonomous driving functions), a fully autonomous vehicle, and / or a semi-autonomous vehicle. In the aspect of being implemented as a part of a standard vehicle, it should be understood that the safety system 200 may perform alternating functions, and therefore, according to such aspects, the safety system 200 may alternatively represent any suitable type of system that can be implemented by a standard vehicle without necessarily utilizing autonomous or semi-autonomous control-related functions.

[0024] Regardless of Figure 1 and Figure 2 Regardless of the specific implementation of the vehicle 100 and accompanying safety system 200 shown, the safety system 200 may include one or more processors 102, one or more image acquisition devices 104 configured to perform image acquisition in any suitable wavelength range (such as, for example, one or more cameras or any other suitable sensors), one or more position sensors 106 that may be implemented as a position and / or location identification system (such as a Global Navigation Satellite System (GNSS), such as a Global Positioning System (GPS)), one or more memories 202, one or more map databases 204, one or more user interfaces 206 (such as, for example, a display, touch screen, microphone, loudspeaker, one or more buttons and / or switches, etc.), and one or more wireless transceivers 208, 210, 212.

[0025] Wireless transceivers 208, 210, and 212 can be configured to operate according to any suitable number and / or type of desired radio communication protocol or standard. For example, a wireless transceiver (e.g., the first wireless transceiver 208) can be configured according to short-range mobile radio communication standards (such as, for example, Bluetooth, Zigbee, etc.). As another example, a wireless transceiver (e.g., the second wireless transceiver 210) can be configured according to medium-range or wide-range mobile radio communication standards (such as, for example, 3G (e.g., Universal Mobile Telecommunications System – UMTS), 4G (e.g., Long Term Evolution – LTE), or 5G mobile wireless communication standards compliant with the corresponding 3GPP (3rd Generation Partnership Project) standard (the latest version at the time of writing is 3GPP Release 16 (2020)).

[0026] As another example, the wireless transceiver (e.g., the third wireless transceiver 212) may conform to a wireless local area network communication protocol or standard (such as, for example, the IEEE 802.11 working group standard, the latest version of which at the time of writing is IEEE Std 802.11, published on February 26, 2021). TM Configured using 2020 standards (e.g., 802.11, 802.11a, 802.11b, 802.11g, 802.11n, 802.11p, 802.11-12, 802.11ac, 802.11ad, 802.11ah, 802.11ax, 802.11ay, etc.). One or more wireless transceivers 208, 210, 212 can be configured to transmit signals via an antenna system (not shown) using an air interface. As an additional example, one or more of transceivers 208, 210, and 212 may be configured to implement one or more vehicle-to-everything (V2X) communication protocols, which may include vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), vehicle-to-network (V2N), vehicle-to-pedestrian (V2P), vehicle-to-device (V2D), vehicle-to-grid (V2G), and any other suitable communication protocols.

[0027] One or more of the wireless transceivers 208, 210, and 212 may be additionally or alternatively configured to enable communication between the vehicle 100 and one or more other remote computing devices via one or more wireless links 140. This may include, for example, communication with a remote server or such Figure 1 Communication with other suitable computing systems 150 shown in the figure. Figure 1The example shown illustrates this remote computing system 150 as a cloud computing system, but this is by way of example and not a limitation, and the computing system 150 can be implemented according to any suitable architecture and / or network, and can constitute one or more physical computers, servers, processors, etc. that include this system. As another example, the computing system 150 can be implemented as an edge computing system and / or network.

[0028] One or more processors 102 may implement any suitable type of processing circuitry, other suitable circuitry, memory, etc., and utilize any suitable type of architecture. One or more processors 102 may be configured by vehicle 100 to act as controllers for performing various vehicle-based functions, which may include control functions, navigation functions, etc. For example, one or more processors 102 may be configured to act as controllers for vehicle 100 to analyze sensor data and received communications, calculate specific actions for vehicle 100 to perform for navigation and / or control of vehicle 100, and cause the corresponding actions to be performed. These one or more processors may be, for example, based on an AV or ADAS system, and are therefore alternatively referred to herein as an AV / ADAS system. One or more processors and / or safety system 200 (e.g., any one of processors 214A, 214B, 216, and / or 218 of one or more processors 102) may form all or part of an advanced driver assistance system (ADAS) or autonomous vehicle (AV) system, which may fully utilize one or more machine vision-based object or feature classification processes that may include any suitable computer vision (CV) algorithms discussed further in detail herein.

[0029] Furthermore, one or more of the processors 214A, 214B, 216, and / or 218 of one or more processors 102 may be configured to work collaboratively with each other and / or with other components of vehicle 100 to collect information about the environment (e.g., sensor data, such as images, depth information (e.g., for lidar)). In this context, one or more of the processors 214A, 214B, 216, and / or 218 of one or more processors 102 may be referred to as a “processor”. Thus, the processor may be implemented (independently or together) to create mapping information from the collected data, such as road segment data (RSD) information that can be used for Road Experience Management (REM) mapping technology, details of which are further described below. As another example, the processor may be implemented to process mapping information (e.g., roadbook information for REM mapping technology) received from a remote server via a wireless communication link (e.g., link 140) to locate vehicle 100 on an AV map, the mapping information being available for the processor to control vehicle 100.

[0030] One or more processors 102 may include one or more application processors 214A, 214B, image processor 216, communication processor 218, and may additionally or alternatively include any other suitable processing devices, circuit systems, components, etc., not shown in the figures for simplicity. Similarly, depending on the needs of a particular application, image acquisition device 104 may include any suitable number of image acquisition devices and components. Image acquisition device 104 may include one or more image capture devices (e.g., a camera, charge-coupled device (CCD), or any other type of image sensor). Security system 200 may also include a data interface for communicatively connecting one or more processors 102 to one or more image acquisition devices 104. For example, a first data interface may include any one or more wired and / or wireless first links 220 for transmitting image data acquired by one or more image acquisition devices 104 to one or more processors 102, for example, to image processor 216.

[0031] Wireless transceivers 208, 210, and 212 may be coupled to one or more processors 102 via a second data interface, for example, to a communications processor 218. The second data interface may include any one or more wired and / or wireless second links 222 for transmitting radio transmission data acquired by wireless transceivers 208, 210, and 212 to one or more processors 102, for example, to the communications processor 218. Such transmissions may also include communication (one-way or two-way) between vehicle 100 and one or more other (target) vehicles in the environment of vehicle 100 (e.g., to facilitate coordination of navigation of vehicle 100 in view of or in conjunction with other (target) vehicles in the environment of vehicle 100), or even broadcast transmissions to unspecified receivers near the transmitting vehicle 100.

[0032] The memory 202 and one or more user interfaces 206 may be coupled to each of the one or more processors 102, for example, via a third data interface. The third data interface may include any one or more wired and / or wireless third links 224. In addition, the position sensor 106 may be coupled to each of the one or more processors 102, for example, via the third data interface.

[0033] Each of the one or more processors 102, 214A, 214B, 216, 218, can be implemented as any suitable number and / or type of hardware-based processing device (e.g., processing circuitry system), and can collectively (i.e., utilizing one or more processors 102) form one or more types of controllers as discussed herein. Provided for ease of explanation and as an example. Figure 2The architecture shown herein, and the vehicle 100 may include any suitable number of one or more processors 102, each of which may be similarly configured to utilize data received via various interfaces and perform one or more specific tasks.

[0034] For example, one or more processors 102 may form a controller configured to perform various control-related functions of the vehicle 100, such as the calculation and execution of specific vehicle following speed, rate, acceleration, braking, steering, trajectory, etc. As another example, in addition to or in place of the one or more processors 102, the vehicle 100 may implement other processors (not shown), which may form different types of controllers configured to perform additional or alternative types of control-related functions. Each controller may be responsible for controlling specific subsystems and / or controls associated with the vehicle 100. According to such aspects, each controller may be able to access, via a corresponding interface (e.g., 220, 222, 224, 232, etc.) from, for example... Figure 2 The corresponding coupled components shown in the figure receive data, wherein wireless transceivers 208, 210 and / or 212 provide data to the corresponding controllers via a second link 222, which in this example serves as a communication interface between the corresponding wireless transceivers 208, 210 and / or 212 and each corresponding controller.

[0035] To provide another example, application processors 214A and 214B may individually represent corresponding controllers that work in conjunction with one or more processors 102 to perform specific control-related tasks. For example, application processor 214A may be implemented as a first controller, while application processor 214B may be implemented as a second and different type of controller configured to perform other types of tasks as further discussed herein. According to such aspects, one or more processors 102 may access the system via various interfaces 220, 222, 224, 232, etc. Figure 2 The corresponding coupled components shown in the diagram receive data, and the communication processor 218 can provide communication data received from other vehicles (or to be transmitted to other vehicles) to each controller via corresponding coupled links 240A, 240B, which in this example serve as communication interfaces between the corresponding application processors 214A, 214B and the communication processor 218.

[0036] One or more processors 102 may be additionally implemented to communicate with any other suitable components of vehicle 100 to determine the state of the vehicle while it is being driven or at any other suitable time. For example, vehicle 100 may include one or more vehicle computers, sensors, ECUs, interfaces, etc., which may be collectively referred to as vehicle components 230, such as... Figure 2As shown in the diagram. One or more processors 102 are configured to communicate with vehicle component 230 via an additional data interface 232, which can represent any suitable type of link and operate according to any suitable communication protocol (e.g., CAN bus communication). Using data received via data interface 232, one or more processors 102 can determine any suitable type of vehicle status information, such as current drive gear, current engine speed, acceleration capability of vehicle 100, etc. As another example, various metrics for controlling speed, acceleration, braking, steering, etc., can be received via vehicle component 230, which may include receiving any suitable type of signal (e.g., braking force, wheel angle, reverse gear, etc.) indicating such metrics or different degrees of change of such metrics over time.

[0037] One or more processors 102 may include any suitable number of other processors 214A, 214B, 216, 218, each of which may include processing circuitry systems such as subprocessors, microprocessors, preprocessors (such as image preprocessors), graphics processors, central processing units (CPUs), support circuitry, digital signal processors, integrated circuits, memory, or any other type of means suitable for running applications and performing data processing (e.g., image processing, audio processing, etc.) and analysis and / or enabling vehicle control to be functionally achieved. In some aspects, each processor 214A, 214B, 216, 218 may include any suitable type of single-core or multi-core processor, microcontroller, central processing unit, etc. These processor types may each include multiple processing units with local memory and instruction sets. Such processors may include video input for receiving image data from multiple image sensors and may also include video output capabilities.

[0038] Any of the processors 214A, 214B, 216, and 218 disclosed herein can be configured to perform certain functions according to program instructions, which may be stored in the local memory of each respective processor 214A, 214B, 216, and 218, or accessed via another memory that is part of or outside the security system 200. This memory may include one or more memories 202. Regardless of the specific type and location of the memory, the one or more memories 202 may store software and / or executable (i.e., computer-readable) instructions that, when executed by the associated processor (e.g., one or more of processors 102, processors 214A, 214B, 216, and 218, etc.), control the operation of the security system 200 and perform other functions, such as those identified by aspects described in further detail below. This may include, for example, identifying the position of the vehicle 100 (e.g., via one or more position sensors 106). These functions may include performing any suitable machine vision-based object or feature classification process (which may include any suitable computer vision (CV) algorithm used in further detail herein), performing vehicle-based functions, implementing image sensor models, any functions associated with the AV / ADAS system of vehicle 100, etc., as further discussed herein.

[0039] The associated memory (e.g., one or more memories 202) accessed by one or more processors 214A, 214B, 216, 218 may also store one or more databases and image processing software, as well as a trained system, such as a neural network or deep neural network, that can be used to perform tasks according to any of the aspects discussed herein. The associated memory (e.g., one or more memories 202) accessed by one or more processors 214A, 214B, 216, 218 may be implemented as any suitable number and / or type of non-transitory computer-readable media, such as random access memory, read-only memory, flash memory, disk drive, optical storage device, magnetic tape storage device, removable storage device, or any other suitable type of storage device.

[0040] With Figure 2 The components associated with the security system 200 shown herein are illustrated for ease of explanation and are intended to be illustrative rather than limiting. The security system 200 may contain additional, fewer, or alternative components, as referenced herein. Figure 2 As demonstrated and discussed. Furthermore, one or more components of the security system 200 may be integrated or otherwise combined into a common processing circuit system, or with... Figure 2The components shown are separated to form different and individual components. For example, one or more components of safety system 200 may be integrated with each other on a common die or chip. As an illustrative example, one or more processors 102 and associated memories (e.g., one or more memories 202) accessed by one or more processors 214A, 214B, 216, 218 may be integrated on a common chip, die, package, etc., and together comprise a controller or system configured to perform one or more specific tasks or functions. Furthermore, this controller or system may be configured to perform various functions related to performing vehicle-based functions based on the analysis of the output of the image sensor model and / or the image sensor performance profiling dataset, as discussed in further detail herein.

[0041] In some aspects, safety system 200 may further include components for measuring the speed of vehicle 100, such as speed sensor 108 (e.g., speedometer). Safety system 200 may also include one or more sensors 105, which may include one or more accelerometers (single-axis or multi-axis) for measuring the acceleration of vehicle 100 along one or more axes, and additionally or alternatively include one or more gyroscope sensors. The one or more sensors 105 may further include additional sensors or different sensor types, such as ultrasonic sensors, infrared sensors, thermal sensors, digital compasses, etc. Safety system 200 may also include one or more radar sensors 110 and one or more lidar sensors 112 (which may be integrated into the headlights of vehicle 100). Radar sensor 110 and / or lidar sensor 112 may be configured to provide preprocessed sensor data, such as a radar target list or lidar target list. A third data interface (e.g., one or more links 224) may couple one or more sensors 105, speed sensor 108, one or more radar sensors 110, and one or more lidar sensors 112 to at least one of one or more processors 102.

[0042] II. Autonomous Vehicle (AV) Map Data and Road Experience Management (REM)

[0043] The data, referred to as REM map data (or alternatively as road book map data or AV map data), may also be stored in associated memory (e.g., one or more memories 202) accessed by one or more processors 214A, 214B, 216, 218, or stored in any suitable location and / or in any suitable format (such as stored locally or in a cloud-based database), accessed via communication between the vehicle and one or more external components (e.g., via transceivers 208, 210, 212), etc. It should be noted that although referred to herein as “AV map data,” the data can be implemented in any suitable vehicle platform, which may include vehicles with any suitable level of automation (e.g., level 0 to 5) as mentioned above.

[0044] Regardless of where the AV map data is stored and / or accessed, it can contain the geographic locations of known landmarks that are easily identifiable in the navigation environment in which vehicle 100 is traveling. The locations of landmarks can be generated from historical accumulations from other vehicles driving on the same road, which collect data on the appearance and / or location of landmarks (e.g., "crowd sourcing"). Therefore, each landmark can be associated with a pre-established set of predetermined geographic coordinates. Thus, in addition to using location-based sensors (such as GNSS), the database of landmarks provided by the AV map data enables vehicle 100 to identify landmarks using one or more image acquisition devices 104. Once identified, vehicle 100 can utilize images from other sensors (such as lidar, accelerometers, speedometers, etc.) or from the image acquisition devices 104 to assess the position and location of vehicle 100 relative to the identified landmark location.

[0045] Furthermore, vehicle 100 can determine its own motion, referred to as "ego-motion." Ego-motion is commonly used in computer vision algorithms and other similar algorithms to represent the movement of a vehicle's camera across multiple frames, thereby providing a baseline (i.e., spatial relationships) that can be used to calculate the 3D structure of a scene based on the corresponding images. Vehicle 100 can analyze its own ego-motion to track its position and orientation relative to identified known landmarks. Because landmarks are identified using predetermined geographic coordinates, vehicle 100 can determine its location and position on a map based on its position relative to the identified landmarks using landmark-related geographic coordinates. This offers the advantage of combining the benefits of smaller-scale location tracking with the reliability of a GNSS positioning system while avoiding the disadvantages of both. It should be further noted that analyzing ego-motion in this way is an example of an algorithm implemented using monocular imaging to determine the relationship between the vehicle's position and the known positions of one or more known landmarks, thereby assisting the vehicle in localizing itself. However, ego-motion is unnecessary or irrelevant for other types of techniques, and therefore, localization using monocular imaging is not required. Therefore, based on the aspects described herein, vehicle 100 can make full use of any suitable type of positioning technology.

[0046] AV map data is typically constructed as part of a series of steps, which may involve selecting any suitable number of vehicles to participate in the data collection process. As each vehicle collects data, the data is categorized into labeled data points and then transmitted to the cloud or another suitable external location. A suitable computing device (e.g., a cloud server such as computing system 150) then analyzes the data points from individual drivers on the same road, aggregating and aligning these data points with each other. After alignment, the data points are used to define a precise profile of the road infrastructure. Next, the relevant semantics that enable vehicles to understand the immediate driving environment are identified; that is, features and objects linked to the categorized data points are defined. Features and objects defined in this way may include, for example, traffic lights, road arrows, signs, road edges, drivable paths, lane dividers, stop lines, lane markings, etc., allowing vehicles to easily identify these features and objects using the AV map data. This information is then compiled into a roadbook map, which constitutes a set of driving paths, semantic road information (such as features and objects), and aggregated driving behaviors.

[0047] For example, a map database 204, which may be stored as part of one or more memories 202 or accessed via one or more links 140 through a computing system 150, may contain any suitable type of database configured to store (digital) map data of vehicle 100 (e.g., safety system 200). One or more processors 102 may use a suitable communication network (e.g., via cellular networks and / or the Internet, etc.) to download information to the map database 204 via wired or wireless data connections (e.g., one or more links 140). Furthermore, the map database 204 may store AV map data containing data relating to the locations of various landmarks (such as items including roads, water features, geographical features, businesses, points of interest, restaurants, gas stations, etc.) in a reference coordinate system.

[0048] Therefore, map database 204 can not only store the locations of such landmarks as part of the AV map data, but also store descriptors associated with these landmarks (including, for example, names associated with any of the stored features), and information related to project details (such as the precise location and orientation of the project). In some cases, the road map data can store a sparse data model containing a polynomial representation of certain road features of vehicle 100 (e.g., lane markings) or a target trajectory. The AV map data can also contain stored representations of various identified landmarks, which can be used to determine or update the known position of vehicle 100 relative to the target trajectory. Landmark representations can contain data fields (such as landmark type, landmark location, etc.) and other potential identifiers. The AV map data can also contain non-semantic features (point clouds containing certain objects or features in the environment) as well as feature points and descriptors.

[0049] Map database 204 can be augmented with data other than AV map data, and / or map database 204 and / or AV map data can reside partially or entirely as part of remote computing system 150. As discussed herein, the location and map database information of known landmarks that can be stored in map database 204 and / or remote computing system 150 can form content referred to herein as “AV map data,” “REM map data,” or “route map data.” Therefore, one or more processors 102 can process sensory information about the vehicle 100’s environment (such as images, radar signals, depth information from lidar, or stereo processing of two or more images) and location information (such as GPS coordinates, vehicle motion, etc.) to determine the vehicle 100’s current location, position, and / or orientation relative to known landmarks using information contained in the AV map. Thus, the determination of vehicle location can be improved in this manner. Certain aspects of this technique may be additionally or alternatively incorporated into positioning techniques such as mapping and routing models.

[0050] III. Safe Driving Model

[0051] Furthermore, safety system 200 may implement a Safe Driving Model (SDM), which may be utilized and / or executed as part of an ADAS system as discussed herein. By way of example, safety system 200 may include (e.g., as part of a driving strategy) a computer implementation of a formal model such as a Safe Driving Model. A Safe Driving Model may include an implementation of a mathematical model that formalizes the interpretation of applicable laws, standards, policies, etc., applicable to autonomous (e.g., ground-based) vehicles. In some embodiments, the SDM may include a standardized driving strategy, such as a Responsibility Sensitive Safety (RSS) model. However, embodiments are not limited to this particular example, and any suitable driving strategy model may be used to implement the SDM, defining various safety parameters that the AV should adhere to to facilitate safe driving.

[0052] For example, an SDM can be designed to achieve, for instance, three objectives: first, the interpretation of the law should be reasonable, in a sense conforming to how humans interpret the law; second, the interpretation should produce useful driving strategies, meaning it will produce flexible driving strategies rather than overly defensive driving, which inevitably confuses other human drivers and obstructs traffic, thereby limiting the scalability of system deployment; and third, the interpretation should be effectively verifiable, in a sense rigorously demonstrating that the autonomous vehicle correctly interprets the law. The implementation of the safe driving model in the primary vehicle (e.g., vehicle 100) can be, or include, the implementation of a mathematical model for safety assurance, which enables the identification and execution of appropriate responses to hazardous situations, thereby preventing self-inflicted accidents.

[0053] The safe driving model can implement logic to apply driving behavior rules, such as the following five rules:

[0054] - Don't rear-end other vehicles.

[0055] - Do not overtake recklessly.

[0056] - Give up, rather than seize, the right of way.

[0057] - Be aware of areas with limited visibility.

[0058] -If you can avoid an accident without causing another one, then you must do so.

[0059] It should be noted that these rules are not restrictive or exclusive, and can be modified in various aspects as needed. Therefore, the rules represent a regionally specific "contract" for social driving and can evolve over time. While these five rules are currently applicable to most countries, they may not be complete or identical in each region or country, and may be subject to revision.

[0060] As described above, vehicle 100 may include, as also referenced Figure 2 The safety system 200 is described herein. Therefore, the safety system 200 can generate data for controlling or assisting in controlling the ECU of vehicle 100 and / or other components of vehicle 100 to directly or indirectly navigate and / or control the driving operations of vehicle 100, including driving vehicle 100 or other suitable operations as further discussed herein. This navigation may optionally include adjusting one or more SDM parameters, which may occur in response to the detection of any suitable type of feedback obtained via image processing, sensor measurements, etc. Feedback used for this purpose may be collectively referred to herein as “environmental data measurement results” and includes any suitable type of data identifying states associated with the external environment, vehicle occupants, vehicle 100, and / or the cabin environment of vehicle 100.

[0061] For example, environmental data measurements can be used to identify longitudinal and / or lateral distances between vehicle 100 and other vehicles, the presence of objects on the road, hazardous locations, etc. Environmental data measurements can be obtained via any suitable component of vehicle 100, and / or can be the result of analysis of data obtained via said component, such as one or more image acquisition devices 104, one or more sensors 105, position sensors 106, speed sensors 108, one or more radar sensors 110, one or more lidar sensors 112, etc. To provide an illustrative example, environmental data can be used to generate an environmental model based on any suitable combination of environmental data measurements. Therefore, vehicle 100 can perform various navigation-related operations within the framework of a driving strategy model using tasks performed via one or more trained models.

[0062] Navigation-related operations can be performed, for example, by generating an environment model and using a driving strategy model and the environment model to determine the actions the vehicle must perform. That is, a driving strategy model can be applied based on the environment model to determine one or more actions (e.g., navigation-related operations) the vehicle must perform. The SDM can represent the driving strategy model, or alternatively, can be used in conjunction with the driving strategy model (as part of or as an additional layer) to ensure the safety of the actions the vehicle must perform at any given time. For example, ADAS can fully utilize or reference SDM parameters defined by a safe driving model to determine the navigation-related operations of vehicle 100 based on environmental data measurements for a specific scenario. Therefore, navigation-related operations can cause vehicle 100 to perform specific actions based on the environment model to conform to the SDM parameters defined by the SDM model as discussed herein. In other words, an environment model can be generated, at least in part, based on sensor data received via various sensors of vehicle 100 as mentioned herein, and a driving strategy model can then be applied together with the environment model to determine the navigation-related operations the vehicle must perform. Examples of SDM parameters may include maximum speed, minimum (e.g., longitudinal) following distance between vehicle 100 and another vehicle, minimum lateral distance between vehicle 100 and other vehicles, etc.

[0063] IV. Image Sensor Analysis

[0064] Furthermore, conventional image sensor testing suffers from various drawbacks and fails to provide a robust and comprehensive evaluation of the image sensor. The embodiments discussed herein address these problems by providing an image sensor profiling process and accompanying architecture that facilitates full-range profiling across a range of different operating parameters of the image sensor. This may include, for example, performing image quality measurements in a single exposure within a range of light levels, and then repeating these measurements for a set of exposure settings for the image sensor. Image quality measurements may optionally be performed taking into account other parameters such as color ratio, contrast target, confidence interval, sensor temperature, etc. These parameters can be adjusted at any suitable granularity to ensure image sensor profiling under a wide range of expected operating conditions.

[0065] Therefore, embodiments include an image sensor performance profiling dataset generated from these measurements. This dataset provides contrast detection probability (CDP) measurements and other optional data (such as color ratio data) across the full range of operating parameters of the image sensor. CDP measurements are also referred to as contrast transfer accuracy, a known metric. Therefore, although the term "contrast detection probability" is used herein, it will be understood to be equivalent to contrast transfer accuracy or other suitable terms describing the same metric. CDP measurements may include CDP values, which may represent the probability that the expected contrast between two regions (e.g., one brighter, one darker) of an image can be measured within a specified confidence interval. This probability may decrease for any suitable number of reasons, including detector nonlinearity, noise at the detector, etc. As further discussed below, the image sensor performance profiling dataset thus quantifies the image quality across the full range of sensor operation at different exposure values ​​and light levels. The generation of this image sensor performance profiling dataset can be achieved in hours (while a conventional process would take days or weeks) using the embodiments described herein, and can be fully automated, and can be measured relative to a whole set of parameters of the image sensor as it is expected to operate during its subsequent deployment in a vehicle.

[0066] Figure 3 This describes an instance architecture for generating a full-range image sensor performance profiling dataset, based on one or more aspects of this disclosure. For example... Figure 3As shown, system 300 may include computing device 301, an image sensor 320 to be profiled (e.g., "sensor under test"), pattern 340, and light source 350. As further discussed herein, computing device 301 is configured to configure and / or control image sensor 320 by adjusting various operating parameters (such as, for example, exposure values) used to acquire images of pattern 340, and to receive image frames acquired by image sensor 320. Additionally, computing device 301 is configured to configure and / or control light source 350 to adjust the light level emitted by the light source onto pattern 340. Therefore, and as further detailed below, computing device 301 is configured to capture image frames acquired by image sensor 320 within a range of different exposure values ​​of image sensor 320 and light levels of pattern 340 to generate an image sensor performance profiling dataset. Of course, these are merely example parameters, and image sensor performance profiling datasets can be generated by varying additional or alternative parameters, with further examples of such alternative parameters discussed below. Figure 3 The configurations shown are provided by way of example and not limitation, and other configurations are contemplated. For example, an alternative configuration could replace pattern 340 with a set of neutral density (ND) filters of various densities to represent patches of different NDs in the pattern, or adjust the light level of light source 350 to any suitable level. This configuration would provide a similar light level difference to using pattern 340 to simulate different patches in the pattern. For example, image sensor 320 could capture an image of each ND filter and / or light level, rather than an image of a patch in the pattern.

[0067] like Figure 3 As shown, image sensor 320 can be mounted or otherwise attached to any suitable type of mounting configuration (not shown) and aligned (e.g., aimed) with pattern 340. Pattern 340 can be positioned at a predetermined distance from image sensor 320 and can include any suitable type of test image for profiling image sensor 320. For example, pattern 340 can be transparent, translucent, semi-transparent, semi-translucent, etc., comprising a graph of patches of predetermined size with contrast differences relative to each other. Pattern 340 can represent a standardized graph that can be used to calculate a contrast detection probability (CDP) measurement. For example, pattern 340 can include a TE295 graph that supports small contrast measurements and allows for a 0.3 OD increment between patches. Implementations of pattern 340 are discussed in further detail below.

[0068] The light source 350 can be implemented as any suitable type of light source, which may, for example, be arranged behind the pattern 340 relative to the image sensor 320. That is, the pattern 340 may be positioned between the image sensor 320 and the light source 350. Thus, when the light source 350 is controlled by the computing device 301, the light source 350 enables the image sensor 320 to acquire image frames of the pattern 340 at different light levels. For this purpose, the light source 350 may be implemented as a dedicated light source. For example, the light source 350 may be implemented as any suitable type of stable, constant (e.g., flicker-free) light source containing a diffuser, such that the emitted light is uniformly distributed on the surface, and there is no measurable contrast between areas of the light source. An example of such a light source may include a Vega light source, which is used to perform high-precision measurements of extremely short exposure times common in automotive-grade cameras. According to such embodiments, the light source 350 may be implemented as a flicker-proof LED driven by DC (direct current) technology and may provide a light range, for example, between 0.1 and 50,000 cd / m².

[0069] To perform measurements (e.g., calculation, retrieval, or otherwise acquisition) to generate an image sensor performance profiling dataset as discussed further in detail herein, computing device 301 can be identified using any suitable type of device configured to control image sensor 320 and light source 350, and to acquire, store, and analyze image frames of pattern 340 captured by image sensor 320 as discussed further in detail herein. Therefore, computing device 301 can be configured to perform any of the operations discussed herein to generate an image sensor performance profiling dataset, and can be performed as part of a semi-automated or fully automated process. For example, computing device 301 may include a laptop computer, desktop computer or workstation, mobile phone, server or cloud-based computing device, tablet computer, etc. Therefore, computing device 301 may be configured to perform various operations as discussed herein to generate an image sensor performance profiling dataset of image sensor 320 via one or more components, such as transmitting control signals and / or data to image sensor 320 and / or light source 350 via image sensor data link 330 and light source data link 332, and / or receiving control signals and / or data from image sensor 320 and / or light source 350.

[0070] like Figure 3 As shown, the computing device 301 may include a processing circuit system 302, a communication interface and image sensor control circuit system 304, a communication interface and light source control circuit system 317, and a memory 306. Provided for ease of explanation. Figure 3 The components shown herein, and the computing device 301 may implement additional, fewer, or alternative components, such as Figure 3 The components shown in the document.

[0071] The communication interface and image sensor control circuitry system 304 and the communication interface and light source control circuitry system 317 can be implemented as any suitable number and / or type of components configured to transmit and / or receive data (such as data packets, control signals, etc.) and / or wireless signals according to any suitable number and / or type of communication protocols. The communication interface and image sensor control circuitry system 304 and the communication interface and light source control circuitry system 317 may include any suitable type of components to facilitate this functionality, including components associated with the operation, configuration, and implementation of known transceivers, transmitters, and / or receivers. The communication interface and image sensor control circuitry system 304 and the communication interface and light source control circuitry system 317 may include components typically identified by a communication interface, including, for example, drivers, connectors, terminals, wires, antennas, ports, amplifiers, buffers, buses, etc. Therefore, the communication interface and image sensor control circuit system 304 and the communication interface and light source control circuit system 317 can be configured as any suitable number and / or type of components, configured to facilitate the reception and / or transmission of signals and / or data according to any suitable number and / or type of communication protocol, which may be a standardized or non-standardized protocol.

[0072] For example, the communication interface and image sensor control circuitry system 304 can be implemented as any suitable number and / or type of hardware and / or software components configured to enable communication between the computing device 301 and a specific image sensor 320, thereby generating a corresponding image sensor performance profiling dataset. For example, the communication interface and image sensor control circuitry system 304 can enable the computing device 301 to receive image frames acquired by the image sensor 320, such as those of pattern 340 discussed herein. Alternatively or additionally, the communication interface and image sensor control circuitry system 304 can enable the computing device 301 to control various settings and / or parameters (such as, for example, exposure settings) of the image sensor 320 and / or trigger the acquisition of image frames via the image sensor 320.

[0073] In any case, any communication and / or control between the computing device 301 and the image sensor 320 in this manner can be facilitated by data and / or control signals transmitted via the image sensor data link 330, which may include any suitable number and / or type of buses, wires, interconnects, etc. Therefore, the communication interface and image sensor control circuitry system 304 may include any suitable type of data interface configured to transmit data to and / or from the image sensor 320 via a corresponding communication interface 320.2 of the image sensor 320 as shown. For example, the communication interface and image sensor control circuitry system 304 may include a serializer, and the corresponding communication interface 320.2 of the image sensor 320 may include a deserializer, or vice versa. Alternatively or additionally, the image sensor control circuitry system 304 may be implemented as a frame capturer that implements control and acquisition of images acquired via the image sensor 320 as discussed herein.

[0074] The communication interface and light source control circuitry system 317 can be implemented as any suitable number and / or type of hardware and / or software components configured to enable communication between the computing device 301 and the light source 350. For example, the communication interface and light source control circuitry system 317 can enable the computing device 301 to adjust the light level settings of the light source 350 to illuminate the pattern 340 at various lighting levels according to any suitable granularity. Any such communication and / or control can be facilitated by data and / or control signals transmitted via a light source data link 332, which may include any suitable number and / or type of buses, wires, interconnects, etc. Therefore, the communication interface and light source control circuitry system 317 may include any suitable type of data interface configured to transmit data to and / or from the light source 350 via a corresponding communication interface 320.1 as shown for the light source 350. For example, the communication interface and light source control circuitry system 317 may include a controller configured to control the light source 350. For embodiments where the light source 350 includes a Vega light source, the communication interface and light source control circuit system 317 may include a Vega light source controller.

[0075] The processing circuitry system 302 may be configured as any suitable number and / or type of processors for controlling the computing device 301 and / or other components of the computing device 301. The processing circuitry system 302 may be identified by one or more processors (or suitable portions thereof) implemented by the computing device 301 or a host system. The processing circuitry system 302 may be identified by one or more processors, such as a host processor, digital signal processor, one or more microprocessors, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.

[0076] In any case, the processing circuitry 302 may be configured to execute instructions to perform arithmetic, logical, and / or input / output (I / O) operations, and / or control the operation of one or more components of the computing device 301 to perform various functions as described herein. The processing circuitry 302 may include one or more microprocessor cores, memory registers, buffers, clocks, etc., and may generate electronic control signals associated with the components of the computing device 301 to control and / or modify the operation of those components. The processing circuitry 302 may communicate with and / or be associated with the functions of any of the components of the computing device 301.

[0077] Memory 306 may store data and / or instructions such that, when executed by processing circuitry system 302, computing device 301 performs various functions as described herein with respect to the generation of image sensor performance profiling datasets, such as control, transmission, monitoring, regulation, etc., as well as one or more components of system 300 as further described herein. Memory 306 may be implemented as any suitable volatile and / or non-volatile memory, including read-only memory (ROM), random access memory (RAM), flash memory, magnetic storage media, optical disk, erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), etc. Memory 306 may be non-removable, removable, or a combination of both. Memory 306 may be implemented as a non-transitory computer-readable medium storing one or more executable instructions (such as, for example, logic, algorithms, code, etc.).

[0078] As further discussed herein, the instructions, logic, code, etc., stored in memory 306 are represented by various modules as shown, which enable the functional implementation of any or all of the functionality disclosed herein with respect to system 300. Alternatively, as Figure 3 One or more modules associated with memory 306, as shown herein, may contain instructions and / or code to facilitate control and / or monitoring of the operation of hardware components implemented via computing device 301. In other words, Figure 3 One or more modules shown are provided to facilitate the explanation of the functional relationships between hardware and software components.

[0079] Therefore, the processing circuitry system 302 can be combined with one or more hardware components to execute instructions stored in these respective modules to perform various functions as discussed herein. Although modules are described and further illustrated separately herein, this is for ease of explanation and is not intended to be limiting. Instructions in modules of memory 306 may be stored in any suitable manner and may include instructions identifiable by executing a single program, an application program, one or more algorithms supporting such programs and / or applications, etc.

[0080] The communication interface and image sensor control circuitry system 304, and the communication interface and light source control circuitry system 317, enable the computing device 301 to communicate with and / or control the image sensor 320 and the light source 350 via corresponding application programming interfaces (APIs), respectively. Therefore, the image sensor control module 313 can store computer-readable instructions that, when executed via the processing circuitry system 302, enable the computing device 301 to communicate with and / or control the image sensor 320 via the corresponding API (as discussed herein). Additionally, the light source control module 315 can store computer-readable instructions that, when executed via the processing circuitry system 302, enable the computing device 301 to communicate with and / or control the light source 250 via the corresponding API (as discussed herein).

[0081] Control interface 311 may store computer-readable instructions that, when executed via processing circuitry system 302, enable computing device 301 to execute the API as described herein and store control parameters of image sensor 320. For example, control interface 311 may be implemented as any suitable type of software interface, such as, for example, a Python interface. Additionally, control interface 311 may store image sensor parameters and exposure control parameters for a specific image sensor 320, which can be used according to the API to perform various measurements as discussed herein. For example, control interface 311 may store image sensor parameters in JavaScript Object Representation (JSON) format, initialization sequence parameters defining the measurements to be performed in text format, and exposure tables for image sensor 320 in tabular (e.g., CSV) format.

[0082] The data recording and exposure control module 309 can store computer-readable instructions that, when executed via the processing circuitry system 302, enable the computing device 301 to control the exposure value of the image sensor 320, and store received image frames acquired by the image sensor 320 at a set of different exposure and light level settings, as discussed in further detail herein. The image frames can be stored in any suitable memory, including memory 306 or other suitable memory accessible by the computing device 301.

[0083] The test and measurement module 307 may store computer-readable instructions that, when executed via the processing circuitry system 302, enable the computing device 301 to perform corresponding analyses for generating an image sensor performance profiling dataset as discussed herein. Therefore, the test and measurement module 307 facilitates the analysis of stored image frames under a set of different exposure settings and light levels, as well as additional or alternative operating parameters, as discussed herein. This may include, for example, calculating contrast detection probability (CDP) measurements within a predetermined contrast target range, within a predetermined confidence interval, within a predetermined range of different exposure settings, and within a range of different light levels, as discussed herein. The test and measurement module 307 may store computer-readable instructions that, when executed via the processing circuitry system 302, enable the presentation of a suitable user interface via a display (not shown) of the computing device 301, allowing the user to set and / or modify any parameters used in the tests for generating the image sensor performance profiling dataset as discussed herein.

[0084] Figure 5 This describes an example process flow according to one or more aspects of this disclosure. The process can be performed, for example, via a suitable computing device and / or processing circuitry system identified by the AV / ADAS system of system 300 and / or vehicle 100, as referenced herein. Figure 5 The functionality associated with the process flow 500 discussed. For example, process flow 500 can be implemented by executing instructions stored in memory 306 (which may be locally stored instructions and / or part of the processing circuitry) via processing circuitry system 302. Process flow 500 may include, for the sake of brevity, instructions not listed here. Figure 5 The alternative or additional steps shown in the text, and can be combined with Figure 5 The steps shown are executed in different orders.

[0085] Process flow 500 may begin when one or more processor settings (block 502) are configured to acquire current parameters for an image of pattern 340 via image sensor 320. Parameters may include any suitable variables that can be adjusted at any appropriate granularity to produce an image sensor performance profiling dataset as discussed herein. For example, block 502 may include computing device 301 setting the initial exposure value of image sensor 320 and the initial illumination settings of light source 350 (as mentioned above) to the current parameters in this example. Parameters may also include, for example, a predetermined contrast target and a predetermined confidence interval, as discussed in further detail below.

[0086] It should be noted that the exposure values, initial illumination settings, predetermined contrast targets, and predetermined confidence intervals of the image sensor 320 provided herein by way of example rather than limitation, and which can be adjusted in further detail herein to generate the image sensor performance profiling dataset, may include any suitable parameters that may affect the CDP of the images acquired via the image sensor 320. As another example, these parameters may include the temperature of the image sensor 320. To adjust the temperature, the computing device 301 may additionally perform (box 502) environmental control of the area where the image sensor 320 is housed when acquiring the image of pattern 340 as discussed herein. This control may be implemented in any suitable manner, for example, as a separate environmental control API and accompanying communication interface and link (not shown).

[0087] For example, embodiments discussed herein allow the resulting image sensor performance profiling dataset to be used to calculate or otherwise evaluate image sensor performance based on several different confidence intervals, which can be adjusted based on the intended application of the image sensor. For instance, images generated by image sensor 320 may be implemented as part of a safety system 200 for vehicle 100 as mentioned above, and these images can be used to perform various CV processes, the results of which are used to perform various vehicle-based functions of vehicle 100. Further examples of vehicle-based functions and how image sensor performance profiling datasets can be used to influence these vehicle-based functions are provided in more detail below.

[0088] It should be noted that the target contrast level (CTL) and confidence interval (CI) can be defined depending on the specific application and / or the background of the image acquired via image sensor 320. As an illustrative example, in some scenarios, image sensor 320 may be used to capture low-contrast road surface images, and therefore may require a smaller CTL and a smaller CI range (e.g., CTL of 0.1 and CI of 0.1). It should be noted that in this example, CI of 0.1 means + / - 10%. Furthermore, CTL of 0.1 refers to the Michelson contrast calculated as “(S1-S0) / (S1+S0)”, where S1 is the brighter surface and S0 is the darker surface. As another illustrative example, when the image acquired by image sensor 320 is used to detect dark objects on a road, the resulting image may be very noisy, thus requiring a larger CTL and a larger CI. In this way, in addition to exposure values ​​and illumination settings, the parameters adjusted to generate the image sensor performance profiling dataset may also include different CIs defined to generate the image sensor performance profiling dataset. This can be used to gain insights into the performance of the image sensor 320 under different usage conditions associated with these different CI values, as discussed in further detail below.

[0089] Regardless of the specific parameters adjusted to generate the image sensor performance profiling dataset, process flow 500 may include receiving (block 504) an image of pattern 340 acquired via image sensor 320. This may include, for example, computing device 301 receiving the image of pattern 340 acquired by image sensor 320, the image being received, for example, via image sensor data link 330, as mentioned above. Computing device 301 may optionally store the acquired image in memory 306 or another suitable memory accessible by computing device 301.

[0090] Process flow 500 may include calculating (box 506) the contrast detection probability (CDP) of the acquired image using currently established parameters (e.g., exposure settings, illumination settings, CI, contrast target, etc.). CDP is a key performance indicator (KPI) that measures the ability of an image sensor to identify the contrast of objects within its field of view. For example, safety system 200, as mentioned herein, may include AV or ADAS, which should be configured to identify different objects within the vehicle's field of view, such as, for example, identifying the difference between the road and the road edge.

[0091] To measure CDP from the acquired image, refer to Figure 4 This may include instances of pattern 340. Therefore, and as... Figure 4 As shown, the image acquired in box 504 may belong to pattern 340 under a specific exposure value of image sensor 320 and a specific illumination setting provided by light source 350. The CDP of the acquired image can be calculated (box 506) in any suitable manner, and this process can be repeated when parameters are adjusted to acquire another image, and, where applicable, repeated for the same acquired image.

[0092] For example, parameters can be adjusted without necessarily acquiring a new image of pattern 340 when they affect the CDP calculation (e.g., adjusting the contrast target and / or confidence interval) but not the way the image is acquired (e.g., exposure value and illumination). Therefore, box 504 can be identified by receiving a new image to perform a CDP calculation on said image or alternatively by accessing a stored image from a suitable memory to recalculate the CDP of said image based on a different set of parameters (e.g., different contrast targets and / or CI). In this way, the resulting image sensor performance profiling dataset can include a set of CDP measurements across a range of parameters adjusted in box 502, such as different exposure values, light levels, contrast targets, confidence intervals, etc., as discussed in further detail herein. Furthermore, these different measurements can provide a range of key performance indicators (KPIs) covering different types of objects or algorithm requirements, such as detecting road surfaces requiring lower texture compared to other scenarios.

[0093] As an example, the CDP can be calculated by first performing a calibration procedure (box 506) based on a contrast target defined according to established parameters (box 502). This contrast target may include, for example, a desired Michelson contrast relative to the various areas in pattern 340. To provide some illustrative examples, this predetermined contrast target may include 10%, 20%, 30%, 50%, etc. Continuing with this example, as... Figure 4 The pattern 340 shown may comprise multiple pairs of patches with a predetermined contrast difference relative to each other. For example, such as Figure 4 The TE95 chart shown typically contains patches that have a predetermined contrast difference of 0.3% relative to each other, moving diagonally downwards and across the pattern. The result is that patch pairs are identified to measure the contrast target level (CTL) required for each measurement, such that these patch pairs are referenced for each set of images received.

[0094] Therefore, as part of this calibration process, box 506 may include first selecting a subset of patches identified as being used to identify contrast targets as described in the previous paragraphs, and then finding rectangular coordinates that will crop a small region of the patch into a predetermined number of pixels at each cropping. This may include, for example, cropping a predetermined portion (e.g., a predetermined number of pixels) within this subset of patches in pattern 340. For example, each of the subset of patches may be cropped into N×M groups of pixels centered on each of the subset of patches (e.g., 20×20, 30×30, etc.). Next, box 506 may include computing device 301 selecting a subset of pixel pair combinations within the image of pattern 340 (e.g., from the subset of patches), the subset of pixel pair combinations having a difference in brightness level (e.g., contrast level) relative to each other within a predetermined threshold deviation of the predetermined contrast target.

[0095] Next, box 506 includes calculating a set of Michelson contrast values ​​between each possible combination of pixels sampled from the brighter and darker patches to generate statistical pairwise comparisons to compare against a target contrast level measured by the Michelson contrast calculations. Then, based on the total set of Michelson contrast calculations from which pixel values ​​between the two patches are compared in this way, box 506 includes measuring the percentage of density combinations of the calculated contrast values ​​with the target contrast values ​​within the current confidence interval, which can also be established as part of the initial parameters (box 502). Specifying the percentage of calculated pairwise values ​​within the confidence interval that match the target contrast level provides the probability of detecting contrast between the two patches. This process can then be repeated for each combination of two patches within the test pattern that satisfy the current predetermined contrast target, thereby allowing the CDP of the image sensor 320 to be calculated based on current parameters (e.g., current contrast target, confidence interval, exposure setting, and light level).

[0096] However, to ensure the generation of a complete image sensor performance profiling dataset, process flow 500 includes determining (box 508) whether to adjust any of the parameters, and if so, repeating the CDP measurement. For example, it may be necessary to generate an image sensor performance profiling dataset relative to multiple different predetermined contrast targets, and therefore box 508 may include determining whether to consider additional contrast targets, which may be defined by the user as part of the initial testing process or as part of a set of predetermined parameters. In this case, the process described above may be repeated for different predetermined contrast targets. For example, the calibration process may be repeated for updated predetermined contrast target values ​​to identify new patch subsets in the stored images, which are then cropped and their corresponding pixels compared to measure the density combination percentage and the resulting CDP of the image sensor 320.

[0097] Next, box 508 may include determining whether to perform additional measurements for additional confidence intervals (CIs) and repeating the measurements discussed above until the CDP of the image sensor is calculated for each predetermined contrast target to be measured. As an illustrative example, and as further discussed herein, different predetermined CTLs are 0.1, 0.2, and 0.5, and the CIs for each CTL may include 0.1, 0.2, and 0.5. Therefore, the CDP of the image sensor 320 can be calculated for any suitable combination of the predetermined contrast target value and the predetermined IS.

[0098] Furthermore, as process flow 500 continues in this manner, the iterative determination performed at block 508 can result in adjustments to the exposure value of image sensor 320 and the illumination level provided to pattern 340 by light source 350 (block 502). As an illustrative example, process flow 500 can be performed based on two or three different exposure values ​​and 10 to 20 different illumination levels. Therefore, as each of these parameters is adjusted, computing device 301 can receive different images acquired by the image sensor, and can repeat the same process described above for each newly acquired image in each case. For example, the CDP of the image acquired via each combination of different predetermined contrast targets and CI can be calculated for each newly acquired image having different exposure values ​​and illumination levels of pattern 340. Therefore, process flow 500 can be repeated in this manner by adjusting parameters and acquiring additional images based on the updated parameters.

[0099] This process can continue until the decision at box 506 is "No," resulting in the calculation of the CDP of image sensor 320 relative to a set of images of pattern 340 for each combination of parameters adjusted in box 502 (e.g., each combination of various predetermined contrast target values, confidence intervals, exposure values, and illumination levels). Therefore, CDP measurements are repeated for each of the acquired images for different light levels and different exposure values ​​of image sensor 320 for the predetermined pattern to generate (box 510) an image sensor performance profiling dataset, which provides CDP measurements calculated for each combination of adjusted parameters. Any suitable number of such parameters can be adjusted in this manner to generate (box 508) the image sensor performance profiling dataset. In this way, the image sensor performance profiling dataset can represent the CDP of image sensor 320 at various light levels, which can represent most of the light data points across the dynamic range of the image sensor, combined with each of the other parameters adjusted as discussed herein.

[0100] Once the image sensor performance profiling dataset is computed in this manner, process flow 500 includes using the image sensor performance profiling dataset to generate (box 512) an image sensor model. The image sensor model can be generated in any suitable manner, fully utilizing the entire set of CDP measurements across all combinations of parameters implemented and iteratively adjusted throughout process flow 500, as mentioned above. The image sensor model may also include additional data that can be measured for known and / or relative to the operation of image sensor 320. For example, in addition to the image sensor performance profiling dataset, the image sensor model can be generated in this manner using data from sensor datasheets and documentation, sensor data measured during operation, etc. The sensor model can be generated in this manner using any suitable machine learning technique (such as, for example, deep learning techniques) such that the sensor model can produce synthetic sensor data that is very similar to real-world data generated by image sensor 320 under similar conditions. Therefore, this sensor model can be generated by using the image sensor performance profiling dataset as part of a training dataset, which may optionally include other sensor data, such as data obtained via the operation of image sensor 320.

[0101] Additionally, process flow 500 includes modifying (block 514) vehicle-based operations based on the output of the sensor model. Thus, once the sensor model is generated (block 512), it can be deployed as, for example, part of a safety system 200 for vehicle 100. This may involve storing the sensor model in one or more memories 202, which can then be accessed by the AV / ADAS system of vehicle 100, and its output being implemented to modify various vehicle-based functions that can be performed by the AV / ADAS system as discussed above. As an illustrative example, the output of this sensor model can be used to provide, for example, a 2D lookup table, which can in turn be input into another model to implement this implementation. The image sensor model can be configured to output data that can be used for various purposes (e.g., to enable modification of vehicle-based operations), which are discussed in further detail below with respect to various scenarios.

[0102] V. Explaining the Image Sensor Performance Profile Dataset

[0103] Furthermore, the image sensor performance profiling dataset may include CDP measurements of images acquired via image sensor 320 within any suitable combination of parameters used to obtain these measurements, as mentioned above. Therefore, the image sensor performance profiling dataset can not only be used to generate sensor models as mentioned above, but it can also be additionally or alternatively stored by computing device 301 and / or the vehicle's AV / ADAS system (e.g., stored in one or more memories 202) and subsequently accessed in various ways to obtain valuable information about the operation of image sensor 320 during its operation. For example, image sensor performance profiling datasets can be acquired for various image sensors, and these datasets can be used to compare the performance of the image sensors under various operating conditions. As another example, the image sensor performance profiling dataset can be fully utilized to provide performance data about the operation of image sensor 320 under various operating conditions, and thus can be used as a guide for evaluating the performance of image sensors and their suitability for specific applications. In other words, if the imager performance has not yet been optimized (or has been misconfigured), this can be determined early in the evaluation process. Another example is that measurement results can be compared with the expected performance of a sensor based on device knowledge, models of imaging components, etc., where undesired results can indicate sensor problems.

[0104] To provide some illustrative examples, Figure 6A and 6B This describes an instance performance plot derived from the image sensor performance profiling dataset generated according to process flow 500. For this purpose, it should be noted that... Figure 6A The plot shown in the image uses the x-axis to represent the range of illumination values, while the y-axis represents the probability values ​​for a specific contrast and confidence interval (CI). Therefore, Figure 6AIt can provide data on the probability of detecting an object at a specific contrast across the entire range of light levels.

[0105] refer to Figure 6B The x-axis represents the range of target contrast values, while the y-axis represents the probability value of a specific target contrast value and CI. Therefore, Figure 6B It can provide data on the probability of detecting an object within a specific confidence interval at various contrast target values.

[0106] Figures 7A to 7C Plots of CDP versus light range are shown for different contrast targets (corresponding to 0.1, 0.2, and 0.5). The light range, as discussed herein, can be expressed in illuminance or lux, as this represents the amount of light falling on the sensor. This can then be used to understand the sensor's or camera's response to the light levels described herein. Figures 7A to 7C The plot shown represents the CDP under light level in a single exposure. For example... Figures 7A to 7C Each of the plots shown illustrates a CDP versus light range trajectory for a set of confidence intervals (corresponding to 0.1%, 0.2%, and 0.5%) at each contrast target and exposure value, as illustrated by the inset and corresponding trajectory labels. Such plots can be generated by fully utilizing image sensor performance profiling datasets, thereby providing valuable information about the performance of the image sensor 320 under various conditions.

[0107] For example, such as Figure 7A The CDP traces shown can identify light levels associated with error detection or non-linearity recognition of texture. As another example, Figure 7B The plot shown illustrates CDP at low light levels, which results in the inability to detect dark objects in low light. As yet another example, Figure 7C The plotting diagrams shown are based on the light levels required for CDP to detect objects at those light levels, and the high probability of detecting high-contrast objects within a range of light levels. Figure 7C The text also presents an example of light levels under saturation in an image sensor, indicating that saturation will occur at a specific threshold light level.

[0108] Figure 8 This describes the CDP (Critical Discharge Point) mapping with light level plots and 2D heatmaps, where the 2D heatmaps are mapped to CDP values ​​represented in an image sensor performance profiling dataset under a set of different light levels and exposure values. Therefore, Figure 7A The plot is shown as an example. Figure 8On the upper left, it again illustrates the CDP of a set of CIs within a light level range at a single exposure value under a contrast target of 0.1. However, by repeating the measurements as discussed above with respect to process flow 500 within the exposure value range, the image sensor performance profiling dataset can contain CDP measurements repeatedly performed within the light level range at each exposure value. This makes it possible to generate, as... Figure 8 The 2D heatmap plot shown illustrates mapping a contrast target of 0.1 and a CI of 10% to a CDP range as shown for both light levels and exposure values. Therefore, through... Figure 8 The value identified at the intersection of the exposure value and the light level represents the corresponding CDP value of the image sensor 320 at those specific exposure values ​​and light levels. In this example, the CDP metric is shown as a range of 0 to 1 (or 0% to 100%).

[0109] Therefore, 2D CDP heatmaps can be used to identify the overall performance of the image sensor 320. For example, such as Figure 8 The 2D CDP heatmap shown illustrates specific areas (in terms of light level and exposure value) where contrast detection needs improvement. Compared to the 2D CDP heatmap shown, it can be observed that exposure values ​​less than 256 and light levels greater than 16 result in a steady increase in CDP. Furthermore, light saturation level, noise level, and nonlinearity can be identified, which create “gaps” in the CDP under specific combinations of lower exposure values ​​and higher light levels.

[0110] Generating a set of 2D CDP heatmaps of contrast targets and CI also allows for objective and standardized comparisons of the performance of different sensors. For example, Figures 9A to 9B This describes an example 2D CDP heatmap of two different image sensors according to one or more aspects of this disclosure, which plots the probability of a specified contrast detection within a specified confidence interval of exposure value and light level. Figure 9A This illustrates two instances of 2D CDP heatmaps for the first sensor, while... Figure 9B This illustrates two instances of 2D CDP thermal maps of the second sensor. In this instance, corresponding to... Figure 9A The first image sensor 9A for 2D CDP thermal mapping includes a first 8.3MP automotive image sensor, while corresponding to Figure 9B The second image sensor for the 2D CDP thermal image includes a second 8.3MP automotive image sensor. Figure 9A and Figure 9B The upper CDP heatmap in each of them corresponds to a 0.1% contrast target and a 10% CI, while Figure 9A and 9BThe lower CDP heatmap in each of these corresponds to a contrast ratio target of 0.2% and a CI of 20%. Therefore, when the tested parameters include contrast ratio targets of 0.1, 0.2, and 0.5 and CIs of 10%, 20%, and 50%, as... Figure 9A and Figure 9B The two 2D CDP heatmaps shown represent two of a total of nine 2D CDP heatmaps that can be generated from the image sensor performance profiling dataset for each corresponding image sensor. However, for the sake of simplicity and ease of interpretation, only two example 2D CDP heatmaps are shown.

[0111] Comparing these two 2D CDP heatmaps from each of two different image sensors, the heatmaps appear similar, except for minor visible differences. Of course, the 2D CDP heatmaps can be compared numerically by calculating the CDP value at each point in each heatmap, providing a direct comparison of this metric between the two different sensors under the same operating conditions. In this way, the generation of image sensor performance profiling datasets, as discussed herein, facilitates automated and empirical processes for benchmarking, measuring, and / or comparing image sensor performance within the expected range of operating conditions.

[0112] Figure 10A and Figure 10B The diagram illustrates that instance plots for determining low-light performance can be generated from the image sensor performance profiling dataset of image sensor 320. For example, Figure 10A This illustrates the plotting of CDP versus light level range under specific contrast targets and CI. By repeating the CDP measurements as discussed above with respect to process flow 500 (box 506), the image sensor performance profiling dataset can include CDP measurements acquired within the exposure values ​​and light levels of the image sensor 320. Therefore, as Figure 10B The plot shown provides a plot of the light range versus CDP value at a specific contrast target and CI (both 20% in this example). Therefore, Figure 10A This indicates that a minimum light level of approximately 1200 lux is required to achieve a corresponding contrast detection probability of 50%. Furthermore, as... Figure 10B The plot shown illustrates that at a specific exposure value of approximately 850, the lowest light level at 50% CDP corresponds to a light level of approximately 5 lux.

[0113] Figure 11A and Figure 11B The explanation above is relative to Figure 9A and Figure 9B The set of plotted images for the two different image sensors discussed. For example... Figure 11AThe plot shown corresponds to a light level versus exposure value plot generated using the image sensor performance profiling dataset from the first 8.3MP automotive image sensor. For example... Figure 11B The plots shown correspond to light level and exposure value plots generated using the image sensor performance profiling dataset from the second 8.3MP automotive image sensor. Each set of plots contains a set of four light level and exposure value plots; as shown, each different combination of contrast target and CI corresponds to one light level and exposure value plot.

[0114] like Figure 11A and Figure 11B Each illustration in the plot shown in this example demonstrates how intercept tracking techniques can be used to determine the light level required to achieve a 50% CDP at corresponding contrast targets of 10%, 20%, or 50%. Therefore, for example... Figure 11A The plot of the first 8.3MP automotive image sensor shown in the figure, with similar light levels of 12.56 lux, 2.16 lux, 0.86 lux, and 0.76 lux, represents the minimum possible light levels at possible sensor exposure values ​​to achieve 50% CDP in each of the contrast target scenarios shown. Furthermore, for... Figure 11B The plotting of the second 8.3MP automotive image sensor shown in the figure requires similar light levels of 12.56 lux, 2.29 lux, 0.91 lux and 0.75 lux to achieve 50% CDP in each of the contrast target scenarios shown.

[0115] In other words, the light level required for 50% CDP at the highest exposure setting is the same for both image sensors, and the sensitivity / noise ratio is also similar between them. In other words, in this scenario, both sensors exhibit better performance at the highest exposure. Upon further analysis of these results, it should be noted that this is because each sensor supports two independent modes: a normal mode, which operates better in low light; and an auxiliary mode, which provides better HDR noise but at the cost of poorer low-light performance. Therefore, these plots confirm that the control scheme implemented for both sensors works as expected, utilizing the auxiliary mode, except at the highest exposure.

[0116] VI. Incorporate color ratio data into the image sensor performance profiling dataset.

[0117] Furthermore, the image sensor performance profiling dataset may include CDP measurements performed within a variety of parameter ranges. In the examples provided above, when the image sensor 320 acquires an image of pattern 340, CDP measurements are performed by varying the contrast target, confidence interval, exposure value, and light level. In additional embodiments discussed in more detail in this section, color ratio data measurements may also be included as part of the image sensor performance profiling dataset, and thus such measurements may be performed together with the CDP measurements discussed in block 506 of process flow 500.

[0118] To achieve this, each patch in the image of pattern 340 can be analyzed to determine the average color value of all pixels in each patch. The number and type of color values ​​averaged in this way can be a function of the specific color filter array (CFA) implemented by image sensor 320. For example, pattern 340 in Figure 4 The display comprises 36 distinct patches with varying contrast relative to each other, as mentioned above. According to an embodiment of the invention, for a specific set of primary color values ​​(e.g., yellow 1, yellow 2, red, and cyan), an average color value can first be calculated from all pixels (or any predetermined subset of all pixels) in each corresponding patch.

[0119] Therefore, continuing with this example, for such Figure 4 The pattern 340 shown can calculate 36 average yellow 1 (Y1) color values, 36 average yellow 2 (Y2) color values, 36 average red (R) values, and 36 average cyan (Cy) values. In this way, each patch is identified using the corresponding average red, two average yellow, and average cyan values. In other words, for each different part of the image of pattern 340 (e.g., different patches), the average color value of each of a set of different colors is calculated.

[0120] Next, color ratio data is calculated from each combination of these average color values. That is, the "per patch" color ratio is calculated by calculating the ratio of each combination of average colors in each corresponding patch to each other. For example, and continuing the previous example, color ratios can be calculated for Y1 / Y2, Y1 / R, Y1 / Cy, Y2 / R, Y2 / Cy, and R / Cy. Of course, it is alternatively possible to calculate a smaller number of color ratios. In any case, this may involve repeated color ratio measurements for each of multiple images at different light levels and different exposure values ​​of the image sensor 320 for pattern 340 to produce an image sensor performance dataset. As a result, in addition to the CDP values ​​discussed above, the image sensor performance dataset can further provide color ratio data at different light levels and exposure values.

[0121] Measuring color ratio values ​​in this manner across different light levels and exposure values ​​can be used to generate a 2D color ratio heatmap in a manner similar to that discussed above with respect to 2D CDP heatmaps. However, in this case, the 2D color ratio heatmap will show areas in the image where color response can vary. For an illustrative example, see below. Figures 12A to 12B For a specific patch in the image of pattern 340, the diagram illustrates two separate 2D color ratio heatmaps representing instance color ratio values ​​within the exposure value range (y-axis) and the light level range (x-axis). Therefore, the intersection of the exposure value and the light level in each 2D color ratio heatmap provides the resulting color ratio value between the two average colors of the patch.

[0122] For example, Figure 12A This represents an example 2D color ratio heatmap, where the intersection of exposure values ​​and light levels provides the resulting color ratio between two average yellow values ​​of the patch within both the exposure value range and the light level range. As another example, Figure 12B This represents an example 2D color ratio heatmap, where the intersection of exposure values ​​and light levels provides the resulting color ratio values ​​between the average yellow and red color values ​​of the patches across the exposure value range and light level range. Based on this information, the expected error in the color ratio can be observed to evaluate the color response of the image sensor 320.

[0123] To provide additional examples, please refer to [link / reference]. Figures 13A to 13B .like Figure 13A and Figure 13B Each of the four 2D color ratio heatmaps shown represents a different color ratio value mapping, where each map represents a different color ratio value across different exposure and light level ranges. For example... Figure 13A The 2D colorimetric thermal image shown corresponds to the first 8.3MP automotive image sensor, while... Figure 13B The 2D color ratio heatmaps shown correspond to the second 8.3MP automotive image sensor. Therefore, the comparison of these 2D maps illustrates that the median color ratios of the two sensors are similar, but the color response varies differently between the sensors under different light levels. In the examples shown in Figures 12 and 13, sensors using small and large photodiodes within the pixel structure (referred to as "segmented pixel photodiodes") demonstrate the variation in color response (measured by color ratio) at points in the HDR response, where the large photodiode (more sensitive) is saturated, while the smaller photodiode (with a different color response) is the primary detector within the pixel structure. Such results can be particularly useful, for example, to identify locations where subtle (but noticeable) variations in color detection may exist within the camera's response range.

[0124] Therefore, process flow 500 describes the overall process for generating an image sensor performance profiling dataset for each image sensor profiled in this manner. The image sensor performance profiling dataset can then be used in various ways, as discussed in further detail herein. For example, any portion of the image sensor performance profiling dataset, as discussed herein, can be compared with a corresponding set of predetermined thresholds that can be used as a portion of the image sensor testing and / or certification process. For example, a set of reference 2D CDP heatmaps, reference 2D color ratio heatmaps, etc., can be generated as tabulated reference data points with various exposure values, light levels, contrast targets, confidence intervals, temperatures, etc. Data points from these reference 2D maps can represent thresholds for the expected performance of a particular image sensor and can be compared with 2D heatmaps obtained from the image sensor performance profiling dataset. In this way, the operation of the image sensor can be verified by comparing empirical data derived from the image sensor performance profiling dataset with such reference datasets before the image sensor is installed in and used in the vehicle. Furthermore, the image sensor performance profiling dataset can be implemented according to various applications in vehicle AV / ADAS systems, as discussed in further detail below.

[0125] VII. Applying Image Sensor Performance Profiling Datasets to AV / ADAS Applications

[0126] Furthermore, an image sensor model can be generated from an image sensor performance profiling dataset, and this model can be implemented as part of the safety system 200 of vehicle 100. Although a single image sensor model is described herein, it should be understood that the image sensor model described herein may include several image sensor models that form part of the overall image sensor model construction. For example, a separate image sensor model can be generated from an image sensor performance profiling dataset for a range of light levels and / or exposure values, said data being measured within said light levels and / or exposure values ​​to generate the image sensor performance profiling dataset.

[0127] Image sensor 320, along with its corresponding image sensor model and / or corresponding image sensor performance profiling dataset, may also be deployed as part of vehicle 100. The image sensor model and / or corresponding image sensor performance profiling dataset can be used by the AV / ADAS system of vehicle 100 to modify various vehicle-based operations in conjunction with images acquired via image sensor 320. For example, the AV / ADAS system of vehicle 100 may utilize, for instance, the safety system 200 and / or its components as described above for identification. Therefore, the AV / ADAS system may be referred to herein as performing various vehicle-based functions, which may include performing CV algorithms (which may include machine vision-based object or feature classification processes) and / or performing other suitable functions that control one or more processes performed by vehicle 100. This can be achieved, for example, via one or more processors 102 of safety system 200 executing instructions stored in suitable memory, such as the local memory of one or more processors 102 mentioned above, one or more memories 202, etc.

[0128] Therefore, the AV / ADAS system of vehicle 100 can receive images generated by image sensor 320, the output of the image sensor model, and any images and / or other data generated by additional sensors (e.g., lidar, radar, etc.) that may be implemented as part of safety system 200 during operation of vehicle 100, and use these various inputs to perform any suitable vehicle-based functions, as discussed in further detail herein. To this end, the AV / ADAS system can transmit controls, instructions, commands, etc., to any suitable component of vehicle 100, which can then control one or more operations in vehicle-based operation according to the embodiments discussed herein.

[0129] For example, image sensor 320 may be implemented as one of image acquisition devices 104, as discussed above with respect to vehicle 100. As part of this implementation, an image sensor model may have the same inputs as image sensor 320 during operation, and the output of the image sensor may be used as a predicted value; the AV / ADAS system compares these outputs with the actual outputs of image sensor 320 during operation. As another example, the image sensor model may generate simulated data for specific operating conditions that match the current operating conditions of image sensor 320 during operation, and the AV / ADAS system may then compare the simulation results with the output of image sensor 320. Therefore, for example, the comparison between the image sensor model output and the actual output of image sensor 320 via the AV / ADAS system may be performed using the output of an image sensor model having the same operating conditions as the current operation of image sensor 320 (e.g., the same light level and / or exposure value).

[0130] In any case, comparing the output of the image sensor model with the output of image sensor 320 can be used to perform or modify the performance of vehicle-based operations of vehicle 100. For example, if the output of the image sensor model and the output of image sensor 320 deviate from each other by more than a corresponding predetermined threshold relative to one or more metrics (e.g., CDP, color value ratio, etc.), the AV or ADAS system can trigger a specific vehicle-based operation, which may include generating an error or warning, issuing audible and / or visual alarms in the vehicle, etc. Additional examples and scenarios of how vehicle-based operations can be modified are discussed below. As an illustrative example, it is contemplated that a roadbook model implemented via the ADAS / AV system of vehicle 100 detects the road ahead of vehicle 100. The sensor model can indicate that the road should be detectable for a given sensor exposure level and input light level. Therefore, it can be determined that failure to detect the road surface should trigger a warning regarding the functionality of the sensors in the camera.

[0131] Furthermore, embodiments include modifying vehicle-based operations using an image sensor model and / or an image sensor performance profiling dataset, which may be done, for example, in response to the output of the image sensor model and / or a comparison of the image sensor model's output with an image output via image sensor 320. For example, during operation of vehicle 100, image sensor 320 may acquire images of the environment, which may be used according to any suitable machine vision-based object or feature classification algorithm implemented via safety system 200, as discussed herein. Embodiments include modifying any suitable portion of this machine vision-based object or feature classification process based on a comparison of the conditions under which image sensor 320 acquires images during operation with the output of an image sensor model for the same conditions (e.g., the same light level and / or exposure value).

[0132] As an example, the image acquired via image sensor 320 may be a specific scene representing a view of the environment when vehicle 100 is navigating. Therefore, the image acquired via image sensor 320 can be used to perform object or feature classification about this scene. This may include, for example, classifying road objects such as pedestrians or other vehicles, identifying road signs or road features, etc.

[0133] Continuing this example, as part of this classification process, the color values ​​of the acquired image can be used as part of a CV algorithm implemented by the AV / ADAS system to identify specific types of traffic signals (e.g., green, yellow, and red stop lights). Furthermore, since the image sensor performance profiling dataset contains data measured over various regions of the acquired image (e.g., patches of pattern 340), the output of the image sensor model can also be analyzed regarding specific locations within the image acquired by image sensor 320. Thus, the CV algorithm can initially identify red light at specific regions within the image acquired by image sensor 320. However, it is now assumed that a comparison between the image sensor model and the output of image sensor 320 indicates that, for this same region of the acquired image, with the same light level and exposure value, one or more color ratio values ​​deviate from a predetermined threshold color ratio value. In this scenario, the AV / ADAS system can issue a warning, switch from autonomous control of the vehicle to manual control, etc.

[0134] As another example, it should be noted that when color ratio data is included in an image sensor performance profiling dataset, this dataset again contains color ratio values ​​mapped to different regions of the image, both exposure and light levels. This allows the computing device 301 or the AV / ADAS system (depending on the specific case) to generate a mapping of color accuracy to each exposure and light level. Therefore, the AV / ADAS system (e.g., via its implemented CV algorithm) can use this color ratio mapping data to adjust the confidence level of color detection determination. For example, the AV / ADAS system can verify from the output of the image sensor model whether the expected error of a set of current operating parameters (e.g., exposure and light levels) is outside a predetermined color ratio error range that the CV algorithm can use to classify colors. Therefore, when the measurement error is outside this predetermined color ratio error range, the CV algorithm can classify the color detection as a failure.

[0135] Vehicle-based functions performed by the AV / ADAS system of vehicle 100 can be triggered in response to any suitable event or decision determined from the output of the image sensor performance profiling dataset and / or the image sensor model, as discussed herein. For example, as mentioned above, the AV / ADAS system of vehicle 100 can determine from the output of the image sensor model that, for the current exposure value and light level, the measurement error is outside a predetermined color ratio error range. Therefore, the AV / ADAS system can predict that subsequent color classification may fail, and the AV / ADAS system can modify the exposure value implemented via image sensor 320 for subsequent image acquisition. This adjustment may include, for example, changing the exposure value of image sensor 320 to a level where the object can then be measured with a lower (e.g., acceptable) color ratio error based on the adjusted parameters.

[0136] As another example, computing device 301 or AV / ADAS system (depending on the specific application) can generate a mapping of CDP data values ​​to each exposure and light level, as mentioned above. Therefore, the AV / ADAS system can also modify vehicle-based functions based on the output of the image sensor model; in this example, the AV / ADAS system can identify the CDP at the current exposure and light level. The AV / ADAS system (e.g., a CV algorithm) can then adjust the probability of classifying a specific object detected from an image acquired via image sensor 320 at the same exposure value and light level implemented by the image sensor model.

[0137] As an illustrative example, the CV algorithm can reduce the likelihood of classifying features and / or objects (e.g., road backgrounds, such as road textures or objects) that rely more heavily on distinguishing contrast values ​​in an image and therefore require a specific threshold CDP to ensure this result. Furthermore, as mentioned above regarding the use of color ratio data, the CV algorithm can additionally or alternatively adjust one or more parameters (e.g., exposure settings) used by the image sensor 320 to obtain subsequent images for this purpose, such that the classification of the road background or object now appears within a range measured to have a lower error, as indicated in the image sensor performance profiling dataset.

[0138] Furthermore, since CDP data is available across a wide range of exposure settings and light levels, the output of the image sensor model can be particularly useful for identifying acquired images that are more suitable for classification and detection via CV algorithm analysis. For example, image sensor 320 may generate multiple images, each with different exposure values, as is typically the case with HDR imagers. Alternatively, image sensor 320 may come from several image sensors in vehicle 100, each providing images to an AV / ADAS system. In any case, such images can be acquired at different exposure values, and the AV / ADAS system can receive these images simultaneously or nearly simultaneously. In this scenario, the AV / ADAS system can use the output of the image sensor model to determine which image has a higher CDP and / or lower color error for each image with different exposure values. In response, the AV / ADAS system can then select the image with the exposure value associated with the acquired image identified from the image sensor model output, which has a preferred metric (e.g., higher CDP and / or lower color error), for use by the CV algorithm.

[0139] As another illustrative example, CDP data may be implemented additionally or alternatively as part of redundant features that may exist in vehicle 100. For example, the safety system 200 of vehicle 100 may include a safety architecture based on an automated driving system (SDS). However, as with all complex systems, such safety architectures may be susceptible to failures related to hardware and software defects, perception failures, planning failures, actuation failures, adversarial interruptions, and so on.

[0140] To address such failures, safety system 200 may include a primary-monitor-backup (PGF) fusion system, which may implement any suitable number of different independent or overlapping sensor systems. The outputs of these sensor systems can be used for CV algorithm processing and / or determination of various vehicle-based functions to be performed by vehicle 100, such as object detection, classification, trajectory planning, actuation, and other vehicle-based functions discussed herein. For example, the primary system of safety system 200 may implement one or more lidar sensors 112, while the backup system of safety system 200 may implement one or more radar sensors 110. Of course, this is merely one example, and in various embodiments, the primary and backup systems may implement different sensors, which may be independent of or overlap with each other.

[0141] In any case, the primary and backup systems of security system 200 can output independent data streams, and security system 200 can also implement a separate model (e.g., SDM) for each data stream / system. Furthermore, as part of this PGF fusion system, the monitoring system can function as a binary system, outputting a Boolean value indicating whether the output of the primary or backup system is the "better" choice. For example, if the primary system output is better than the backup system output, the monitoring system can output '1'; otherwise, it outputs '0'. Therefore, the output of the monitoring system controls which output and / or model security system 200 uses. In this context, "better" can be determined based on any suitable metric that can be detected, measured, identified, etc., in the primary and backup systems. This may include, for example, identifying faults and / or errors in one of the primary or backup systems, determining that one of the primary or backup systems is outputting data that produces inaccurate or inconsistent results, etc. Alternatively or alternatively (which may include the use of a PGF system or an alternative redundant system), comparisons can be made between two data streams, where consistency between these different sensors (and optional other data sources, such as REM map data) is used to make decisions and / or determine vehicle-based functions.

[0142] As part of the operation of the PGF or another alternative redundant system, embodiments include utilizing CDP data from one or more sensors within the primary and / or backup systems to facilitate determination of whether the output of the primary or backup system should be implemented. For example, when the primary system does not provide satisfactory results from one or more lidar sensors 112 (e.g., based on comparisons with predetermined thresholds, output history, etc.), the PGF system may rely on the output of the backup system utilizing one or more lidar sensors 100, as mentioned above. For alternative redundant systems, CDP data from one or more sensors may be implemented to determine whether there is consistency between the outputs of each system.

[0143] As another example, the output of an image sensor model can be used to perform specific, predetermined vehicle-based functions in response to predicted values ​​obtained from an image sensor performance profiling dataset. For instance, one or more parameters of image sensor 320 (e.g., temperature, exposure settings, light level, etc.) can be used to determine the corresponding CDP, color ratio, etc., of an image acquired by image sensor 320 based on these same parameters. Therefore, continuing this example, when a CDP less than a predetermined threshold is predicted for an image acquired via image sensor 320 in this manner, the AV / ADAS system can identify this CDP as too low to be used for one or more predetermined key use cases for images acquired under these conditions. For example, the AV / ADAS system can determine that for a specific combination of exposure value and light level, the resulting CDP is too low, making it impossible to accurately identify pedestrians under these conditions. In this scenario, the AV / ADAS system can enable vehicle 100 to perform specific vehicle-based functions to avoid using CV algorithms that rely on such acquired images. This may include, for example, stopping the car at the side of the road, instructing the driver to take over manual control of the vehicle, ignoring images acquired via image sensor 320, and alternatively relying on other sensor inputs such as lidar and / or radar for use by the CV algorithm.

[0144] Furthermore, the AV / ADAS system can collect statistical data during operation, which may include data about various objects within the scene and / or at specific geographic locations. The AV / ADAS can compare this statistical data with the output of the image sensor model to identify or predict faults in the image sensor 320. For example, the AV / ADAS system can aggregate and store statistical data about regions with known identities within the acquired image, which can be determined, for example, using road map data of locations as discussed above. The AV / ADAS system can then, for example, compare any suitable known metrics obtained in this way (e.g., color ratios of objects in the scene) with color ratio data obtained from a CV algorithm to identify whether a camera malfunction has occurred.

[0145] As an illustrative example, suppose road map data indicates the location of an upcoming red traffic light, and then the location of the traffic light is identified within an image acquired via image sensor 320 (or another sensor). The AV / ADAS system can then measure the color ratio of the pixels within the traffic light to determine if the color ratio matches the expected red value. If a predetermined number of such pixels correspond to color information that is not aligned with red (or a high color ratio error for red is detected for this predetermined number of pixels), the AV / ADAS system can determine that image sensor 320 is operating incorrectly. In response, the AV / ADAS system can perform one or more vehicle-based functions, such as disabling image sensor 320 and / or taking corrective actions, such as pulling the vehicle to the side of the road, instructing the driver to take manual control of the vehicle, etc., to prevent such erroneous image data from causing unsafe conditions.

[0146] As another illustrative example, one or more vehicle-based functions may include adjusting the machine vision-based object or feature classification process of the CV algorithm to reduce the classification probability of traffic signals. Furthermore, it should be noted that the image sensor model used for this comparison can be generated under the same conditions under which the image sensor 320 operates when acquiring such images. Therefore, the machine vision-based object or feature classification process can be adjusted in this way based on the position of the object or feature in the acquired image and other parameters common to both the image sensor model and the image sensor 320 (e.g., the exposure value and / or light level of the image sensor 320 when acquiring the image).

[0147] Alternatively, the machine vision-based object or feature classification process of the CV algorithm can be adjusted to prevent the detection of features and / or objects in the acquired image and / or the classification of features and / or objects in the acquired image when a deviation exceeding a predetermined threshold is detected between the image sensor model and the output of the image sensor 320. For example, and using the above example with color ratio values, the CV of the AV or ADAS system of vehicle 100 can avoid using the acquired image data and / or regions within the acquired image data to detect traffic lights, because predicting such classifications would be unreliable.

[0148] Alternatively, if the discrepancy between the identified image sensor model and the output of image sensor 320 exceeds a threshold CDP, the CV of the AV or ADAS system of vehicle 100 can avoid using the acquired image data and / or regions within the acquired image data to detect features that rely on contrast differentiation. This implementation may be particularly useful, for example, in preventing false alarms detected via the AV / ADAS system.

[0149] As another illustrative example, instead of using color information as described above, the collected statistics can be CDP data. In other words, the AV / ADAS system can compare the expected contrast detection of an object in a scene with the CDP probability derived from the image sensor model output and / or from the image sensor performance profiling dataset. For example, different sensors (e.g., radar) can identify objects in a scene expected to have high contrast (e.g., another vehicle in front of vehicle 100). The AV / ADAS system can then calculate the difference between the average pixels of this other vehicle and the pixels outside the other vehicle to calculate the contrast between the two. Then, if the contrast is less than a predetermined threshold, and the image sensor model output for the same corresponding conditions (e.g., the same light level, exposure value, temperature, etc.) indicates that the expected CDP is higher than this predetermined threshold, the AV / ADAS system can perform one or more vehicle-based functions. This may include, for example, disabling image sensor 320 and / or taking corrective actions, such as, for example, pulling the car to the side of the road, instructing the driver to take manual control of the vehicle, etc., to avoid such erroneous image data leading to unsafe conditions.

[0150] Alternatively, when a deviation exceeding a predetermined threshold is detected between the image sensor model and the output of image sensor 320, the AV / ADAS system can modify vehicle-based operation by modifying the sensor weighting system implemented by the AV / ADAS system. For example, image sensor 320 may come from a series of sensors, which may include additional image sensors and other sensor types (e.g., radar, lidar, etc.) implemented by the AV / ADAS system to detect and / or classify objects during vehicle operation. This "sensor fusion" technique may involve applying different weights to images and / or data acquired via each of these sensors to perform such determinations. For example, an image acquired via a forward-facing camera may be given more weight than one acquired via a side-facing camera for detecting nearby objects, since the latter acquires images that are more easily blurred. Therefore, when a deviation exceeding a predetermined threshold is detected between the image sensor model and the output of image sensor 320, the AV / ADAS system can modify vehicle-based operation by further reducing the weighting of the images acquired via image sensor 320 compared to other sensors in vehicle 100 when performing machine vision object or feature classification processes.

[0151] Alternatively or, as mentioned above, sensor fusion techniques (e.g., the PGF system mentioned above) can utilize redundancy methods. Such redundancy methods can, for example, implement data streams received from different sensors, where consistency between these different sensors (and optionally other data sources, such as REM map data) is used to make decisions and / or determine vehicle-based functions. In other cases, such as when safety constraints are involved, a single input type can be utilized according to this redundancy method. In any case, when utilizing such redundancy methods, the AV / ADAS system can compare any suitable metrics derived from CDP data to determine whether consistency exists between redundant systems and / or which redundant system output should be used with the CV algorithm. For example, for each individual system, safety system 200 can compare the expected contrast detection of objects in the scene with CDP probabilities derived from a specific image sensor model output and / or from an image sensor performance profiling dataset. This data can then be used to identify which redundant system is more reliable and / or whether consistency is achieved between two different systems, depending on the specific circumstances.

[0152] As another example, when a deviation exceeding a predetermined threshold is detected between the image sensor model and the output of image sensor 320, the AV / ADAS system can modify vehicle-based operation by adjusting the parameters of the Safe Driving Model (SDM). Furthermore, the SDM parameters of vehicle 100 can be implemented by the vehicle to navigate the environment, and therefore parameters such as minimum following distance, maximum lateral distance, and maximum speed can be defined. Therefore, modifications to vehicle-based operation can include, for example, adjusting one or more of these SDM parameters to compensate for unreliability in the image currently acquired via image sensor 320. This could include temporarily increasing the minimum following distance, decreasing the maximum speed, etc.

[0153] Example

[0154] The following examples involve other aspects.

[0155] Example (e.g., Example 1) relates to a method for profiling an image sensor for use in a vehicle, comprising: receiving from the image sensor a plurality of images containing a predetermined pattern; controlling a light source to adjust illumination of the predetermined pattern at a plurality of different light levels; controlling the image sensor to adjust exposure values; performing a contrast detection probability (CDP) measurement on each of the plurality of images according to a predetermined contrast target and a predetermined confidence interval, wherein the CDP measurement is repeated for each of the plurality of images for different light levels of the predetermined pattern and different exposure values ​​of the image sensor to generate an image sensor performance profiling dataset, the image sensor performance profiling dataset providing CDP measurements for different exposure values ​​and different light levels; using the image sensor performance profiling dataset to generate an image sensor model; and modifying vehicle-based operations based on the output of the image sensor model.

[0156] Another example (e.g., Example 2) relates to a previously described example (e.g., Example 1), wherein the image sensor is configured to acquire one or more images during operation of the vehicle, and wherein the modification of the vehicle-based operation includes adjusting a machine vision-based object or feature classification process performed using the acquired one or more images.

[0157] Another example (e.g., example 3) relates to the previously described examples (e.g., any combination of one or more of examples 1 to 2), wherein the adjustment of the machine vision-based object or feature classification process is based on a comparison of the output of the image sensor model with the output of the image sensor model under the same conditions.

[0158] Another example (e.g., example 4) relates to the previously described examples (e.g., any combination of one or more of examples 1 to 3), wherein the adjustment of the machine vision-based object or feature classification process is based on the position of the object or feature in the acquired one or more images, the exposure value of the image sensor when the one or more images are acquired, and the light level when the one or more images are acquired.

[0159] Another example (e.g., example 5) relates to the previously described examples (e.g., any combination of one or more of examples 1 to 4), wherein the modification of the vehicle-based operation includes modifying the weighting of one or more images acquired by the image sensor during vehicle operation compared with other sensors in the vehicle to perform a machine vision object or feature classification process.

[0160] Another example (e.g., example 6) relates to the previously described examples (e.g., any combination of one or more of examples 1 to 5), wherein the modification of the vehicle-based operation includes modifying parameters of the Safe Driving Model (SDM) parameters implemented by the vehicle to navigate the environment.

[0161] Another example (e.g., example 7) relates to the previously described examples (e.g., any combination of one or more of examples 1 to 6), wherein the CDP measurement is repeated for each of a plurality of different predetermined confidence intervals to generate an image sensor performance dataset to provide CDP measurements for different light levels, exposure values, and confidence intervals.

[0162] Another example (e.g., example 8) relates to the previously described examples (e.g., any combination of one or more of examples 1 to 7), wherein the CDP measurement is repeated for each of a plurality of different predetermined target contrasts to generate the image sensor performance dataset to provide CDP measurements for different light levels, exposure values, confidence intervals, and target contrasts.

[0163] Another example (e.g., example 9) relates to the previously described examples (e.g., any combination of one or more of examples 1 to 8), wherein the contrast detection probability (CDP) measurement is performed by selecting a subset of pixel pair combinations within each of the plurality of images, the subset of pixel pair combinations having a brightness level difference relative to each other within a predetermined threshold deviation of the predetermined contrast target.

[0164] Another example (e.g., example 10) relates to the previously described examples (e.g., any combination of one or more of examples 1 to 9), further comprising: performing a color ratio measurement on each of the plurality of images by: calculating the average color value of pixels contained within different portions of the predetermined pattern to provide an average color value of each of a set of different colors for each of the different portions, and calculating the ratio of each combination of the average color values ​​of each of the set of different colors; and repeating the color ratio measurement on each of the plurality of images for different light levels of the predetermined pattern and different exposure values ​​of the image sensor to generate the image sensor performance dataset to further provide color ratio data for different light levels and exposure values.

[0165] Example (e.g., Example 11) relates to a computing device for profiling an image sensor for use in a vehicle, comprising: a memory configured to store instructions; and a processing circuitry system configured to execute the instructions stored in the memory such that the computing device: receives from the image sensor a plurality of images comprising a predetermined pattern; controls a light source to adjust illumination of the predetermined pattern at a plurality of different light levels; controls the image sensor to adjust exposure values; performs a contrast detection probability (CDP) measurement on each of the plurality of images according to a predetermined contrast target and a predetermined confidence interval, wherein the CDP measurement is repeated for each of the plurality of images for different light levels of the predetermined pattern and different exposure values ​​of the image sensor to generate an image sensor performance profiling dataset, the image sensor performance profiling dataset providing CDP measurements for different exposure values ​​and different light levels; and uses the image sensor performance profiling dataset to generate an image sensor model, wherein the image sensor model is configured to output data to modify vehicle-based operation.

[0166] Another example (e.g., example 12) relates to a previously described example (e.g., example 11), wherein the image sensor is configured to acquire one or more images during operation of the vehicle, and wherein the modification of the vehicle-based operation includes adjusting a machine vision-based object or feature classification process performed using the acquired one or more images.

[0167] Another example (e.g., example 13) relates to the previously described examples (e.g., any combination of one or more of examples 11 to 12), wherein the adjustment of the machine vision-based object or feature classification process is based on a comparison of the output of the image sensor model with the output of the image sensor model under the same conditions.

[0168] Another example (e.g., example 14) relates to the previously described examples (e.g., any combination of one or more of examples 11 to 13), wherein the adjustment of the machine vision-based object or feature classification process is based on the position of the object or feature in the acquired one or more images, the exposure value of the image sensor when the one or more images are acquired, and the light level when the one or more images are acquired.

[0169] Another example (e.g., example 15) relates to the previously described examples (e.g., any combination of one or more of examples 11 to 14), wherein the modification of the vehicle-based operation includes modifying the weighting of one or more images acquired by the image sensor during vehicle operation compared with other sensors in the vehicle to perform a machine vision object or feature classification process.

[0170] Another example (e.g., example 16) relates to the previously described examples (e.g., any combination of one or more of examples 11 to 15), wherein the modification of the vehicle-based operation includes modifying parameters of the Safe Driving Model (SDM) parameters implemented by the vehicle to navigate the environment.

[0171] Another example (e.g., example 17) relates to the previously described examples (e.g., any combination of one or more of examples 11 to 16), wherein the processing circuitry is configured to repeat the CDP measurement for each of a plurality of different predetermined confidence intervals to generate an image sensor performance dataset to provide CDP measurements for different light levels, exposure values, and confidence intervals.

[0172] Another example (e.g., example 18) relates to the previously described examples (e.g., any combination of one or more of examples 11 to 17), wherein the processing circuitry is configured to repeat the CDP measurement for each of a plurality of different predetermined target contrasts to generate the image sensor performance dataset to provide CDP measurements for different light levels, exposure values, confidence intervals, and target contrasts.

[0173] Another example (e.g., example 19) relates to the previously described examples (e.g., any combination of one or more of examples 11 to 18), wherein the processing circuitry is configured to perform the contrast detection probability (CDP) measurement by selecting a subset of pixel pair combinations within each of the plurality of images, the subset of pixel pair combinations having a brightness level difference relative to each other within a predetermined threshold deviation of the predetermined contrast target.

[0174] Another example (e.g., example 20) relates to the previously described examples (e.g., any combination of one or more of examples 11 to 19), wherein the processing circuitry is configured to perform a color ratio measurement on each of the plurality of images by: calculating the average color value of pixels contained within different portions of the predetermined pattern to provide an average color value of each of a set of different colors for each of the different portions, and calculating the ratio of each combination of the average color values ​​of each of the set of different colors; and repeating the color ratio measurement on each of the plurality of images for different light levels of the predetermined pattern and different exposure values ​​of the image sensor to generate the image sensor performance dataset to further provide color ratio data for different light levels and exposure values.

[0175] A device, as shown and described.

[0176] One method, as shown and described.

[0177] in conclusion

[0178] The foregoing description of specific aspects so fully discloses the general nature of this disclosure that, without requiring excessive experimentation and without departing from the general concept of this disclosure, others can readily modify and / or adapt various applications of such specific aspects by applying knowledge within the art. Therefore, based on the teachings and guidance given herein, such adaptations and modifications are intended to fall within the meaning and scope of equivalents of the disclosed aspects. It should be understood that the wording or terminology used herein is for descriptive rather than limiting purposes, and that the terminology or terminology of this specification will be interpreted by those skilled in the art in light of the teachings and guidance.

[0179] The references to "an aspect," "one aspect," "an exemplary aspect," etc., in this specification indicate that the described aspect may include a particular feature, structure, or characteristic, but each aspect need not necessarily include the stated particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same aspect. Additionally, when an aspect is used to describe a particular feature, structure, or characteristic, it should be understood that, whether explicitly described or not, the combination of other aspects affecting that feature, structure, or characteristic is within the knowledge of those skilled in the art.

[0180] The exemplary aspects described herein are provided for illustrative purposes and are not intended to be limiting. Other exemplary aspects are possible, and modifications may be made to the exemplary aspects. Therefore, this specification is not intended to limit this disclosure. Rather, the scope of this disclosure is defined only by the appended claims and their equivalents.

[0181] The aspect can be implemented in hardware (e.g., circuitry), firmware, software, or any combination thereof. The aspect can also be implemented as instructions stored on a machine-readable medium, which can be read and executed by one or more processors. The machine-readable medium can contain any mechanism for storing or transmitting information in a machine-readable form (e.g., a computing device). For example, a machine-readable medium can include read-only memory (ROM); random access memory (RAM); disk storage media; optical storage media; flash memory devices; electrical, optical, acoustic, or other forms of propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.). Furthermore, firmware, software, routines, and instructions can be described herein as performing certain actions. However, it should be understood that such descriptions are merely for convenience, and such actions are actually generated by a computing device, processor, controller, or other means of executing firmware, software, routines, instructions, etc. Additionally, any of the embodiments can be implemented by a general-purpose computer.

[0182] For the purposes of this discussion, the term "processor circuit system" or "processor circuit system" should be understood as one or more circuits, one or more processors, logic, or combinations thereof. For example, a circuit may include analog circuits, digital circuits, state machine logic, other structured electronic hardware, or combinations thereof. A processor may include a microprocessor, a digital signal processor (DSP), or other hardware processor. A processor may be "hard-coded" with instructions to perform one or more corresponding functions according to the aspects described herein. Alternatively, a processor may access internal and / or external memory to retrieve instructions stored in memory that, when executed by the processor, perform one or more corresponding functions associated with the processor, and / or one or more functions and / or operations related to the operation of components containing the processor.

[0183] In one or more of the exemplary aspects described herein, the processing circuitry system may include memory for storing data and / or instructions. The memory may be any well-known volatile and / or non-volatile memory, including, for example, read-only memory (ROM), random access memory (RAM), flash memory, magnetic storage media, optical discs, erasable programmable read-only memory (EPROM), and programmable read-only memory (PROM). The memory may be non-removable, removable, or a combination of both.

Claims

1. A method for profiling an image sensor used in a vehicle, comprising: Receive multiple images containing a predetermined pattern from an image sensor; The light source is controlled to adjust the illumination of the predetermined pattern at multiple different light levels; Control the image sensor to adjust the exposure value; Perform a contrast detection probability (CDP) measurement on each of the plurality of images according to a predetermined contrast target and a predetermined confidence interval. The CDP measurement is repeated for each of the plurality of images for different light levels of the predetermined pattern and different exposure values ​​of the image sensor to generate an image sensor performance profiling dataset, which provides CDP measurements for different exposure values ​​and different light levels. The image sensor performance profiling dataset is used to generate an image sensor model; as well as The vehicle-based operation is modified based on the output of the image sensor model.

2. The method of claim 1, wherein the image sensor is configured to acquire one or more images during operation of the vehicle, and The modification of the vehicle-based operation includes adjusting the machine vision-based object or feature classification process performed using one or more acquired images.

3. The method of claim 2, wherein the adjustment of the machine vision-based object or feature classification process is based on a comparison of the output of the image sensor model with the output of the image sensor model under the same conditions.

4. The method of claim 2, wherein the adjustment of the machine vision-based object or feature classification process is based on the position of the object or feature in the acquired one or more images, the exposure value of the image sensor when the one or more images are acquired, and the light level when the one or more images are acquired.

5. The method according to any one of claims 1 to 4, wherein the modification of the vehicle-based operation comprises modifying the weighting of one or more images acquired by the image sensor during vehicle operation compared with other sensors in the vehicle to perform a machine vision object or feature classification process.

6. The method according to any one of claims 1 to 4, wherein modifying the vehicle-based operation includes modifying parameters of a Safe Driving Model (SDM) parameter implemented by the vehicle to navigate the environment.

7. The method according to any one of claims 1 to 4, wherein the CDP measurement is repeated for each of a plurality of different predetermined confidence intervals to generate an image sensor performance dataset to provide CDP measurements for different light levels, exposure values ​​and confidence intervals.

8. The method of claim 7, wherein the CDP measurement is repeated for each of a plurality of different predetermined target contrasts to generate the image sensor performance dataset, to provide CDP measurements for different light levels, exposure values, confidence intervals, and target contrasts.

9. The method according to any one of claims 1 to 4, wherein the contrast detection probability (CDP) measurement is performed by selecting a subset of pixel pair combinations within each of the plurality of images, the subset of pixel pair combinations having a brightness level difference relative to each other within a predetermined threshold deviation of the predetermined contrast target.

10. The method according to any one of claims 1 to 4, further comprising: Color ratio measurement is performed on each of the plurality of images by: calculating the average color value of the pixels contained in different portions of the predetermined pattern, providing an average color value of each of a set of different colors for each of the different portions, and calculating the ratio of each combination of the average color values ​​of each of the set of different colors. as well as The color ratio measurement is repeated for each of the plurality of images for different light levels and different exposure values ​​of the image sensor for the predetermined pattern to generate the image sensor performance dataset, in order to further provide color ratio data for different light levels and exposure values.

11. A computing device for profiling an image sensor used in a vehicle, comprising: A memory configured to store instructions; as well as A processing circuitry system configured to execute the instructions stored in the memory to cause the computing device to: Receive multiple images containing a predetermined pattern from an image sensor; The light source is controlled to adjust the illumination of the predetermined pattern at multiple different light levels; Control the image sensor to adjust the exposure value; Perform a contrast detection probability (CDP) measurement on each of the plurality of images according to a predetermined contrast target and a predetermined confidence interval. The CDP measurement is repeated for each of the plurality of images for different light levels of the predetermined pattern and different exposure values ​​of the image sensor to generate an image sensor performance profiling dataset, which provides CDP measurements at different exposure values ​​and different light levels; and The image sensor performance profiling dataset is used to generate an image sensor model; The image sensor model is configured to output data to enable modifications to vehicle-based operations.

12. The computing device of claim 11, wherein the image sensor is configured to acquire one or more images during operation of the vehicle, and The modification of the vehicle-based operation includes adjusting the machine vision-based object or feature classification process performed using one or more acquired images.

13. The computing device of claim 12, wherein the adjustment of the machine vision-based object or feature classification process is based on a comparison of the output of the image sensor model with the output of the image sensor model under the same conditions.

14. The computing device of claim 12, wherein the adjustment of the machine vision-based object or feature classification process is based on the position of the object or feature in the acquired one or more images, the exposure value of the image sensor when the one or more images are acquired, and the light level when the one or more images are acquired.

15. The computing device of claim 11, wherein the modification of the vehicle-based operation comprises modifying the weighting of one or more images acquired by the image sensor during vehicle operation compared with other sensors in the vehicle to perform a machine vision object or feature classification process.

16. The computing device according to any one of claims 11 to 14, wherein modifying the vehicle-based operation includes modifying parameters of a Safe Driving Model (SDM) parameter implemented by the vehicle to navigate the environment.

17. The computing device according to any one of claims 11 to 14, wherein the processing circuitry is configured to repeat the CDP measurement for each of a plurality of different predetermined confidence intervals to generate an image sensor performance dataset, providing CDP measurements for different light levels, exposure values, and confidence intervals.

18. The computing device of claim 17, wherein the processing circuitry is configured to repeat the CDP measurement for each of a plurality of different predetermined target contrasts to generate the image sensor performance dataset, to provide CDP measurements for different light levels, exposure values, confidence intervals, and target contrasts.

19. The computing device according to any one of claims 11 to 14, wherein the processing circuitry is configured to perform the contrast detection probability (CDP) measurement by selecting a subset of pixel pair combinations within each of the plurality of images, the subset of pixel pair combinations having a brightness level difference relative to each other within a predetermined threshold deviation of the predetermined contrast target.

20. The computing device according to any one of claims 11 to 14, wherein the processing circuitry is configured to: Color ratio measurement is performed on each of the plurality of images by: calculating the average color value of pixels contained in different portions of the predetermined pattern, providing an average color value of each of a set of different colors for each of the different portions, and calculating the ratio of each combination of the average color values ​​of each of the set of different colors; and The color ratio measurement is repeated for each of the plurality of images for different light levels and different exposure values ​​of the image sensor for the predetermined pattern to generate the image sensor performance dataset, in order to further provide color ratio data for different light levels and exposure values.

Images