IMAGE SENSOR WITH RED-TRANSPARENT-TRANSPARENT-Green (RCCG) COLOR FILTER ARRAY (CFA) FOR VEHICLE SENSING APPLICATIONS

Abstract

This disclosure provides systems, methods, and apparatus for a vehicle driving assistance system that supports image processing for vehicle monitoring operations. In a first aspect, a method of image processing includes receiving image data from an image sensor including a red-transparent-transparent-green (RCCG) color filter, where the image data includes a first channel corresponding to red, a second channel corresponding to transparency, and a third channel corresponding to green, where the image data is in a logarithmic domain; and processing the image data by performing operations including linearization, tone mapping, demosaicing, and color mapping. Other aspects and features are also claimed and described.

Description

Technical Field

[0001] Aspects of the present disclosure relate generally to driver-operated vehicles, and more particularly, to methods and systems suitable for providing driver assistance or suitable for automated driving. Background Art

[0002] Vehicles come in many shapes and sizes, are propelled by a variety of propulsion technologies, and carry cargo, including people, animals, or objects. These machines are capable of moving cargo over long distances, moving cargo at high speeds, and moving cargo that is larger than human power can move. Vehicles were originally driven by humans to control the speed and direction of the cargo to its destination. Human operation of vehicles has led to many unfortunate accidents caused by vehicle-to-vehicle, vehicle-to-object, vehicle-to-human, or vehicle-to-animal collisions. As research into vehicle automation has progressed, a variety of driver assistance systems have been produced and introduced. These include navigation guidance via GPS, adaptive cruise control, lane change assistance, collision avoidance systems, night vision, parking assistance, and blind spot detection. Summary of the invention

[0003] The following summarizes some aspects of the present disclosure to provide a basic understanding of the technology discussed. This summary is not an exhaustive overview of all the expected features of the present disclosure, and is neither intended to identify the key or important elements of all aspects of the present disclosure, nor to delineate the scope of any or all aspects of the present disclosure. The sole purpose of this summary is to present some concepts of one or more aspects of the present disclosure in a summarized form as a prelude to more specific embodiments presented later.

[0004] A human operator of a vehicle may be distracted, which is a factor in many vehicle crashes. Driver distraction may include, among other things, changing radio components, observing events outside the vehicle, and using electronic devices. Sometimes situations create situations that even a careful driver cannot identify in time to prevent a vehicle crash. Aspects of the present disclosure provide improved systems for assisting a driver in a vehicle to enhance situational awareness while traveling on a road.

[0005] Example embodiments provide driver behavior monitoring systems and methods capable of detecting driver inattention, whether the inattention is the result of distraction or untrained vehicle operation. The driver behavior analysis results can be combined with other vehicle sensor information (e.g., engine information, speed sensor, location information, etc.) to determine the occurrence and location of accidents, near misses of accidents, and / or to score the performance of vehicle operators. The system can report accidents to authorities, emergency responders, or other nearby drivers based on the location of the car by utilizing global positioning system (GPS) information and / or wireless multicast technology.

[0006] In some embodiments, a red-clear-clear-green (RCCG) color filter array (CFA) may be included in a camera module of a vehicle system to improve the operation of a vehicle processing system for processing images of the vehicle's surroundings, and used in the example applications described above. The RCCG CFA may have improved performance in vehicle applications, such as improved accuracy, signal-to-noise ratio (SNR), and / or yellow-red discrimination, each of which may improve the accuracy of detecting traffic signs and traffic lights, among other features.

[0007] In one aspect of the present disclosure, a method for vehicle monitoring includes: receiving image data from an image sensor including a red-clear-clear-green (RCCG) filter, wherein the image data includes a first channel corresponding to red, a second channel corresponding to transparent, and a third channel corresponding to green, wherein the image data is in a logarithmic domain; and processing the image data by performing operations including linearization, tone mapping, demosaicing, and color mapping.

[0008] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations including: receiving image data from an image sensor including a red-clear-clear-green (RCCG) color filter, wherein the image data includes a first channel corresponding to red, a second channel corresponding to clear, and a third channel corresponding to green, wherein the image data is in a logarithmic domain; and processing the image data by performing operations including linearization, tone mapping, demosaicing, and color mapping.

[0009] In additional aspects of the present disclosure, an apparatus includes components for receiving image data from an image sensor including a red-clear-clear-green (RCCG) filter, wherein the image data includes a first channel corresponding to red, a second channel corresponding to transparent, and a third channel corresponding to green, wherein the image data is in a logarithmic domain; and components for processing the image data by performing operations including linearization, tone mapping, demosaicing, and color mapping.

[0010] In an additional aspect of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including: receiving image data from an image sensor including a red-clear-clear-green (RCCG) color filter, wherein the image data includes a first channel corresponding to red, a second channel corresponding to clear, and a third channel corresponding to green, wherein the image data is in a logarithmic domain; and processing the image data by performing operations including linearization, tone mapping, demosaicing, and color mapping.

[0011] The features and technical advantages of examples according to the present disclosure have been outlined quite extensively above so that the following specific embodiments may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be easily used as a basis for modifying or designing other structures for the same purpose of achieving the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. When considered in conjunction with the accompanying drawings, the characteristics of the concepts disclosed herein (both their organization and methods of operation) and the associated advantages will be better understood according to the following description. Each of the figures in the accompanying drawings is provided for the purpose of illustration and description, and not as a definition of limitations to the claims.

[0012] In various specific implementations, the techniques and devices may be used in wireless communication networks, such as code division multiple access (CDMA) networks, time division multiple access (TDMA) networks, frequency division multiple access (FDMA) networks, orthogonal FDMA (OFDMA) networks, single carrier FDMA (SC-FDMA)ng networks, LTE networks, GSM networks, fifth generation (5G) or new radio (NR) networks (sometimes referred to as "5G NR" networks, systems or devices), and other communication networks. As described herein, the terms "network" and "system" may be used interchangeably.

[0013] A CDMA network may implement, for example, a radio technology such as Universal Terrestrial Radio Access (UTRA), cdma2000, etc. UTRA includes Wideband-CDMA (W-CDMA) and Low Chip Rate (LCR). CDMA2000 covers IS-2000, IS-95, and IS-856 standards.

[0014] For example, a TDMA network can implement radio technologies such as the Global System for Mobile Communications (GSM). The Third Generation Partnership Project (3GPP) defines the standards for the GSM EDGE (Enhanced Data Rate GSM Evolution) Radio Access Network (RAN) (also known as GERAN). GERAN is the radio component of the network of GSM / EDGE together with connecting base stations (e.g., Ater and Abis interfaces) and base station controllers (A interfaces, etc.). The radio access network represents the component of the GSM network, through which phone calls and packet data are routed from the public switched telephone network (PSTN) and the Internet to subscriber phones (also known as user terminals or user equipment (UE)) and from subscriber phones to the PSTN and the Internet. The network of a mobile phone operator may include one or more GERANs, which may be coupled with UTRAN in the case of a UMTS / GSM network. Additionally, the operator network may also include one or more LTE networks, or one or more other networks. Various network types may use different radio access technologies (RATs) and RANs.

[0015] OFDMA networks can implement radio technologies such as Evolved UTRA (E-UTRA), Institute of Electrical and Electronics Engineers (IEEE) 802.11, IEEE 802.16, IEEE 802.20, flash-OFDM, etc. UTRA, E-UTRA, and GSM are part of the Universal Mobile Telecommunications System (UMTS). Specifically, Long Term Evolution (LTE) is a version of UMTS that uses E-UTRA. UTRA, E-UTRA, GSM, UMTS, and LTE are described in documents provided by an organization named "3rd Generation Partnership Project" (3GPP), and cdma2000 is described in documents from an organization named "3rd Generation Partnership Project 2" (3GPP2). 5G networks include diverse deployments, diverse spectrums, and diverse services and devices that can be implemented using a unified air interface based on OFDM.

[0016] The present disclosure may describe certain aspects with reference to LTE, 4G or 5G NR technology; however, the description is not intended to be limited to a particular technology or application, and one or more aspects described with reference to one technology may be understood to be applicable to another technology. Additionally, one or more aspects of the present disclosure may relate to shared access to wireless spectrum between networks using different radio access technologies or radio air interfaces.

[0017] Devices, networks, and systems may be configured to communicate via one or more portions of the electromagnetic spectrum. The electromagnetic spectrum is typically subdivided into various categories, bands, channels, etc. based on frequency / wavelength. In 5G NR, two initial operating bands have been identified as frequency ranges designated FR1 (410 MHz-7.125 GHz) and FR2 (24.25 GHz-52.6 GHz). Frequencies between FR1 and FR2 are typically referred to as mid-band frequencies. Although a portion of FR1 is greater than 6 GHz, FR1 is typically (interchangeably) referred to as the "below 6 GHz" band in various documents and articles. Similar naming issues sometimes arise for FR2, which is typically (interchangeably) referred to as the "millimeter wave" (mmWave) band in documents and articles, although it is different from the extremely high frequency (EHF) band (30 GHz-300 GHz) identified as the "mmWave" band by the International Telecommunication Union (ITU).

[0018] In view of the above aspects, unless otherwise specifically stated, it should be understood that if the term "below 6 GHz" or the like is used herein, it can broadly represent frequencies that can be less than 6 GHz, can be within FR1, or can include mid-band frequencies. In addition, unless otherwise specifically stated, it should be understood that if used herein, the term "mmWave" or the like can broadly represent frequencies that can include mid-band frequencies, can be within FR2, or can be within the EHF band.

[0019] 5G NR devices, networks, and systems can be implemented to use optimized OFDM-based waveform features. These features may include scalable parameter sets and transmission time intervals (TTIs); a common flexible framework that efficiently multiplexes services and features using dynamic, low-latency time division duplex (TDD) designs or frequency division duplex (FDD) designs; and advanced wireless technologies such as massive multiple-input multiple-output (MIMO), robust mmWave transmission, advanced channel decoding, and device-centric mobility. The scalability of parameter sets in 5G NR and the scaling of subcarrier spacing can efficiently address various services operating across different spectrums and different deployments. For example, in various outdoor and macro coverage deployments of less than 3GHz FDD or TDD implementations, the subcarrier spacing may appear at 15kHz, such as on bandwidths of 1MHz, 5MHz, 10MHz, 20MHz, etc. For various other outdoor and small cell coverage deployments with TDD greater than 3GHz, the subcarrier spacing may appear at 30kHz on 80MHz / 100MHz bandwidths. For various other indoor broadband implementations, using TDD on the unlicensed portion of the 5GHz band, the subcarrier spacing may occur at 60kHz over a 160MHz bandwidth. Finally, for various deployments transmitting over mmWave components at 28GHz TDD, the subcarrier spacing may occur at 120kHz over a 500MHz bandwidth.

[0020] For clarity, certain aspects of the devices and techniques may be described below with reference to example 5G NR implementations or in a 5G-centric manner, and 5G terminology may be used as illustrative examples in portions of the description below; however, the description is not intended to be limited to 5G applications.

[0021] Furthermore, it should be understood that in operation, a wireless communication network adapted according to the concepts herein can operate using any combination of licensed spectrum or unlicensed spectrum depending on load and availability. Therefore, it will be apparent to one of ordinary skill in the art that the systems, devices, and methods described herein can be applied to other communication systems and applications beyond the specific examples provided.

[0022] Although various aspects and implementations are described in this application by way of illustration of some examples, it will be appreciated by those skilled in the art that additional implementations and use cases may be generated in many different arrangements and scenarios. The innovations described herein may be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, implementations or uses may be implemented via integrated chip implementations or other non-module component-based devices (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail devices or purchasing devices, medical devices, AI-enabled devices, etc.). Although some examples may or may not specifically point to use cases or applications, the applicability of a wide variety of described innovations may occur.

[0023] Implementations may range from chip-level or modular components to non-modular, non-chip-level implementations, and further to aggregated, distributed, or original equipment manufacturer (OEM) devices or systems that incorporate one or more of the described aspects. In some practical environments, devices incorporating the various aspects and features described may also necessarily include additional components and features for implementing and practicing the various aspects claimed and described. It is intended that the innovations described herein may be implemented in a wide variety of implementations of different sizes, shapes, and configurations, including both large and small devices, chip-level components, multi-component systems (e.g., radio frequency (RF) chains, communication interfaces, processors), distributed arrangements, end-user devices, and the like.

[0024] In the following description, a large number of specific details (such as examples of specific components, circuits and processes) are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of the present disclosure. However, it will be appreciated by those skilled in the art that these specific details may not be required for implementing the teachings disclosed herein. In other examples, in order to avoid blurring the teachings of the present disclosure, known circuits and devices are shown in block diagram form.

[0025] Some portions of the following detailed description are presented in terms of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, procedures, logic blocks, processes, etc. are conceived as a self-consistent sequence of steps or instructions leading to a desired result. These steps are steps requiring physical manipulations of physical quantities. Typically, although not necessarily, these physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0026] In the accompanying drawings, a single box may be described as performing one or more functions. The one or more functions performed by the box may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, the following generally describes each exemplary component, box, module, circuit, and step in terms of its functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Moreover, the example device may include components other than those shown, including known components such as processors, memories, etc.

[0027] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "transmit," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "set," "generate," etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission, or display devices.

[0028] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more components that can implement at least some portions of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the described operations.

[0029] As used herein, including in the claims, the term "or" used in a list of two or more items means that any one of the listed items may be employed alone, or any combination of two or more of the listed items may be employed. For example, if a composition is described as containing components A, B, or C, the composition may contain A alone; B alone; C alone; A and B combined; A and C combined; B and C combined; or A, B, and C combined.

[0030] Furthermore, as used herein (including in the claims), “or” as used in a list of items beginning with “at least one of” indicates a separate list, so that, for example, a list of “at least one of A, B, or C” means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these.

[0031] In addition, as used herein, the term "substantially" is defined as being largely, but not necessarily completely, what is specified (and including what is specified; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any disclosed implementation, the term "substantially" may be replaced with "within [percentage] of" what is specified, where percentages include 0.1%, 1%, 5%, or 10%.

[0032] In addition, as used herein, relative terms, unless otherwise specified, can be understood as a certain amount relative to a reference. For example, terms such as "higher" or "lower" or "more" or "less" can be understood as a threshold amount higher, lower, more or less than a reference value. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] A further understanding of the nature and advantages of the present disclosure may be achieved by reference to the following drawings. In the drawings, similar components or features may have the same reference label. In addition, various components of the same type may be distinguished by adding a dash and a second label to distinguish between similar components after the reference label. If only the first reference label is used in the specification, the description applies to any one of the similar components having the same first reference label, regardless of the second reference label.

[0034] Figure 1 is a perspective view of a motor vehicle having a driver monitoring system according to an embodiment of the present disclosure.

[0035] Figure 2 A block diagram of an example image processing configuration for a vehicle is shown according to one or more aspects of the present disclosure.

[0036] Figure 3 is a block diagram illustrating details of an example wireless communication system in accordance with one or more aspects.

[0037] Figure 4 is a block diagram illustrating a red-clear-clear-green (RCCG) filter array for an image sensor according to one or more aspects of the present disclosure.

[0038] Figure 5 is a block diagram illustrating a camera system with an RCCG filter array according to one or more aspects of the present disclosure.

[0039] Figure 6 is a graph illustrating a logarithmic response in an image sensor according to one or more aspects of the present disclosure.

[0040] Fig. 7A is a block diagram illustrating an image signal processing system according to one or more aspects of the present disclosure.

[0041] Figure 7B is a block diagram illustrating an image signal processing system according to one or more aspects of the present disclosure.

[0042] Figure 7C is a block diagram illustrating an image signal processing system according to one or more aspects of the present disclosure.

[0043] Figure 8 is a flow chart illustrating an example method for obtaining image data for a vehicle monitoring system.

[0044] The same reference numbers and designations in different drawings indicate the same elements. DETAILED DESCRIPTION

[0045] The specific embodiments described below in conjunction with the accompanying drawings are intended as descriptions of various configurations and are not intended to limit the scope of the present disclosure. Instead, the specific embodiments include specific details for providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in every case, and in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0046] The present disclosure provides systems, devices, methods, and computer-readable media that support capturing an image using a red-clear-clear-green (RCCG) color filter array (CFA) having an image sensor configured in a logarithmic domain; and processing the RCCG image, wherein at least a portion of the processing is performed in the logarithmic domain.

[0047] Specific implementations of the subject matter described in the present disclosure can be implemented to achieve one or more of the following potential advantages or benefits. In some aspects, the present disclosure provides techniques for image processing that may be particularly beneficial in intelligent vehicle applications. For example, the RCCG color filter array may provide a trichromatic color response similar to human vision, an improved signal-to-noise ratio (SNR) in a broadband luminance channel, an increased dynamic range with consistent color reproduction, a spectral eigenmode of a system of opposing colors used in human visual sensing, and / or an increased degree of color discrimination of objects in the yellow-orange spectral range. Discrimination in the yellow-orange space is particularly important in traffic environments because many lane markings, traffic signals, and traffic lights use orange and yellow.

[0048] Figure 1 1 is a perspective view of a motor vehicle with a driver monitoring system according to an embodiment of the present disclosure. The vehicle 100 may include a front camera 112 mounted in the cab for viewing through the windshield 102. The vehicle may also include a cab-facing camera 114 mounted in the cab for viewing by the occupants of the vehicle 100, and in particular the driver of the vehicle 100. Although a set of mounting locations for the cameras 112 and 114 is shown for the vehicle 100, other mounting locations may also be used for the cameras 112 and 114. For example, one or more cameras may be mounted on one of the driver or passenger B-pillars 126 or one of the driver or passenger C-pillars 128, such as near the top of the pillars 126 or 128. As another example, one or more cameras may be mounted at the front of the vehicle 100, such as behind the radiator grille 130 or integrated with the bumper 132. As a further example, one or more cameras may be mounted as part of the driver or passenger side mirror assembly 134.

[0049] The camera 112 may be oriented such that the field of view of the camera 112 captures the scene in front of the vehicle 100 in the direction that the vehicle 100 is moving when in the drive mode or forward direction. In some embodiments, an additional camera may be located at the rear of the vehicle 100 and oriented such that the field of view of the additional camera captures the scene behind the vehicle 100 in the direction that the vehicle 100 is moving when in the reverse direction. Although embodiments of the present disclosure may be described with reference to a “forward facing” camera (reference camera 112), aspects of the present disclosure may be similarly applied to a “rearward facing” camera facing the reverse direction of the vehicle 100. Thus, the benefits obtained when the operator is driving the vehicle 100 in the forward direction may be similarly obtained when the operator is driving the vehicle 100 in the reverse direction.

[0050] Furthermore, while embodiments of the present disclosure may be described with reference to a "forward facing" camera (reference camera 112), aspects of the present disclosure may be similarly applied to inputs received from an array of cameras mounted around the vehicle 100 to provide a larger field of view, which may be up to 360 degrees around parallel to the ground and / or up to 360 degrees around a vertical direction perpendicular to the ground. For example, additional cameras may be mounted around the outside of the vehicle 100, such as mounted on or integrated into a door, mounted on or integrated into a wheel, mounted on or integrated into a bumper, mounted on or integrated into a hood, and / or mounted on or integrated into a roof.

[0051] Camera 114 may be oriented so that the field of view of camera 114 captures the scene in the cabin of the vehicle and includes a user operator of the vehicle, and in particular the face of the user operator of the vehicle, with sufficient detail to resolve the direction of gaze of the user operator.

[0052] Each of the cameras 112 and 114 may include one, two or more image sensors, such as including a first image sensor. When there are multiple image sensors, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a telephoto image sensor. In another example, the first sensor is configured to obtain an image through a first lens having a first optical axis, and the second sensor is configured to obtain an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. This configuration may occur in a camera module with a lens group, wherein multiple image sensors and associated lenses are located in offset positions within the camera module. Additional image sensors with larger, smaller, or the same field of view may be included.

[0053] Each image sensor may include components for capturing data representing a scene, such as an image sensor (including a charge coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal oxide semiconductor (CMOS) sensor), and / or a time of flight detector. The device may also include one or more components for gathering and / or focusing light into one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components may be controlled to capture a first image frame, a second image frame, and / or more image frames. The image frames may be processed to form a single output image frame (such as, by a fusion operation), and the output image frame may be further processed according to aspects described herein.

[0054] As used herein, an image sensor may refer to the image sensor itself and any specific other components coupled to the image sensor for generating image frames for processing by an image signal processor or other logic circuitry or for storage in a memory (whether a short-term buffer or a long-term non-volatile memory). For example, an image sensor may include other components of a camera, including a shutter, a buffer, or other readout circuitry for accessing individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of an image frame, which is provided to a digital circuit coupled to the image sensor.

[0055] Figure 2 A block diagram of an example image processing configuration for a vehicle according to one or more aspects of the present disclosure is shown. The vehicle 100 may include or otherwise be coupled to an image signal processor 212 for processing image frames from one or more image sensors (such as a first image sensor 201, a second image sensor 202, and a depth sensor 240). In some specific implementations, the vehicle 100 also includes or is coupled to a processor (e.g., a CPU) 204 and a memory 206 storing instructions 208. The device 100 may also include or be coupled to a display 214 and an input / output (I / O) component 216. The I / O component 216 may be used to interact with a user, such as a touch screen interface and / or physical buttons. The I / O component 216 may also include a network interface for communicating with other devices such as other vehicles, an operator's mobile device, and / or a remote monitoring system. The network interface may include one or more of a wide area network (WAN) adapter 252, a local area network (LAN) adapter 253, and / or a personal area network (PAN) adapter 254. An example WAN adapter 252 is a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 253 is an IEEE 802.11 WiFi wireless network adapter. An example PAN adapter 254 is a Bluetooth wireless network adapter. Each of the adapters 252, 253, and / or 254 may be coupled to an antenna including multiple antennas configured for primary and diversity reception and / or configured to receive a particular frequency band. The vehicle 100 may also include or be coupled to a power source 218, such as a battery or an alternator. The vehicle 100 may also include or be coupled to a Figure 2 In one example, a wireless interface, which may include one or more transceivers and an associated baseband processor, may be coupled to or included in a WAN adapter 252 for wireless communication devices. In yet another example, an analog front end (AFE) for converting analog image frame data to digital image frame data may be coupled between image sensors 201 and 202 and image signal processor 212.

[0056] The vehicle 100 may include a sensor hub 250 for interfacing with sensors to receive data about the movement of the vehicle 100, data about the environment surrounding the vehicle 100, and / or other non-camera sensor data. One example non-camera sensor is a gyroscope, i.e., a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which can also be used to determine speed and distance traveled by appropriately integrating the measured acceleration, and one or more of the acceleration, speed, and / or distance can be included in the generated motion data. In other examples, the non-camera sensor may be a global positioning system (GPS) receiver, a light detection and ranging (LiDAR) system, a radio detection and ranging (RADAR) system, or other ranging system. For example, sensor hub 250 may interface to a vehicle bus for transmitting configuration commands and / or receiving information from vehicle sensors 272, such as distance (e.g., ranging) sensors or vehicle-to-vehicle (V2V) sensors (e.g., sensors for receiving information from nearby vehicles).

[0057] Image signal processor (ISP) 212 may receive image data such as for forming an image frame. In one embodiment, the local bus connector couples image signal processor 212 to the corresponding Figure 1 The first camera 203 of the camera 112 and the first camera 203 may correspond to Figure 1 The image sensors 201 and 202 of the second camera 205 of the camera 114 can be connected to each other. In another embodiment, the wire interface can couple the image signal processor 212 to the external image sensor. In yet another embodiment, the wireless interface can couple the image signal processor 212 to the image sensors 201 and 202.

[0058] The first camera 203 may include a first image sensor 201 and a corresponding first lens 231. The second camera 205 may include a second image sensor 202 and a corresponding second lens 232. Each of the lenses 231 and 232 may be controlled by an associated autofocus (AF) algorithm 233 executed in the ISP 212, which adjusts the lenses 231 and 232 to focus on a specific focal plane at a certain scene depth from the image sensors 201 and 202. The AF algorithm 233 may be assisted by a depth sensor 240. In some implementations, the lenses 231 and 232 may have a fixed focal length.

[0059] The first image sensor 201 and the second image sensor 202 are configured to capture one or more image frames. The lenses 231 and 232 focus light at the image sensors 201 and 202, respectively, through one or more apertures for receiving light, one or more shutters for blocking light when outside the exposure window, one or more color filter arrays (CFA) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, and / or other suitable components for imaging.

[0060] In some embodiments, the image signal processor 212 may execute instructions from a memory, such as instructions 208 from the memory 206, instructions stored in a separate memory coupled to or included in the image signal processor 212, or instructions provided by the processor 204. In addition or in an alternative form, the image signal processor 212 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in the present disclosure. For example, the image signal processor 212 may include one or more image front ends (IFEs) 235, one or more image post-processing engines (IPEs) 236, and one or more automatic exposure compensation (AEC) 234 engines. The AF 233, AEC 234, IFE 235, IPE 236 may each include a dedicated circuit that is embodied as software code executed by the ISP 212 and / or a combination of hardware within the ISP 212 and software code executed on the ISP.

[0061] In some implementations, the memory 206 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 208 to perform all or part of one or more operations described in the present disclosure. In some implementations, the instructions 208 include a camera application (or other suitable application) to be executed during the operation of the vehicle 100 for generating images or videos. The instructions 208 may also include other applications or programs for execution by the vehicle 100, such as an operating system, a mapping application, or an entertainment application. The execution of the camera application by the processor 204 may enable the vehicle 100 to generate images using the image sensors 201 and 202 and the image signal processor 212. The memory 206 may also be accessed by the image signal processor 212 to store processed frames, or may be accessed by the processor 204 to obtain processed frames. In some embodiments, the vehicle 100 includes a system on a chip (SoC) that combines the image signal processor 212, the processor 204, the sensor hub 250, the memory 206, and the input / output component 216 into a single package.

[0062] In some embodiments, at least one of the image signal processor 212 or the processor 204 executes instructions to perform various operations described herein, including object detection, risk map generation, driver monitoring, and driver alert operations. For example, execution of the instructions may instruct the image signal processor 212 to start or end capturing an image frame or sequence of image frames. In some embodiments, the processor 204 may include one or more general-purpose processor cores 204A capable of executing scripts or instructions of one or more software programs (such as instructions 208 stored in the memory 206). For example, the processor 204 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 206.

[0063] When executing the camera application, the processor 204 may be configured to instruct the image signal processor 212 to perform one or more operations with reference to the image sensor 201 or 202. For example, the camera application may receive a command to start a video preview display, and upon receiving the command, capture and process a video including a sequence of image frames from one or more image sensors 201 or 202, and display the video on an information display on the display 114 in the cabin of the vehicle 100.

[0064] In some embodiments, the processor 204 may include an IC or other hardware (e.g., an artificial intelligence (AI) engine 224) in addition to the ability to execute software to cause the vehicle 100 to perform multiple functions or operations, such as those described herein. In some other embodiments, the vehicle 100 does not include a processor 204, such as when all of the described functionality is configured in the image signal processor 212.

[0065] In some embodiments, the display 214 may include one or more suitable displays or screens that allow user interaction and / or present items (such as previews of image frames captured by the image sensors 201 and 202) to the user. In some embodiments, the display 214 is a touch-sensitive display. The I / O component 216 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from the user and provide output to the user through the display 214. For example, the I / O component 216 may include (but is not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (such as a power button), a slider, a switch, etc. In some embodiments involving autonomous driving, the I / O component 216 may include an interface to a vehicle bus for providing commands and information to and receiving information from the vehicle system 270, including propulsion (e.g., commands to increase or decrease speed or apply brakes) and steering systems (e.g., commands to turn wheels, change routes, or change final destinations).

[0066] Although shown coupled to each other via processor 204, components such as processor 204, memory 206, image signal processor 212, display 214, and I / O components 216 may be coupled to each other in various other arrangements, such as via one or more local buses, which are not shown for simplicity. Although image signal processor 212 is illustrated as being separate from processor 204, image signal processor 212 may be the core of processor 204, which is an application processor unit (APU), included in a system on a chip (SoC), or otherwise included in processor 204. Although vehicle 100 is mentioned in the examples herein to include aspects of the present disclosure, some device components may not be present. Figure 2 100 to prevent obscuring the various aspects of the present disclosure. Additionally, other components, numbers of components, or combinations of components may be included in a suitable vehicle for performing various aspects of the present disclosure. Therefore, the present disclosure is not limited to a particular configuration of devices or components, including vehicle 100.

[0067] The vehicle 100 may communicate as a user equipment (UE) within the wireless network 300, such as through the WAN adapter 252, such as Figure 3 shown. Figure 3 is a block diagram illustrating details of an example wireless communication system according to one or more aspects. Wireless network 300 may, for example, include a 5G wireless network. As will be appreciated by those skilled in the art, Figure 3 Components appearing in are likely to have related corresponding components in other network arrangements, including, for example, cellular-style network arrangements as well as non-cellular-style network arrangements (e.g., device-to-device or peer-to-peer or ad hoc network arrangements, etc.).

[0068] Figure 3The illustrated wireless network 300 includes a base station 305 and other network entities. A base station may be a station that communicates with a UE, and may also be referred to as an evolved Node B (eNB), a next generation eNB (gNB), an access point, etc. Each base station 305 may provide communication coverage for a specific geographic area. In 3GPP, the term "cell" may refer to a specific geographic coverage area of ​​a base station or a base station subsystem serving the coverage area, depending on the context in which the term is used. In the specific implementation of the wireless network 300 herein, the base station 305 may be associated with the same operator or different operators (e.g., the wireless network 300 may include multiple operator wireless networks). Additionally, in the specific implementation of the wireless network 300 herein, the base station 305 may provide wireless communication using one or more frequencies (e.g., one or more frequency bands in a licensed spectrum, an unlicensed spectrum, or a combination thereof) in the same frequency as an adjacent cell. In some examples, a separate base station 305 or UE 315 may be operated by more than one network operating entity. In other examples, each base station 305 and UE 315 may be operated by a single network operating entity.

[0069] A base station may provide communication coverage for a macro cell or a small cell (e.g., a pico cell or a femto cell), or other types of cells. A macro cell generally covers a relatively large geographic area (e.g., several kilometers in radius) and may allow unrestricted access by UEs with a service subscription with a network provider. A small cell (such as a pico cell) will generally cover a relatively small geographic area and may allow unrestricted access by UEs with a service subscription with a network provider. A small cell (such as a femto cell) will generally also cover a relatively small geographic area (e.g., a home), and in addition to unrestricted access, may also provide restricted access by UEs associated with the femto cell (e.g., UEs in a closed subscriber group (CSG), UEs of users in a home, etc.). A base station for a macro cell may be referred to as a macro base station. A base station for a small cell may be referred to as a small cell base station, a pico base station, a femto base station, or a home base station. In Figure 3 In the example shown, base stations 305d and 305e are conventional macro base stations, while base stations 305a-305c are macro base stations implemented using one of three-dimensional (3D), full-dimensional (FD) or massive MIMO. Base stations 305a-305c use their higher-dimensional MIMO capabilities to employ 3D beamforming in elevation and azimuth beamforming to increase coverage and capacity. Base station 305f is a small cell base station, which can be a home node or a portable access point. The base station can support one or more (e.g., two, three, four, etc.) cells.

[0070] Wireless network 300 may support synchronous or asynchronous operation. For synchronous operation, base stations may have similar frame timings, and transmissions from different base stations may be approximately aligned in time. For asynchronous operation, base stations may have different frame timings, and transmissions from different base stations may not be aligned in time. In some cases, the network may be enabled or configured to handle dynamic switching between synchronous or asynchronous operation.

[0071] UE 315 is dispersed throughout wireless network 300, and each UE may be stationary or mobile. It should be understood that although in the standards and specifications promulgated by 3GPP, mobile devices are often referred to as UEs, such devices may be additionally or otherwise referred to by those skilled in the art as mobile stations (MS), subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals (AT), mobile terminals, wireless terminals, remote terminals, handsets, terminals, user agents, mobile clients, clients, gaming devices, augmented reality devices, vehicle components, vehicle devices or vehicle modules, or some other suitable terminology.

[0072] Some non-limiting examples of mobile devices, such as those that may include specific implementations of one or more of UE 315, include mobile phones, cellular (cell) phones, smart phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, laptop computers, personal computers (PCs), notebooks, netbooks, smart books, tablet computers, personal digital assistants (PDAs), and vehicles. Although UE 315a-j is specifically shown as a vehicle, the vehicle may employ the communication configuration described with reference to any of UE 315a-315k.

[0073] In one aspect, the UE may be a device including a universal integrated circuit card (UICC). In another aspect, the UE may be a device that does not include a UICC. In some aspects, a UE that does not include a UICC may also be referred to as an IoE device. Figure 3 The UE 315a-315d of the specific implementation illustrated in the figure are examples of mobile smart phone type devices accessing the wireless network 300. The UE may also be a machine specifically configured for connected communications, including machine type communications (MTC), enhanced MTC (eMTC), narrowband IoT (NB-IoT), etc. Figure 3 The UEs 315e-315k illustrated in FIG. 3 are examples of various machines configured for communication that access the wireless network 300.

[0074] A mobile device, such as UE 315, may be able to communicate with any type of base station, whether macro, pico, femto, relay, etc. Figure 3In the figure, the communication link (represented as a lightning ball) indicates the wireless transmission between the UE and the serving base station (which is a base station designated to serve the UE on the downlink or uplink) or the expected transmission between the base stations and the backhaul transmission between the base stations. The UE can operate as a base station or other network node in some scenarios. The backhaul communication between the base stations of the wireless network 300 can be carried out using a wired or wireless communication link.

[0075] In operation, at the wireless network 300, base stations 305a-305c use 3D beamforming and collaborative spatial techniques (e.g., coordinated multipoint (CoMP) or multi-connectivity) to serve UEs 315a and 315b. Macro base station 305d performs backhaul communications with base stations 305a-305c and small cells (base station 305f). Macro base station 305d also transmits multicast services subscribed to and received by UEs 315c and 315d. Such multicast services may include mobile TV or streaming video, or may include other services for providing community information, such as weather emergencies or alerts, such as Amber alerts or gray alerts.

[0076] The wireless network 300 of the specific implementation supports mission-critical communications with ultra-reliable and redundant links for mission-critical devices such as UE 315e. The redundant communication links with UE 315e include links from macro base stations 305d and 305e and small cell base station 305f. Other machine-type devices such as UE 315f (thermometer), UE 315g (smart meter), and UE 315h (wearable device) can communicate directly with base stations such as small cell base station 305f and macro base station 305e through the wireless network 300, or in a multi-hop configuration by communicating with another user device that relays its information to the network, such as UE 315f communicating temperature measurement information to smart meter UE 315g, which then reports it to the network through small cell base station 305f. The wireless network 300 may also provide additional network efficiency through dynamic, low-latency TDD communications or low-latency FDD communications, such as in a vehicle-to-vehicle (V2V) mesh network between UEs 315i-315k communicating with a macro base station 305e.

[0077] Reference Figure 1 , Figure 2 and Figure 3Various aspects of the vehicle system described and shown therein may include a camera sensor having a color filter array (CFA) and appropriate image processing circuitry. In some embodiments, the image sensor may have a color filter array (CFA) configured to produce a trichromatic color response of a system similar to human visual sensing. For example, the CFA may be a red-clear-clear-green (RCCG) filter array. Image processing for the RCCG output of the image sensor may include appropriate exposure control and image preprocessing.

[0078] The RCCG image sensor configuration provides higher sensitivity in a broadband spectral channel (e.g., luminance), which is important for providing similar color responses in both daytime and nighttime use of the vehicle. Much of the morphological information representing the vehicle's surroundings is primarily carried by this channel. The red (R) and green (G) color channels enhance luminance information because the color response is similar to that of the human eye, as many objects in the traffic environment have been designed to match the color response of the human eye. When the signal is displayed in real time in the vehicle or for recording, the red (R) and green (G) color channels also provide a better representation of the surroundings while still providing useful luminance information in combination with the transparent channel.

[0079] Figure 4 4 is a block diagram illustrating a red-transparent-transparent-green (RCCG) filter array for an image sensor according to one or more aspects of the present disclosure. The color filter 400 may include a 4×4 unit cell 410. Although an example is shown as a unit 410, the pattern may be shifted in a vertical column or horizontal row, such as shifting a row and / or a column. The unit cell is repeated on an array of a single pixel sensor. Each unit cell includes a combination of red, transparent, and green filters. In some embodiments, the unit cell 410 is an RCCG unit, which includes a first transparent pixel 410A, a red pixel 410B, a green pixel 410C, and a second transparent pixel 410D. The order of pixels in the unit cell may follow a zigzag pattern of transparent, red, green, and transparent, but different arrangements may also be used for the RCCG filter array.

[0080] A color filter array may be included with other components to form a camera module, which may be included in an image processing system. Figure 5 5 is a block diagram illustrating a camera system with an RCCG filter array according to one or more aspects of the present disclosure. Camera module 510 may include a lens 512 that directs light toward an image sensor 514. Image sensor 514 includes a sensor array of individual pixels corresponding to image data. Superimposed on image sensor 514 is a color filter array (CFA) 516, such as Figure 4RCCG filter array shown. In some embodiments, array 516 is a red-green-transparent color filter array having color filters organized in a pattern similar to a Bayer color filter array, with red in the same position, transparent replacing green, and green replacing the blue filter position in the Bayer color filter.

[0081] The camera module 510 may output image data to an image signal processor (ISP) 520. The ISP may process the raw output of the camera module 510 to form image frames that may be saved, displayed, or sent for viewing by a user. The ISP 520 may include a pipeline of logic for performing a series of operations on the image data. In some embodiments, portions of the circuitry in the ISP 520 may be reused for different processing tasks in the image processing pipeline.

[0082] The image sensor 514 may operate based on an exposure control 530, such as the AEC 234. Controlling the exposure may control settings in the camera module 510, such as gain, exposure time, frame rate, and / or other parameters that change how the image data output from the camera module 510 represents light captured from the scene at the image sensor 514. For example, the exposure control 530 may increase the gain and / or exposure time in a dark scene to increase the brightness of the image data. In certain embodiments, the exposure control 530 may configure the camera module 510 to have different gains on the red and green channels to achieve a signal strength similar to that present in the clear channel given a spectrally neutral signal input to the system.

[0083] In certain embodiments, lens 512 can be a lens assembly with a specific cutoff filter at longer wavelengths, thereby limiting spectral transmission in near infrared wavelengths. The lens assembly can act as a bandpass filter. The filter can be tuned to cut off blue light.

[0084] The output of camera module 510 may be an image signal having a logarithmic response to the input light signal at image sensor 514 . Figure 6 An example logarithmic relationship is shown in . Figure 6 600 is a graph showing a logarithmic response in an image sensor according to one or more aspects of the present disclosure. Graph 600 includes line 610 showing the relationship between an input light signal on an x-axis 602 and an output image sensor signal on a y-axis 604. The logarithmic output differs from a linear output relationship in that the logarithmic relationship is more sensitive with low input signal values. The logarithmic relationship may not be strictly logarithmic, but rather is logarithmic for a portion of the input range, with the lowest end not approximating a logarithmic mapping but a linear mapping with a particular gain. In certain embodiments, the logarithmic output may be applied to certain channels of the R, G, and C channels. For example, a linear gain may be applied to a green channel, and a logarithmic gain may be applied to a transparent and red channel.

[0085] Operating with a logarithmic response can reduce the amount of data sent between the camera module 510 and the ISP 520 to reduce transmission and signaling issues on the bus between the camera module 510 and the ISP 520. For example, in the linear domain, a desired level of dynamic range (DR) may require a 20-bit image signal, which would be formatted as a 32-bit word to be sent over the bus to the ISP 520. Accommodating a 32-bit wide bus would require significant complexity in the packaging of the camera module 510 and the ISP 520, increase the size required for the connectors and / or traces on the printed circuit board (PCB), and increase the complexity of the ISP 520 processing the 32-bit wide signal. Using a logarithmic response (such as a 32-bit word) for the output of the camera module 510 may reduce the amount of data sent between the camera module 510 and the ISP 520. Figure 6 ) can enable a dynamic range (DR) similar to that of a 12-bit or 16-bit image signal, which can be sent in 16-bit wide words over the connection to ISP 520.

[0086] In certain embodiments, HDR merging can be applied to image data captured by camera module 510 by mapping a high linear dynamic range of intensities to a logarithmic response. The mapping can include a linear mapping with a predefined gain at very low signal ranges.

[0087] The output of ISP 520 may be used in various driver assistance systems including navigation, adaptive cruise control, lane change assist, collision avoidance systems, night vision, parking assist, and blind spot detection, self-parking, and / or autonomous driving.

[0088] An example image signal processing chain for ISP 520 is shown in FIG. Fig. 7A Shown in. Fig. 7A is a block diagram illustrating an image signal processing system according to one or more aspects of the present disclosure. The signal chain 700 may be implemented in an ISP having a computer vision (CV) and / or human vision (HV) image. The first portion 710 of the signal chain may perform CV processing by receiving a logarithmic signal response as a raw image input. The first portion 710 may further perform demosaicing 712, color mapping 714 from a specifically tuned color space to a new color space, tuning based on recorded data, and / or bit depth adjustment 716 for each channel in the new color space.

[0089] The output of the first portion 710 of processing is a computer vision (CV) image that can be input to the second portion 720 of the signal chain 700. The second portion 720 may include processing color mapping 722 from the new color space back to a red-transparent-green color space, linearization 724 of the image data, tone mapping 726 (to obtain consistent color balance across the entire dynamic range), and color mapping 728 into a color space suitable for display, recording, or other intended use by a human observer (e.g., a color space such as sRGB or YUV). The output of the ISP may be coupled to a processor or memory and may be processed via a processor such as a processor or memory. Figure 3 The wireless communication system of the wireless communication system is sent.

[0090] Fig. 7A is an example signal processing chain, but other processing chain configurations may be used to process image data. Figure 7B is a block diagram illustrating an image signal processing system according to one or more aspects of the present disclosure. A first portion 740 of the signal chain 730 may include linearization 742 of the logarithmic data (or another predefined grayscale mapping) of the input signal, local tone mapping 744 (to obtain consistent color balance across the entire dynamic range of the image signal), bit depth reduction, demosaicing 746, color mapping 748 according to a particular tuned color space (e.g., by mapping to a color matrix in the spectral eigenmodes of the system), tone mapping 749, and / or gamma adjustment. A second portion 750 of the signal chain 730 may include tone mapping 752, color mapping 754, and / or tone mapping 756 into a color space suitable for display, recording, or other intended use by a human observer (e.g., a color space such as sRGB or YUV).

[0091] Fig. 7A and Figure 7B A series of sequential processing steps are shown that are performed on image data received from, or derived from, an image sensor. Certain processes in the image signal processing system may be performed in parallel. Figure 7C is a block diagram illustrating an image signal processing system according to one or more aspects of the present disclosure. Figure 7C Shown with Fig. 7A A similar processing chain is shown, but the input to the second portion 720 is image data from an image sensor, rather than the output CV image from the first portion 710. In one or more additional aspects, the input to the second portion 740 may also be implemented using the input from the image sensor. Figure 7C In one or more additional aspects, different arrangements of portions of the first portion and the second portion are configured to provide different combinations of parallel operations.

[0092] Figure 8A method of performing image processing according to the above-mentioned embodiment is shown in FIG. Figure 8 800 is a flow chart illustrating an example method for obtaining image data for a vehicle monitoring system. Method 800 includes: at box 802, receiving image data from an image sensor including a red-clear-clear-green (RCCG) color filter, wherein the image data includes a first channel corresponding to red, a second channel corresponding to clear, and a third channel corresponding to green, wherein the image data is in a logarithmic domain. At box 804, processing the image data by performing operations including linearization, tone mapping, demosaicing, and color mapping. In some embodiments, at least some of the processing at box 804 may be performed in the logarithmic domain. In some embodiments, prior to box 802, the ISP may configure a camera module including an image sensor using a programmable lookup table for a piecewise linear function to configure the image sensor for output in the logarithmic domain.

[0093] It should be noted that reference Figure 5 One or more blocks (or operations) described in FIG. 1 may be combined with one or more blocks (or operations) described in another figure in the reference figure. Figure 8 One or more boxes (or operations) of Figure 4 As another example, with Figure 8 One or more boxes associated with Figure 5 As another example, Figure 8 One or more boxes associated with Figure 6 To one or more blocks (or operations) associated with Figure 7. Additionally or alternatively, the above reference Figures 1 to 3 One or more of the operations described may be combined with reference to Figures 4 to 8 A combination of one or more of the operations described.

[0094] In one or more aspects, the technology for supporting vehicle operations may include additional aspects, such as any single aspect or any combination of aspects described below or in combination with one or more other processes or devices described elsewhere herein. In a first aspect, image processing (such as for vehicle monitoring operations) may be supported, and may include a device including one or more of a camera module, the camera module including a lens, an image sensor, a color filter array (CFA); and an image signal processor (e.g., a processor). Additionally, the device may be performed or operated according to one or more aspects as described below. In some specific implementations, the device includes a wireless device, such as a UE. In some specific implementations, the device may include at least one processor, and a memory coupled to the processor. The processor may be configured to perform the operations described herein for the device. In some other specific implementations, the device may include a non-transitory computer-readable medium having a program code recorded thereon, and the program code may be executable by a computer for causing the computer to perform the operations described herein with reference to the device. In some specific implementations, the device may include one or more components configured to perform the operations described herein. In some implementations, a method of wireless communication can include one or more operations described herein with reference to the apparatus.

[0095] In a second aspect, in combination with the first aspect, image data is received from an image sensor including a red-clear-clear-green (RCCG) filter, wherein the image data includes a first channel corresponding to red, a second channel corresponding to transparent, and a third channel corresponding to green, wherein the image data is in a logarithmic domain; and the image data is processed by performing operations including linearization, tone mapping, demosaicing, and color mapping.

[0096] In a third aspect, in combination with one or more of the first or second aspects, the apparatus is further configured to perform operations including configuring the image sensor using a programmable lookup table for a piecewise linear function to configure the image sensor for output in the logarithmic domain.

[0097] In a fourth aspect, in combination with one or more of the first to third aspects, at least some of the operations are performed in the logarithmic domain.

[0098] In a fifth aspect, in combination with one or more of the first to fourth aspects, processing the image data comprises: demosaicing the image data to obtain demosaiced data; and color mapping the demosaiced data to second image data.

[0099] In the sixth aspect, in combination with one or more of the first to fifth aspects, processing the image data also includes: linearizing the second image data to obtain linearized data; and color mapping the linearized data to obtain an output image frame in an RGB or YUV color space.

[0100] In a seventh aspect, in combination with one or more of the first to sixth aspects, processing the image data includes: linearizing the image data to obtain linearized data; demosaicing the linearized data to obtain demosaiced data; and color mapping the demosaiced data to second image data.

[0101] In the eighth aspect, in combination with one or more of the first to seventh aspects, processing the image data further includes: performing color mapping and tone mapping based on the second image data to obtain an output image frame in an RGB or YUV color space.

[0102] It should be understood by those skilled in the art that any of a variety of different technologies and techniques may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0103] This article is about Figures 1 to 8 The components, functional blocks and modules described include processors, electronic devices, hardware devices, electronic components, logical circuits, memories, software codes, firmware codes, etc. or any combination thereof. Software should be broadly interpreted to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, and / or functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language or other terms. In addition, the features discussed herein may be implemented via dedicated processor circuits, via executable instructions, or a combination thereof.

[0104] The technician will further understand that the various exemplary logic boxes, modules, circuits and algorithm steps described in conjunction with the disclosure of this article can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, boxes, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. Those skilled in the art can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of the present disclosure. The technician will also easily recognize that the order or combination of components, methods or interactions described herein are only examples, and the components, methods or interactions of various aspects of the present disclosure can be combined or performed in a manner other than the manner illustrated and described herein.

[0105] The various illustrative logical components, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. The interchangeability of hardware and software has been generally described in terms of functionality and is illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0106] The hardware and data processing apparatus for implementing the various illustrative logics, logic blocks, modules and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller or state machine. In some specific implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. In some specific implementations, specific processes and methods may be performed by circuits specific to a given function.

[0107] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed in this specification and their structural equivalents, or any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on computer storage media for execution by data processing apparatus or for controlling the operation of data processing apparatus.

[0108] If implemented in software, the function may be stored as one or more instructions or codes on a computer-readable medium or sent via a computer-readable medium. The process of the method or algorithm disclosed herein may be implemented in a processor executable software module that may reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, and the communication media include any media that can be implemented to transfer a computer program from one place to another. The storage medium may be any available medium that a computer can access. As an example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection may be appropriately referred to as a computer-readable medium. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as a code and instruction set, or any combination of code and instruction sets, on a machine-readable medium or computer-readable medium, which may be incorporated into a computer program product.

[0109] Various modifications to the specific implementations described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to some other specific implementations without departing from the spirit or scope of the disclosure. Therefore, the claims are not intended to be limited to the specific implementations shown herein, but are to be consistent with the broadest scope consistent with the disclosure, the principles and novel features disclosed herein.

[0110] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0111] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all illustrated operations to achieve the desired result. In addition, the accompanying drawings can schematically depict one or more example processes in the form of a flow chart. However, other operations that are not depicted can be combined in the example processes schematically illustrated. For example, one or more additional operations can be performed before, after, simultaneously or between any illustrated operations. In some environments, multitasking and parallel processing are advantageous. In addition, the separation of various system components in the specific implementation described above should not be understood as requiring such separation in all specific implementations, and it should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other specific implementations also fall within the scope of the appended claims. In some cases, the actions described in the claims can be performed in different orders and still achieve the desired result.

[0112] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Therefore, the disclosure is not intended to be limited to the examples and designs described herein, but to be consistent with the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for image processing in a vehicle assistance system, the method comprising: receiving image data from an image sensor including a red-clear-clear-green (RCCG) color filter, wherein the image data includes a first channel corresponding to red, a second channel corresponding to clear, and a third channel corresponding to green, wherein the image data is in a logarithmic domain; and The image data is processed by performing operations including demosaicing and color mapping.

2. The method according to claim 1, further comprising: The image sensor is configured using a programmable lookup table for a piecewise linear function to configure the image sensor for output in the logarithmic domain.

3. The method of claim 1, wherein at least some of the operations are performed in the logarithmic domain.

4. The method of claim 3, wherein processing the image data comprises: De-mosaicing the image data to obtain demosaiced data; as well as The demosaiced data is color mapped to second image data.

5. The method of claim 4, wherein processing the image data further comprises: linearizing the second image data to obtain linearized data; as well as The linearized data is color mapped to obtain an output image frame in RGB or YUV color space.

6. The method of claim 1 , wherein processing the image data comprises: linearizing the image data to obtain linearized data; De-mosaicing the linearized data to obtain de-mosaiced data; as well as The demosaiced data is color mapped to second image data.

7. The method of claim 6, wherein processing the image data further comprises: Color mapping and tone mapping are performed based on the second image data to obtain an output image frame in an RGB or YUV color space.

8. A device, comprising: a memory storing processor readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations including: receiving image data from an image sensor including a red-clear-clear-green (RCCG) color filter, wherein the image data includes a first channel corresponding to red, a second channel corresponding to clear, and a third channel corresponding to green, wherein the image data is in a logarithmic domain; and The image data is processed by performing operations including demosaicing and color mapping.

9. The apparatus of claim 8, wherein the at least one processor is further configured to perform operations comprising: The image sensor is configured using a programmable lookup table for a piecewise linear function to configure the image sensor for output in the logarithmic domain.

10. The apparatus of claim 8, wherein at least some of the operations are performed in the logarithmic domain.

11. The apparatus of claim 10, wherein processing the image data comprises: De-mosaicing the image data to obtain demosaiced data; as well as The demosaiced data is color mapped to second image data.

12. The apparatus of claim 11, wherein processing the image data further comprises: linearizing the second image data to obtain linearized data; as well as The linearized data is color mapped to obtain an output image frame in RGB or YUV color space.

13. The apparatus of claim 8, wherein processing the image data comprises: linearizing the image data to obtain linearized data; De-mosaicing the linearized data to obtain de-mosaiced data; as well as The demosaiced data is color mapped to second image data.

14. The apparatus of claim 13, wherein processing the image data further comprises: Color mapping and tone mapping are performed based on the second image data to obtain an output image frame in an RGB or YUV color space.

15. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving image data from an image sensor including a red-clear-clear-green (RCCG) color filter, wherein the image data includes a first channel corresponding to red, a second channel corresponding to clear, and a third channel corresponding to green, wherein the image data is in a logarithmic domain; and The image data is processed by performing operations including demosaicing and color mapping.

16. The non-transitory computer readable medium of claim 15, wherein the operations further comprise: The image sensor is configured using a programmable lookup table for a piecewise linear function to configure the image sensor for output in the logarithmic domain.

17. The non-transitory computer readable medium of claim 16, wherein at least some of the operations are performed in the logarithmic domain.

18. The non-transitory computer readable medium of claim 17, wherein processing the image data comprises: De-mosaicing the image data to obtain demosaiced data; as well as The demosaiced data is color mapped to second image data.

19. The non-transitory computer readable medium of claim 18, wherein processing the image data further comprises: linearizing the second image data to obtain linearized data; as well as The linearized data is color mapped to obtain an output image frame in RGB or YUV color space.

20. The non-transitory computer readable medium of claim 15, wherein processing the image data comprises: linearizing the image data to obtain linearized data; De-mosaicing the linearized data to obtain de-mosaiced data; color mapping the demosaiced data to second image data; as well as Color mapping and tone mapping are performed based on the second image data to obtain an output image frame in an RGB or YUV color space.

Images