Image sensor, imaging apparatus, and image processing method

Abstract

The present invention relates to an image sensor, an imaging apparatus, and an image processing method. The image sensor includes a plurality of photoelectric conversion elements (2204) and a plurality of monochrome filters for generating a plurality of colors. The plurality of monochrome filters are arranged in correspondence with the plurality of photoelectric conversion elements, respectively. At least one of the plurality of monochrome filters includes a primary color type monochrome filter (2303). The primary color type monochrome filter allows transmission of a corresponding primary color. The primary color type monochrome filter has a first given transmittance for one of other primary colors than the corresponding primary color, which one of the other primary colors is transmitted by the primary color type monochrome filter at the first given transmittance. The first given transmittance is higher than a lower limit of transmittance for improving sensitivity of the image sensor.

Description

Technical Field

[0001] This disclosure relates to an image sensor including a color filter, an imaging apparatus including an image sensor, and an image processing method used in the imaging apparatus. Background Technology

[0002] Imaging devices that include color filters and photoelectric conversion elements are known. Typically, a three-color filter, comprising red, green, and blue filters, is used as the color filter. To improve sensitivity, known systems use either a yellow filter or a transparency filter instead of the green filter.

[0003] However, the quality of images generated by known systems is not good enough. This is because the signal level output from a pixel input by a beam passing through either a transparency filter or a yellow filter is different from the signal level output from a pixel input by a beam passing through either a red filter or a blue filter.

[0004] This disclosure aims to address and resolve such problems, and its purpose is to provide novel image sensors, imaging devices, and image processing methods that can reduce the difference in signal levels detected by each photoelectric conversion element, thereby improving sensitivity. Summary of the Invention

[0005] Therefore, one aspect of this disclosure provides a novel image sensor comprising a plurality of photoelectric conversion elements (2204) and a plurality of monochromatic filters for generating multiple colors. The plurality of monochromatic filters are arranged corresponding to the respective plurality of photoelectric conversion elements. At least one of the plurality of monochromatic filters includes a primary color type monochromatic filter (2303). The primary color type monochromatic filter transmits light of the corresponding primary color. The primary color type monochromatic filter also transmits light of at least one of other primary colors besides the corresponding primary color. The primary color type monochromatic filter has a first given transmittance for one of the other primary colors besides the corresponding primary color, and the other primary color passes through the primary color type monochromatic filter at this first given transmittance. The first given transmittance is higher than a lower limit of transmittance for improving the sensitivity of the image sensor. According to one aspect of this disclosure, the sensitivity of the image sensor is improved more effectively than that of a conventional image sensor having a transmittance for a primary color besides the corresponding primary color that is less than or equal to a lower effective transmittance.

[0006] Another aspect of this disclosure provides a novel imaging apparatus comprising: the aforementioned image sensor; and a processing circuit (2400) for generating a color image by processing signals output from the image sensor. The processing circuit generates the color image using at least one of a first set of signals and a second set of signals, the first set of signals being output from one or more photoelectric conversion elements arranged corresponding to primary color type monochromatic filters, and the second set of signals being output from one or more photoelectric conversion elements arranged corresponding to one or more sub-primary color filters. The correction coefficients used when correcting the signals output from the one or more photoelectric conversion elements arranged corresponding to the one or more primary color type monochromatic filters are different from the correction coefficients used when correcting the signals output from the one or more photoelectric conversion elements arranged corresponding to the sub-primary color filters.

[0007] Another aspect of this disclosure provides a novel image processing method. The method includes the following steps: receiving incident light using multiple color monochromatic filters; generating primary colors using primary color filter sections (4304, 5304, 6304); and transmitting a portion of the incident light through high-sensitivity filter sections (4305, 5305) that have a higher sensitivity than the primary color filter sections. The high-sensitivity filter sections are divided into multiple sub-high-sensitivity filter sections (5305s). The multiple sub-high-sensitivity filter sections are respectively arranged corresponding to multiple photoelectric conversion elements.

[0008] The method further includes the following steps: adjusting the number of photoelectric conversion elements used in generating the color of a single pixel according to the ambient brightness; performing multiple photoelectric conversions using multiple photoelectric conversion elements (2204) arranged respectively corresponding to multiple color monochrome filters to obtain an electrical signal; and correcting the electrical signal.

[0009] The method also includes the step of generating a color image based on the corrected electrical signal.

[0010] Therefore, according to another aspect of this disclosure, even if the ambient brightness changes, the difference in the level of the signal output from the photoelectric conversion element corresponding to the pixel can be reduced. Attached Figure Description

[0011] Since this disclosure becomes generally better understood when considered with reference to the accompanying drawings, and by referring to the following detailed description, a more complete understanding of the disclosure and its many accompanying advantages will be more readily obtained, in which:

[0012] Figure 1 This is a block diagram illustrating an imaging system 100 according to one embodiment of the present disclosure;

[0013] Figure 2 This is an illustrative representation of one embodiment of the present disclosure. Figure 1 A side view of an exemplary vehicle for the system;

[0014] Figure 3 It is shown schematically. Figure 2 The diagram shows a vehicle with a system.

[0015] Figure 4 It also schematically shows a plan view of a vehicle having another system according to another embodiment of the present disclosure;

[0016] Figure 5 It also schematically shows a plan view of a vehicle including yet another system according to another embodiment of the present disclosure;

[0017] Figure 6 It also schematically shows a plan view of a vehicle including yet another system according to another embodiment of the present disclosure;

[0018] Figure 7 This is a block diagram illustrating an exemplary vehicle control system according to one embodiment of the present disclosure;

[0019] Figure 8 This is a schematic diagram illustrating the interior of a vehicle, including a rearview mirror and a user interface to a vehicle imaging system, according to one embodiment of this disclosure.

[0020] Figure 9 This is a schematic diagram illustrating a camera bracket arranged behind a rearview mirror and facing the vehicle windshield according to one embodiment of the present disclosure.

[0021] Figure 10 It is shown that from and Figure 9 Viewing from different viewpoints Figure 9 A diagram of a camera bracket;

[0022] Figure 11 This is a schematic diagram illustrating one embodiment of the present disclosure of a camera bracket arranged behind a rearview mirror and facing the vehicle windshield.

[0023] Figure 12 This is a block diagram illustrating a memory storing one or more instructions for performing one or more operations, according to one embodiment of the present disclosure.

[0024] Figure 13 This is a flowchart illustrating an exemplary process for evoking one or more navigation responses based on monocular image analysis according to one embodiment of the present disclosure;

[0025] Figure 14This is a flowchart illustrating a process for detecting one or more vehicles and / or pedestrians in a set of images according to one embodiment of the present disclosure;

[0026] Figure 15 This is a flowchart illustrating a process for detecting road markings and / or lane geometry information in a set of images according to one embodiment of the present disclosure;

[0027] Figure 16 This is a flowchart illustrating a process for detecting traffic lights in a set of images according to one embodiment of the present disclosure;

[0028] Figure 17 This is a flowchart illustrating a process by which a vehicle route triggers one or more navigation responses according to one embodiment of the present disclosure;

[0029] Figure 18 This is a flowchart illustrating a process for determining whether a vehicle ahead is changing lanes according to one embodiment of the present disclosure;

[0030] Figure 19 This is a flowchart illustrating a process by which stereoscopic image analysis causes one or more navigation responses according to one embodiment of the present disclosure;

[0031] Figure 20 This is a flowchart illustrating a process for causing one or more navigation responses based on analysis performed on three image sets, according to one embodiment of the present disclosure;

[0032] Figure 21 This is a cross-sectional view showing components of a vehicle-mounted camera according to one embodiment of the present disclosure;

[0033] Figure 22 This is a first table illustrating an exemplary design rule for providing weighting based on the wavelength of a lens system, according to one embodiment of this disclosure;

[0034] Figure 23 This is a second table illustrating exemplary design rules for a multicolor MTF (modulation transfer function) of a lens system according to one embodiment of the present disclosure;

[0035] Figure 24 This is a third table illustrating exemplary design rules for parameters of a cutoff filter attached to a lens system according to one embodiment of this disclosure;

[0036] Figure 25 This is a diagram schematically illustrating the configuration of an image sensor according to one embodiment of the present disclosure;

[0037] Figure 26This is a cross-sectional view showing a front illumination pixel according to one embodiment of the present disclosure;

[0038] Figure 27 This is a cross-sectional view showing a rear-side illumination pixel according to one embodiment of the present disclosure;

[0039] Figure 28 This is a diagram illustrating a color filter array and the smallest repeating unit of a color filter according to one embodiment of the present disclosure;

[0040] Figure 29 It is shown Figure 28 A diagram showing the configuration of the smallest repeating unit of the color filter;

[0041] Figure 30 This is a graph showing the relationship between the transmittance and wavelength of incident light transmitted through a pale red monochromatic filter according to one embodiment of the present disclosure.

[0042] Figure 31 This is a graph showing the relationship between the transmittance and wavelength of incident light transmitted through a green monochromatic filter according to one embodiment of the present disclosure.

[0043] Figure 32 This is a graph showing the relationship between the transmittance and wavelength of incident light transmitted through a blue monochromatic filter according to one embodiment of the present disclosure.

[0044] Figure 33 This is a diagram illustrating the smallest repeating unit of a color filter according to a second embodiment of the present disclosure;

[0045] Figure 34 This is a block diagram schematically illustrating an imaging apparatus according to a second embodiment of the present disclosure;

[0046] Figure 35 This is a diagram illustrating the smallest repeating unit of a color filter according to a third embodiment of the present disclosure;

[0047] Figure 36 This is a diagram illustrating the smallest repeating unit of a color filter according to a fourth embodiment of the present disclosure;

[0048] Figure 37 This is a diagram illustrating the smallest repeating unit of a color filter according to a fifth embodiment of the present disclosure;

[0049] Figure 38 This is a block diagram schematically illustrating an imaging apparatus according to a fifth embodiment of the present disclosure; and

[0050] Figure 39This is a block diagram schematically illustrating an imaging apparatus according to a sixth embodiment of the present disclosure. Detailed Implementation

[0051] Now refer to the accompanying drawings, in which the same reference numerals denote the same or corresponding parts in several views, and refer to... Figure 1 First, an overview of the system is given. That is to say, Figure 1 This is a block diagram illustrating an exemplary imaging system 100 according to one embodiment of the present disclosure. The imaging system 100 may include various components to meet the requirements of a particular implementation. In some embodiments, the imaging system 100 may include a processing unit 110, an image acquisition unit 120, and a position sensor 130. The imaging system 100 may also include one or more memories 140 and 150, a map database 160, and a user interface 170. The imaging system 100 may also include a wireless transceiver 172. The processing unit 110 may include one or more processors. In some embodiments, the processing unit 110 may include an application processor 180, an image processor 190, and any other optional suitable processor. Similarly, the image acquisition unit 120 may include any number of image acquirers and components to meet the requirements of a particular application. That is, the image acquisition unit 120 may include one or more image acquirers 122, 124, and 126. For example, each image acquirer is a camera. The imaging system 100 may also include a data interface 128 that connects the processing unit 110 to the image acquisition unit 120 to enable communication between the processing unit 110 and the image acquisition unit 120. For example, the data interface 128 may include any wired and / or wireless link for transmitting image data acquired by the image acquisition unit 120 to the processing unit 110.

[0052] Furthermore, the wireless transceiver 172 may include one or more devices configured to exchange transmissions with one or more networks (e.g., cellular networks, the Internet) via a wireless interface using radio frequency or infrared frequencies in a magnetic or electric field. The wireless transceiver 172 may use any known standard to send and / or receive data.

[0053] Furthermore, each of the application processor 180 and the image processor 190 may include various types of processors. For example, the application processor 180 and / or the image processor 190 may include a microprocessor, a preprocessor (e.g., an image preprocessor), and a graphics processor. The application processor 180 and / or the image processor 190 may also include a central processing unit (hereinafter sometimes referred to as a CPU), support circuitry, and a digital signal processor. The application processor 180 and / or the image processor 190 may also include integrated circuits, memory, and any other type of device suitable for performing applications, image processing, and analysis. In some embodiments, the application processor 180 and / or the image processor 190 may include any type of single-core or multi-core processor, mobile device microcontroller, and CPU, etc. Furthermore, various processors can be used. Various architectures may also be included.

[0054] In some implementations, application processor 180 and / or image processor 190 may include multiple processing units with local memory and instruction sets. One or more such processors may include video input functionality for receiving image data from multiple image sensors. One or more processors may also include video output functionality. As an example, one or more processors may use 90nm-class micrometer technology capable of operating at approximately 332MHz. Furthermore, the architecture includes two floating-point-decimal-point hyper-threaded 32-bit RISC (Reduced Instruction Set Computer) CPUs, five Visual Computing Engines (VCEs), and three vector microcode processors. The architecture may also include a 64-bit mobile DDR (Dual Data Rate) controller, 128-bit internal acoustic interconnects, and dual 16-bit video inputs. The architecture may also include an 18-bit video output controller, a 16-channel DMA (Direct Memory Access), and multiple peripheral devices.

[0055] Furthermore, any of the processing units discussed in this disclosure can be configured to perform a specific function. To construct a processor (e.g., a processor, controller, microprocessor, etc.) that performs such a specific function, computer-executable instructions can be programmed and executed by the processor to perform those instructions during processor operation. In other embodiments, the processor can be directly programmed using architectural instructions. In other embodiments, the processor can store executable instructions in its accessible memory during its operation. For example, the processor can retrieve and execute instructions stored in memory by accessing memory during its operation.

[0056] Furthermore, although Figure 1The processing unit 110 shown includes two separate processors, but more or fewer processors may be used. For example, in some embodiments, a single processor may be used to perform the corresponding tasks of the application processor 180 and the image processor 190. In other embodiments, these tasks may be performed by two or more processors. Furthermore, in some embodiments, the imaging system 100 may include one or more processing units 110 that do not include other components such as the image acquisition unit 120.

[0057] Furthermore, the processing unit 110 can be configured with various types of devices. For example, the processing unit 110 may include a controller, an image preprocessor, and a CPU. The processing unit 110 may also include support circuitry, a digital signal processor, and integrated circuits. The processing unit 110 may also include memory and any other types of devices for image processing and analysis, etc. The image preprocessor may include a video processor for receiving images from an image sensor and digitizing and processing the images. The CPU may include any number of microcontrollers or microprocessors. The support circuitry may be any number of circuits generally known in the applicable technical fields, such as cache circuitry, power supply circuitry, clock circuitry, input / output circuitry, etc. The memory may store software that controls the operation of the system when executed by the processor. The memory may also include database or image processing software. Such memory may include any number of RAM (random access memory), ROM (read-only memory), and flash memory. The memory may also be configured with any number of disk drives, optical storage devices, and magnetic tape storage devices. The memory may also be configured with any number of removable storage devices and other types of storage devices. In one example, the memory may be separate from the processing unit 110. In other embodiments, the memory may be integrated into the processing unit 110.

[0058] More specifically, each of the memories 140 and 150 may include (i.e., store) software instructions executed by a processor (e.g., application processor 180 and / or image processor 190) to control various aspects of the imaging system 100. These memories 140 and 150 may also include various databases and image processing software. Each of the memories may include random access memory, read-only memory, and flash memory as previously described. Each of the memories may also include disk drives, optical storage devices, and magnetic tape storage devices. Each of the memories may also include removable storage devices and / or any other type of storage device. In some embodiments, each of the memories 140 and 150 may be decoupled from the application processor 180 and / or image processor 190. In another embodiment, each of the memories may be integrated into the application processor 180 and / or image processor 190.

[0059] Furthermore, the position sensor 130 may include any type of device suitable for determining the position of components of the imaging system 100, such as an image acquirer. In some embodiments, the position sensor 130 may include a GPS (Global Positioning System) receiver. Such a receiver can determine the user's position and speed by processing signals broadcast by GPS satellites. The application processor 180 and / or the image processor 190 may utilize the position information output from the position sensor 130.

[0060] In addition, in some embodiments, the imaging system 100 may include a speed sensor (e.g., a tachometer) for measuring the speed of the vehicle 200 and / or an acceleration sensor for measuring the degree of acceleration of the vehicle 200.

[0061] Furthermore, the user interface 170 may include any device suitable for the imaging system 100 to provide information to one or more users or to receive input from one or more users. In some embodiments, the user interface 170 may include, for example, user input devices such as a touchscreen, microphone, keyboard, etc. The user input device may also be a pointer device, a tracking wheel, and a camera. The user input device may also be a knob and a button, etc. Therefore, using such input devices, a user can input instructions or information as well as voice commands. Furthermore, a user can select menu options displayed on the screen by using buttons, pointer devices, or eye-tracking functionality. Users can also input information or provide commands to the imaging system 100 using any other suitable technology for communicating with the imaging system 100.

[0062] More specifically, the user interface 170 may include one or more processors configured to provide information to a user, receive information from a user, and process the information for use in, for example, an application processor 180. In some embodiments, such a processor may execute instructions to recognize and track eye movements, receive and interpret voice commands, and recognize and interpret touches and / or gestures made on a touchscreen. The processor may also execute instructions to respond to keyboard input or menu selection, etc. In some embodiments, the user interface 170 may include a display, speakers, and haptic devices for outputting information to the user. The user interface 170 may also include any other devices.

[0063] Furthermore, map database 160 may include any type of database for storing map data useful to imaging system 100. For example, in some embodiments, map database 160 may include data relating to the location in a reference coordinate system of various items such as roads, water features, geographic features, etc. Various items also include businesses, points of interest, and restaurants. Various items also include gas stations, etc. In addition to these locations of such items, map database 160 may store descriptors associated with such items, the descriptors including names associated with any stored features. In some embodiments, map database 160 may be physically arranged together with other components of imaging system 100. Alternatively or additionally, at least a portion of map database 160 may be located at a remote location away from other components of imaging system 100 (e.g., processing unit 110). In such embodiments, information can be downloaded from map database 160 via a wired or wireless data connection to a network (e.g., via a cellular network and / or the Internet).

[0064] Furthermore, image acquirers 122, 124, and 126 may each include any type of acquirer suitable for capturing at least a single image from the environment. Additionally, any number of image acquirers can be used to acquire the image input to the image processor. In some embodiments, only a single image acquirer may be included. In other embodiments, two or more image acquirers may be included. Referring later... Figures 2 to 6 Image acquirers 122, 124, and 126 are described in more detail.

[0065] Furthermore, the imaging system 100 or its various components can be incorporated into various platforms. In some embodiments, the imaging system 100 may be included in, for example... Figure 2 In the vehicle 200 shown. For example, vehicle 200 may include processing unit 110 and as previously referred to. Figure 1 Any other component of the imaging system 100 described. In some embodiments, the vehicle 200 may include only a single image acquirer (e.g., a camera). In other embodiments, multiple image acquirers may be used, as described in reference... Figures 3 to 20 As described. For example, such as Figure 2 As shown, any one of the image acquisition units 122 to 124 of the vehicle 200 may be part of an ADAS (Advanced Driver Assistance System) imaging device.

[0066] The image acquisition unit 120, which is included in the vehicle 200 as part of the image acquisition unit 120, can be positioned at any suitable location within the vehicle. Specifically, in some embodiments, the image acquisition unit 122 can be positioned near the rearview mirror 310, such as... Figures 2 to 19 as well as Figures 8 to 10As shown. This position provides the same line of sight as the driver of vehicle 200, which helps the driver determine whether things are visible or invisible. The image acquirer 122 can be positioned anywhere near the rearview mirror 310. In particular, when the image acquirer 122 is placed on the driver's side near the rearview mirror 310, such a position also helps the driver acquire an image representing the driver's field of vision and / or his or her line of sight.

[0067] Furthermore, the image acquirer of the image acquisition unit 120 can be located in other locations. For example, the image acquirer 124 can be mounted on or within the bumper of the vehicle 200. This is because such a location is particularly suitable for an image acquirer with a wide field of view. However, the line of sight of an image acquirer placed in the bumper may differ from that of the driver. Therefore, the bumper image acquirer and the driver do not always see the same object. Additionally, the image acquirers (e.g., image acquirers 122, 124, and 126) can be located elsewhere. For example, the image acquirer can be placed on one or both side mirrors of the vehicle 200, on the roof, or on the hood. The image acquirer can also be placed in the trunk and on the sides of the vehicle 200. Furthermore, the image acquirer can be attached to one of the windows of the vehicle 200, placed at the rear or front of the vehicle 200, and mounted on or near the headlights and / or taillights of the vehicle 200.

[0068] In addition to the image acquisition unit, vehicle 200 may include various other components of imaging system 100. For example, processing unit 110 may be integrated into vehicle 200 with electronic control unit (ECU) or included separately from electronic control unit (ECU) of vehicle 200. Furthermore, vehicle 200 may include position sensor 130 such as GPS receiver, as well as map database 160 and memories 140 and 150.

[0069] Furthermore, as previously described, the wireless transceiver 172 can receive data via one or more networks. For example, the wireless transceiver 172 can upload data collected by the imaging system 100 to one or more servers. Additionally, the wireless transceiver 172 can download data from one or more servers. For example, via the wireless transceiver 172, the imaging system 100 can periodically or on-demand receive and update data stored in the map database 160, memory 140, and / or memory 150. Similarly, the wireless transceiver 172 can upload any data, such as images captured by the image acquisition unit 120, data received by the position sensor 130, other sensors, and the vehicle control system, from the imaging system 100 to one or more servers. The wireless transceiver 172 can also upload any data processed by the processing unit 110 from the imaging system 100 to one or more servers.

[0070] Furthermore, the imaging system 100 can upload data to a server (e.g., a cloud computer) based on privacy level settings. For example, the imaging system 100 can incorporate privacy level settings to adjust or limit the data types (including metadata) sent to the server, which can uniquely identify the vehicle and / or its driver or owner. Such privacy level settings can be implemented, for example, by the user via the wireless transceiver 172 or as factory default settings in the initial state. Additionally, privacy level settings can be implemented using data received by the wireless transceiver 172.

[0071] More specifically, in some implementations, the imaging system 100 can upload data according to a privacy level. For example, depending on such a privacy level setting, the imaging system 100 can send data such as location information of the route, captured images, etc., that do not include details about a specific vehicle and / or the driver / owner of the vehicle. Specifically, in order to upload data with high privacy settings, the imaging system 100 can send data such as captured images that do not include the vehicle identification number (VIN) or the name of the driver or owner, and / or limited location information of the vehicle's route.

[0072] In addition, other privacy levels can also be used. For example, the imaging system 100 sends data to a server with a medium privacy level by including additional information excluded from the "high" privacy level (e.g., vehicle manufacturer, vehicle model, vehicle type (e.g., bus, SUV, truck)). Furthermore, in some embodiments, the imaging system 100 can upload data with a low privacy level. That is, in the "low" privacy level setting, the imaging system 100 can upload data including sufficient information to uniquely identify a particular vehicle, owner / driver, and / or part or all of the route traveled by the vehicle. For example, such "low" privacy level data may include one or more information items such as VIN (Vehicle Identification Number), driver / owner's name, and the vehicle's starting point before departure. One or more information items may also be the vehicle's intended destination, vehicle manufacturer and / or model, and vehicle type, etc.

[0073] Figure 2 This is a schematic side view of a representative imaging system 100 according to one embodiment of the present disclosure. Figure 3 yes Figure 2 An exemplary plan view of the illustrated embodiment. Figure 3 As shown, in this embodiment, the main body of the vehicle 200 may include an imaging system 100, which has a first image acquisition device 122 near the rearview mirror and / or the driver driving the vehicle 200, a second image acquisition device 124 above or in the bumper portion of the vehicle 200 (e.g., one of the bumper portions 210), and a processing unit 110.

[0074] In addition, such as Figure 4 As shown, both image acquisition units 122 and 124 can be located near the rearview mirror and / or the driver of the vehicle 200. Although in Figure 3 and Figure 4 Each of the examples shows two image acquirers 122 and 124, but other implementations may employ three or more image acquirers. For example, in Figure 5 and Figure 6 In each of the embodiments shown, a first image acquirer 122, a second image acquirer 124, and a third image acquirer 126 are employed in the imaging system 100.

[0075] Specifically, such as Figure 5 As shown, image acquisition device 122 can be located near the rearview mirror and / or driver of vehicle 200. Image acquisition devices 124 and 126 can then be positioned on or within a bumper portion of vehicle 200 (e.g., one of bumper portions 210). Additionally, as... Figure 6As shown, image acquirers 122, 124, and 126 can be positioned near the rearview mirror and / or driver's seat of vehicle 200. However, according to embodiments of this disclosure, the number and configuration of image acquirers are not limited to a specific number and configuration, and the image acquirers can be located in and / or above vehicle 200 at any suitable location.

[0076] Furthermore, the embodiments of this disclosure are not limited to vehicles and can be applied to other moving bodies. Moreover, the embodiments of this disclosure are not limited to a specific type of vehicle 200, but are applicable to all types of vehicles, including cars, trucks, trailers, and other types of vehicles.

[0077] Furthermore, the first image acquirer 122 can include any suitable type of image acquirer. Specifically, the image acquirer 122 includes an optical axis. As an example, the image acquirer 122 can include a WVGA (Wide Video Graphics Array) sensor with a global shutter. In other embodiments, the image acquirer 122 can have a resolution defined by 1280 × 960 pixels. The image acquirer 122 can also include a rolling shutter. The image acquirer 122 can include various optical elements. For example, in some embodiments, one or more lenses are included to provide the image acquirer with a given focal length and field of view. For example, in some embodiments, the image acquirer 122 can employ a 6mm lens or a 12mm lens. Furthermore, in some embodiments, the image acquirer 122 can be configured to capture an image within a given field of view (FOV) 202, such as... Figure 5As shown. For example, image acquisition device 122 can be configured to have a regular FOV ranging from 40 degrees to 56 degrees, including 46-degree FOV, 50-degree FOV, and 52-degree FOV, and more. Alternatively, image acquisition device 122 can be configured to have a narrow FOV ranging from about 23 degrees to about 40 degrees, such as 28-degree FOV, 36-degree FOV, etc. Furthermore, image acquisition device 122 can be configured to have a wide FOV ranging from about 100 degrees to about 180 degrees. For example, in some embodiments, image acquisition device 122 may include a wide-angle bumper camera or a camera with a maximum FOV of about 180 degrees. Furthermore, in some embodiments, image acquisition device 122 may have a resolution of 7.2 megapixels, a horizontal FOV of about 100 degrees, and an aspect ratio of about 2:1 (e.g., H×V = 3800×1900 pixels). Therefore, such an image acquisition device can be used instead of the three image acquisition devices 122, 124, and 126. When an image acquisition device uses a lens that is rotationally symmetrical about the optical axis, the vertical FOV of such an image acquisition device may become significantly less than about 50 degrees due to the large distortion of the lens. Such a lens cannot be radially symmetrical, which results in a vertical FOV greater than about 50 degrees when the horizontal FOV is about 100 degrees.

[0078] Furthermore, the first image acquirer 122 can acquire multiple first images of the scene viewed from the vehicle 200. Each of the multiple first images can be acquired as a series of image scan lines, or captured using a global shutter. Each scan line can include multiple pixels.

[0079] The first image acquirer 122 can acquire a first series of image data on an image scan line at a given scan rate. Here, the scan rate can sometimes refer to the rate at which an image sensor can acquire image data of pixels included in a given scan line.

[0080] Therefore, each of the image acquisition units 122, 124, and 126 can include any suitable type and number of image sensors, such as CCD (charge-coupled diode) sensors, CMOS (complementary metal-oxide-semiconductor) sensors, etc. In one embodiment, the CMOS image sensor can be used in conjunction with a rolling shutter, reading each line of pixels at a time and scanning line by line until the image frame is completely captured. Thus, lines are captured sequentially from top to bottom within the frame.

[0081] In some implementations, one or more of the image acquirers (e.g., image acquirers 122, 124, and 126) may be one or more high-resolution imagers, each having a resolution of one of 5M pixels, 7M pixels, and 10M pixels or greater.

[0082] Here, when using a rolling shutter, pixels in different columns can be exposed and captured at different times, potentially causing skew and image artifacts in the captured image frame. In contrast, when the image acquirer 122 is configured to operate using a global shutter or a sync shutter, all pixels can be exposed simultaneously during a common exposure period. Therefore, the image data in a frame collected by a system using a global shutter fully represents a snapshot of the FOV (e.g., FOV 202) over a given time period. In contrast, with a system using a rolling shutter, each column in the frame image is exposed, and its data is acquired from each row at different timings. Therefore, in an image acquirer with a rolling shutter, as described in more detail later, moving objects may sometimes appear distorted.

[0083] Furthermore, the second image acquirer 124 and the third image acquirer 126 can be any type of image acquirer. That is, similar to the first image acquirer 122, each of the image acquirers 124 and 126 includes an optical axis. In one embodiment, each of the image acquirers 124 and 126 may include a WVGA sensor with a global shutter. Alternatively, each of the image acquirers 124 and 126 may include a rolling shutter. Similar to image acquirer 122, each of the image acquirers 124 and 126 may be configured to include various lenses and optical elements. In some embodiments, each lens employed in the image acquirers 124 and 126 may have the same FOV as that employed in the image acquirer 122 (e.g., FOV 202) or a narrower FOV than that employed in the image acquirer 122 (e.g., FOV 204 and FOV 206). For example, each of the image acquisition units 124 and 126 can have a field of view (FOV) of 40 degrees, 30 degrees, 26 degrees, 23 degrees, and 20 degrees or less.

[0084] Furthermore, each of image acquirers 124 and 126 can acquire multiple images of the second and third images of the scene viewed from vehicle 200. Each of the second and third images can be captured using a rolling shutter. Each of the second and third images can be acquired as a second series of image scan lines and a third series of image scan lines. Each scan line or line can have multiple pixels. Each of image acquirers 124 and 126 can acquire each image scan line included in the second and third series at a second scan rate and a third scan rate.

[0085] Each image acquirer 122, 124, and 126 can be positioned on the vehicle 200 at any suitable location facing a given direction. The positional relationships between the image acquirers 122, 124, and 126 can be selected to effectively perform information fusion on the information acquired by these image acquirers. For example, in some embodiments, the field of view (FOV) of image acquirer 124 (e.g., FOV 204) may partially or completely overlap with the FOV of image acquirer 122 (e.g., FOV 202) and the FOV of image acquirer 126 (e.g., FOV 206).

[0086] Furthermore, each of the image acquirers 122, 124, and 126 can be positioned on the vehicle 200 at any suitable relative height. For example, the heights of the image acquirers 122, 124, and 126 can differ to provide sufficient parallax information for stereoscopic analysis. For example, as... Figure 2 As shown, the two image acquirers 122 and 124 are arranged at different heights. Furthermore, a lateral displacement difference is allowed between the image acquirers 122, 124, and 126 to provide additional parallax information for stereo analysis, for example, performed by the processing unit 110. Figure 4 and Figure 5 As shown, the lateral displacement difference is indicated by the reference numeral dx. In some embodiments, forward or backward displacement (e.g., range displacement) may be permitted between image acquirers 122, 124, and 126. For example, image acquirer 122 may be located approximately 0.5 meters to approximately 2 meters or more behind image acquirers 124 and / or 126. This type of displacement of the image acquirers allows one of the image acquirers to cover potential blind spots caused by the other multiple image acquirers.

[0087] Furthermore, image acquirer 122 can have any suitable resolution capability (e.g., a given number of pixels employed in the image sensor). The resolution of the image sensor of image acquirer 122 can be the same as, higher than, or lower than the resolution of each image sensor employed in image acquirers 124 and 126. For example, in some embodiments, the image sensors of image acquirers 122 and / or image acquirers 124 and 126 can have resolutions of approximately 640×480, approximately 1024×768, and approximately 1280×960, respectively, or any other suitable resolution.

[0088] Furthermore, the frame rate can be controllable. Here, the frame rate is defined as the rate at which the image acquirer acquires the set of pixel data constituting one image frame per unit time. Therefore, the image acquirer moves to a stage where it acquires pixel data for the next image frame at this rate. The frame rate of image acquirer 122 can be changed to be higher than, lower than, or even the same as the frame rates of image acquirers 124 and 126. The timing of each frame rate of image acquirers 122, 124, and 126 can be determined based on various factors. For example, a pixel delay may be included before or after acquiring image data of one or more pixels from one or more image acquirers 122, 124, and 126. Typically, image data corresponding to each pixel can be acquired at the acquirer's clock rate (e.g., one pixel per clock cycle). Furthermore, in some embodiments employing a rolling shutter, a horizontal blanking period may be selectively included before or after acquiring image data from one or more of the image acquirers 122, 124, and 126 in a column of pixels of the image sensor. In addition, a vertical blanking period may be optionally included before or after acquiring image data of image frames from one or more of the image acquirers 122, 124 and 126.

[0089] Even with different scan rates for each line, these timing controls enable the frame rates of image acquirers 122, 124, and 126 to be synchronized. Furthermore, as described in more detail later, even if the field of view (FOV) of image acquirer 122 differs from that of image acquirers 124 and 126, these selectable timing controls enable the synchronization of image capture from the FOV of image acquirer 122 with the overlapping areas of one or more FOVs of image acquirers 124 and 126.

[0090] The timing of the frame rate used in each of the image acquirers 122, 124, and 126 can be determined depending on the resolution of the corresponding image sensor. For example, when assuming similar line scan rates are used in the two acquirers and one acquirer includes an image sensor with a resolution of 640×480 while the other acquirer includes an image sensor with a resolution of 1280×960, it will take longer to acquire a frame of image data from the sensor with the higher resolution.

[0091] Another factor that may affect (or change) the acquisition timing of image data in each of the image acquirers 122, 124, and 126 is the maximum line scan rate. For example, a minimum amount of time is required to acquire one line of image data from an image sensor arranged in each of the image acquirers 122, 124, and 126. Therefore, assuming no additional use (or adoption) of pixel delay periods, the minimum amount of time required to acquire one line of image data will affect the maximum line scan rate of a given device. In such a case, a device providing a higher maximum line scan rate is able to provide a higher frame rate compared to a device providing a lower maximum line scan rate. Therefore, in some embodiments, the maximum line scan rate of one or more of the image acquirers 124 and 126 may be higher than the maximum line scan rate of image acquirer 122. In some embodiments, the maximum line scan rate of image acquirers 124 and / or 126 may be one of approximately 1.25 times, approximately 1.5 times, and approximately 1.75 times the maximum line scan rate of image acquirer 122. Otherwise, the maximum line scan rate of image acquirers 124 and / or 126 can be greater than twice the maximum line scan rate of image acquirer 122.

[0092] In another embodiment, image acquirers 122, 124, and 126 can operate at the same maximum line scan rate. Furthermore, only image acquirer 122 can operate at a scan rate lower than the maximum scan rate. Additionally, the system can be configured such that one or more of image acquirers 124 and 126 operate at a line scan rate equal to that of image acquirer 122. In another embodiment, the system can be configured such that the line scan rate of image acquirer 124 and / or image acquirer 126 is one of approximately 1.25 times, approximately 1.5 times, and approximately 1.75 times the line scan rate of image acquirer 122. Furthermore, the system can be configured such that the line scan rate of image acquirer 124 and / or image acquirer 126 is more than twice the line scan rate of image acquirer 122.

[0093] Furthermore, in some embodiments, image acquirers 122, 124, and 126 may be asymmetrical. That is, these image acquirers 122, 124, and 126 may include cameras with different fields of view (FOV) and focal lengths from each other. For example, the field of view of each of image acquirers 122, 124, and 126 may be any given area of ​​the environment of vehicle 200. For example, in some embodiments, one or more of image acquirers 122, 124, and 126 may be configured to acquire image data from the front, rear, and sides of vehicle 200. Furthermore, one or more of image acquirers 122, 124, and 126 may be configured to acquire image data from a combination of these directions.

[0094] Furthermore, the focal length of each image acquirer 122, 124, and / or 126 can be determined by selectively incorporating appropriate lenses so that each acquirer acquires an image of an object at a given distance from vehicle 200. For example, in some embodiments, image acquirers 122, 124, and 126 can acquire images of nearby objects within several meters of vehicle 200. Image acquirers 122, 124, and 126 can also be configured to acquire images of objects at a greater distance from vehicle 200 (e.g., 25 meters, 50 meters, 100 meters, 150 meters, or more). Additionally, one of the image acquirers 122, 124, and 126 (e.g., image acquirer 122) can have a given focal length capable of acquiring images of objects relatively close to vehicle 200, for example, objects located within 10 meters or 20 meters of vehicle 200. In such a case, the remaining image acquirers (e.g., image acquirers 124 and 126) may have a given focal length capable of acquiring images of objects located at a distance of 200 from the vehicle (e.g., at one of the distances of 20m, 50m, 100m, and 150m or more).

[0095] Furthermore, in some embodiments, the field of view (FOV) of each of the image acquirers 122, 124, and 126 may have a wide angle. In particular, a 140-degree FOV may be advantageous for each of the image acquirers 122, 124, and 126 to capture images near the vehicle 200. For example, image acquirer 122 may be used to capture images of the left and right regions of the vehicle 200. In such cases, it is sometimes preferred that image acquirer 122 has a wide FOV. That is, the FOV may be at least 140 degrees.

[0096] Furthermore, the field of view of each of the image acquirers 122, 124, and 126 depends on each focal length. For example, the longer the focal length, the narrower the corresponding field of view.

[0097] Therefore, image acquirers 122, 124, and 126 can be configured to have any suitable field of view. In a given example, image acquirer 122 can have a horizontal FOV of 46 degrees. Image acquirer 124 can have a horizontal FOV of 23 degrees. Image acquirer 126 can have a horizontal FOV between 23 and 46 degrees. In other examples, image acquirer 122 can have a horizontal FOV of 52 degrees. Image acquirer 124 can have a horizontal FOV of 26 degrees. Image acquirer 126 can have a horizontal FOV between 26 and 52 degrees. In some embodiments, the ratio between the FOVs of image acquirer 122 and image acquirer 124 and / or image acquirer 126 can vary from about 1.5 to about 2.0. In other embodiments, this ratio can vary between about 1.25 and about 2.25.

[0098] Imaging system 100 can be configured such that the field of view of image acquirer 122 at least partially or completely overlaps with the field of view of image acquirer 124 and / or image acquirer 126. For example, in some embodiments, imaging system 100 can be configured such that the field of view of image acquirers 124 and 126 (e.g., these are narrower fields of view) fits within the field of view of image acquirer 122 and shares a common center with the field of view of image acquirer 122. In other embodiments, image acquirers 122, 124, and 126 can capture adjacent fields of view (FOV). Furthermore, partial duplication (i.e., overlap) may exist in the FOVs of image acquirers 122, 124, and 126. In some embodiments, the field of view of image acquirers 122, 124, and 126 can be positioned such that the center of each other narrower FOV image acquirer 124 and / or 126 is located in the lower half of the field of view of the wider FOV image acquirer 122.

[0099] Figure 7 This is a block diagram illustrating an exemplary vehicle control system according to one embodiment of the present disclosure. Figure 7 As shown, vehicle 200 may include a throttle system 220, a braking system 230, and a steering system 240. Imaging system 100 may provide inputs (e.g., control signals) to one or more of the throttle system 220, braking system 230, and steering system 240 via one or more data links (e.g., links for transmitting data or any wired and / or wireless links). For example, based on analysis of images acquired by image acquirers 122, 124, and / or 126, imaging system 100 may provide control signals to one or more of the throttle system 220, braking system 230, and steering system 240 to control these systems to perform actions such as acceleration, cornering, and lane changing, respectively. Furthermore, imaging system 100 may receive inputs from one or more of the throttle system 220, braking system 230, and steering system 240 indicative of the operating status of vehicle 200 (e.g., vehicle 200 speed, braking, and / or cornering). See below. Figures 12 to 20 Describe other details.

[0100] In addition, such as Figure 8As shown, vehicle 200 may include a user interface 170 for interacting with a driver or occupant in vehicle 200. For example, user interface 170 is included in a vehicle application and may include a touchscreen 320, a knob 330, and a button 340. User interface 170 may also include a microphone 350. Furthermore, the driver or occupant in vehicle 200 can interact with imaging system 100 using a steering wheel and buttons, etc. Handles include, for example, turn signal handles located on or near the steering column of vehicle 200. Buttons are, for example, located on the steering wheel of vehicle 200. In some embodiments, microphone 350 may be positioned adjacent to rearview mirror 310. Similarly, in some embodiments, image acquisition unit 122 may be located near rearview mirror 310. In some embodiments, user interface 170 may include one or more speakers 360 (e.g., speakers used in a vehicle audio system). Imaging system 100 may provide various notifications (e.g., alarms) to the driver via speakers 360.

[0101] Figures 9 to 11 An exemplary camera mount 370, disposed behind a rearview mirror (e.g., rearview mirror 310) and facing the vehicle windshield, is shown as one embodiment of the present disclosure. Specifically, as Figure 9 As shown, the camera bracket 370 may include image acquisition units 122, 124, and 126. Image acquisition units 124 and 126 may be positioned behind a light shield 380. The light shield 380 may have the same height as the vehicle windshield and comprises a composition of film and / or anti-reflective material. For example, the light shield 380 may be arranged to face the vehicle windshield. The light shield 380 and the windshield have the same angle of inclination. In some embodiments, each of the image acquisition units 122, 124, and 126 may be positioned behind the light shield 380, for example, as shown in the image acquisition unit 126. Figure 1 As shown. However, this disclosure is not limited to any particular configuration of the image acquisition devices 122, 124 and 126, the camera bracket 370 and the light shield 380. Figure 10 It is shown Figure 9 The front view of the camera bracket 370 shown.

[0102] As those skilled in the art will understand, many variations and / or modifications can be made to the present disclosure as described above. For example, not all components are required when operating the imaging system 100. Furthermore, any component can be located in any other suitable part of the imaging system 100. Components can also be repositioned while providing the same functionality performed in embodiments of the present disclosure. Therefore, the configurations mentioned above are merely examples, and the imaging system 100 can provide a wide range of functions to analyze images of the environment surrounding the vehicle 200 and navigate the vehicle 200 based on that analysis.

[0103] Furthermore, as will be described in more detail below, according to various embodiments of this disclosure, the imaging system 100 can provide various functions related to autonomous driving and / or driver assistance technologies. For example, the imaging system 100 can analyze image data, location data (e.g., GPS positioning information), map data, speed data, and / or data transmitted from sensors included in the vehicle 200. For example, the imaging system 100 can collect data from the image acquisition unit 120, the position sensor 130, and other sensors for analysis. Furthermore, the imaging system 100 can analyze the collected data and, based on this, determine whether the vehicle 200 should take certain actions, and automatically take the determined actions without human intervention. For example, when navigating the vehicle 200 without human intervention, the imaging system 100 can automatically control the braking, acceleration, and / or steering of the vehicle 200 by sending control signals to one or more systems, respectively, the throttle system 220, the braking system 230, and the steering system 240. Furthermore, the imaging system 100 can analyze the collected data and, based on its analysis, issue warnings and / or alarms to the occupants of the vehicle. Further details regarding the various functions provided by the imaging system 100 are described below.

[0104] Specifically, as described above, the imaging system 100 can provide driver assistance functions by using a multi-camera system. The multi-camera system can use one or more cameras facing forward of the vehicle. In other embodiments, the multi-camera system may include one or more cameras facing the side or rear of the vehicle. For example, in one embodiment, the imaging system 100 can use a two-camera imaging system, wherein a first camera and a second camera (e.g., image acquirers 122 and 124) can be positioned in front of and / or on the side of the vehicle 200. The first camera may have a larger (wider) or smaller (narrower) field of view than the second camera. Alternatively, the field of view of the first camera may partially overlap with that of the second camera. Furthermore, the first camera may be connected to a first image processor to perform monocular image analysis on the images provided by the first camera. The second camera may be connected to a second image processor to provide images and enable the second image processor to perform monocular image analysis on the images. The outputs of the first and second image processors (e.g., processed information) may be combined with each other. In some embodiments, the second image processor may receive images from both the first and second cameras and perform stereoscopic analysis on the images. In other embodiments, the imaging system 100 may use a three-camera imaging system, where each camera has a different field of view than the others. In such a system, determinations are made based on information from objects located at different distances in front of and to the sides of the vehicle. Here, monocular image analysis refers to analyzing images taken from a single viewpoint (e.g., a single camera). In contrast, stereo image analysis refers to image analysis performed based on two or more images taken using one or more image-capturing parameters. For example, images suitable for stereo image analysis are those taken from two or more different positions or in different fields of view. Furthermore, images suitable for stereo image analysis are those taken at different focal lengths or utilizing parallax information, etc.

[0105] Furthermore, in one embodiment, the imaging system 100 can employ a three-camera system using, for example, image acquirers 122, 124, and 126. In such a system, image acquirer 122 can provide a narrow field of view (e.g., a value of 34 degrees, selected from a range of about 20 degrees to about 45 degrees). Image acquirer 124 can provide a wide field of view (e.g., a value of 150 degrees, selected from a range of about 100 degrees to about 180 degrees). Image acquirer 126 can provide an intermediate field of view (e.g., a value of about 46 degrees, selected from a range of about 35 degrees to about 60 degrees). In some embodiments, image acquirer 126 can act as a master camera or primary camera. These image acquirers 122, 124, and 126 can be placed substantially side-by-side behind the rearview mirror 310 at intervals (e.g., about 6 cm). Furthermore, in some embodiments, as previously described, one or more of the image acquisition devices 122, 124, and 126 may be attached to the rear side of a sunshade 380 located on the same plane as the windshield of the vehicle 200. Such a sunshade 380 can be used to minimize any reflection of light from inside the vehicle, thereby reducing its impact on the image acquisition devices 122, 124, and 126.

[0106] Furthermore, in another embodiment, as previously referred to Figure 9 and Figure 10 As described, the wide field-of-view camera (e.g., image acquirer 124 in the example above) can be attached to a position lower than the narrow field-of-view camera and the main field-of-view camera (e.g., image acquirers 122 and 126 in the example above). With such a configuration, the line of sight from the wide field-of-view camera can be freely provided. Here, the camera can be mounted near the windshield of the vehicle 200, or include a polarizer to reduce the amount of reflected light.

[0107] Furthermore, a three-camera system can provide a given performance (i.e., characteristics). For example, in some embodiments, as a function, the detection of an object performed by the first camera is verified by another second camera based on the result of the object detection performed by the second camera. Additionally, for a three-camera system, processing unit 110 may include, for example, three processors (i.e., first processor to third processor). Each processor is dedicated to processing images captured by one or more of the image acquirers 122, 124, and 126.

[0108] Using a three-camera system, the first processor can receive images from both the main camera and the narrow field-of-view camera. The first processor can then apply visual processing to the image transmitted from the narrow field-of-view camera and detect other vehicles, pedestrians, and lane markings. The first processor can also detect traffic signs, traffic lights, and other road objects. The first processor can also calculate the pixel disparity between the image transmitted from the main camera and the image transmitted from the narrow field-of-view camera. The first processor can then create a 3D (three-dimensional) reconstruction (image) of the vehicle's environment. The first processor can combine this 3D reconstruction with 3D map data or 3D information calculated based on information transmitted from the other cameras.

[0109] The second processor can receive images from the main camera, apply visual processing to the images, and detect other vehicles, pedestrians, and lane markings. The second processor can also detect traffic signs, traffic lights, and other road objects. Furthermore, the second processor can calculate the camera's displacement and, based on this displacement, calculate the disparity of pixels between successive images. The second processor can then create a 3D reconstruction of the scene (e.g., based on the structure of motion). The second processor can then send the 3D reconstruction generated based on the structure of motion to the first processor, and synthesize the 3D reconstruction with the stereoscopic 3D image.

[0110] The third processor can receive images from a wide-angle camera. It can then process the images and detect objects on the road (e.g., vehicles, pedestrians, lane markings, traffic signs, traffic lights, etc.). Furthermore, the third processor can execute additional processing instructions and analyze the images to identify moving objects (e.g., vehicles, pedestrians, etc.) changing lanes within the image.

[0111] In some implementations, the system can be redundant by independently receiving and processing image-based information streams. For example, such redundancy includes information obtained by using a first image acquirer and processing the image with the first image acquirer for verification and / or supplementation by capturing image information from at least a second image acquirer and applying a given processing to that image information.

[0112] Furthermore, in some embodiments, when the imaging system 100 performs navigation assistance for the vehicle 200, the imaging system 100 can provide redundancy by using a third image acquirer (e.g., image acquirer 126) to verify the analysis of data received from the other two image acquirers (e.g., image acquirers 122 and 124). For example, using such a system, image acquirers 122 and 124 can provide images for stereo analysis performed by the imaging system 100 while navigating the vehicle 200. Simultaneously, to provide redundancy and verification based on information obtained from images captured by and transmitted from image acquirers 122 and / or 124, image acquirer 126 can provide images to the imaging system 100 for monocular analysis therein. That is, image acquirer 126 and its corresponding processor can be considered as a system that provides a redundant subsystem (e.g., an automatic emergency braking (AEB) system) for examining the analysis of images obtained from image acquirers 122 and 124.

[0113] Here, the above configuration, arrangement, and number of cameras are merely examples. Furthermore, the camera positions described above are also just examples. Specifically, without departing from the spirit of the above embodiments, these components of the entire system previously described can be assembled and used in various ways. Additionally, without departing from the spirit of the above embodiments, other configurations not previously described can be assembled and used. Below, the system and method using a multi-camera system that provides driver assistance and autonomous vehicle operation functions are described in more detail.

[0114] Figure 12 This is a block diagram illustrating exemplary functionality of each of memories 140 and 150, which store programs and instructions for performing one or more operations, according to one embodiment of this disclosure. Hereinafter, although memory 140 is generally referred to, instructions may be stored in both memories 140 and 150.

[0115] Specifically, such as Figure 12As shown, memory 140 may store monocular image analysis module 402, stereo image analysis module 404, and velocity-acceleration module 406. Memory 140 may also store navigation response module 408. However, this disclosure is not limited to any particular configuration of memory 140. For example, application processor 180 and / or image processor 190 may execute instructions stored in any one or more of the modules 402, 404, 406, and 408 included in memory 140. In the following description, although processing unit 110 is generally described, application processor 180 and image processor 190 may operate similarly, individually or jointly. That is, any one or more steps of the process described below may be executed by one or more processors.

[0116] In one embodiment of this disclosure, the monocular image analysis module 402 may store instructions, such as computer vision software, which, when executed by the processing unit 110, perform monocular image analysis on a set of images acquired by one of the image acquisition units 122, 124, and 126. In some embodiments, the processing unit 110 may perform monocular image analysis based on a combination formed by combining information from the image set with additional sensor information (e.g., information obtained from radar). Reference is made below. Figures 13 to 16 As described, the monocular image analysis module 402 may include (i.e., store) instructions for detecting a set of features (e.g., lane markings, vehicles, pedestrians, etc.) included in the image set. This set of features may also include road signs, highway exit ramps, and traffic lights. The set of features may also include hazardous materials and other features related to the vehicle's environment. For example, based on this analysis, the imaging system 100 may control the vehicle 200 via the processing unit 110 to induce one or more navigation responses (e.g., turning, lane changing, and acceleration changes, etc.), as described later with reference to the navigation response module 408.

[0117] In one embodiment, the stereo image analysis module 404 may store instructions, such as computer vision software, to perform stereo image analysis on a first image set and a second image set obtained by a combination of any two or more image acquirers selected from image acquirers 122, 124, and 126. In some embodiments, the processing unit 110 may perform stereo image analysis based on information from the first and second image sets combined with additional sensor information (e.g., information obtained from radar). For example, the stereo image analysis module 404 may include instructions for performing stereo image analysis based on the first image set acquired by image acquirer 124 and the second image set acquired by image acquirer 126. Reference will be made below. Figure 19As described, the stereo image analysis module 404 may include instructions for detecting a set of features (e.g., lane markings, vehicles, pedestrians, etc.) in a first image set and a second image set. This set of features may also include road signs, highway exit ramps, and traffic lights. It may also include hazardous materials, etc. Based on this analysis, the processing unit 110 may control the vehicle 200 to induce one or more navigation responses (e.g., turning, lane changing, and acceleration changes), as described later with respect to the navigation response module 408.

[0118] Furthermore, in some embodiments, the speed-acceleration module 406 may store software configured to analyze data received from one or more computers and electromechanical devices installed in the vehicle 200 to cause changes in the speed and / or acceleration of the vehicle 200. For example, the processing unit 110 may execute instructions stored in the speed-acceleration module 406 and calculate the target speed of the vehicle 200 based on data obtained by executing instructions from the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include target position, speed, and / or acceleration. The data may also include the position and / or speed of the vehicle 200 relative to nearby vehicles, pedestrians, and / or road objects. The data may also include position information of the vehicle 200 relative to road lane markings, etc. In addition, the processing unit 110 may calculate the target speed of the vehicle 200 based on sensor inputs (e.g., information from radar) and inputs from other systems installed in the vehicle 200 (e.g., throttle system 220, braking system 230, steering system 240, etc.). Therefore, based on the calculated target speed, the processing unit 110 can send electronic signals to the throttle system 220, braking system 230, and / or steering system 240 of the vehicle 200 to change the speed and / or acceleration of these systems, for example, by physically pressing the brakes of the vehicle 200 or releasing (i.e., releasing) the accelerator.

[0119] Furthermore, in one embodiment, the navigation response module 408 may store software that can be executed by the processing unit 110 to determine a given navigation response based on data obtained by executing the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include position and speed information about nearby vehicles, pedestrians, and road objects. The data may also include position and speed information about the target location targeted by the vehicle 200, etc. Furthermore, in some embodiments, the navigation response may be generated partially or entirely based on map data, the position of the vehicle 200, and / or the relative velocity or acceleration of the vehicle 200 with respect to one or more objects detected by executing the monocular image analysis module 402 and / or the stereo image analysis module 404. The navigation response module 408 may also determine a given navigation response based on sensor inputs (e.g., information from radar) and inputs from other systems installed in the vehicle 200 (e.g., throttle system 220, braking system 230, steering system 240, etc.). Then, in order to trigger a given navigation response of vehicle 200 and cause vehicle 200 to rotate its steering wheel by a given angle, processing unit 110 may send electronic signals to throttle system 220, braking system 230, and steering system 240, for example. Here, in some embodiments, processing unit 110 may use the output of navigation response module 408 (e.g., a given navigation response) as input to execute instructions from speed-acceleration module 406 to calculate the speed change of vehicle 200.

[0120] Figure 13 This is a flowchart illustrating an exemplary process 500A for generating one or more navigation responses based on monocular image analysis according to one embodiment of the present disclosure. As shown, in step S510, processing unit 110 may receive multiple images via data interface 128 located between processing unit 110 and image acquisition unit 120. For example, a camera included in image acquisition unit 120 (e.g., image acquirer 122 having a field of view 202) may capture multiple images, for example, in one of the front, two side, and rear regions of vehicle 200, and transmit these images to processing unit 110 via a data connection. Here, the data connection may be a wired or wireless connection. See later. Figures 14 to 16 In more detail, processing unit 110 can execute instructions from monocular image analysis module 402 and analyze multiple images in step 520. Subsequently, by performing analysis in this manner, processing unit 110 can detect a range of features included in a series of images, such as lane markings, vehicles, pedestrians, road signs, etc. Processing unit 110 can also detect a range of features included in a series of images, such as highway exit ramps, traffic lights, etc.

[0121] Subsequently, in step S520, processing unit 110 can also execute instructions from monocular image analysis module 402 to detect various road hazards, such as truck tire debris, fallen road signs, unsecured cargo, small animals, etc. Since the structure, shape, and size of such road hazards may vary, detection of such hazards may become more difficult. Furthermore, since the color of road hazards may also vary, detection of such hazards may again become more difficult. In some embodiments, processing unit 110 can execute instructions from monocular image analysis module 402 and perform multi-frame analysis of multiple images to detect such road hazards. For example, processing unit 110 can estimate the camera movement caused between successive image frames, calculate the parallax of pixels between frame images, and construct a 3D map of the road. Subsequently, processing unit 110 can detect the road surface and hazards present on the road surface based on the 3D map.

[0122] Subsequently, in step S530, the processing unit 110 may execute the instructions of the navigation response module 408, and based on the analysis performed in step S520, simultaneously use the previously referenced... Figure 12 The described techniques enable vehicle 200 to generate one or more navigation responses. Navigation responses may include, for example, turning, lane changing, and acceleration changes. Here, in some embodiments, processing unit 110 can induce one or more navigation responses using data obtained as a result of the execution of instructions to speed-acceleration module 406. Furthermore, multiple navigation responses can occur simultaneously or sequentially. Moreover, multiple navigation responses can occur in any combination of these response generation methods. For example, processing unit 110 can cause vehicle 200 to accelerate after changing lanes by sequentially sending control signals to steering system 240 and throttle system 220 of vehicle 200. Alternatively, processing unit 110 can cause vehicle 200 to simultaneously brake and change lanes by simultaneously sending control signals to braking system 230 and steering system 240 of vehicle 200.

[0123] Figure 14Also shown is a flowchart of an exemplary process 500B for detecting one or more vehicles and / or pedestrians in a series of images according to another embodiment of the present disclosure. Specifically, processing unit 110 may execute instructions from monocular image analysis module 402 for the purpose of performing process 500B. That is, in step S540, processing unit 110 may select a set of candidate objects that may represent one or more vehicles and / or pedestrians. For example, processing unit 110 may scan one or more images and compare the images with one or more given patterns. Processing unit 110 may then identify the location in each image that may include the object of interest (e.g., a vehicle, a pedestrian, or a portion thereof). The given pattern may be designed to increase the percentage of false hits while reducing the percentage of omissions (e.g., missed identifications). For example, for the purpose of identifying an object as a possible candidate for a vehicle or pedestrian, processing unit 110 may utilize a threshold indicating less similarity to a given pattern. In this way, processing unit 110 may reduce the probability of ignoring candidate objects representing vehicles or pedestrians.

[0124] Subsequently, in step S542, for the purpose of excluding a given candidate (e.g., irrelevant or unrelated objects), processing unit 110 may filter the set of candidate objects based on one or more classification criteria. Such one or more criteria can be derived from various characteristics related to the type of objects stored in a database (e.g., a database stored in memory 140). Here, various characteristics may include the shape, size, and texture of the object. Various characteristics may also include the position of the object (e.g., its position relative to vehicle 200), etc. Therefore, processing unit 110 can reject false candidates from the set of object candidates by using one or more sets of criteria.

[0125] Subsequently, in step S544, processing unit 110 can analyze images from multiple frames and determine whether one or more objects in the set of candidate objects represent vehicles and / or pedestrians. For example, processing unit 110 can track candidate objects detected in successive frames and accumulate object data (e.g., size, position relative to vehicle 200) for each frame. Furthermore, processing unit 110 can estimate parameters of one or more detected objects and compare the position data of one or more objects included in each frame with one or more estimated positions.

[0126] Subsequently, in step S546, processing unit 110 can generate a set of measurements for one or more detected objects. Such measurements may include, for example, the position, velocity, and acceleration values ​​of one or more detected objects relative to vehicle 200. In some embodiments, processing unit 110 may generate measurements based on estimation techniques using a series of time-based observations (e.g., Kalman filter, linear quadratic estimation (LQE), etc.). Furthermore, processing unit 110 may generate measurements based on available modeling data for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs). Kalman filters may be based on measurements of object proportions. Such proportion measurements are proportional to the collision time (e.g., the time period until vehicle 200 reaches the object). Therefore, by performing steps from S540 to S546, processing unit 110 can identify vehicles and pedestrians appearing in a series of captured images and obtain information about the vehicles and pedestrians (e.g., position, velocity, size). Then, based on the information identified and obtained in this way, processing unit 110 can cause vehicle 200 to generate one or more navigation responses, as previously referenced. Figure 13 As described.

[0127] Subsequently, in step S548, processing unit 110 may perform optical flow analysis on one or more images and detect false hits, thereby reducing the probability of missing candidate objects representing vehicles or pedestrians. Here, optical flow analysis may be an analysis of the pattern of movement relative to vehicle 200, which differs from the movement of the road surface in one or more images of other vehicles and pedestrians. Furthermore, processing unit 110 may calculate the movement of one or more candidate objects by observing changes in the position of one or more candidate objects in multiple image frames captured at different times. Here, processing unit 110 may use position and time as inputs to a mathematical model for calculating the movement of one or more candidate objects. In this way, optical flow analysis can provide another method for detecting vehicles and pedestrians present near vehicle 200. Subsequently, in step S548, processing unit 110 may combine the processing of steps S540 to S546 to perform optical flow analysis to provide redundancy for the purpose of detecting vehicles and pedestrians, thereby improving the reliability of imaging system 100.

[0128] Figure 15This is a flowchart illustrating an exemplary process 500C for detecting road markings and / or lane geometry information in a set of images according to one embodiment of the present disclosure. Specifically, for the purpose of performing process 500C, processing unit 110 may execute instructions in monocular image analysis model 402. Thus, in step S550, processing unit 110 may detect a series of objects by scanning one or more images. To detect lane marking segments, lane geometry information, and other suitable road markings, processing unit 110 may filter a series of objects and exclude given objects determined to be irrelevant (e.g., holes, small stones). Subsequently, in step S552, processing unit 110 may group the segments detected in step S550 into those belonging to the same road marking or lane marking. Based on such grouping, processing unit 110 may develop a model, such as a mathematical model, representing the detected segments.

[0129] Subsequently, in step S554, processing unit 110 can generate a set of measurements for the detected segment. In some embodiments, processing unit 110 can generate a projection of the detected segment by projecting the segment from an image plane to a real-world plane. The projection can be characterized using a third-order polynomial composed of coefficients corresponding to the physical characteristics of the detected road (e.g., position, inclination, curvature, differential curvature, etc.). When generating the projection, processing unit 110 can use information about changes in the road surface and the pitch and roll rates of vehicle 200. Furthermore, processing unit 110 can model the road height by analyzing positional and movement cues present on the road surface. Here, positional cues can be the detected road position, inclination, and curvature. Additionally, the differential curvature value of the detected road can serve as a cue. Movement cues include the vehicle's pitch and / or roll rates, etc. That is, based on these cues, the road height and inclination are estimated.

[0130] Furthermore, the processing unit 110 can estimate the pitch and roll rates of the vehicle 200 by tracking a set of feature points included in one or more images.

[0131] Subsequently, in step S556, processing unit 110 can perform multi-frame analysis, for example, by tracking segments detected successively in image frames and accumulating data for each segment in each image frame. When processing unit 110 performs multi-frame analysis, the set of measurements generated in step S554 can become more reliable. Therefore, increasingly higher confidence levels can be assigned to this set of measurements. Thus, by performing steps from S550 to S556, processing unit 110 can identify road markings appearing in the captured image set, thereby making it possible to obtain lane geometry information. Based on the information identified and obtained in this way, processing unit 110 can, as previously referred to... Figure 13The description enables vehicle 200 to generate one or more navigation responses.

[0132] Subsequently, in step S558, processing unit 110 can utilize additional information sources to further develop a safety model for vehicle 200, taking into account surrounding conditions. Specifically, processing unit 110 can define the conditions under which imaging system 100 can safely perform autonomous control of vehicle 200 by using the safety model. For example, in some embodiments, to develop the safety model, processing unit 110 can utilize the positions and movements of other vehicles, information on detected road edges and barriers, and / or descriptions of general road shapes obtained from map data, such as data in map database 160. Furthermore, by using additional information sources, processing unit 110 can provide redundancy when detecting road markings and lane shapes, thereby enhancing the reliability of imaging system 100.

[0133] Figure 16 This is a flowchart illustrating an exemplary process 500D for detecting traffic signals in a series of images according to one embodiment of the present disclosure. Specifically, processing unit 110 can execute instructions from monocular image analysis module 402 and perform process 500D. Thus, in step S560, processing unit 110 can scan a series of images and identify objects appearing in the images at locations that may include traffic signals. For example, processing unit 110 can generate a set of candidate objects by applying filtering processing to the identified objects and excluding (i.e., filtering out) applicable objects that are unlikely to correspond to traffic lights. Such filtering can be performed based on various characteristics of traffic lights, such as shape, size, texture, position (e.g., position relative to vehicle 200), etc. Such characteristics can be stored in a database as multiple examples of traffic signals and traffic control signals. Here, in some embodiments, processing unit 110 can perform multi-frame analysis based on a set of candidate objects that may reflect traffic signals. For example, processing unit 110 can track candidate objects on successive image frames and estimate the real-world positions of candidate objects, thereby filtering out moving objects that are unlikely to be traffic lights. Furthermore, in some embodiments, the processing unit 110 may perform color analysis of the candidate objects and identify the relative positions of colors that appear and are detected in the applicable traffic lights.

[0134] Subsequently, in step S562, processing unit 110 can analyze the shape of the intersection. The analysis can be performed based on any combination of the first to third information listed below. The first information is the number of lanes detected on both sides of vehicle 200. The second information is the markings detected on the road, such as arrow markings. The third information is a description of the intersection extracted from map data (e.g., data extracted from map database 160). Then, processing unit 110 can analyze the information obtained by executing instructions from monocular image analysis module 402. Then, in step S560, processing unit 110 can determine whether the traffic lights detected in step S560 correspond to one or more lanes appearing near vehicle 200.

[0135] Subsequently, in step 564, as vehicle 200 approaches the intersection, processing unit 110 can update the confidence level of the geometry assigned to the analyzed intersection and the detected traffic lights. That is, the confidence level can be changed by comparing the estimated number of traffic lights appearing at the intersection with the actual number appearing. Therefore, based on the reliability level, processing unit 110 can delegate control to the driver of vehicle 200 to improve safety. Thus, processing unit 110 can identify traffic lights appearing in the captured image set and analyze the intersection's geometric information by executing steps S560 to S564. Subsequently, based on the identification and analysis, processing unit 110 can cause vehicle 200 to generate one or more navigation responses, as previously described. Figure 13 As described.

[0136] Figure 17This is a flowchart illustrating an exemplary process 500E of a vehicle-based route control vehicle 200 generating one or more navigation responses according to one embodiment of the present disclosure. Specifically, in step S570, processing unit 110 can establish a vehicle route in the initial stage of vehicle 200. The vehicle route can be represented by a set of points indicated by coordinates (x, z). The distance "d" between two points in the set can range from approximately 1 meter to approximately 5 meters. In one embodiment, processing unit 110 can establish (i.e., construct) the initial stage vehicle route using two polynomials: a left-road polynomial and a right-road polynomial. Processing unit 110 can calculate a geometric midpoint, defined between two points obtained by calculating these two polynomials, to obtain the vehicle route as a result of the calculation. A given offset (e.g., a so-called intelligent lane offset) can be assigned to each of the geometric midpoints (if a geometric midpoint exists). Thus, if the offset is zero, the vehicle can accordingly travel along the middle of the lane. Here, the offset can be perpendicular to the segment orientation defined between any two points in the vehicle route. In another implementation, processing unit 110 may define the vehicle route based on a polynomial and half of the estimated lane width. A given offset (e.g., a so-called intelligent lane offset) is then added to each half-point in the vehicle route.

[0137] Subsequently, in step S572, processing unit 110 can update the vehicle route established in step S570. Specifically, processing unit 110 can use a higher resolution to make the distance d between two points in the set of points representing the vehicle route more consistent. k The distance di(d) is less than the distance described above to reconstruct (i.e., re-establish) the vehicle route established in step S570. For example, distance d k The range can be from approximately 0.1 meters to approximately 0.3 meters. More specifically, processing unit 110 can reconstruct the vehicle route using a parabolic spline algorithm. That is, using this algorithm, processing unit 110 can obtain the cumulative distance vector S based on the set of points representing the total length of the vehicle route.

[0138] Subsequently, in step S574, processing unit 110 can determine the predictive point represented by (X1, Z1) in coordinates based on the vehicle route updated in step S572. Here, processing unit 110 can extract the predictive point based on the cumulative distance vector S. The predictive point can be a predictive distance and a predictive time. The predictive distance can be calculated as the product of the vehicle 200's speed and the predictive time, with a lower limit ranging from approximately 10m to approximately 20m. For example, when the speed of vehicle 200 decreases, the predictive distance can also be reduced to a lower limit. Here, the predictive time can range from approximately 0.5 seconds to approximately 1.5 seconds. The predictive time can be inversely proportional to the gain of one or more control loops used to generate a navigation response in vehicle 200, such as a heading error tracking control loop. For example, the gain of the heading error tracking control loop can be determined based on the bandwidth of each of the yaw rate loop, the steering actuator loop, and the vehicle's dynamics in its lateral direction. Therefore, the higher the gain of the heading error tracking control loop, the shorter the predictive time.

[0139] Subsequently, in step S576, processing unit 110 can determine the values ​​of the heading error and yaw rate command based on the predictability point determined in step S574. Here, for example, processing unit 110 can determine the presence of heading error by calculating the arctangent of the predictability point, such as arctan(X1 / Z1). Furthermore, processing unit 110 can determine the yaw rate command as the product of the bearing error and the advanced control gain. If the predictability distance is not a lower limit, the advanced control gain can be equal to a value calculated as 2 / predictability time. In contrast, if the predictability distance is a lower limit, the advanced control gain can be a value calculated using the following formula: 2 × vehicle speed 200 / predictability distance.

[0140] Figure 18 This is a flowchart illustrating an exemplary process 500F for determining whether a preceding vehicle is changing lanes according to one embodiment of the present disclosure. Specifically, in step S580, processing unit 110 can select navigation information of the preceding vehicle (e.g., another vehicle traveling in front of this vehicle 200). Processing unit 110 can then use the previously referenced... Figure 13 and Figure 14 The described technique determines the position, velocity (i.e., direction and speed), and / or acceleration of the vehicle in front. Furthermore, the processing unit 110 can use the aforementioned reference... Figure 17 The described technique determines one or more road polynomials, predictive points of vehicle 200 (i.e., preceding vehicles), and / or snail trajectories (e.g., a set of points describing the path the preceding vehicle travels).

[0141] Subsequently, in step S582, processing unit 110 can analyze the navigation information selected in step S580. In one embodiment, processing unit 110 can calculate the distance along the road between the serpentine trajectory and the road polynomial. If such a distance difference along the serpentine trajectory exceeds a given threshold, processing unit 110 can determine that the vehicle ahead may be changing lanes. Here, for example, the given threshold may be from approximately 0.1 meters to approximately 0.2 meters on a straight road, from approximately 0.3 meters to approximately 0.4 meters on a moderately curved road, and from approximately 0.5 meters to approximately 0.6 meters on a sharply curved road. Otherwise, if multiple vehicles are detected traveling in front of vehicle 200, processing unit 110 can compare the serpentine trajectories of these vehicles among them. Then, based on the comparison results, processing unit 110 can determine that a vehicle whose serpentine trajectory does not match the serpentine trajectories of other vehicles is likely changing lanes. Furthermore, processing unit 110 can compare the curvature of the serpentine trajectory of the vehicle ahead with the expected curvature of the road segment along which the vehicle ahead is traveling. The expected curvature can be extracted from map data (e.g., data from map database 160), the polynomial of the road, and the zigzag trajectory of other vehicles. The expected curvature can also be extracted from prior knowledge about the road. Then, if the difference between the curvature of the zigzag trajectory and the expected curvature of the road segment exceeds a given threshold, the processing unit 110 can determine that the vehicle ahead may be changing lanes.

[0142] In another embodiment, processing unit 110 can compare the instantaneous position of the preceding vehicle with the predictive point of vehicle 200 for a given time period (e.g., approximately 0.5 seconds to approximately 1.5 seconds). Then, if the distance between the instantaneous position of the preceding vehicle and the predictive point varies during the given time period, and the cumulative sum of the distance fluctuations exceeds a given threshold, processing unit 110 can determine that the preceding vehicle may be changing lanes. Here, the given threshold may be, for example, from approximately 0.3 meters to approximately 0.4 meters on a straight road, from approximately 0.7 meters to approximately 0.8 meters on a moderately curved road, and from approximately 1.3 meters to approximately 1.7 meters on a sharply curved road. In yet another embodiment, processing unit 110 can analyze the geometry of the worm track by comparing the lateral distance the preceding vehicle has traveled along the worm track with the expected curvature of the worm track. Here, the expected radius of curvature can be calculated using the formula listed below: (δ... z 2 +δ x 2 ) / 2 / (δ x ), where δ x Indicates the horizontal driving distance, and δ zThis represents the longitudinal travel distance. Therefore, when the difference between the lateral travel distance and the expected radius of curvature exceeds a given threshold (e.g., from approximately 500 meters to approximately 700 meters), the processing unit 110 can determine that the vehicle ahead may be changing lanes. In another embodiment, the processing unit 110 can analyze the position of the vehicle ahead. Specifically, when the position of the vehicle ahead obscures the road polynomial (e.g., the vehicle ahead is superimposed on the road polynomial as a result of calculation), the processing unit 110 can determine that the vehicle ahead may be changing lanes. In yet another embodiment, when another vehicle is detected in front of the vehicle ahead and the zigzag trajectories of the two vehicles are not parallel to each other, the processing unit 110 can determine that the vehicle closer to it may be changing lanes.

[0143] Therefore, in step S584, processing unit 110 can determine whether the preceding vehicle 200 is changing lanes based on the analysis performed in step S582. Here, processing unit 110 can make this determination by weighting and averaging the various analyses performed in step S582. For example, in such a method, a value 1 (a) can be assigned to the determination by processing unit 110 that the preceding vehicle may be changing lanes based on a given type of analysis. In contrast, a value 0 (zero) can be assigned to the determination by processing unit 110 that the preceding vehicle is unlikely to be changing lanes. Furthermore, different weights can be assigned to the different analyses performed in step S582. That is, each of the embodiments of this disclosure is not limited to any particular combination of analysis and weights.

[0144] Figure 19 This is a flowchart illustrating an exemplary process 600, according to one embodiment of the present disclosure, where stereoscopic image analysis leads to the generation of one or more navigation responses. Specifically, in step S610, processing unit 110 may receive a first plurality of images and a second plurality of images via data interface 128. For example, a camera included in image acquisition unit 120 (e.g., image acquirer 122 or image acquirer 124 having a field of view 202 or 204) may capture multiple images of an area in front of vehicle 200 and transmit the images to processing unit 110 via a digital connection system. Here, the digital connection system may be a wired connection system or a wireless connection system. In some embodiments, processing unit 110 may receive the first plurality of images and the second plurality of images via two or more data interfaces. However, the present disclosure is not limited to any particular data interface. Furthermore, the present disclosure is not limited to any particular protocol.

[0145] Subsequently, in step S620, processing unit 110 can execute the instructions of stereo image analysis module 404 and perform stereo image analysis on the first plurality of images and the second plurality of images. Processing unit 110 can then create a 3D map of the road area in front of the vehicle and detect features included in the image, such as lane markings, vehicles, pedestrians, etc. Processing unit 110 can also detect road signs, highway exit ramps, and traffic lights as features in the image based on the 3D map. Processing unit 110 can also detect road hazards as features in the image based on the 3D map. Stereo image analysis can be substantially similar to the analysis described above. Figures 13 to 16 The applicable steps described are performed similarly. For example, processing unit 110 executes the instructions of stereo image analysis module 404 and detects candidate objects (e.g., vehicles, pedestrians, road markings, traffic lights, road hazards, etc.) included in the first plurality of images and the second plurality of images. Processing unit 110 then filters out a subset of candidate objects using one of various criteria and performs a multi-frame analysis to analyze the remaining candidate objects. Processing unit 110 then obtains the measurement results and determines their confidence level. Here, in step S620 (i.e., analysis of navigation information), processing unit 110 may utilize information from both the first plurality of images and the second plurality of images, rather than information from only one set of plurality of images. For example, processing unit 110 may analyze the differences in pixel data between candidate objects appearing in each of the first plurality of images and the second plurality of images. Alternatively, processing unit 110 may analyze the differences in data subsets between two captured image streams of candidate objects appearing in each of the first plurality of images and the second plurality of images. As another example, processing unit 110 can estimate the position and / or velocity of a candidate object relative to vehicle 200 by observing events in which the candidate object appears in one of several images but not in the other several images. Alternatively, the position and / or velocity of the candidate object relative to vehicle 200 can be estimated based on other differences between objects appearing in two image streams. For example, the position, velocity, and / or acceleration relative to vehicle 200 can be determined based on the trajectory, position, and movement characteristics of objects appearing as features in two image streams, etc.

[0146] Subsequently, in step 630, processing unit 110 may, based on the analysis performed in step 620 and the preceding references, Figure 4The described techniques execute instructions from navigation response module 408 to cause vehicle 200 to generate one or more navigation responses. Here, navigation responses may include, for example, turning, lane changing, and changes in acceleration. Navigation responses may also include speed changes and braking, etc. In some embodiments, processing unit 110 may use data obtained as a result of executing instructions from speed-acceleration module 406. Therefore, data obtained as a result of executing instructions from speed-acceleration module 406 can be used to cause vehicle 200 to generate one or more navigation responses. Here, multiple navigation responses may be generated simultaneously, sequentially, or in any combination thereof.

[0147] Figure 20 This is a flowchart illustrating an exemplary process 700 in which a vehicle 200 generates one or more navigation responses based on the analysis of three image sets according to one embodiment of the present disclosure. Specifically, in step 710, the processing unit 110 may receive a first plurality of images, a second plurality of images, and a third plurality of images via data interface 128. For example, cameras included in the image acquisition unit 120 (e.g., image acquirers 122, 124, and 126, etc., having fields of view 202, 204, and 206) may capture multiple images of the front and / or side regions of the vehicle 200 and transmit the images to the processing unit 110 via a digital connection. In some embodiments, the processing unit 110 may receive a plurality of first, second, and third images via three or more data interfaces. For example, each of the image acquirers 122, 124, and 126 may have a data interface for transmitting data to the processing unit 110. However, the present disclosure is not limited to any given data interface or even protocol.

[0148] Subsequently, in step 720, processing unit 110 can analyze the first plurality of images, the second plurality of images, and the third plurality of images, and detect features included in the images such as lane markings, vehicles, pedestrians, etc. Processing unit 110 also detects features included in the images such as road signs, highway exit ramps, traffic lights, etc. Processing unit 110 also detects features included in the images such as road hazards. Such analysis can be substantially similar to the analysis described above. Figures 13 to 16 as well as Figure 19 The steps described are performed similarly. That is, processing unit 110 can, for example, perform monocular image analysis of each of the first plurality of images, the second plurality of images, and the third plurality of images. Here, this can be achieved by executing the instructions of monocular image analysis module 402 and performing the steps described above. Figures 13 to 16The steps described are used to perform monocular image analysis. Alternatively, processing unit 110 can perform stereo image analysis of a first combination of a first plurality of images and a second plurality of images, a second combination of a second plurality of images and a third plurality of images, and / or a third combination of a first plurality of images and a third plurality of images. Here, this is achieved by executing the instructions of stereo image analysis module 404 and performing the steps described above. Figure 19 The steps described are used to perform stereoscopic image analysis. Information processed corresponding to the analysis of a first plurality of images, a second plurality of images, and / or a third plurality of images can be combined. Furthermore, in some embodiments, processing unit 110 can perform a combination of monocular image analysis and stereoscopic image analysis. For example, processing unit 110 can perform monocular image analysis by analyzing a first plurality of images and stereoscopic image analysis by analyzing a second plurality of images and a third plurality of images. Here, monocular image analysis is performed by executing instructions from monocular image analysis module 402. Furthermore, stereoscopic image analysis is performed by executing instructions from stereoscopic image analysis module 404. Here, the positions of image acquirers 122, 124, and 126 and their fields of view 202, 204, and 206 can influence the selection of the type of analysis performed on the first plurality of images, the second plurality of images, and the third plurality of images. However, the previously disclosed content is not limited to a specific image acquirer 122, 124, or 126, or the type of analysis performed on the first plurality of images, the second plurality of images, and the third plurality of images.

[0149] In some implementations, processing unit 110 may test imaging system 100 based on the images acquired and analyzed in steps S710 and S720. Such testing can provide an indicator of the overall performance of imaging system 100 relative to image acquirers 122, 124, and 126 with a given configuration. For example, processing unit 110 may determine the percentage of each of false hits and errors. Here, a false hit indicates a situation where imaging system 100 incorrectly determines the presence of a vehicle or pedestrian. An error represents ignoring such an object.

[0150] Subsequently, in step S730, processing unit 110 may generate one or more navigation responses for vehicle 200 based on information obtained from all the first plurality of images, the second plurality of images, and the third plurality of images, or from any two of the first plurality of images, the second plurality of images, and the third plurality of images. Here, selecting two such sets of images from the first plurality of images, the second plurality of images, and the third plurality of images may depend at least on one of the object-related factors. For example, factors include the number, type, and size of objects detected in each of the plurality of images. Furthermore, processing unit 110 may select two such sets of images based on the effective field of view, image quality, and resolution reflected in the images. Processing unit 110 may also select such two sets based on the number of frames captured and the degree of actual presence (i.e., occurrence) of one or more objects of interest in the frames. Here, the degree of actual presence in a frame refers to the frequency of frames in which objects appear, or the ratio of the size of the object to the overall size of the frames in which the object appears, etc.

[0151] Furthermore, in some embodiments, the processing unit 110 may select two groups of images from a first plurality of images, a second plurality of images, and a third plurality of images based on the degree of matching between information obtained from one image source and information obtained from another image source. For example, the processing unit 110 may process information obtained from each of the image acquirers 122, 124, and 126, and identify visual indicators that consistently appear in the plurality of image groups captured by the image acquirers 122, 124, and 126 based on a combination of this information. Here, visual indicators include lane markings, detected vehicles and their positions and / or routes, and detected traffic lights, etc.

[0152] For example, processing unit 110 can combine information obtained and processed from each of image acquirers 122, 124, and 126. Processing unit 110 can then determine the presence of visual indicators that are consistent with each other in a group of multiple images captured from image acquirers 122, 124, and 126. Specifically, processing unit 110 combines information obtained from each of image acquirers 122, 124, and 126 that is processed independently of performing monocular analysis, stereo analysis, or any combination of both (i.e., multiple image groups). Here, consistent visual indicators included in the images captured from image acquirers 122, 124, and 126 represent lane markings, detected vehicles, vehicle positions, and / or vehicle routes. Such visual indicators could also be detected traffic lights, etc. Furthermore, processing unit 110 can exclude information that is inconsistent with other information (i.e., multiple image groups). Here, inconsistent information could be vehicles changing lanes, lane models indicating that a vehicle is too close to vehicle 200, etc. In this way, the processing unit 110 can select information obtained from two groups (i.e., multiple image groups) from the first plurality of images, the second plurality of images, and the third plurality of images based on the determination of consistency and inconsistency.

[0153] Here, the navigation response may include turning, lane changing, and changes in acceleration, etc. Processing unit 110 may base its response on the analysis performed in step S720 and as previously described. Figure 4 The described techniques enable vehicle 200 to generate one or more navigation responses. Processing unit 110 can also enable vehicle 200 to generate one or more navigation responses using data obtained by executing instructions from velocity-acceleration module 406. In some embodiments, processing unit 110 can enable vehicle 200 to generate one or more navigation responses based on the relative position, relative velocity, and / or relative acceleration between vehicle 200 and objects detected in any of a first plurality of images, a second plurality of images, and a third plurality of images. These plurality of navigation responses can occur simultaneously, sequentially, or in any combination of these sequences.

[0154] Figure 21This diagram schematically illustrates components of an exemplary imaging device 2500 used as an image acquirer. The exemplary imaging device 2500 includes a lens system 1200 coupled to an image sensor 2100 used as an imager system. The lens system 1200 is housed in a lens barrel 1202 having a front cover glass 1204 and a rear cover glass 1218. The exemplary lens system 1200 includes a composite lens comprising a first biconvex lens 1206, a second biconvex lens 1208, a first positive meniscus lens 1210, and a biconvex lens 1212. The exemplary lens system 1200 also includes a second positive meniscus lens 1214. The lens system 1200 also includes a cutoff filter 1216 that attenuates IR (infrared) and UV (ultraviolet) light in the spectral range projected from the lens system 1200 onto the image sensor 2100. Because the lens system 1200 is configured to provide a relatively large MTF (modulation transfer function) when receiving light in the spectral range from red to green, the cutoff filter 1216 can be configured to attenuate at least a portion of light in the blue spectral range in addition to light in the IR and UV spectral ranges. Here, at least one of the biconvex lenses 1206, 1208, and 1212, the first positive meniscus lens 1210, and the second positive meniscus lens 1214 can be spherical or aspherical as lens elements. The lens elements constituting the lens—i.e., the positive meniscus lens 1210 and the biconvex lens 1212—can be bonded together by optical adhesive or separated from each other by air. However, such a lens configuration is only an example and can be optionally changed. That is, other configurations that satisfy the design rules described below with reference to Tables 1, 2, and 3 can be used instead of the lens system 1200.

[0155] Furthermore, the imaging device 2500 may also include a housing 1222, a color filter array 2300, and an APS image sensor (hereinafter simply referred to as the image sensor) 1226. The image sensor may be a CMOS (Complementary Metal-Oxide-Semiconductor) sensor. Here, for ease of understanding, the relative sizes of each of the color filter array 2300 and the image sensor 1226 are exaggerated. The image sensor 1226 is positioned in the housing 1222 relative to the lens system 1200 such that an image from the scene is focused onto the upper surface of the image sensor 1226 via the color filter array 2300. Pixel data captured by the image sensor 1226 is provided to the processing circuitry 2400. The processing circuitry 2400 enables control of the operation of the image sensor 1226.

[0156] In automotive applications, image data in the blue spectral range may sometimes be less important than image data in the red to green spectral range. Generally, a way to improve the quantum efficiency of an imager without increasing the number of lens apertures is to design a lens that, when used with a color filter, produces a clearer image in the red to green spectral range than in the blue spectral range. However, it is not always necessary to improve the quantum efficiency of an imager by designing lenses in this way. That is, when the importance of image data in the blue spectral range is no less than that of image data in the red to green spectral range, the cutoff filter 1216 does not need to be configured to attenuate light in the blue spectral range.

[0157] Here, as an example Figure 21 The lens system 1200 shown can be based on Figure 22 Table 1 Figure 23 Table 2 and Figure 24 The design follows the design rules shown in Table 3. In these tables, Table 1 shows the weighted wavelengths used in the lens design. Table 2 shows the multicolor MTF used in the lens design with wavelengths weighted as shown in Table 1. Table 3 shows the parameters of cutoff filter 1216, which is configured to attenuate UV light with wavelengths less than the cutoff wavelength, ranging from approximately 395 nm to approximately 410 nm, and IR light with wavelengths greater than the cutoff wavelength, ranging from approximately 690 nm to approximately 705 nm. As described above, in some embodiments, cutoff filter 1216 can be configured to attenuate light with wavelengths less than approximately 500 nm, thereby attenuating not only light in the blue to violet spectral range but also light in the UV spectral range.

[0158] Furthermore, the aforementioned design rules specify a lens system in which the optical focus is emphasized in a field of view of approximately 60 degrees, emphasizing light in the spectral range from red to green compared to others. Additionally, the weighting design rules in Table 1 assign higher wavelength values ​​for yellow than for red and blue. In this way, the field-of-view design rules shown in Tables 1 to 3 specify a relatively high MTF for light in at least the spectral range from red to green throughout the entire field of view of the lens system. Such a lens system is used by the processing circuitry 2400 included in the imaging device 2500 and can identify items of interest included throughout the entire field of view of the imaging device 2500.

[0159] Figure 25An example of an image sensor 2100 is shown, which includes a pixel array 2105, a readout circuit 2170 coupled to the pixel array 2105, and a control circuit 2120 coupled to the pixel array 2105. Here, the pixel array 2105 includes various image sensors having X pixel columns and Y pixel rows, or including a two-dimensional (2D) array of pixels (e.g., pixels P1, P2, ..., Pn). The pixel array 2105 is used as such... Figure 26 The image sensor shown has front-side illumination, or as... Figure 27 The image sensor shown has rear-side illumination. As illustrated, each pixel P of array 2105 is arranged in rows (e.g., rows from R1 to Ry) and columns (e.g., columns from C1 to Cx) to obtain image data of individuals, locations, and / or objects. These pixels P are then used to present a 2D image of the person, location, and / or object based on the image data. As described further later, pixel array 2105 can assign color to each pixel P using a color filter array 2300 coupled to pixel array 2105. Here, a single pixel is used as a single point in a color image consisting of a set of points. The unit (single / one) color filter, described in detail later, corresponds to a single pixel.

[0160] Therefore, when each pixel of the pixel array 2105 has acquired image data (i.e., image charge), the image data is read by the readout circuit 2170. The image data is then transmitted to the processing circuit 2400 for purposes such as storage and additional processing therein. The readout circuit 2170 includes amplifier circuitry and analog-to-digital converter (ADC) circuitry or other circuitry. The processing circuit 2400 is coupled to the readout circuit 2170. The processing circuit 2400 executes functional logic. The processing circuit 2400 can process or manipulate the image data while storing it by applying cropping, rotation, and red-eye removal as post-image actions. The processing circuit 2400 can also process (or manipulate) the image data while storing it by applying brightness adjustment and contrast adjustment as post-image actions. In one embodiment, the processing circuit 2400 is also used to process the image data to correct (i.e., reduce or remove) fixed pattern noise. Furthermore, a control circuit 2120 coupled to the pixel array 2105 is used for the purpose of controlling the operating characteristics of the pixel array 2105. For example, control circuit 2120 generates a shutter signal for controlling image acquisition by pixel array 2105.

[0161] Figure 26This is a cross-sectional view showing a pair of exemplary front-side illumination pixels (hereinafter referred to as FSI pixels) 2200 included in a CMOS image sensor. Here, the front side of the FSI pixel 2200 is the following side of the substrate 2202: both the photoelectric conversion element 2204, which serves as the optical sensing area, and the corresponding pixel circuitry, as well as a metal stack 2206 for signal redistribution, are formed on this side in this order. The metal stack 2206 includes metal layers M1 and M2—each of which is patterned to form an optical channel. Thus, light incident on the FSI pixel 2200 reaches the photoelectric conversion element 2204 through this channel. For use as a color image sensor, the front side of the FSI pixel 2200 includes a color filter array 2300. The color filter array 2300 includes a primary color monochromatic filter 2303. The primary color monochromatic filter 2303 is disposed below a microlens 2207, which effectively focuses the incident light onto the photoelectric conversion element 2204. Here, for simplicity, Figure 26 The cross-sectional view shows only two primary color monochromatic filters 2303. The color filter array 2300 includes a minimum repeating unit 2302, as described in more detail later.

[0162] Figure 27 This is a cross-sectional view illustrating a pair of exemplary rear-side illumination pixels (hereinafter referred to as BSI pixels) 2250 included in a CMOS image sensor according to one embodiment of the present disclosure. Figure 26 Compared to the case shown, the front side of the BSI pixel 2250 is the side below the color filter 2303: the substrate 2202, the photoelectric conversion element 2204 and the corresponding pixel circuit, and the metal stack 2206 for redistributing signals are formed on this side in this order.

[0163] To construct a color image sensor, a color filter array 2300 is included behind the BSI pixel 2250. The color filter array 2300 includes a primary color monochromatic filter 2303. The primary color monochromatic filter 2303 is arranged below the microlens 2207. However, for simplicity... Figure 27 The cross-sectional view shows only two primary color monochromatic filters 2303. Here, the color filter array 2300 is a color filter array formed by one of the smallest repeating units described in more detail below. The microlens 2207 effectively focuses the incident light onto the photoelectric conversion element 2204. Utilizing such back-side illumination of the BSI pixel 2250, the metal interconnects of the metal stack 2206 do not interfere with the path formed between the object to be imaged and the photoelectric conversion element 2204, allowing the photoelectric conversion element 2204 to generate a larger signal.

[0164] Figure 28A color filter array 2300 and a single set of minimum repeating units 2302 in a tiled state to form the color filter array 2300 are shown. The color filter array 2300 includes a number of primary color monochromatic filters 2303 that substantially correspond to the number of individual pixels P in the pixel array 2105 to which the color filter array 2300 is coupled or to be coupled. Each primary color monochromatic filter 2303 is optically coupled to a corresponding individual pixel P in the pixel array 2105 and has a given spectral light responsivity selected from a single set of spectral light responsivity. The given spectral light responsivity has high sensitivity to a given portion of the electromagnetic spectrum and low sensitivity to other portions of the spectrum. Although pixel P itself does not have color, pixel P is generally considered to have a given light response because the color filter array 2300 assigns the light response individually to each pixel P by placing the primary color monochromatic filters 2303 on pixel P. Therefore, when pixel P is combined with a blue filter, it is called a blue pixel. Similarly, when another pixel P is combined with a green filter, it is called a green pixel. Furthermore, when another pixel P is combined with a red filter, it is referred to as a red pixel.

[0165] The primary color monochromatic filters 2303 of the color filter array 2300 are grouped into minimum repeating units 2302. The primary color monochromatic filters 2303 are color filters arranged corresponding to individual photoelectric conversion elements 2204. The minimum repeating units 2302 are tiled vertically and horizontally as indicated by arrows to form the color filter array 2300. Here, the minimum repeating unit 2302 is a repeating unit without fewer individual filters. The color filter array 2300 may include many different repeating units. However, if another repeating unit with fewer individual filters exists in the array, then the repeating unit is not the minimum repeating unit. In other examples of the color filter array 2300, the minimum repeating unit may be larger or smaller than the minimum repeating unit 2302 in this example.

[0166] Figure 29 The configuration of the color filter for the minimum repeating unit 2302 is shown. Figure 29 The minimum repeating unit 2302 shown includes four primary color monochromatic filters 2303 that are independent of each other. Specifically, the minimum repeating unit 2302 includes a single red monochromatic filter 2303R, a single blue monochromatic filter 2303B, and two green monochromatic filters 2303G.

[0167] As shown in the figure, each primary color monochromatic filter 2303 is square in shape, and the four primary color monochromatic filters 2303 are arranged in two rows and two columns. Therefore, the smallest repeating unit 2302 also has a square shape. However, this disclosure is not limited to this, and the shape of the primary color monochromatic filter 2303 is not necessarily square.

[0168] In addition, such as Figure 29 As shown, in the minimum repeating unit 2302, a red monochromatic filter 2303R, a green monochromatic filter 2303G, and a blue monochromatic filter 2303B are arranged by forming a Bayer array.

[0169] Therefore, the red monochromatic filter 2303R transmits red light, which is used as one of the three primary colors. Additionally, the red monochromatic filter 2303R also transmits light of a primary color different from the corresponding primary color (i.e., red), although its transmittance is not as high as that of red.

[0170] Figure 30 This is a graph showing the relationship between the transmittance and wavelength of the red monochromatic filter 2303R. Figure 30 In the diagram, the solid line shows the transmittance of the red monochromatic filter 2303R. The dashed line represents the transmittance of a typical red filter for comparison. Although depending on the definition, the transmittance of red in the graph is approximately 650 nm. The red monochromatic filter 2303R has approximately 100% transmittance for red light and therefore allows red light to pass through. However, since this is merely an example, this disclosure is not limited to approximately 100% transmittance for red light. That is, it may be sufficient for the red monochromatic filter 2303R to have a higher transmittance for red light than for other primary colors.

[0171] Furthermore, green light has a wavelength of approximately 540 nm, and blue light has a wavelength of approximately 400 nm. The dashed line in the graph, used for comparison, shows a standard red filter that almost never allows light of other primary colors to pass through. In contrast, the red monochromatic filter 2303R transmits light of primary colors other than red, even though its transmittance is not as high as red. Specifically, as... Figure 30 As an example, the red monochromatic filter 2303R has approximately 30% transmittance for other primary colors.

[0172] That is, since almost all imaging targets have a broader spectrum of light than a single wavelength (i.e., monochromatic), the amount of light from the object detected by pixels of each color in the RGB color spectrum can be increased, and the sensitivity is correspondingly improved if the wavelength range of light detected by pixels of each color in the RGB color spectrum is expanded. Therefore, one embodiment of this disclosure expands the wavelength range of light detected by pixels of each color in the RGB color spectrum to satisfy the following inequality.

[0173] Black and white > One implementation method > Ordinary RGB

[0174] Here, the sensitivity is calculated as follows. First, it is assumed that the object is white and the light intensity (L) is uniform in the wavelength range from 380nm to 680nm. It is also assumed that the image sensor with filters has 0% transmittance in the wavelength range other than 380nm to 680nm and 100% transmittance in the wavelength range from 380nm to 680nm. Further, it is assumed that the output wavelength range of the RGB color filter is from 380nm to 680nm. Specifically, color filter B has 100% transmittance in the wavelength range from 380nm to 480nm, color filter G has 100% transmittance in the wavelength range from 480nm to 580nm, and color filter G' has 100% transmittance in the wavelength range from 580nm to 680nm. It is also assumed that the RGB type filters of this embodiment transmit 30% of the other wavelengths respectively. Therefore, the sensitivity of a normal RGB pixel is calculated as follows:

[0175] R = 100nm × L × 1.0,

[0176] G = 100nm × L × 1.0

[0177] B = 100nm × L × 1.0.

[0178] In contrast, the sensitivity of each of the RGB type filters in this embodiment is calculated by the following equation and is 1.9 times that of each of ordinary RGB pixels.

[0179] R=100nm×L×1.0+300nm×L×0.3=100nm×L×1.9,

[0180] G=100nm×L×1.0+300nm×L×0.3=100nm×L×1.9,

[0181] B=100nm×L×1.0+300nm×L×0.3=100nm×L×1.9.

[0182] However, the 30% ratio is merely an example of comparing low effective transmittance with high transmittance.

[0183] Furthermore, a lower effective transmittance is a lower limit on the transmittance that can be effectively used to improve the sensitivity of the image sensor 2100. The lower effective transmittance can be appropriately determined according to the required specifications of the image sensor 2100, etc. Furthermore, the lower effective transmittance is at least a level that can distinguish transmittance from noise level. Therefore, for example, the lower effective transmittance can be one of 10%, 15%, and 20%. Furthermore, the lower effective transmittance can be, for example, 25%.

[0184] In addition, such as Figure 30 As shown in the graph, according to the red monochromatic filter 2303R, the transmittance is essentially the same not only near the wavelengths of the corresponding blue and green light, but also near the wavelengths of colors other than red. Similarly, the transmittance is essentially the same (i.e., uniform) within the wavelength range that a typical red filter does not allow light to pass through.

[0185] Figure 31 This is a schematic graph illustrating the relationship between the transmittance and wavelength of the green monochromatic filter 2303G. Figure 31 In the diagram, the solid line shows the transmittance of the green monochromatic filter 2303G. The dashed line indicates the transmittance of a standard green filter for comparison. The green monochromatic filter 2303G has approximately 100% transmittance for green light and therefore allows green light to pass through. However, since the 100% ratio is merely an example, the green monochromatic filter 2303G may only be suitable for situations where its transmittance for green is higher than its transmittance for other primary colors.

[0186] In view of this, based on as an example Figure 31 The green monochrome filter 2303G shown has a transmittance of approximately 30% for each of the other primary colors besides green. Therefore, the green monochrome filter 2303G also has a higher effective transmittance for other primary colors than the aforementioned lower transmittance.

[0187] Figure 32 The diagram schematically illustrates the relationship between the transmittance and wavelength of the blue monochromatic filter 2303B. Figure 32 In the diagram, the solid line shows the transmittance of the blue monochromatic filter 2303B. The dashed line represents the transmittance of a standard blue filter for comparison. The blue monochromatic filter 2303B has approximately 100% transmittance for blue, and therefore allows blue light to pass through. However, since the 100% ratio is merely an example, the blue monochromatic filter 2303B may only be suitable for situations where the transmittance for blue is higher than that for other primary colors.

[0188] Furthermore, as an example Figure 32As shown, the blue monochromatic filter 2303B has approximately 30% transmittance for colors other than blue, including other primary colors. Therefore, the blue monochromatic filter 2303B also has a higher transmittance for other primary colors than its lower effective transmittance. Furthermore, as shown, each of the red monochromatic filter 2303R, green monochromatic filter 2303G, and blue monochromatic filter 2303B has substantially the same transmittance for colors other than their corresponding primary colors.

[0189] Therefore, in each of the red monochromatic filter 2303R, blue monochromatic filter 2303B, and green monochromatic filter 2303G, the transmittance over the entire visible light region is higher than the lower effective transmittance, while particularly increasing the transmittance of the corresponding primary color. Since the image sensor 2100 includes the red monochromatic filter 2303R, blue monochromatic filter 2303B, and green monochromatic filter 2303G, the image sensor 2100 can effectively improve its sensitivity when compared to a system with color filters that do not transmit colors other than the corresponding primary color.

[0190] Furthermore, sensitivity can be improved by using a filter that has a higher transmittance than the effective transmittance for primary colors other than the corresponding primary color. Therefore, as a monochromatic filter, it can more effectively reduce the difference in signal level compared to a primary color filter that does not allow primary colors other than the corresponding primary color to pass through, or a system equipped with a separate transparent filter.

[0191] Next, refer to the following text. Figure 33 A second embodiment of the present disclosure is described. In the second and subsequent embodiments, unless otherwise specifically mentioned, the devices or components having the same reference numerals are the same as those already described. Furthermore, where the previously described configuration is partially modified and described, the remainder of the configuration may adopt portions of the configuration of the already described embodiments.

[0192] like Figure 33As shown, in this embodiment, the minimum repeating unit 3302 can be used instead of the minimum repeating unit 2302 described in the first embodiment. That is, the minimum repeating unit 3302 includes a red monochromatic filter 2303R, a green monochromatic filter 2303G, and a blue monochromatic filter 2303B, each having the same characteristics as the monochromatic filter described in the first embodiment. However, unlike the first embodiment, these filters in the second embodiment have a rectangular shape. This is because the minimum repeating unit 3302 can additionally accommodate a rectangular red sub-primary color filter portion 3304R, a green sub-primary color filter portion 3304G, and a blue sub-primary color filter portion 3304B while maintaining a square shape. That is, by forming the primary color monochromatic filter 2303 and the sub-primary color filter portion 3304 in the above-described shape, the minimum repeating unit 3302 becomes a square. However, this disclosure is not limited to a square shape, and the shape of the minimum repeating unit 3302 can be changed. That is, the shape of each of the primary color monochromatic filter 2303 and the sub-primary color filter portion 3304 is merely an example and can be changed to various other shapes.

[0193] Here, the sub-primary color filter portion 3304 and the primary color type monochrome filter 2303 of the same color type form a set. However, the sub-primary color filter portion 3304 is smaller than the primary color filter portion 2303. For example, the area of ​​the sub-primary color filter portion 3304 is less than half the combined area obtained by combining the sub-primary color filter portion 3304 and the primary color filter portion 2303. More specifically, the area of ​​the sub-primary color filter portion 3304 is less than half of the primary color filter portion 2303.

[0194] The red sub-primary color filter section 3304R and the red-type monochromatic filter 2303R form a set. The green sub-primary color filter section 3304G also forms a set together with the green-type monochromatic filter 2303G. Similarly, the blue sub-primary color filter section 3304B forms a set together with the blue-type monochromatic filter 2303B. Therefore, a single monochromatic filter includes a set of primary color monochromatic filters 2303 and sub-primary color filter sections 3304.

[0195] The sub-primary color filter section 3304 has a lower transmittance for primary colors other than the corresponding primary color than the primary color type monochromatic filter 2303. An example of the relationship between wavelength and transmittance for the sub-primary color filter section 3304 can be found in... Figure 30 , Figure 31 and Figure 32 The ordinary primary color filter shown by the dashed line in any of the diagrams is the same.

[0196] As shown in the figure, each sub-primary color filter section 3304 is arranged adjacent to the primary color type monochrome filter 2303 to jointly form a filter set. That is, jointly forming a set means that these filters have the same color. However, the sub-primary color filter section 3304 does not need to be arranged adjacent to the primary color type monochrome filter 2303 to jointly form a filter set.

[0197] Reference Figure 34 An exemplary structure of an imaging apparatus 2500 according to a second embodiment of the present disclosure is described. As shown in the figures, the imaging apparatus 2500 includes a photoelectric conversion element 2204 (i.e., on the left side) disposed corresponding to a primary color type monochromatic filter 2303 and another photoelectric conversion element 2204 (i.e., on the right side) disposed corresponding to a sub-primary color filter portion 3304.

[0198] The readout circuit 2170 can separate the signal into a signal output from the photoelectric conversion element 2204 corresponding to the primary color monochromatic filter 2303 and a signal output from the photoelectric conversion element 2204 corresponding to the sub-primary color filter section 3304. The readout circuit 2170 can then output these signals to the processing circuit 2400.

[0199] The primary color monochromatic filter 2303 has a higher transmittance than the sub-primary color filter portion 3304. Therefore, the photoelectric conversion element 2204 corresponding to the primary color monochromatic filter 2303 is more sensitive than the photoelectric conversion element 2204 corresponding to the sub-primary color filter portion 3304. In the following text, the photoelectric conversion element 2204 corresponding to the primary color monochromatic filter 2303 is referred to as a high-sensitivity pixel 2204H. The photoelectric conversion element 2204 corresponding to the sub-primary color filter portion 3304 can be referred to as a low-sensitivity pixel 2204L.

[0200] Furthermore, the processing circuit 2400 enables the generation of the color of each pixel of a color image using only one of the signals output from the high-sensitivity pixel 2204H and the low-sensitivity pixel 2204L. Alternatively, the processing circuit 2400 can generate the color of each pixel of a color image using both types of signals.

[0201] Here, the low-sensitivity pixel 2204L is considered to be a pixel that is more difficult to saturate than the high-sensitivity pixel 2204H. Since the image sensor 2100 includes both the high-sensitivity pixel 2204H and the low-sensitivity pixel 2204L, the image sensor 2100 can more effectively widen the dynamic range compared to a system that only sets the high-sensitivity pixel 2204H.

[0202] Furthermore, when generating a color image using the signal output from the high-sensitivity pixel 2204H, the processing circuit 2400 uses a different correction coefficient than when generating a color image using the signal output from the low-sensitivity pixel 2204L. An example of the correction coefficient could be a white balance setting value (i.e., a preset white balance) or a color matrix setting value. Another example of the correction coefficient could be a correction coefficient used when calculating brightness values. The white balance setting value is a value that significantly corrects the signal output from the low-sensitivity pixel 2204L to be greater than the signal output from the high-sensitivity pixel 2204H. In contrast, the color matrix setting value is a coefficient that significantly corrects the signal output from the high-sensitivity pixel 2204H to be greater than the signal output from the low-sensitivity pixel 2204L. Furthermore, the correction coefficient used when calculating brightness values ​​is a coefficient that significantly corrects the signal output from the low-sensitivity pixel 2204L to be greater than the signal output from the high-sensitivity pixel 2204H. Additionally, the correction coefficients used when correcting the outputs from the low-sensitivity pixel 2204L and the high-sensitivity pixel 2204H can be adjusted individually by the user.

[0203] Next, the following text will refer to Figure 35 A third embodiment of the present disclosure is described. Figure 35 The smallest repeating unit 4302 of the color filter array 4300 according to the third embodiment of this disclosure is shown. Specifically, the size of the smallest repeating unit 4302 can be the same as that of the smallest repeating unit 2302 in the first embodiment. As shown, the smallest repeating unit 4302 has a shape formed by arranging four primary color type monochromatic filters 4303 in two rows and two columns. The configuration other than the color filter array 4300 is basically the same as that in the first embodiment. Therefore, a single photoelectric conversion element 2204 is arranged corresponding to each primary color type monochromatic filter 4303.

[0204] Furthermore, the primary color type monochromatic filter 4303 includes a red type monochromatic filter 4303R, a green type monochromatic filter 4303G, and a blue type monochromatic filter 4303B. Specifically, the minimum repetition unit 4302 has a Bayer array arranged with a single red type monochromatic filter 4303R, two green type monochromatic filters 4303G, and a single blue type monochromatic filter 4303B.

[0205] Furthermore, each primary color type monochrome filter 4303 includes a primary color filter portion 4304 and a transparency filter portion 4305. Specifically, in this embodiment, the primary color type monochrome filter 4303 is formed in a square shape and is divided into two rectangular quarter portions, such that one quarter portion is the primary color filter portion 4304 and the other quarter portion is the transparency filter portion 4305.

[0206] Furthermore, the red-type monochromatic filter 4303R includes a red filter portion 4304R as a primary color filter portion 4304. Similarly, the green-type monochromatic filter 4303G includes a green filter portion 4304G as a primary color filter portion 4304. The blue-type monochromatic filter 4303B also includes a blue filter portion 4304B as a primary color filter portion 4304. Here, the characteristics of the red filter portion 4304R are substantially the same as those of the red sub-primary color filter portion 3304R. Likewise, the characteristics of the green filter portion 4304G are substantially the same as those of the green sub-primary color filter portion 3304G. Furthermore, the characteristics of the blue filter portion 4304B are substantially the same as those of the blue sub-primary color filter portion 3304B.

[0207] Here, each of the transparent filter sections 4305 includes a colorless transparent filter. Therefore, since the transparent filter section 4305 is colorless and transparent, it is more sensitive than the primary color filter section 4304. Here, a filter with higher sensitivity than the primary color filter section 4304 is either a filter that increases sensitivity even when using substantially the same photoelectric conversion element 2204, or a filter with higher transmittance than the primary color filter section 4304.

[0208] Furthermore, as shown in the figure, according to the third embodiment, the minimum repeating unit 4302 includes four primary color type monochromatic filters 4303. Each primary color type monochromatic filter 4303 includes a primary color filter portion 4304 and a transparency filter portion 4305. Therefore, compared to the case where the primary color type monochromatic filter 4303 is entirely composed of the primary color filter portion 4304, the third embodiment more effectively improves sensitivity. As a result, since the sensitivity is improved by providing the transparency filter portion 4305, the difference in signal levels between pixels P can be reduced compared to a system where the transparency filter and the primary color filter (which is a monochromatic filter) are provided separately.

[0209] Next, the following text will refer to Figure 36 The fourth embodiment is described. For example... Figure 36 As shown, each primary color monochromatic filter 5303 constituting the minimum repeating unit 5302 is divided into four small squares. Then, within these four small squares, a pair of squares that are perpendicularly and horizontally shifted (i.e., arranged to the right and downwards) serve as sub-primary color filter sections 5304. The remaining two squares serve as sub-transparency filter sections 5305s. Specifically, within a primary color monochromatic filter 5303, the transparency filter section 5305 is divided into two (i.e., multiple) sub-transparency filter sections 5305s. The sub-transparency filter sections 5305s have high sensitivity to serve as sub-high sensitivity filter sections.

[0210] Furthermore, the primary color filter section 5304 is further divided into multiple sub-primary color filter sections 5304. However, although each primary color type monochrome filter 5303 is composed of, for example... Figure 36 Each part shown is configured, but the area ratio between the primary color filter part 5304 and the transparency filter part 5305 in each primary color type monochrome filter 5303 is... Figure 35 The area ratios shown are the same. Therefore, with Figure 35 Similar to the system shown, this embodiment improves sensitivity more effectively compared to a system where the primary color type monochrome filter 5303 is entirely composed of the primary color filter portion 5304. Furthermore, compared to a system where the transparency filter and the primary color filter (which is a monochrome filter) are set separately, the difference in signal levels between pixels P can be reduced more effectively.

[0211] Next, refer to the following text. Figure 37 The fifth embodiment is described. For example... Figure 37 As shown, the minimum repeating unit 6302 of this embodiment includes a plurality of primary color type monochromatic filters 6303, each having substantially the same characteristics as the filters used in the third embodiment. That is, the minimum repeating unit 6302 includes a red type monochromatic filter 6303R, a green type monochromatic filter 6303G, and a blue type monochromatic filter 6303B, each having substantially the same characteristics as the filters described in the third embodiment. However, in the fifth embodiment, the shape of each of the portions and filters is rectangular. This is because the minimum repeating unit 6302 can accommodate the red sub-primary color filter portion 6306R, the green sub-primary color filter portion 6306G, and the blue sub-primary color filter portion 6306B while maintaining a square shape overall. Each of the red sub-primary color filter portion 6306R, the green sub-primary color filter portion 6306G, and the blue sub-primary color filter portion 6306B has a shape substantially the same as the sub-primary color filter portion 3304. Here, the shapes of the primary color type monochromatic filters 6303 and the sub-primary color filter portions 6306 are merely examples and can be changed to various other shapes.

[0212] Each primary color type monochrome filter 6303 includes a primary color filter portion 6304 and a transparency filter portion 6305. Therefore, the red type monochrome filter 6303R includes a red filter portion 6304R and a red sub-primary color filter portion 6306R, both commonly used as the primary color filter portion 6304. Similarly, the green type monochrome filter 6303G includes a green filter portion 6304G and a green sub-primary color filter portion 6306G, both commonly used as the primary color filter portion 6304. Furthermore, the blue type monochrome filter 6303B includes a blue filter portion 6304B and a blue sub-primary color filter portion 6306B, both commonly used as the primary color filter portion 6304.

[0213] Here, the characteristics of the red filter section 6304R and the red sub-primary color filter section 6306R are essentially the same as those of the red sub-primary color filter section 3304R. Furthermore, the characteristics of the green filter section 6304G and the green sub-primary color filter section 6306G are essentially the same as those of the green sub-primary color filter section 3304G. Similarly, the characteristics of the blue filter section 6304B and the blue sub-primary color filter section 6306B are the same as those of the blue sub-primary color filter section 3304B.

[0214] Figure 38 The configuration of the imaging apparatus 6500 according to the fifth embodiment is partially shown. For example... Figure 38 As shown, the imaging device 6500 includes a photoelectric conversion element 2204 corresponding to the primary color monochromatic filter 6303 and a photoelectric conversion element 2204 corresponding to the sub-primary color filter section 6306.

[0215] The readout circuit 2170 separates the signal into a signal output from the photoelectric conversion element 2204, which is disposed corresponding to the primary color type monochromatic filter 6303, and a signal output from the photoelectric conversion element 2204, which is disposed corresponding to the sub-primary color filter section 6306. Then, the readout circuit 2170 outputs these signals to the processing circuit 6400.

[0216] The primary color monochromatic filter 6303 has a higher transmittance than the sub-primary color filter portion 6306. Therefore, the photoelectric conversion element 2204, which is provided corresponding to the primary color monochromatic filter 6303, has a high-sensitivity pixel 2204H. In contrast, the photoelectric conversion element 2204, which is provided corresponding to the sub-primary color filter portion 6306, has a low-sensitivity pixel 2204L.

[0217] Furthermore, similar to processing circuit 2400, processing circuit 6400 can also generate a color image using one or both of high-sensitivity pixel 2204H and low-sensitivity pixel 2204L. Therefore, processing circuit 6400 can use a different correction coefficient when correcting the signal output from high-sensitivity pixel 2204H than when correcting the signal output from low-sensitivity pixel 2204L.

[0218] Here, one or more correction coefficients used in calculating the white balance setting value, color matrix setting value, and brightness value are used as correction coefficients. The magnitude relationship of the correction coefficients between the high-sensitivity pixel 2204H and the low-sensitivity pixel 2204L is the same as that between the high-sensitivity pixel 2204H and the low-sensitivity pixel 2204L in the second embodiment.

[0219] Next, we will refer to the following. Figure 39 The sixth embodiment of this disclosure is described. Figure 39 The configuration of the imaging apparatus 6500 according to the sixth embodiment is partially shown. As shown, the image sensor 2100 of the sixth embodiment includes... Figure 36 The color filter array 5300 is shown. Therefore, a single pixel P includes two sub-primary color filter portions 5304s and two sub-transparency filter portions 5305s.

[0220] Therefore, the image sensor 2100 of the sixth embodiment includes two photoelectric conversion elements 2204 corresponding to the two sub-transparency filter sections 5305s, respectively. Furthermore, two photoelectric conversion elements 2204 are provided corresponding to the two sub-primary color filter sections 5304, respectively.

[0221] The processing circuit 7400 is employed, and it enables the signals output from the photoelectric conversion element 2204 to be acquired individually by controlling the readout circuit 2170. The processing circuit 7400 executes an image processing method for generating a color image. As a step in executing the image processing method, the processing circuit 7400 adjusts the number of photoelectric conversion elements 2204 used in generating the color of a single pixel P from (i.e., by selective use) two photoelectric conversion elements 2204 corresponding to the two sub-transparency filter sections 5305s, based on the ambient brightness of the imaging device 7500.

[0222] Because two sub-transparent filter sections 5305s are used, the number of effective photoelectric conversion elements 2204 corresponding to the sub-transparent filter sections 5305s for a single pixel P can be 0, 1, and 2 (i.e., there are three ways). Similarly, when preparing to divide the brightness level into one or two thresholds, the brightness of the environment of the imaging device 7500 can be divided into two or three.

[0223] Furthermore, the number of photoelectric conversion elements 2204 used in generating the color of a single pixel P, which are provided in the processing circuit 7400 corresponding to the sub-transparency filter section 5305s, increases as the ambient brightness of the imaging device 7500 decreases (i.e., as the environment darkens). Therefore, the illuminance sensor 7600 is mounted around the imaging device 7500, and the processing circuit 7400 detects the ambient brightness of the imaging device 7500 based on the illuminance detection signal generated by the illuminance sensor 7600. Additionally, the processing circuit 7400 can acquire the brightness detection signal from the illuminance sensor 7600. Furthermore, the processing circuit 7400 can acquire a value indicating the degree of brightness determined by other processors based on the illuminance detection signal generated by the illuminance sensor 7600.

[0224] Furthermore, the processing circuit 7400 changes the correction coefficient used when correcting the signal output from the photoelectric conversion element 2204 according to the number of photoelectric conversion elements 2204 used in generating the color of a single pixel P. The correction coefficient can correct one or more values, such as white balance setting, color matrix setting, and brightness value.

[0225] Here, the more photoelectric conversion elements 2204 corresponding to the sub-transparency filter section 5305s, the paler the color before correction. Then, as the number of photoelectric conversion elements 2204 corresponding to the sub-transparency filter section 5305s increases, the processing circuit 7400 gradually adjusts the correction coefficient to a level that can solve the problem of color fading. Exemplary adjustments are described in more detail below.

[0226] Specifically, the amount of correction coefficient used when correcting the white balance setting value decreases as the number of photoelectric conversion elements 2204 corresponding to the sub-transparency filter section 5305s increases. Furthermore, the amount of correction coefficient used when correcting the color matrix setting value increases as the number of photoelectric conversion elements 2204 corresponding to the sub-transparency filter section 5305s increases. This is because the more photoelectric conversion elements 2204 corresponding to the sub-transparency filter section 5305s, the paler the color before correction. Additionally, the correction coefficient used to correct the brightness value is specified as a value that decreases as the number of photoelectric conversion elements 2204 corresponding to the sub-transparency filter section 5305s increases. This is because the more photoelectric conversion elements 2204 corresponding to the sub-transparency filter section 5305s, the higher the brightness of the signal before correction. The correction coefficient, which varies according to the ambient brightness of the imaging device 7500, is predetermined based on the initial setting measured in reality. In addition, the correction factor used when calibrating the white balance settings is adjusted so that white objects appear white in sufficiently bright areas to prevent over-calibration. For example, such adjustments are performed in an area illuminated by the headlights of vehicle 200 and which is sufficiently bright.

[0227] Furthermore, the imaging apparatus 7500 of the sixth embodiment adjusts the number of low-sensitivity pixels 2204L used when generating the color of a single pixel P from (i.e., by selective use) a plurality of low-sensitivity pixels 2204L provided corresponding to a plurality of sub-transparency filter sections 5305s, according to the level of ambient brightness of the imaging apparatus 7500. Thus, even if the ambient brightness of the imaging apparatus 7500 changes, the difference in signal levels between signals output from the photoelectric conversion element 2204 provided corresponding to pixel P can be reduced.

[0228] While various embodiments of this disclosure have been described so far, this disclosure is not limited thereto and may include at least the following modifications. Therefore, various other changes and modifications may be included within the scope of this disclosure without departing from its spirit.

[0229] Various modifications to the above-described embodiments are briefly described below. First, a first modification is briefly described below. In the above-described embodiments, all primary color monochromatic filters 2303, 4303, 5303, and 6303 employed in the respective minimum repeating units 2302, 3302, 4302, and 6302 are arranged by forming a Bayer array. However, this disclosure is not limited to this, and various arrangements can be adopted. That is, for example, the primary color monochromatic filters 2303, 4303, 5303, and 6303 included in the minimum repeating unit can adopt various arrays, such as oblique Bayer arrays, quad Bayer arrays, etc.

[0230] The second modification is briefly described below. The minimum repeating unit is valid (i.e., suitable) if it includes at least one primary color type monochromatic filter 2303, 4303, 5303, or 6303. Furthermore, the minimum repeating unit may include monochromatic filters other than the primary color type monochromatic filters 2303, 4303, 5303, and 6303. For example, a transparent standalone filter as a colorless transparent monochromatic filter is illustrated as such a monochromatic filter other than a primary color type monochromatic filter. Additionally, a yellow monochromatic filter as a monochromatic filter that allows yellow to pass through can be illustrated. Furthermore, complementary color type monochromatic filters can be used as monochromatic filters. Here, examples of cyan and magenta as complementary colors can be illustrated.

[0231] Furthermore, the minimum repetition unit can be a combination of the following monochrome filters, where R represents a red monochrome filter, G represents a green monochrome filter, and B represents a blue monochrome filter. Additionally, C represents a transparent monochrome filter, Ye represents a yellow monochrome filter, and Cy represents a cyan monochrome filter. That is, the minimum repetition unit can be RGCB, RYeYeB, and RYeYeCy. Alternatively, the minimum repetition unit can be RYeYeG, RYeYeC, and RYeYeYe. Furthermore, the minimum repetition unit can be RCCB, RCCCy, and RCCG. Additionally, the minimum repetition unit can be RCCC and RCCYe, etc.

[0232] Next, the third modification will be briefly described below. In the above embodiment, the transparent filter sections 4305, 5305, and 6305, which serve as the high-sensitivity filter section, are colorless and transparent. However, the filter used in the high-sensitivity filter section does not necessarily have to be colorless and transparent. That is, if the sensitivity of the filter used in the high-sensitivity filter section is higher than the sensitivity of each of the primary color filter sections 4304, 5304, and 6304, then the filter used in the high-sensitivity filter section does not need to be colorless and transparent. For example, a yellow filter can be used in the high-sensitivity filter section.

[0233] Next, the fourth modification is briefly described below. In the fifth embodiment, the minimum repeating unit 6302 can be replaced by the minimum repeating unit 5302. In this case, the low-sensitivity pixel 2204L is arranged at a given position that allows the low-sensitivity pixel 2204L to receive light transmitted through one of the two sub-primary color filter portions 5304. Similarly, the high-sensitivity pixel 2204H is arranged at a given position that allows the high-sensitivity pixel 2204H to receive light transmitted through the remaining primary color type monochromatic filters 5303. Here, a plurality of high-sensitivity pixels 2204H can be provided according to the shape of the remaining portions of the primary color type monochromatic filters 5303.

[0234] Next, the fifth modification is briefly described below. The imaging devices 2500, 6500, or 7500 of the above embodiments are used to enable the vehicle 200 to generate a navigation response. However, the imaging devices 2500, 6500, or 7500 can be used for other applications, such as driving recorder applications. Furthermore, the imaging devices 2500, 6500, or 7500 can be used for a variety of applications. For example, the imaging devices 2500, 6500, or 7500 can be used to enable the vehicle 200 to generate a navigation response while simultaneously operating a driving recorder.

[0235] The sixth modification is briefly described below. The processing unit 110, control circuitry 2120, and processing circuits 2400, 6400, or 7400 as described in this disclosure can be implemented by a dedicated computer configured as a processor to perform multiple functions. Furthermore, the methods of operating the processing unit 110, control circuitry 2120, and processing circuits 2400, 6400, or 7400 can be implemented by a dedicated computer configured as a processor to perform multiple functions. Alternatively, the processing unit 110, processing circuitry 2400, 6400, or 7400 as described in this disclosure, and the methods of operating these circuits, can be implemented by dedicated hardware logic circuitry. Additionally, the processing unit 110, processing circuitry 2400, 6400, or 7400 as described in this disclosure, and the methods of operating these circuits, can be implemented by one or more dedicated computers consisting of a processor executing a computer program and a combination of one or more hardware logic circuits. The hardware logic circuitry can be, for example, an ASIC and an FPGA.

[0236] Furthermore, the storage medium used to store computer programs is not limited to ROM. That is, the storage medium can be a computer-readable, non-transitory tangible recording medium that enables a computer to read and execute programs stored therein as instructions. For example, flash memory can be used as a storage medium to store the aforementioned programs.

Claims

1. An image sensor, comprising: Multiple photoelectric conversion elements; as well as Multiple monochrome filters are used to generate multiple colors, and the multiple monochrome filters are arranged corresponding to the multiple photoelectric conversion elements. At least one of the plurality of monochrome filters includes a primary color monochrome filter. The primary color type monochromatic filter transmits light of the corresponding primary color and light of at least one of the other primary colors besides the corresponding primary color. The primary color type monochromatic filter has a first given transmittance for light of one of the primary colors other than the corresponding primary color, and light of one of the other primary colors passes through the primary color type monochromatic filter with the first given transmittance. The first given transmittance is higher than a lower limit of transmittance that improves the sensitivity of the image sensor. At least one of the plurality of monochrome filters includes: The sub-primary color filter section, together with the primary color type monochromatic filter, forms a set. The sub-primary color filter section has a second given transmittance for light of primary colors other than the corresponding primary color. For light of primary colors other than their corresponding primary colors, the second given transmittance is lower than the first given transmittance of the primary color type monochromatic filter. The plurality of photoelectric conversion elements are respectively set separately from the plurality of primary color monochromatic filters and the sub-primary color filters.

2. An imaging device, comprising: The image sensor according to claim 1; as well as Processing circuitry is used to generate a color image by correcting and processing the signal output from the image sensor. The processing circuit generates the color image based on the following: The first signal output from the photoelectric conversion element arranged corresponding to the primary color monochromatic filter, and The second signal output from the photoelectric conversion element arranged corresponding to the sub-primary color filter section, The correction coefficient used when correcting the signal output from the photoelectric conversion element arranged corresponding to the primary color type monochromatic filter is different from the correction coefficient used when correcting the signal output from the photoelectric conversion element arranged corresponding to the sub-primary color filter section.

3. An image sensor, comprising: Multiple photoelectric conversion elements, corresponding to multiple pixels; as well as Multiple color monochromatic filters, each color monochromatic filter being arranged on a corresponding photoelectric conversion element among the multiple photoelectric conversion elements. Each of the plurality of color monochrome filters includes a primary color monochrome filter. The primary color type monochromatic filter includes: The primary color filter part that generates the corresponding primary color; and A high-sensitivity filter section that is more sensitive than the primary color filter section. Furthermore, in each of the plurality of color monochrome filters, the primary color filter portion and the high-sensitivity filter portion are arranged on the same corresponding photoelectric conversion element among the plurality of photoelectric conversion elements.

4. The image sensor according to claim 3, wherein, The primary color filter section is divided into multiple blocks.

5. The image sensor according to claim 4, wherein, One of the blocks in the primary color filter portion constitutes a sub-primary color filter portion having an area less than half that of the primary color filter portion, and the image sensor further includes: Low-sensitivity pixels, which serve as photoelectric conversion elements arranged corresponding to the sub-primary color filter section; and High-sensitivity pixels, which serve as photoelectric conversion elements arranged corresponding to one or more of the remaining blocks of the primary color filter section.

6. An imaging device, comprising: The image sensor according to claim 5; as well as A processing circuit is used to generate a color image by processing the signal output from the image sensor; The processing circuit generates the color image by correcting the signals output from both the low-sensitivity pixel and the high-sensitivity pixel. The correction coefficients used when correcting the signal output from the low-sensitivity pixel are different from the correction coefficients used when correcting the signal output from the high-sensitivity pixel.

7. An imaging device, comprising: The image sensor according to any one of claims 3 to 5; as well as Processing circuitry is used to generate a color image by processing the signal output from the image sensor. The high-sensitivity filter section is divided into multiple sub-high-sensitivity filter sections. The plurality of sub-high-sensitivity filter sections are respectively arranged corresponding to the plurality of photoelectric conversion elements. The processing circuit adjusts the number of photoelectric conversion elements used when generating the color of a pixel based on the ambient brightness.

8. The imaging apparatus according to claim 7, in, The processing circuit generates the color image by correcting the signal output from the photoelectric conversion element. The processing circuit changes the correction coefficient used when correcting one or more signals output from the photoelectric conversion elements according to the number of photoelectric conversion elements used when generating the color of a single pixel.

9. An image processing method, comprising the following steps: Incident light is received using multiple color monochromatic filters, each of which includes a primary color monochromatic filter, and the primary color monochromatic filter includes a primary color filter section and a high-sensitivity filter section; Primary colors are generated through the primary color filter section; A portion of the incident light is transmitted through a high-sensitivity filter section, the high-sensitivity filter section having a higher sensitivity than the primary color filter section, the high-sensitivity filter section being divided into multiple sub-high-sensitivity filter sections, the multiple sub-high-sensitivity filter sections being arranged to correspond to corresponding photoelectric conversion elements among multiple photoelectric conversion elements; The number of corresponding photoelectric conversion elements among the plurality of photoelectric conversion elements that correspond to the plurality of sub-high sensitivity filter sections, used to generate the color of a single pixel, is adjusted according to the ambient brightness. Multiple photoelectric conversions are performed using the plurality of photoelectric conversion elements arranged corresponding to the plurality of color monochrome filters to obtain an electrical signal; Correct the electrical signal; as well as A color image is generated based on the corrected electrical signal.

Images