Multi-distance single camera for vehicle vision system

Abstract

The imaging system includes a multi-distance camera having an imager and a lens. The lens includes a plurality of regions, and each region is physically different from each other region. The controller is in communication with the multi-range camera. The controller includes a processor and a memory. The memory stores instructions configured to cause the processor to process an image received from the imager by separating the image into a plurality of different region images, processing each different region image using a processing program corresponding to a region of the region image being processed, the region images are recombined into a single processed image, and the single processed image is provided to at least one vision system.

Description

Technical Field

[0001] The present disclosure relates to vehicle vision systems, and more particularly to configurations for capturing multirange images using a single physical imager, such as a camera. Background Art

[0002] Modern vehicles utilize imaging to monitor and respond to their surroundings. Vehicle systems that use imaging in this manner can include driver assistance systems (such as backup cameras), semi-autonomous driving systems (such as parallel parking assist systems), and fully autonomous driving systems. Systems that use imaging to assist in vehicle operation are often referred to as vehicle vision systems.

[0003] To facilitate these systems, multiple cameras are incorporated throughout the vehicle. These cameras include different physical configurations and orientations, allowing for various viewing distances and required fields of view for each system requiring imaging. Each required camera adds weight and complexity to the vehicle. Therefore, it would be desirable to reduce the number of cameras required on a vehicle while still providing all the necessary views to achieve the desired operation of the vehicle's vision systems. Summary of the Invention

[0004] In one exemplary embodiment, an imaging system includes a multi-range camera having an imager and a lens. The lens includes a plurality of regions, and each region is physically distinct from each other region. A field of view generated by the camera includes the plurality of regions. A controller communicates with the multi-range camera. The controller includes a processor and a memory. The memory stores instructions configured to cause the processor to process an image received from the imager by separating the image into a plurality of different region images, separately processing each different region image using a processing routine corresponding to the region of the region image being processed, recombining the different region images into a single processed image, and providing the single processed image to at least one vision system.

[0005] In addition to one or more features described herein, each zone includes a lens distortion that is different from the lens distortion in every other zone.

[0006] In addition to one or more features described herein, each region includes a pixel density that is different from a pixel density of every other region.

[0007] In addition to one or more features described herein, each region is a single continuous shape.

[0008] In addition to one or more features described herein, at least one region is a plurality of discrete shapes.

[0009] In addition to one or more features described herein, separating the image into a plurality of different region images includes providing the image and a set of X, Y blanking regions to a SerDes module of the controller and outputting the plurality of region images from the SerDes module.

[0010] In addition to one or more of the features described herein, a number of region images in the plurality of region images is equal to a number of regions in the plurality of regions.

[0011] In addition to one or more features described herein, the imaging system further includes sequentially ordering the plurality of different region images using a frame concatenation module.

[0012] In addition to one or more features described herein, processing each unique region image using a processing procedure corresponding to the processed region image includes at least one of edge enhancement of the region image and pixel density normalization of each unique region image.

[0013] In addition to one or more features described herein, processing each different region image using a processing procedure corresponding to the region image being processed includes each of edge enhancement of the region image and pixel density normalization of the region image.

[0014] In addition to one or more features described herein, the at least one vision system is a vehicle vision system.

[0015] In another exemplary embodiment, a method for providing an image to a vision system includes receiving, at a controller, a base image from a multi-range camera. The multi-range camera has an imager and a lens, wherein the lens has multiple regions. Each of the multiple regions is physically distinct from one another. The method, using the controller, separates the base image into multiple distinct region images. The method, using the controller, processes each distinct region image using a processing procedure corresponding to the region image being processed. The method reassembles the region images into a single processed image and provides the single processed image to at least one vision system.

[0016] In addition to one or more features described herein, separating the base image into the plurality of regional images includes providing the base image and a set of X, Y blanking regions to a serializer / deserializer module of the controller, and outputting the plurality of regional images from the serializer / deserializer module.

[0017] In addition to one or more of the features described herein, a number of region images in the plurality of region images is equal to a number of regions in the plurality of regions.

[0018] In addition to one or more of the features described herein, the method further includes sequentially sorting the plurality of different region images using a frame concatenation module of the controller.

[0019] In addition to one or more of the features described herein, processing each different region image using a processing procedure corresponding to the region image being processed includes at least one of edge enhancement of the region image and pixel density normalization of the region image.

[0020] In addition to one or more features described herein, processing each different region image using a processing procedure corresponding to the region image being processed includes each of edge enhancement of the region image and pixel density normalization of the region image.

[0021] In addition to one or more features described herein, the at least one vision system comprises a vehicle vision system.

[0022] In addition to one or more features described herein, each region includes at least one of a lens distortion that is different from a lens distortion in each other region and a pixel density that is different from a pixel density in each other region.

[0023] In one exemplary embodiment, a vehicle includes an imaging system having a multi-range camera having an imager and a lens. The lens includes multiple regions, and each region is physically distinct from each other region. A field of view generated by the camera includes multiple regions. A controller communicates with the multi-range camera. The controller includes a processor and a memory. The memory stores instructions configured to cause the processor to process an image received from the imager by separating the image into multiple different region images, separately processing each different region image using a processing routine corresponding to the region of the region image being processed, recombining the different region images into a single processed image, and providing the single processed image to at least one vision system.

[0024] The above features and advantages and other features and advantages of the present disclosure are readily apparent from the following detailed description when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Additional features, advantages, and details appear, by way of example only, in the following detailed description, which refers to the accompanying drawings, in which:

[0026] Figure 1 depicts a top view of a motor vehicle including a multi-range camera for use with a vehicle vision system;

[0027] Figure 2Depicts the side view and corresponding field of view of a multi-range camera;

[0028] Figure 3 Depicts from Figure 1 and Figure 2 The X, Y blocked area of ​​the image received by the multi-range camera;

[0029] Figure 4 Depicts Figure 1 and Figure 2 Alternative example X, Y blocking zone configuration for multiple range cameras;

[0030] Figure 5 Describes a general approach for processing images from multiple range cameras used in a vehicle vision system;

[0031] Figure 6 Describes a specific exemplary method for integrating images from a given multi-range camera into a vehicle vision system; and

[0032] Figure 7 Depicts the front view of a camera lens. DETAILED DESCRIPTION

[0033] The following description is merely exemplary in nature and is in no way intended to limit the present disclosure, its application, or uses. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features.

[0034] As used herein, a "vehicle vision system" refers to any vehicle system that utilizes or employs digitally generated images of the environment. Vehicle vision systems may include, but are not limited to, driver perception systems, driver monitoring systems, driver assistance systems, and autonomous vehicle operating systems.

[0035] According to exemplary embodiments, methods, devices, and systems are provided for configuring and implementing a multi-range camera-based vision system within a vehicle. In one example, a single camera includes a lens configured with two or more distinct range zones (e.g., a close range zone, a mid-range zone, and a long-range zone). Each distinct range zone is configured to provide the sharpest resolution of objects at the distance designated by the zone. For example, the portion of the lens configured for the close range zone provides the sharpest resolution for objects within the field of view that are close to the camera, while the portion of the lens configured for the long range zone provides the sharpest resolution for a region of the field of view that includes only long-range objects.

[0036] In addition to one or more cameras including a multi-range lens, one or more controllers within the vehicle also include an image processing process that is configured to isolate each area of ​​the multi-range camera, process the isolated area using corresponding image processing, combine the isolated areas into a resulting partitioned image, and provide the resulting partitioned image to the corresponding vehicle vision system for use.

[0037] The embodiments described herein present numerous advantages and technical effects, including a reduction in the number and size of cameras required to implement various vehicle vision systems. As the number and size of cameras are reduced, the weight and complexity of the entire vehicle containing the multi-range cameras are also reduced.

[0038] Embodiments are not limited to use with any particular vehicle and may be applicable to a variety of other environments besides vehicle systems. For example, multi-range cameras and corresponding processing may be used in automated agricultural equipment, fixed monitoring equipment (e.g., security cameras), or any similar application where it is desirable to effectively utilize multiple ranges within a single field of view.

[0039] Figure 1 An embodiment of a motor vehicle 10 is shown, including a body 12 that at least partially defines a passenger compartment 14. Vehicle 10 may be an electric vehicle (EV) or a hybrid vehicle. In one embodiment, vehicle 10 is an electric vehicle that includes at least one electric motor assembly. Body 12 also supports various vehicle subsystems, including a propulsion system and other subsystems that support the functions of the propulsion system and other vehicle components, such as a braking subsystem, a suspension system, a steering subsystem, and the like.

[0040] In addition, the vehicle 10 includes a plurality of multi-range cameras 20 (or collectively referred to as "cameras") disposed about the vehicle body 12. Each of the cameras 20 defines a corresponding field of view 22. The field of view 22 is the area visible to the camera 20. Although shown as four exterior-facing cameras 20, it should be understood that the vehicle 10 may include a plurality of additional exterior-facing cameras, one or more interior-facing cameras, or any other configuration of additional cameras as may be desired for a corresponding vehicle vision system, and that any or all of the cameras may benefit from being configured as multi-range cameras.

[0041] Each camera 20 communicates with a corresponding vehicle controller 30. Communication may be via a direct digital link, a wireless link, a combination of direct digital and wireless, an analog link, indirect communication through a vehicle communication bus, or any other form of communication configured to provide generated images from the camera 20 to the controller 30.

[0042] In one example, controller 30 is a dedicated vision system controller that is configured to receive and analyze image feeds from camera 20 and provide the necessary image processing to allow images from camera 20 to be utilized by a corresponding vehicle vision system. In other examples, controller 30 can be one or more control modules running within a common controller, multiple modules distributed across multiple controllers, or any other controller configuration.

[0043] Continue to refer Figure 1 , Figure 2 A side view of an example camera 20 is shown defining a field of view 22, which includes three regions 210, 220, and 230. Within field of view 22 are a plurality of objects 206, 208, and 209. Field of view 22 includes a short-range range 201 defining a short-range region 210, a mid-range range 203 defining a mid-range region 220, and a long-range range 205 defining a long-range region 230. Some objects 206 are present in short-range range 201, some objects 208 are present in mid-range range 203, and some objects 209 are present in long-range range 205. As used herein, the distance of an object refers to the real-world distance of the object from camera 20, which defines field of view 22 including the object.

[0044] For image processing purposes, the field of view 22 seen by the camera 20 is divided into three regions 210, 220, and 230. The close-range region 210 includes close objects 206, medium-range objects 208, and long-range objects 209. The intermediate-range region 220 includes medium-range objects 208 and long-range objects 209. The long-range region 230 includes only long-range objects 209.

[0045] Figure 3 An example field of view 22 is shown, wherein the field of view 22 is divided into nested rectangular regions 210', 220', 230', wherein rectangular region 210' corresponds to the short distance region 210, rectangular region 220' corresponds to the medium distance region 220, and rectangular region 230' corresponds to the long distance region 230. It should be understood that the regions 210, 220, 230 may be arranged in any shape or configuration, including any number of distance regions desired for a given camera placement.

[0046] In another example conceptualization, regions 210, 220, 230 are Figure 42 shows three different frames 302, 304, and 306 of the same field of view 22. The portion of a given frame 302, 304, or 306 that is omitted from analysis at the corresponding distance is referred to as an X,Y blanking region. The first frame 302 shows a short-distance region 210', where no portion of the frame 302 is an X,Y blanking region. The second frame 304 shows an intermediate-distance region 220', where the remainder of the frame 304 is blanked as an X,Y blanking region, and the third frame 306 shows a long-distance region 230', where the remainder of the frame 306 is blanked as an X,Y blanking region.

[0047] refer to Figure 1-4 It should be understood that certain physical camera structures (such as lens distortion and pixel density) are suitable for generating ideal images of objects 206, 208, 209 at certain distances. Therefore, a given lens distortion and a given pixel density across the entire lens 21 are not ideal for a multi-range camera. To accommodate this aspect, the lens 21 of the camera 20 is physically divided into regions 702, 704, 706 corresponding to the regions 210, 220, 230 of the field of view 22.

[0048] Each region 702, 704, 706 of the lens 21 has a unique and different physical, specific distortion and a corresponding unique pixel density that corresponds to the distance of the object 206, 208, 209 captured within that region 702, 704, 706. This different structure allows for better image quality and greater accuracy at longer distances when tracking and detecting moving objects (e.g., objects 206, 208, 209) using the vehicle's vision system. The image processing performed by the controller 30 also allows the multi-range camera to produce a single image (e.g., a single frame) from multiple regions 210, 220, 230.

[0049] Continue to refer Figures 1 to 4 , Figure 5 A high-level image signal processing (ISP) process 500 is shown, by which the controller 30 processes images received from the camera 20 for use with one or more vehicle vision systems. Initially, the camera 20 generates a zoned image using a zoned lens 21 and a conventional digital imager. In an "Acquire Zoned Image" step 502, the zoned image is provided to the controller 30. In this step 502, the image provided from the camera 20 is separated by zones 210, 220, and 230, such that each frame 22 from the camera 20 generates three different images (in the example camera including three zones 210, 220, and 230).

[0050] Once the image is received, the controller 30 calibrates and normalizes the image using image processing in a “Calibrate and Normalize” step 504 .

[0051] In a "Frame Composer" step 506, the calibrated and normalized images are provided to a frame compositor module within the controller 30. The frame compositor combines the calibrated and normalized images into a corresponding single frame, which is then output by the controller 30 to one or more vision systems 510 in a "Perception / Viewing Output" step 508.

[0052] Continue to refer Figure 1-5 , Figure 6 Shown for execution Figure 5 A specific example process 600 of the general process 500 is shown. Initially, the camera 20 generates and outputs an image in a generate image step 602. The generated image includes the entire field of view 22 as received through the partitioned lens 21.

[0053] The image is provided to a serializer / deserializer (SerDes) 606 along with a set of information 604 defining the X, Y blanking regions of the image. The X, Y blanking regions define the X, Y coordinates of each region within the generated image based on the physical configuration of the partitioned lens 21. The X, Y blanking region data 604 can be stored in local memory within the controller 30, stored elsewhere on the vehicle 10, contained within the camera 20 itself, or stored in any other accessible memory location.

[0054] SerDes 606 divides each frame of an image into multiple regional images. In the example camera 20 having a lens 21 defining three regions 210, 220, and 230, SerDes 606 divides the image into three regional images 607, 608, and 609. In a process regional image step 610, each of regional images 607, 608, and 609 is processed into a corresponding image array. The processing of each regional image is different and corresponds to the classification of the region 210, 220, and 230 corresponding to that regional image 607, 608, and 609. For example, if regional image 607 corresponds to the near-range region 210, the image processing of regional image 607 during process regional image step 610 is near-range image processing.

[0055] Once each region image 607, 608, 609 has been processed, all of the region images 607, 608, 609 are provided to a frame concatenation processing block 612 where they are sorted for subsequent processing steps by the controller 30. Sorting places the region images 607, 608, 609 in a linear order, allowing for sequential processing of the region images 607, 608, 609 rather than parallel processing.

[0056] Each region image 607, 608, 609 is further processed in an edge enhancement step 614. During the edge enhancement step 614, the edges of each region image 607, 608, 609 are manipulated using an existing edge enhancement algorithm, and anti-aliasing is applied to the edges of the region images 607, 608, 609. The edge enhancement facilitates region-based merging that occurs later in a region-based merging step 618.

[0057] The pixel density of each regional image 607, 608, 609 is then normalized in a normalize pixel density step 616. Because the pixel density in each region of the camera 20 is different, the resulting regional images 607, 608, 609 do not have a naturally uniform pixel density. Normalization can increase and / or decrease the pixel density of one or more regions within the regional images 607, 608, 609 so that after normalization, each regional image 607, 608, 609 includes the same pixel density.

[0058] After normalization, the region images corresponding to the individual frames 22 are merged by the controller 30 in a region-based merging step 618, and a single synthesized frame 620 corresponding to the original frame 22 is generated. The single synthesized frame 620 has been fully processed into a single image frame to remove artifacts and variations that may be caused by the different regions 607, 608, 609 within the lens 21.

[0059] The synthesized frame 620 is provided to a standard image signal processing module 622, where the entire synthesized frame 620 is processed using an image signal processing system 624 within the vehicle controller 30 and then provided to a vehicle vision system 626. The vehicle vision system 626 then utilizes the synthesized frame 620 in any capacity, including but not limited to object detection, collision avoidance, driver alerting, operating assistance, and / or any other vehicle vision system.

[0060] While described entirely in the context of a multi-range camera 20 having three zones 210, 220, and 230, it should be understood that any number of zones may be utilized, depending on the specific vehicle vision system being fed the image, and that the zones within a given field of view 22 are not limited to rectangular and / or nested zone shapes. In some practical embodiments, the specific zones and their shapes may be determined through empirical testing and / or through computer modeling. Furthermore, as with the shape and number of zones, the specific delineations between the zones need not be uniformly distributed, and the relative area of ​​each zone again depends on the specific vehicle vision system to which the resulting image will be fed.

[0061] The terms "a" and "an" do not indicate a limitation of quantity, but rather indicate the presence of at least one of the referenced item. Unless the context clearly indicates otherwise, the term "or" means "and / or." References to "aspects" throughout this specification mean that a particular element (e.g., a feature, structure, step, or characteristic) described in conjunction with that aspect is included in at least one aspect described herein and may or may not be present in other aspects. In addition, it should be understood that the described elements may be combined in any suitable manner in each aspect.

[0062] When an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may also be present. In contrast, when an element is referred to as being "directly on" another element, there are no intervening elements present.

[0063] Unless otherwise indicated herein, all test standards are the most current standards in effect as of the filing date of this application, or, if priority is claimed, the filing date of the earliest priority application in which the test standards appear.

[0064] Unless defined otherwise, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0065] Although the above disclosure has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope thereof. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present disclosure without departing from the basic scope of the present disclosure. Therefore, it is intended that the present disclosure is not limited to the particular embodiments disclosed, but is intended to include all embodiments falling within its scope.

Claims

1. An imaging system comprising: a multi-range camera having an imager and a lens, wherein the lens includes a plurality of regions, and wherein each region of the plurality of regions is physically distinct from each other region, and wherein a field of view generated by the camera includes the plurality of regions; a controller in communication with the multi-range camera, the controller comprising a processor and a memory, wherein the memory stores instructions configured to cause the processor to process an image received from the imager by separating the image into a plurality of distinct region images, individually processing each distinct region image using a processing procedure corresponding to the region image being processed, recombining the region images into a single processed image, and providing the single processed image to at least one vision system.

2. The imaging system according to claim 1, wherein Each region includes lens distortion that is different from lens distortion in every other region.

3. The imaging system according to claim 1, wherein: Each region includes a pixel density that is different from the pixel density of every other region.

4. The imaging system according to claim 1, wherein: Each region has a single continuous shape.

5. The imaging system according to claim 1, wherein: At least one region has a plurality of discrete shapes.

6. The imaging system of claim 1 , wherein separating the image into a plurality of different region images comprises providing the image and a set of X, Y blanking regions to a serializer / deserializer module of the controller and outputting a plurality of region images from the serializer / deserializer module, and wherein the number of region images in the plurality of region images is equal to the number of regions in the plurality of regions. 7 . The imaging system according to claim 1 , further comprising sequentially sorting the plurality of different region images using a frame concatenation module.

8. The imaging system according to claim 1, wherein: Processing each different region image using a processing procedure corresponding to the processed region image includes at least one of edge enhancement of each different region image and pixel density normalization of each different region image.

9. The imaging system according to claim 8, wherein: Processing each different region image using a processing procedure corresponding to the region image being processed includes each of edge enhancement of the region image and pixel density normalization of the region image.

10. The imaging system of claim 1, wherein the at least one vision system is a vehicle vision system.

Images