Vehicle surround view system and method

Abstract

The present disclosure relates to a method and system for displaying a surround view in a vehicle. A first image is captured at a first time instance (t-1) and a second image is captured at a second time instance (t) from a camera mounted on the vehicle. The first image is different from the second image. During the capture of the first and second images, one or more vehicle motion parameters are received from one or more sensors associated with the vehicle. Further, a corrected image is synthesized using the first image, the second image, and the one or more vehicle motion parameters. The corrected image is then displayed on a display unit of the vehicle.

Description

[Technical Field]

[0001] The present disclosure relates generally to in-vehicle camera systems, and particularly, but not exclusively, to systems and methods for displaying a vehicle surround view. [Background technology]

[0002] Modern vehicles provide different types of assistance to drivers to increase safety and improve the driving experience. Various driver assistance features include route navigation, autonomous or semi-autonomous driving, vehicle infotainment features, and vehicle surround view features. Referring to a vehicle surround view system, various cameras are installed on the vehicle to capture a surround view of the vehicle. The surround view system provides the driver with 3D visualization. Typically, the 3D visualization is a top view synthesized using different images captured by the cameras.

[0003] Generally, fisheye cameras are used to cover a larger field of view. Because fisheye cameras add distortion, the captured images need to be corrected before being rendered on a display screen. However, the outer regions of the images captured from a fisheye camera are radially distorted, resulting in low image quality when corrected and displayed on a display screen. Therefore, there is a need to generate high-quality images from images captured using a fisheye camera.

[0004] The information disclosed in this Background section of the present disclosure is intended only to enhance understanding of the general background of the present invention and should not be construed as an admission or any form of suggestion that this information constitutes prior art already known to those skilled in the art. Summary of the Invention [Means for solving the problem]

[0005] Additional features and advantages are realized through the techniques of the present disclosure.Other embodiments and aspects of the present disclosure are described in detail herein and are considered a part of the claimed disclosure.

[0006] In one embodiment, a method and system for displaying a surround view in a vehicle is disclosed. A first image is captured at a first time instance (t-1) and a second image is captured at a second time instance (t) from a camera mounted on the vehicle. The first image is different from the second image. During the capture of the first and second images, one or more vehicle motion parameters are received from one or more sensors associated with the vehicle. Furthermore, a corrected image is synthesized using the first image, the second image, and the one or more vehicle motion parameters. The corrected image is then displayed on a display unit of the vehicle.

[0007] In one embodiment, the first image and the second image are radially distorted, and the second image is captured after the vehicle has moved from the position where the first image was captured.

[0008] In one embodiment, the one or more vehicle operating parameters include at least one of a vehicle speed and a vehicle steering angle.

[0009] In one embodiment, the corrected image is synthesized by identifying a first slice having an undistorted view in the first image, which corresponds to a second slice having a distorted view in the second image. The slices are identified based on one or more vehicle motion parameters. Further, the corrected image is synthesized such that the first slice replaces the second slice in the corrected image.

[0010] In one embodiment, the corrected image is processed to represent a view selected from the group consisting of a top view of the vehicle, a front view of the vehicle, a side view of the vehicle, and combinations thereof, and the view of the vehicle is then displayed on a display unit.

[0011] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the exemplary aspects, embodiments, and features described above, further aspects, embodiments, and features may become apparent by reference to the drawings and the following detailed description.

[0012] The novel features and characteristics of the present disclosure are set forth in the appended claims. However, the disclosure itself, as well as its preferred modes of use, further objects and advantages, may best be understood by reference to the following detailed description of exemplary embodiments, when read in conjunction with the accompanying drawings. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles. In the drawings, the leftmost digit(s) of a reference number identifies the figure in which that reference number first appears. One or more embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which like reference numbers represent like elements. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is an exemplary block diagram of a vehicle for displaying a surround view, according to some embodiments of the present disclosure. [Figure 2] FIG. 1 is a detailed block diagram of a system for displaying a vehicle surround view, according to some embodiments of the present disclosure. [Figure 3] 1 is a flowchart illustrating method steps for displaying a vehicle surround view according to some embodiments of the present disclosure. [Figure 4a] 1 is an exemplary diagram of a sensor field of view around a vehicle, according to some embodiments of the present disclosure. [Figure 4b] 1 is an exemplary diagram of a sensor field of view around a vehicle, according to some embodiments of the present disclosure. [Figure 5a] FIG. 1 illustrates an exemplary temporal visualization of a fisheye camera, according to some embodiments of the present disclosure. [Figure 5b] FIG. 1 illustrates an exemplary temporal visualization of a fisheye camera, according to some embodiments of the present disclosure. [Figure 6a] FIG. 10 illustrates an exemplary temporal visualization of a radial distortion corrected image, according to one embodiment of the present disclosure. [Figure 6b] FIG. 10 illustrates an exemplary temporal visualization of a radial distortion corrected image, according to one embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates an exemplary top surround view of a vehicle, according to some embodiments of the present disclosure. [Figure 8] FIG. 1 illustrates a general-purpose computer system for displaying a vehicle surround view, according to an embodiment of the present disclosure.

[0014] It should be understood by those skilled in the art that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, any flowcharts, flow diagrams, state transition diagrams, pseudocode, etc. may be understood to represent various processes substantially represented on a computer-readable medium and that may be executed by a computer or processor, whether or not such a computer or processor is explicitly shown. DETAILED DESCRIPTION OF THE INVENTION

[0015] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0016] While the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and may be described in detail below. It is to be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is intended to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure.

[0017] The terms "comprises," "includes," "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a setup, device, or method that includes a list of components or steps does not only include those components or steps, but may also include other components or steps that are not expressly listed or inherent to such setup, or device, or method. In other words, one or more elements in a system or apparatus proceeded by "comprising a" or "including a" does not, without more constraints, exclude the presence of other or additional elements in the system or method.

[0018] In the following detailed description of embodiments of the present disclosure, reference is made to the accompanying drawings which form a part hereof, and which show, by way of illustration, specific embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure, it being understood that other embodiments may be utilized and changes may be made without departing from the scope of the present disclosure. The following description, therefore, is not to be taken in a limiting sense.

[0019] FIG. 1 shows a block diagram of a vehicle for displaying a surround view on a display unit. As shown in FIG. 1, the block diagram includes a system 101, multiple cameras 102a, 102b, ... 102n, an electronic control unit 103, and a display unit 104. The multiple cameras 102a, 102b, ... 102n may be fisheye cameras capable of capturing a large field of view. For example, four cameras 102a, 102b, 102c, 102d may be installed on the vehicle to provide a 360-degree view of the vehicle's surroundings. In an exemplary embodiment, the multiple cameras 102a, 102b, ... 102n may be complementary metal-oxide semiconductor (CMOS) cameras with a wide dynamic range of resolution. Furthermore, the multiple cameras 102a, 102b, ... 102n may have a wide field of view of at least 180 degrees or more. For example, the fisheye cameras may have a field of view of 195 degrees.

[0020] In one embodiment, ECU 103 may be configured to acquire one or more vehicle operating parameters. The one or more vehicle operating parameters may include at least one of a vehicle speed and a vehicle steering angle. In some embodiments, the one or more vehicle operating parameters may be acquired by ECU 103 using sensors (not disclosed in FIG. 1 ) commonly known in the art. For example, the vehicle speed may be acquired using a Hall effect sensor, and the steering angle may be acquired using a steering angle sensor.

[0021] In one embodiment, display unit 104 may be part of an infotainment system (not shown in FIG. 1 ), a vehicle's head-up display (HUD) unit, a mobile device display, or other display associated with the vehicle. Display unit 104 may include its own processing unit, operating system, and communication ports. Display unit 104 is configured to display a surround view of the vehicle provided by system 101. In some embodiments, the displayed content may be a directional view of the vehicle. In some embodiments, the displayed content may be a 3D surround view of the vehicle.

[0022] In one embodiment, the system 101 may be a system-on-chip (SoC). In some embodiments, the system 101 may be part of the ECU 103 or may be a standalone system. The system 101 may be connected to the ECU 103, multiple cameras 102a, 102b, ... 102n, and display 104 using Ethernet or a Controller Area Network (CAN) or FlexRay or low-voltage differential signaling standards.

[0023] The system 101 may be configured to receive, from multiple cameras 102 a, 102 b, ... 102 n, a first image and a second image captured by the camera 102 a. The first image and the second image may be temporally distinct, i.e., the first image may be captured at a first time instance (t-1), and the second image may be captured at a second time instance (t) after the first time instance (t). In some embodiments, the first image and the second image may contain substantially similar objects. Because camera movement in time may be due to vehicle motion, the objects in the second image may be slightly different from the objects in the first image. For example, the first image may be captured when the vehicle was at point A, and the second image may be captured when the vehicle was at point B. Therefore, the objects in the first image may be slightly different from the objects in the second image due to the movement of the vehicle from point A to point B and the passage of time from (t-1) to (t). In one embodiment, the first and second images may be geometrically distorted, for example, radially distorted. Due to the geometric distortion, the outermost regions of the first and second images may be unclear for display. The distortion may also complicate processing for detecting objects within the images. For example, in a vehicle, it is essential to detect all objects around the vehicle while displaying them on the display unit 104 while parking. However, due to the geometric distortion, if an object is not accurately detected, serious damage to the vehicle may occur. The system 101 utilizes the first and second images along with one or more vehicle operating parameters to compensate for the geometric distortion.

[0024] In one embodiment, the system 101 receives one or more vehicle operating parameters from the ECU 103. The system may configure the camera 102a to capture a first image and a second image based on the one or more vehicle operating parameters. For example, the system 101 configures the camera 102a to capture an image every time the vehicle moves forward by 1 meter. Similarly, the system 101 configures the camera 102a to capture an image every time the vehicle moves backward by 1 meter. Similarly, the system 101 configures the camera 102a to capture an image every time the vehicle steers by one degree. In another example, the system 101 configures the camera 102a to capture an image every 16 ms when the vehicle is moving forward or backward. The examples provided in this disclosure should not be considered limiting. The values considered are merely exemplary, and the system may be configured to perform actions for different values.

[0025] Furthermore, the system 101 may synthesize a corrected image using the first image, the second image, and one or more vehicle motion parameters, and display the corrected image on the display unit 104. Similarly, the system 101 may synthesize corrected images from multiple cameras 102a, 102b, ... 102n to generate a 3D view of the surroundings of the vehicle, and display the 3D view on the display unit 104.

[0026] 2 shows a detailed block diagram of system 101. System 101 may include a central processing unit ("CPU" or "processor") 203 and a memory 202 that stores instructions executable by processor 203. Processor 203 may include at least one data processor for executing program components for carrying out user- or system-generated requests. Memory 202 may be communicatively coupled to processor 203. System 101 further includes an input / output (I / O) interface 201. I / O interface 201 may be coupled to processor 203 through which input and / or output signals may be communicated.

[0027] In some implementations, system 101 may include data 204 and modules 209. As an example, data 204 and modules 208 may be stored in memory 202 configured within system 101. In one embodiment, data 204 may include, for example, image data 205, vehicle operating parameters 206, and other data 207.

[0028] In one embodiment, the image data 205 may include a first image and a second image. As described above, the first image was captured at time (t-1) and the second image was captured at time (t). In one embodiment, the first image and the second image are stored after applying image correction. Various image processing techniques may be used to correct geometric distortions in the first image and the second image.

[0029] In one embodiment, the vehicle operating parameters 206 may include the vehicle speed and the vehicle steering angle. The vehicle operating parameters 206 may be obtained from the ECU 103 at regular intervals.

[0030] In one embodiment, the other data 207 may include image processing parameters for correcting the first image and the second image. Additionally, the other data 207 may also include image processing parameters for synthesizing a 3D surround view from images acquired from multiple cameras 102a, 102b, ... 102n.

[0031] In some embodiments, data 204 may be stored in memory 202 in the form of various data structures. Additionally, data 204 may be organized using a data model, such as a relational or hierarchical data model. Other data 207 may store data, including temporary data and files, generated by modules 208 to perform various functions of computing system 102.

[0032] In some embodiments, data 204 stored in memory 202 may be processed by module 208 of system 101. Module 208 may be stored within memory 202. In one example, module 208 communicatively coupled to processor 203 configured within system 101 may also reside outside of memory 202 and be implemented as hardware, as shown in FIG. 2. As used herein, the term module 208 may refer to an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), electronic circuitry, processor 203 (shared, dedicated, or group) and memory 202 executing one or more software or firmware programs, combinatorial logic circuitry, and / or other suitable components that provide the described functionality. In some other embodiments, module 209 may be implemented using at least one of an ASIC and an FPGA.

[0033] In one implementation, the modules 208 may include, for example, a communication module 209, an image correction module 210, an image synthesis module 211, a rendering module 212, and other modules 213. It may be understood that such aforementioned modules 208 may be represented as a single module or a combination of different modules 208.

[0034] In one embodiment, the communication module 209 is used to communicate with the multiple cameras 102a, 102b, ... 102n, the ECU 103, and the display unit 104. The communication module 209 may be an Ethernet module, a CAN module, a FlexRay module, or an LVDS module. The communication module 209 is configured to receive a first image and a second image from at least one camera 102a of the multiple cameras 102a, 102b, ... 102n. The communication module 209 may also be used to communicate instructions to the multiple cameras 102a, 102b, ... 102n about when to capture images. In some embodiments, only a specific camera (e.g., 102a) of the multiple cameras 102a, 102b, ... 102n may be operated. In such a situation, the communication module 209 may signal the remaining cameras to operate in an off state or place the remaining cameras in a power-saving state. In one embodiment, the first image is stored in memory 202 while the second image is being captured. After the second image is captured, the first image and the second image are processed to derive a corrected image. The communication module 209 can further communicate with the ECU 103 to receive vehicle operating parameters 206 and can communicate with the display unit 104 to provide a view (corrected image) of the vehicle (2D or 3D view).

[0035] In one embodiment, the image correction module 210 is configured to correct first and second images received from a camera (e.g., 102a) that have geometric distortion. In one embodiment, the first and second images may be radially distorted. In one embodiment, the image correction module 210 may use existing techniques to correct the radial distortion in the first and second images. Typically, barrel-shaped radial distortion occurs in fisheye lens cameras due to their wide-angle lenses. Few techniques used to correct radially distorted images involve obtaining the 3D coordinates of the distorted image and mapping it to a 2D plane. Few other techniques use images captured from different orientations. Few techniques also use geometric invariants such as lines or vanishing points. In one aspect, the radial distortion may be flattened in the corrected images. However, flattening the images may add noise and blur objects that exist in the outer regions of the first and second images. In some embodiments, the first and second images are processed together to correct the distortion. In some other embodiments, the first image is corrected and stored in memory 202 while the second image is being captured.

[0036] In one embodiment, the image synthesis module 211 is configured to synthesize a corrected image using the first image and the second image and the vehicle motion parameters 206. Because the present invention is performed in real time, the image synthesis module 211 retrieves the first image from the memory 202 and the second image from the camera (e.g., 102a). In one embodiment, the image synthesis module 211 can receive a corrected first image and a corrected second image. Although the images are corrected, distortion may still be present, causing objects to blur. The image synthesis module 211 further identifies a first slice in the first image where the view is undistorted and a corresponding second slice in the second image where the view is distorted. An image slice is a logical portion of an image. In one embodiment, image slicing techniques may be used to generate the first slice and the second slice. Slicing involves cutting an image into smaller logical images. For example, an object may be closer to the vehicle in the first image, while the same object may be farther away in the second image due to the forward motion of the vehicle. Thus, an object may be undistorted in the first image but distorted in the second image. Similarly, during a vehicle reversing motion, an object may be distorted in the first image but not in the second image. In one embodiment, the first and second slices are based on vehicle motion parameters 206. In one embodiment, a linear relationship between vehicle motion and pixel distance may be used. For example, the distance traveled by the vehicle between capturing the first and second images is determined, and the corresponding distance is converted to pixel distance to identify similar objects in the first and second images. The first and second slices are thereby identified. Furthermore, the image synthesis module 211 can synthesize a corrected image by substituting the second slice for the first slice. The same principle can be applied when the vehicle is reversing or steering left or right. The vehicle motion parameters 206 are used to determine the slices in the first and second images.The corrected composite image can therefore be displayed on the display unit 104 without radial distortion.

[0037] In one embodiment, the image synthesis module 211 can use multiple cameras 102a, 102b, . . . 102n to synthesize corrected images for each view (front, rear, side) to generate a 3D top view.

[0038] In one embodiment, the rendering module 212 is configured to render the corrected composite image on the display unit 104. In one embodiment, the rendering module 212 may perform basic image processing for display on the display unit 104, such as noise reduction, mapping the image resolution to the resolution of the display unit 104, setting the frame rate for displaying the video, setting the saturation, hue, and contrast parameters of the image, etc.

[0039] In one embodiment, the other modules 213 may include an object detection module, a driver assistance module, a notification module, etc. The object detection module may detect any object near the vehicle. For example, an obstacle near the vehicle may be detected using this module. The driver assistance module may assist the driver of the vehicle in navigation. For example, the driver assistance module may guide the driver while parking the vehicle. The notification module may notify the driver when an object is detected near the vehicle.

[0040] 3 shows a flowchart illustrating a method for displaying a vehicle surround view on the display unit 104 according to some embodiments of the present disclosure. The order in which the method 300 may be described is not intended to be construed as limiting, and any number of the described method blocks may be combined in any order to implement the method. Additionally, individual blocks may be deleted from the method without departing from the spirit and scope of the subject matter described herein. Furthermore, the method may be implemented in any suitable hardware, software, firmware, or combination thereof.

[0041] In step 301, the system 101 receives a first image and a second image captured by a camera (e.g., 102a) at time (t-1) and time (t), respectively. In one embodiment, the system 101 configures the camera 102a to capture the first image and the second image based on the vehicle operating parameters 206. Based on the situation, the system 101 may configure the camera 102a to capture an image. In one scenario when the vehicle is moving forward, the front camera (e.g., 102b) may be configured to capture an image every second, and the rear camera (e.g., 102a) may be configured to capture an image every three seconds. In another scenario when the vehicle is moving backward, the front camera 102b may be configured to capture an image every three seconds, and the rear camera 102a may be configured to capture an image every second. In another scenario, when the vehicle is parking, all the cameras 102, 102b, 102c, 102d may be configured to capture an image every second. Thus, the system 101 receives a first image and a second image based on the configuration of each camera. Because the first image is captured at time (t-1) and the second image is captured at time (t), the two images are different but contain substantially similar objects. The similarity of the objects may be due to small movements of the vehicle from its position from time (t-1) to time (t).

[0042] Figures 4a and 4b show exemplary diagrams of sensor fields of view around a vehicle. In Figure 4a, front camera 102b and rear camera 102a are shown. Lines may represent fields of view, but this is for illustrative purposes only and should not be considered limiting. Rear camera 102a has a field of view 401a, and front camera 102b has a field of view 401b. Figure 4b shows a scenario in which side cameras 102c and 102d are installed on a vehicle. Side camera 102c has a field of view 401c, and side camera 102d has a field of view 401d. A combination of images from cameras 102a, 102b, 102c, and 102d is used to generate a surround view. Figures 5a and 5b show exemplary temporal visualizations of a fisheye camera. Figures 5a and 5b show rear fisheye camera views of a first image and a second image, respectively. As can be seen, the objects in region 501 in Figure 5a are closer to the vehicle, and the objects in region 502 in Figure 5b are farther from the vehicle, indicating that the vehicle has moved forward since the first image was captured. Additionally, the first and second images are radially distorted.

[0043] In step 302, the system 101 receives the vehicle operating parameters 206 from the ECU 103, which in turn receives the vehicle operating parameters 206 from other sensors as described above. In one embodiment, the vehicle operating parameters 206 may be received at regular time intervals or at regular distance intervals traveled by the vehicle. For example, the vehicle speed and steering angle may be received every 2 seconds or every 10 meters traveled by the vehicle. The receipt of the vehicle operating parameters 206 may depend on the vehicle scenario. For example, while the vehicle is parked, the vehicle operating parameters 206 may be received every 50 milliseconds, and while the vehicle is on the highway, the vehicle operating parameters 206 may be received every second.

[0044] In step 303, the system 101 synthesizes a corrected image using the first image, the second image, and the vehicle motion parameters 206. Because the first image and the second image contain radial distortion, the system 101 processes the images to remove or reduce the effects of the distortion. Image flattening techniques, such as mapping 3D image space to 2D space, may be employed. Other techniques, such as using patterns acquired from different orientations to generate the corrected image. A distortion model may be used with additional techniques to estimate distortion parameters for generating the corrected image. In one embodiment, any existing technique may be employed to generate the corrected first image and the corrected second image. Furthermore, the system 101 synthesizes a single corrected image using the corrected first image, the corrected second image, and the vehicle motion parameters 206. The system 101 identifies a first slice in the corrected first image having an undistorted view and a second slice in the corrected second image having a distorted view. Regions in the first slice correspond to regions in the second slice, i.e., objects in the first slice may be substantially similar to objects in the second slice. Figures 6a and 6b show exemplary temporal visualizations of radial distortion-corrected images. As seen in Figure 6a, slice 601 shows a region in the corrected first image that has no distortion. However, in Figure 6b, slice 602 of the corrected second image has distortion. As explained, in images captured by a fisheye camera, more distant objects generally tend to be curved, so more distant objects may be distorted. Furthermore, when a vehicle is close to an object, the object may appear clear (without distortion), and when the vehicle moves away from the object, the object may appear distorted. Therefore, the proposed solution uses both images to synthesize the corrected image. The system 101 then correlates the vehicle motion parameters 206 with the second image to determine how far the vehicle has traveled since capturing the first image, in what direction the vehicle has traveled, and by how much the vehicle has steered since capturing the first image.This determination provides the system 101 with information to identify the first slice 601 and the second slice 602 .

[0045] In one embodiment, multiple images captured at small time intervals may also be used to synthesize a single corrected image. For example, five images captured at different time intervals may be stitched together to synthesize a corrected image. For example, a first image captured at a first time instance (t), a second image captured at a second time instance (t+1), a third image captured at a third time instance (t+2), and a fourth image captured at a fourth time instance (t+3) may be stitched together. In one embodiment, the time instances may be determined based on vehicle ego parameters. The time instances may be determined based on vehicle operating parameters. The number of images needed to stitch together may be based on image distortion. If the fisheye camera introduces distortion closer to the center of the image, a greater number of images may be needed. Furthermore, the time images are also based on vehicle movement. The above scenario is described considering a case where the fisheye camera is installed on the rear side of the vehicle while the vehicle is moving forward. While the vehicle is moving in reverse, objects behind the vehicle may move closer to the vehicle, causing a first slice to have distorted regions and a second slice to have corresponding undistorted regions. Similarly, based on the vehicle's movement, the system 101 determines similarities that identify slices in the images to be stitched together. In one embodiment, slices 601 and 602 are further identified based on vehicle motion parameters 206. For example, once the first image is captured, it is stored in memory 202.

[0046] Furthermore, the system 101 replaces the second slice 602 with the first slice 601 to synthesize a distortion-free corrected image. In one embodiment, the system 101 may use an image stitching technique to replace the second slice 602 with the first slice 601. The synthesized corrected image is distortion-free and objects appear clear without blurring.

[0047] Referring back to FIG. 3 , in step 304, the system 101 displays the rectified composite image on the display unit 104. In one embodiment, the system 101 can display a rectified composite view of one side view of the vehicle (rear view, front view, or left or right view) or a rectified composite view of all sides of the vehicle (surround view). In one embodiment, the system 101 can stitch together the rectified composite images generated for all sides (front, rear, left, and right) and render a surround view of the vehicle on the display unit 104. FIG. 7 shows an example diagram of a top surround view 700 of a vehicle. Similarly, a perspective view may also be provided. In one embodiment, existing techniques may be used to generate the top surround view 700. For example, four perspective images (matrices) may be acquired and converted to a 2D ground plane so that the top view provides a 2D visualization. 7, a top surround view 700 is obtained by stitching together four top views 701, 702, 703, 704 obtained by processing the respective sensor data. In one embodiment, the stitching is performed such that the overlapping regions are considered for reference, and the images are stitched together in the overlapping regions. For the stitching, the camera placement may be taken into account, or the images themselves may be processed to determine the overlapping regions. In the overlapping regions, data may be blended, or data from one of the images may be considered. Furthermore, the surround top view is displayed on the display unit 104.

[0048] In one embodiment, the proposed solution is useful for driver assistance systems, as objects are clearly identified and displayed. Furthermore, parking a vehicle becomes more convenient, as obstacles are clearly detected. Furthermore, the driving experience is enriched, as the surroundings are clearly displayed. 3D visualization provides high-quality synthesized information of the surroundings. The synthesized corrected image consists of a higher pixel density compared to existing techniques.

[0049] Computer Systems 8 illustrates a block diagram of an exemplary computer system 800 for implementing embodiments consistent with the present disclosure. In one embodiment, the computer system 800 may be used to implement a method for generating a filter sequence for training a model. The computer system 800 may include a central processing unit ("CPU" or "processor") 802. The processor 802 may include at least one data processor for executing program components for dynamic resource allocation at runtime. The processor 802 may include special-purpose processing units such as an integrated system (bus) controller, a memory management control unit, a floating-point unit, a graphics processing unit, a digital signal processing unit, etc.

[0050] The processor 802 may be arranged to communicate with one or more input / output (I / O) devices (not shown) via an I / O interface 801. The I / O interface 801 may employ communication protocols / methods such as, but not limited to, audio, analog, digital, mono, RCA, stereo, IEEE-(1394), serial bus, universal serial bus (USB), infrared, PS / 2, BNC, coaxial, component, composite, digital visual interface (DVI), high-definition multimedia interface (HDMI), RF antenna, S-video, VGA, IEEE 802.n / b / g / n / x, Bluetooth, cellular (e.g., code division multiple access (CDMA), high-speed packet access (HSPA+), global system for mobile communications (GSM), long-term evolution (LTE), WiMax, etc.).

[0051] Using I / O interface 801, computer system 800 can communicate with one or more I / O devices. For example, input device(s) 810 can be an antenna, keyboard, mouse, joystick, (infrared) remote control, camera, card reader, fax machine, dongle, biometric reader, microphone, touch screen, touch pad, trackball, stylus, scanner, storage device, transceiver, video device / source, etc. Output device(s) 811 can be a printer, fax machine, video display (e.g., cathode ray tube (CRT), liquid crystal display (LCD), light emitting diode (LED), plasma, plasma display panel (PDP), organic light emitting diode display (OLED), etc.), audio speaker, etc.

[0052] In some embodiments, computer system 800 is connected to a service operator via a communications network 809. Processor 802 may be arranged to communicate with communications network 809 via a network interface 803. Network interface 803 may communicate with communications network 809. Network interface 803 may employ connection protocols including, but not limited to, direct connections, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base-T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 802.11a / b / g / n / x, etc. Communications network 809 may include, but is not limited to, direct interconnections, e-commerce networks, peer-to-peer (P2P) networks, local area networks (LANs), wide area networks (WANs), wireless networks (e.g., using wireless application protocols), the Internet, Wi-Fi, etc. Using network interface 803 and communications network 809, computer system 800 may communicate with one or more service operators.

[0053] In some embodiments, the processor 802 may be arranged to communicate with memory 805 (e.g., RAM, ROM, etc., not shown in FIG. 7 ) via a storage interface 804. The storage interface 804 may be connected to memory 805 including, but not limited to, memory drives, removable disk drives, etc. employing connection protocols such as Serial Advanced Technology Attachment (SATA), Integrated Drive Electronics (IDE), IEEE-1394, Universal Serial Bus (USB), Fibre Channel, Small Computer System Interface (SCSI), etc. The memory drives may further include drum, magnetic disk drives, magneto-optical drives, optical drives, redundant array of independent disks (RAID), solid state memory devices, solid state drives, etc.

[0054] Memory 805 can store a collection of program or database components, including, but not limited to, a user interface 806, an operating system 807, a web server 808, etc. In some embodiments, computer system 800 can store user / application data 806, such as data, variables, records, etc., as described in this disclosure. Such a database can be implemented as a fault-tolerant, relational, scalable, and secure database, such as Oracle or Sybase.

[0055] Operating system 807 may facilitate resource management and operation of computer system 800. Examples of operating systems include, but are not limited to, APPLE® MACINTOSH® OS X®, UNIX®, UNIX-like system distributions (e.g., BERKELEY SOFTWARE DISTRIBUTION® (BSD), FREEBSD®, NETBSD®, OPENBSD, etc.), LINUX® DISTRIBUTIONS (e.g., REDHAT®, UBUNTU®, KUBUNTU®, etc.), IBM® OS / 2®, MICROSOFT® WINDOWS® (XP®, VISTA® / 7 / 8, 10, etc.), APPLE® IOS®, GOOGLE™ ANDROID™, BLACKBERRY® OS, etc.

[0056] In some embodiments, computer system 800 may implement a web browser (not shown) stored program component. The web browser may be a hypertext browsing application such as MICROSOFT® INTERNET EXPLORER®, GOOGLE™ CHROME™, MOZILLA® FIREFOX®, APPLE® SAFARI®, etc. Secure web browsing may be provided using Secure Hypertext Transport Protocol (HTTPS), Secure Sockets Layer (SSL), Transport Layer Security (TLS), etc. Web browser 808 may utilize facilities such as AJAX, DHTML, ADOBE® FLASH®, JAVASCRIPT®, JAVA®, application programming interfaces (APIs), etc. In some embodiments, computer system 800 may implement a mail server stored program component. The mail server may be an Internet mail server such as Microsoft Exchange. The mail server may utilize facilities such as Active Server Pages (ASP), ACTIVEX®, ANSI® C++ / C#, MICROSOFT®, .NET, CGI SCRIPTS, JAVA®, JAVASCRIPTS®, PERL®, PHP, PYTHON®, WEBOBJECTS®, etc. The mail server may utilize communication protocols such as Internet Message Access Protocol (IMAP), Messaging Application Programming Interface (MAPI), MICROSOFT® Exchange, Post Office Protocol (POP), Simple Mail Transfer Protocol (SMTP), etc. In some embodiments, computer system 800 may implement a mail client storage program component.The email client can be an email viewing application such as APPLE® MAIL, MICROSOFT® ENTOURAGE®, MICROSOFT® OUTLOOK®, MOZILLA® THUNDERBIRD®, or the like.

[0057] Additionally, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present invention. A computer-readable storage medium refers to any type of physical memory 805 in which information or data readable by the processor 802 may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and to exclude carrier waves and transient signals, i.e., non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, compact disc (CD) ROMs, digital video discs (DVDs), flash drives, disks, and any other known physical storage medium.

[0058] In one embodiment, the computer system 800 may include a remote device 812. The computer system 800 may receive the first model 104, the second model 105, and the data set 103 from the remote device 812 via a communication network 809.

[0059] The terms "an embodiment," "embodiment," "embodiments," "the embodiment," "the embodiment," "the embodiments," "one or more embodiments," "some embodiments," and "one embodiment" mean "one or more (but not all) embodiments of the present invention," unless expressly specified otherwise.

[0060] The terms "including," "comprising," and "having," and variations thereof, mean "including but not limited to," unless expressly stated otherwise.

[0061] An enumerated list of items does not imply that any or all of the items are mutually exclusive unless expressly stated otherwise. The terms "a," "an," and "the" mean "one or more" unless expressly stated otherwise.

[0062] A description of an embodiment having several components in communication with each other does not imply that all such components are required, to the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0063] Where a single device or article is described herein, it will be apparent that more than one device / article (whether or not they cooperate) may be used in place of the single device / article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device / article may be used in place of more than one device or article, or that a different number of devices / articles may be used in place of the number of devices or programs shown. The functionality and / or features of a device may alternatively be embodied by one or more other devices not explicitly described as having such functionality / features. Thus, other embodiments of the present invention need not include the device itself.

[0064] The depicted operations in Figures 3 and 5 show certain events occurring in a particular order. In alternative embodiments, certain operations may be performed in a different order, modified, or removed. Furthermore, steps may be added to the logic described above and still be consistent with the embodiments described above. Furthermore, operations described herein may occur sequentially, or certain operations may be processed in parallel. Furthermore, operations may be performed by a single processing unit or by distributed processing units.

[0065] Finally, the language used herein has been chosen primarily for readability and instructional purposes, and not to delineate or limit the subject matter of the present invention. Accordingly, it is intended that the scope of the invention be limited not by this detailed description, but by any claims that issue on an application based thereon. Accordingly, the disclosure of embodiments of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is defined in the following claims.

[0066] While various aspects and embodiments have been disclosed herein, other aspects and embodiments may be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims. The present application relates to the invention described in the claims, but also includes the following as other aspects. 1. 1. A method for displaying a surround view in a vehicle, comprising: receiving, from the vehicle-mounted camera 102a, a first image captured at a first time instance (t-1) and a second image captured at a second time instance (t), wherein the first image is different from the second image; receiving one or more vehicle operating parameters from one or more sensors associated with the vehicle; synthesizing a corrected image using the first image, the second image, and the one or more vehicle motion parameters; displaying the corrected image on a display unit 104 of the vehicle; A method comprising: 2. 2. The method of claim 1, wherein the first image and the second image are radially distorted. 3. 2. The method of claim 1, wherein the second image is captured after the vehicle has moved from the location where the first image was captured. 4. 2. The method of claim 1, wherein the one or more vehicle operating parameters include at least one of a speed of the vehicle and a steering angle of the vehicle. 5. synthesizing the corrected images, identifying a first slice having an undistorted view 501 in the first image, the first slice corresponding to a second slice having a distorted view 502 in the second image, wherein identifying the slice is based on the one or more vehicle motion parameters; combining the corrected images such that the first slice replaces the second slice in the corrected image; 2. The method according to claim 1, comprising: 6. 2. The method of claim 1, wherein the corrected image is processed to represent a view selected from the group consisting of a top view 700 of the vehicle, a front view 704 of the vehicle, a rear view (702) of the vehicle, a side view 701, 703 of the vehicle, and combinations thereof, and the view of the vehicle is displayed on the display unit 104. 7. A system 101 for displaying a surround view 700 in a vehicle, comprising: A memory 202; a processor 203, receiving a first image captured at a first time instance (t-1) and a second image captured at a second time instance (t) from a camera 102a mounted on the vehicle, the first image being different from the second image; receiving one or more vehicle operating parameters from one or more sensors associated with the vehicle; synthesizing a corrected image using the first image, the second image, and the one or more vehicle motion parameters; displaying the corrected image on a display unit 104 of the vehicle; a processor 203 configured to: The system 101 includes: 8. 8. The system 101 of claim 7, wherein the processor 203 receives a first image and a second image that include radial distortion. 9. 8. The system 101 described in claim 7, wherein the processor 203 controls the camera 102a to capture the second image after the vehicle has moved from the position where the first image was captured. 10. 8. The system 101 of claim 7, wherein one or more sensors are configured to measure at least one of the speed of the vehicle and the steering angle of the vehicle. 11. The processor 203: identifying a first slice having an undistorted view 501 in the first image, the first slice corresponding to a second slice having a distorted view 502 in the second image, wherein identifying the slice is based on the one or more vehicle motion parameters; combining the corrected images such that the first slice replaces the second slice in the corrected image; 8. The system 101 according to claim 7, wherein the corrected image is synthesized by performing the steps of: 12. The system 101 described in claim 7, wherein the processor 203 processes the corrected image to represent a view from a group consisting of a top view 700 of the vehicle, a front view 704 of the vehicle, a rear view 702 of the vehicle, side views 701, 703 of the vehicle, and combinations thereof, and the views of the vehicle are displayed on the display unit. [Explanation of symbols]

[0067] 101 System 102 Camera 103 ECU 104 Display Unit 201 I / O interface 202 memory 203 processor 204 Data 205 Image Data 206 Vehicle operating parameters 207 Other Data 208 modules 209 Communication Module 210 Image Correction Module 211 Image Synthesis Module 212 Rendering Module Operators 213 other modules 501, 502, 601, 602 image slices 800 Computer Systems 801 I / O interface 802 processor 803 network interface 804 storage interface 805 memory 806 User Interface 807 Operating System 808 web browser 809 Communication Network 810 Input Devices 811 output devices 812 Remote Device

Claims

1. 1. A method for displaying a surround view in a vehicle, comprising: receiving, from a camera (102a) mounted on the vehicle, a first image captured at a first time instance (t-1) and a second image captured at a second time instance (t), wherein the first image is different from the second image; receiving one or more vehicle operating parameters from one or more sensors associated with the vehicle; synthesizing a corrected image using the first image, the second image, and the one or more vehicle motion parameters; The step of synthesizing the corrected images includes: identifying a first slice having an undistorted view (501) in the first image, the first slice corresponding to a second slice having a distorted view (502) in the second image, wherein identifying the slice is based on the one or more vehicle motion parameters; combining the corrected images such that the first slice replaces the second slice in the corrected image; and displaying the corrected image on a display unit (104) of the vehicle; A method comprising:

2. The method of claim 1 , wherein the first image and the second image are radially distorted.

3. The method of claim 1 , wherein the second image is captured after the vehicle has moved from the location where the first image was captured.

4. The method of claim 1 , wherein the one or more vehicle operating parameters include at least one of a speed of the vehicle and a steering angle of the vehicle.

5. 2. The method of claim 1, wherein the corrected image is processed to represent a view selected from the group consisting of a top view (700) of the vehicle, a front view (704) of the vehicle, a rear view (702) of the vehicle, a side view (701, 703) of the vehicle, and combinations thereof, and the view of the vehicle is displayed on the display unit (104).

6. A system (101) for displaying a surround view (700) in a vehicle, comprising: The system includes a memory (202) and a processor (203), The processor (203) receiving, from a camera (102a) mounted on the vehicle, a first image captured at a first time instance (t-1) and a second image captured at a second time instance (t), the first image being different from the second image; receiving one or more vehicle operating parameters from one or more sensors associated with the vehicle; synthesizing a corrected image using the first image, the second image, and the one or more vehicle motion parameters, wherein the processor (203) performs the steps of: identifying a first slice having an undistorted view (501) in the first image, the first slice corresponding to a second slice having a distorted view (502) in the second image, the identifying of the slice being based on the one or more vehicle motion parameters; and synthesizing the corrected image such that the first slice replaces the second slice in the corrected image; displaying the corrected image on a display unit (104) of the vehicle; The system (101) is configured as follows.

7. The system (101) of claim 6, wherein the processor (203) receives a first image and the second image, the first image including radial distortion.

8. 7. The system (101) of claim 6, wherein the processor (203) controls the camera (102a) to capture the second image after the vehicle has moved from the position where the first image was captured.

9. The system (101) of claim 6, wherein one or more sensors are configured to measure at least one of a speed of the vehicle and a steering angle of the vehicle.

10. 7. The system (101) of claim 6, wherein the processor (203) processes the corrected image to represent a view from the group consisting of a top view (700) of the vehicle, a front view (704) of the vehicle, a rear view (702) of the vehicle, a side view (701, 703) of the vehicle, and combinations thereof, and the view of the vehicle is displayed on the display unit.

Images