System and method for image normalization

Abstract

A system for generating a virtual image view for a vehicle is provided. The system includes: one or more image capture devices configured to capture image data near the vehicle and provide identifiers identifying one or more corresponding image capture devices, wherein the image data is at least partially defined by a first viewpoint parameter; a storage device configured to store a plurality of virtualization records containing conversion information related to a virtualization viewpoint and a plurality of image capture devices; and a processing device. The processing device is configured to receive the captured image data; convert the first viewpoint parameter of the captured image data into a virtual viewpoint parameter based on the conversion information associated with the virtualization record stored by the storage device to generate a virtual image view, wherein the virtualization record is identified based on at least the identifier; and perform at least one driver assistance and / or autonomous driving function based on the virtual image view.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for automotive driver assistance, and more particularly, to virtualization of one or more images obtained from a mounted onboard camera. Background Art

[0002] Vehicles equipped with one or more cameras have become more common in recent years. These cameras have been introduced for various reasons, such as visual feedback, parking assistance, collision avoidance, traffic guidance, etc., and have become an important input modality for autonomous driving and automated driving assistance.

[0003] These cameras typically exhibit different characteristics (eg, optical properties) described by, for example, pinhole and fisheye lens models, as well as different mounting locations on a particular vehicle.

[0004] Software designed and developed for automotive or automation applications typically expects images from a predefined viewpoint assumed for one or more cameras installed in the vehicle, and performance may degrade or even cease when the viewpoint provided by the camera or cameras is not within the assumed specifications.

[0005] Some existing systems implement camera devices for improving recognition accuracy and robustness. Vehicle manufacturers use front camera devices and devices for combining signals with data related to the driving state of the vehicle. Further, devices for combining image signals with navigation device output have also been implemented.

[0006] US 2016 / 360104 discloses systems and methods for generating combined views from a fisheye camera. Summary of the invention

[0007] The inventors have realized that for mass production of multiple vehicle models, many problems arise due to the differences in the viewing angle(s) of the camera setup for each specific and different vehicle type with different geometries. This may also result in viewpoint differences due to installation constraints and / or restrictions during the production process.

[0008] The impact of these issues on the processing system can be complex for traditional camera setups (eg, pinhole), but for fisheye cameras the impact can be even more significant.

[0009] According to an embodiment of the present disclosure, a system for generating a virtual image view for a vehicle is provided. The system includes: one or more image capture devices, which are configured to capture image data near the vehicle and provide identifiers that identify one or more corresponding image capture devices, the image data being at least partially defined by a first viewpoint parameter; a storage device, which is configured to store multiple virtualization records containing conversion information related to virtualization viewpoints and multiple image capture devices; and a processing device. The processing device is configured to receive the captured image data, convert the first viewpoint parameters of the captured image data into virtual viewpoint parameters based on the conversion information associated with the virtualization record stored by the storage device to generate a virtual image view, wherein the virtualization record is identified at least based on the identifier, and perform at least one driver assistance and / or autonomous driving function based on the virtual image view.

[0010] By providing such a system, data associated with one or more captured images and / or scenes around a vehicle can be virtualized and normalized to a view desired by a particular processing application. For example, where a fisheye camera is provided on one side of a first vehicle model and a pinhole camera is provided on one side of a second vehicle model, a single processing application installed on the first and second vehicles can receive nearly the same virtualized view regardless of differences in camera type, camera mounting angle, camera position, etc. Thus, one processing application can be implemented across a full set of mass-produced models.

[0011] In addition, since there may be individual differences between cameras, these differences can be effectively presented to the processing application without being noticed by virtualizing the viewpoint parameters.

[0012] The conversion information may include at least one of: distortion compensation information, image correction information, image refraction information, and rotation information.

[0013] Each virtualized viewpoint may be defined by at least one of a resolution and a field of view and at least one of a pinhole model, a fisheye model, a cylindrical model, a spherical model, and a rectilinear model.

[0014] The storage means may include means for identifying a processing application linked to the virtual image view.

[0015] The storage device may include a lookup table linking the camera identifier with transformation information for generating the virtual image view required by the processing application.

[0016] The conversion information may include calibration parameters for the identified image capture device, and the processing device may be configured to verify the calibration parameters for each of the one or more image capture devices.

[0017] The processing application may include at least one of: an object detection and tracking application, and a lane identifier application.

[0018] According to a further embodiment, a vehicle comprising the system described above is provided.

[0019] According to a further embodiment, a method for generating a virtual image view for a vehicle is provided. The method includes: receiving image data captured by at least one image capture device from a surrounding environment of the vehicle, the image data being at least partially defined by a first viewpoint parameter; receiving an identifier identifying one or more image capture devices providing the image data; determining a virtualization record containing conversion information related to a virtualization viewpoint and a plurality of image capture devices, wherein the virtualization record is determined based on at least the identifier; converting the first viewpoint parameter of the captured image data into a virtual viewpoint parameter based on the conversion information associated with the determined virtualization record to generate a virtual image view; and performing at least one driver assistance and / or autonomous driving function based on the virtual image view.

[0020] The conversion information may include at least one of: distortion compensation information, image correction information, image refraction information, and rotation information.

[0021] Each virtualized viewpoint may be defined by at least one of a resolution and a field of view and at least one of a pinhole model, a fisheye model, a cylindrical model, a spherical model, and a rectilinear model.

[0022] Determining may include identifying a processing application linked to the virtual image view.

[0023] Determining may include searching a lookup table including information linking a camera identifier with transformation information for generating a virtual image view required by a processing application.

[0024] The conversion information may include calibration parameters for the identified image capture device, and the method further includes verifying the calibration parameters for each of the one or more image capture devices.

[0025] The processing application may include at least one of: an object detection and tracking application, and a lane identifier application.

[0026] It is contemplated that combinations of elements described above and within the specification can be made unless otherwise contradicted.

[0027] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure, as claimed.

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram showing an exemplary image virtualization system of a vehicle according to an embodiment of the present disclosure is shown;

[0030] Figure 2A shows an exemplary image captured from a first camera mounted on a vehicle;

[0031] Figure 2B shows a captured image normalized according to an embodiment of the present disclosure;

[0032] Figure 3A shows a logical grouping of information and associated data associated with one or more camera settings;

[0033] Figure 3B shows an exemplary data graph related to transformation information and processing applications; and

[0034] Figure 4 is a flowchart highlighting an exemplary processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals will be used throughout the drawings to refer to the same or like parts.

[0036] Figure 1 A schematic diagram of an exemplary image normalization system for a vehicle is shown. The driver assistance system 1 may include a processing device, such as an electronic control unit (ECU) 10, one or more cameras 15, one or more sensors 20, a system controller 32, and a display 25, among others.

[0037] A global positioning system (GPS) 17 corresponding to a location providing device is also provided for providing map data about the location of the vehicle (e.g., location information such as coordinates) and feature information about features near the vehicle (e.g., adjustment indicators, traffic signals, exit lanes, etc.).

[0038] The image capture device 15 may include, for example, one or more cameras and / or other suitable devices configured to obtain optical data from areas around the vehicle (e.g., in front of, on both sides of, and behind the vehicle). The image capture device 15 may be configured to process the optical data obtained from the surrounding environment of the vehicle and output a signal to transmit the data. Such image capture devices 15 are known in the art, and those skilled in the art will understand that any such image capture device 15 may be implemented in the present system without departing from the scope of the present disclosure. For example, the image capture device 15 may include a pinhole camera, a fisheye camera, a cylindrical camera, a spherical camera, etc.

[0039] The image capture devices 15 may be positioned at various locations on the vehicle to provide an adequate field of view 4 of the vehicle's surroundings (e.g., a front and side view covering nearly 360 degrees around the vehicle). For example, one or more image capture devices 15 may be positioned behind the windshield, on the front bumper, side mirrors, rearview mirrors, rear window, rear bumper, and / or any other suitable mounting location on the vehicle to provide a field of view 4 of surrounding objects, including other vehicles present in the lane of the vehicle currently being driven.

[0040] According to some embodiments, it may be desirable to minimize the visibility of the image capture device 15 on the vehicle for aesthetic reasons, and those skilled in the art will appreciate that finding a mounting location suitable for achieving this goal while also providing an adequate field of view around the vehicle is a legitimate consideration. Further, those skilled in the art will recognize that, depending on the vehicle design, the placement of the image capture device 15 may vary across models, and depending on the placement of each image capture device 15, the output data may have different initial parameters such as rotation, distortion, refraction, etc.

[0041] When referring to the field of view as used herein, the term "sufficient" shall mean a field of view that provides the image capture device 15 with the ability to provide image data at a sufficiently large distance to provide the ECU 10 with sufficient time in an autonomous driving scenario to respond to the presence of objects and other features (e.g., lane lines 210, vehicles 250, road conditions 200, etc.) in the field of view of the image capture device 15. For example, a "sufficient" field of view to the right or left of the vehicle would include a view of the lane immediately adjacent to the vehicle or two or more lanes away from the vehicle, as well as any other vehicles traveling in / belonging to those lanes and lane markings.

[0042] In addition to an identifier that enables identification of the image capture device 15 by the ECU 10, the image capture device 15 may be configured to provide data regarding the vehicle's surroundings to the ECU 10. The image capture device 15 may provide such data to the ECU 10 via a wired connection, a wireless connection, or other suitable method for transmitting data to the ECU 10. For example, the image capture device 15 may include a wireless communication device (e.g., IEEE 802.11 compliant Wi-Fi hardware, Bluetooth, etc.) for sending data to the ECU 10 and / or other devices that may use data from the image capture device 15.

[0043] Alternatively or additionally, a wired connection may be provided, for example for safety purposes.Such a wired connection may be provided, for example, to provide a failsafe when a wireless connection should cease to function, for example due to interference or a transceiver anomaly.

[0044] The identifier for the image capture device 15 may include, for example, one or more digits, numbers, and / or letters, and may be, for example, unique to the image capture device. Alternatively, the identifier may be used more generally to identify the location and type of the camera. For example, one or more digits of the identifier may indicate / correspond to the type of camera (e.g., fisheye, cylindrical, etc.), while one or more digits may indicate the mounting location on the vehicle, and / or the vehicle model on which it is mounted. Those skilled in the art will appreciate that various schemes may be implemented for the identifier, and any such schemes are intended to fall within the scope of the present disclosure.

[0045] As described above, the various image capture devices 15 mounted / installed on the vehicle may vary not only by type and mounting location, but also by various viewpoint parameters, such as, for example, rotation, resolution, distortion, projection model, field of view, etc. The viewpoint parameters for each image capture device 15 may therefore be stored in a table and referenced by, for example, an identifier. For example, some image capture devices 15 may capture image data at a resolution of 1024×768 pixels, while others may capture at full 1080p (i.e., 1900×1080 pixels). A processing application may expect an image resolution of 800×600 pixels, and therefore, it may be desirable to convert each data set for each image capture device 15, which may be converted to a resolution of 800×600 for that particular processing application.

[0046] Figure 2A and Figure 2B An exemplary image captured from a first camera mounted on a vehicle and a captured image normalized according to an embodiment of the present disclosure are respectively shown. Figure 3A and Figure 3BAn exemplary data diagram is highlighted showing logical grouping of information associated with one or more camera settings and associated data (eg, viewpoint parameters) and related transformation information and processing application in accordance with an embodiment of the present disclosure.

[0047] As shown in the figure, many camera settings 161-16 i may exist on any particular vehicle model, as well as vary across vehicle models. Thus, a table 80 may be provided for linking a camera setting / identifier 16 with its intrinsic viewpoint parameters, and subsequently for such camera setting 16 x The conversion parameters are linked with the processing application 90, thereby allowing the image output data in the image capture device 15 to be converted to the viewpoint parameters desired by the processing application.

[0048] For example, as in Figure 2A As shown in FIG. 1 , the mounted image capture device outputs a "non-virtualized" view with first viewpoint parameters including, for example, rotation and lens distortion (e.g., caused by a fisheye lens). An object or lane detection application may request a view from the image capture device. Thus, based on the application, the ECU 10 may check a lookup table based on an identifier of the image capture device.

[0049] Taking the first viewpoint parameters and the camera settings 16, based on the identifier for the camera, conversion information for converting the first viewpoint parameters into virtual viewpoint parameters can be obtained from the database. Such conversion information can indicate, for example, Figure 2B The compensated rotation and distortion values ​​are generated in the virtualized view shown at . The virtualized view can then be used by the object detection application for the desired purpose.

[0050] according to Figure 2A and Figure 2B The example shown here is for Figure 2A The first viewpoint parameters of the image data (e.g., fisheye projection model) can be defined by the following four tables:

[0051] 338.1759919 0 644.4998055 0 338.1759919 403.6153106 0 0 1

[0052] Table 1 - Camera matrix (3×3 matrix)

[0053] -0.006448210017 0.06115093356 -0.01489872136

[0054] Table 2 - Radial Distortion

[0055] -0.3047415517 0.7943601275 -0.5254755698 0.9319508488 0.3625064463 0.007529406374 0.1964693416 -0.4874228803 -0.850775372

[0056] Table 3 - Rotation Matrix

[0057] -0.9463271281 0.7095791917 0.9865831897

[0058] Table 4 - Translation Matrix

[0059] According to an embodiment of the present disclosure, the above first viewpoint parameters can then be converted into rectilinear cylindrical virtual viewpoint parameters (such as Figure 2B ), for example, to facilitate object detection. Conversion of the image virtualized by the virtual viewpoint parameters can be performed using the conversion information from the lookup table 80 based on the processing application 90 and the camera identifier. The resulting converted virtual viewpoint parameters are shown in the following four tables:

[0060] 368.2736511 0 640 0 368.2736511 403 0 0 1

[0061] Table 5 – Transformed camera matrix (3×3)

[0062] 0 0 0

[0063] Table 6 – Transformed radial distortion

[0064] 0.01249270748 0.04229977313 -0.9990268549 0.9999219729 -0.0007382465767 0.0124726821 -0.0002099510197 -0.9991046853 -0.04230563897

[0065] Table 7 – Transformed rotation matrix

[0066] -0.9463271281 0.7095791917 0.9865831897

[0067] Table 8 – Transformed translation vectors

[0068] For the purpose of further describing the table shown above, assume that the pinhole camera is the original projection model camera mounted on the vehicle. For the pinhole camera model, a 3D point M in the scene can be related to its projection m in the image such that:

[0069]

[0070] in:

[0071] M=(XYZ)T is a 3D point M of the scene, m=(xy)T is its projection in the camera image, and ≈ denotes equality up to a non-zero scalar multiple.

[0072] K is a 3×3 calibration matrix, R is a 3×3 rotation matrix representing the orientation of the camera, and t is a translation vector t=(tx ty tz)T representing the position of the camera.

[0073] In this embodiment, it can be assumed that the aspect ratio is 1 and the tilt is 0. The calibration matrix K can be expressed as:

[0074]

[0075] Where: f is the focal length (in pixels), and p = (p x py ) T is the principal point of the camera. Consider a point Q=R corresponding to point M but expressed in a coordinate system attached to the camera (camera coordinate system) T (Mt); q is the "normalized" projection of point Q,

[0076] So, Or m=f.q+p (3)

[0077] For fisheye cameras, image projection is formulated in terms of angles rather than normalized coordinates:

[0078]

[0079] To introduce radial distortion into the camera model, equation (3) can be rewritten as:

[0080] Or m=fD(q)+p (4)

[0081] The distortion function D() radially displaces the normalized projection before it is converted to image coordinates:

[0082] D(q)=d*q,d=1+κ1θ 2 , +κ2θ 4 +κ3θ 6 (+...)

[0083] Where κ1, κ2, κ3, ... are parameters defining the radial distortion in the image of the camera. For this specific example, the first three tables above contain 3 parameters to describe the radial distortion, but it should be noted that the formula is not limited to 3 parameters.

[0084] For a virtual camera, the model projection can be chosen to be much simpler (e.g., radial distortion is eliminated), and a more intuitive image representation can be defined for a given computer vision algorithm. For the fourth to sixth tables above, data is used by an exemplary equidistant rectangular cylindrical projection model. Formula (3) applies, where q is defined as:

[0085]

[0086] According to some embodiments, the first viewpoint parameters and the output image data may facilitate a determination as to whether the calibration of the associated image capture device 15 is correct.

[0087] For example, calibration can be checked using a single image based on a priori known information of the surrounding structure of the vehicle, such as 1) shape and / or contour; and 2) information about the surrounding scene (e.g., road structure, parallel lines, map information, etc.). Therefore, the projection constraints of the image data output can be compared with the first viewpoint parameters (i.e., the first viewpoint parameters expected to be provided by the camera 15) to determine whether the actual data and parameters correspond, and thereby determine whether the camera 15 is calibrated.

[0088] Alternatively or in addition, feature tracking across a sequence of images may also give information about the three-dimensional structure in the context of the structure from the motion.Using these three-dimensional structures, the calibration can be checked in a similar manner to that described above.

[0089] According to another example, since a vehicle typically includes more than one image capture device 15 mounted thereon (e.g., two, four, six, eight cameras, etc.), any overlap between the various image capture devices 15 can be further exploited in terms of feature correspondence tracking and / or triangulation to examine the surrounding 3D structure (using SfM as described above). Thus, calibration can be checked based on the expected first viewpoint parameters compared to the actual parameters across overlapping camera views.

[0090] According to some embodiments, a system for generating virtual image views from image capture device 15 may assume that image capture device 15 and its associated output image data are properly calibrated. Thus, no automatic calibration check may be performed, and image data may be virtualized / normalized according to the techniques described herein.

[0091] One or more sensors 20 may be configured to send status information related to the status of the vehicle to the ECU 10. For example, the status of the vehicle may include the speed at which the vehicle is traveling, the direction in which the vehicle is traveling, the changes in direction that the vehicle is experiencing and / or has experienced, the position of the steering wheel, the distance that the vehicle has traveled, etc.

[0092] Thus, the one or more sensors 20 may include, for example, a steering wheel position sensor, a vehicle speed sensor, a yaw rate sensor, etc. Similar to the image capture device 15, such sensors may be configured to provide such state information to the ECU 10 wirelessly and / or by wire, and may also include duration information. Duration information associated with the state information may be tracked by the ECU 10.

[0093] The ECU 10 may include any suitable device configured to manipulate data, perform calculations, execute code for decision making so that information is displayed to an operator of the vehicle, and / or cause the control system controller 32 to take action on one or more systems of the vehicle (e.g., steering, braking, etc.) in order to perform embodiments of the present disclosure. For example, the ECU 10 may include various analog and / or digital circuits, and may include integrated circuits such as RISC processors, i386 processors, ASIC processors, etc. Typically, onboard computers in modern vehicles include such processors, and those skilled in the art will appreciate that the present ECU 10 may be included by such onboard computers, or may be provided separately. Those skilled in the art will also appreciate that the exemplary circuits and processors described herein are not intended to be limiting, and any suitable device may be implemented.

[0094] The ECU 10 may be linked to one or more databases and / or other memories (e.g., RAM, ROM, etc.) associated with the vehicle to enable storage of vehicle-related data and values ​​(e.g., viewpoint parameter conversion tables, processing applications, camera identifiers, etc.) that may be used during processing of vehicle functions (such as object detection and recognition). Those skilled in the art will recognize that the information discussed herein with respect to any such databases and / or memories is not intended to be limiting.

[0095] The ECU 10 may be configured to receive data from the image capture device 15 in order to provide functionality associated with the present disclosure. For example, the ECU 10 may simultaneously receive image data (eg, streaming data) having first viewpoint parameters from one or more image capture devices 15 and one or more sensors.

[0096] The ECU 10 may also be configured to receive data from the GPS 17, the data including location information and map information related to features in the vicinity of the vehicle's location. The location information may include, for example, global coordinates that enable the vehicle's location to be fixed / determined on a map, current and next link information indicating the link the vehicle is currently traveling on (i.e., the current link) and possible future travel paths (i.e., the next link), and information about these current and next links (e.g., controlled access to highways, urban areas, etc.), etc.

[0097] Features included in the map information may include, for example, lane markings 210, exit lanes, condition indicators (e.g., activation time, prohibition time, weather conditions, seasonality, etc.), terrain, etc. Those skilled in the art will recognize that more or fewer features may be present in the map information as desired, with the level of detail depending on, for example, the map information provider, etc. Those skilled in the art will also recognize that the GPS 17 may form part of the ECU 10, may be separate from the ECU 10, or any level of combination between the GPS 17 and the ECU 10 may be implemented without departing from the scope of the present disclosure.

[0098] The ECU 10 may be linked to one or more interfaces, such as a network interface, which may be configured to wirelessly and / or by wire receive data and information provided by the image capture device 15, GPS 17, sensors 20, etc. Further, while the GPS 17 is described as being present on the vehicle, those skilled in the art will appreciate that certain map data including features such as lane markings 210, exit lanes, etc. may be stored remotely and sent to the GPS 17 and / or ECU 10, such as via 4G, so that the latest information is available.

[0099] According to some embodiments, the vehicle may include one or more system controllers 32 that may be configured to receive information and / or commands from the ECU 10 and execute those commands to control various vehicle systems (e.g., steering, braking, accelerator, etc.) Such devices may be configured to actively manipulate the vehicle's control system 32, for example, to operate a steering system, a braking system, an acceleration system, etc.

[0100] Such devices may include one or more servomotors, actuators, etc., which may receive instructions from one or more systems of the vehicle, such as ECU 10. Based on these instructions, the vehicle may be controlled by the operator, by ECU 10 in combination with system controller 32, or both (e.g., system controller 32 providing steering and braking assistance in an emergency stop situation).

[0101] The display 25 may be configured to display information provided to the driver of the vehicle by the ECU 10. Figure 2 shows an exemplary display 25 that provides information that may be of interest to the driver of the vehicle. As shown at Figure 2, in addition to the information currently displayed to the driver, the effective speed limit is also on the display 25.

[0102] Display 25 can be any suitable device for providing visible and / or audible information to the driver of the vehicle. For example, display 25 can include a head-up display (e.g., on the windshield in front of the driver), a monitor, a display mounted on the dashboard, etc.

[0103] Figure 4 4 is a flowchart 400 highlighting an exemplary processing method according to an embodiment of the present disclosure. The ECU 10 may receive image data from one or more image capture devices 15 and an identifier associated with the image capture device 15 providing the image data (step 405). Such image data may include the surroundings of the vehicle, such as lane lines 210, road conditions 200, and other vehicles 250. The image data may be in video, still, or hybrid form. For example, one or more frames of a video may be provided in conjunction with periodic still photos, for example, where a particular traffic scene becomes complex.

[0104] Upon receiving the image data, the ECU 10 may check the image data to determine whether the image capture device 15 from which the data is received is properly calibrated (step 415). As described above, such a calibration check may be optional, and according to some embodiments, such a calibration check may not be performed.

[0105] Where such a calibration check is performed, and when the ECU 10 determines that a calibration problem may exist (step 415: NO), the operator of the vehicle may be notified (e.g., via the display 25) that a calibration problem exists in one or more of the image capture devices 15 (step 415). Based on the camera identifiers, it may be possible to indicate to the operator which of the image capture devices 15 has the calibration problem, and thus correction / repair of the calibration problem may be facilitated.

[0106] According to some embodiments, it is possible for the image capture device 15 and / or ECU 10 to provide additional information regarding diagnostics, such as calibration-related, via an interface (e.g., display 25) to help diagnose and correct calibration problems, such as at a manufacturer-certified repair facility.

[0107] After verification of the calibration (if performed), the ECU 10 may then use the camera identifier provided by the image capture device 15 providing the image data at step 405 to determine first viewpoint parameters (i.e., default output parameters) for the image data from the associated image capture device 15, and virtualized viewpoint parameters for converting the first viewpoint parameters to a virtualized set (step 415). For example, using Figure 2A and Figure 2B As an example, the camera has been identified as Figure 2A After providing the first viewpoint parameters (e.g., fisheye, 20 degree rotation, peripheral distortion), a processing application (e.g., lane tracking application) is identified that uses / expects a view with specific viewpoint parameters. Based on the correspondence stored in the database / lookup table, virtualization parameters for converting the raw output image of the identified camera to the desired viewpoint parameters of the processing application can be identified and provided to the ECU 10.

[0108] After identifying the transformation information associated with the identified camera 15 and the processing application, the image data (i.e., the raw output of the camera 15) may be transformed into a virtualized viewpoint (e.g., as in Figure 2B (as shown in step 425).

[0109] Those skilled in the art will appreciate that applying the transformation information to the camera output image data is well understood and does not require detailed explanation. However, for purposes of example, it may be noted that where the first viewpoint parameter includes a rotation parameter of 30 degrees, for example, the transformation information may indicate that a rotation of -30 degrees, or 390 degrees, for example, is applied to the output image data to generate a virtual viewpoint. An example of such a transformation may be described above with respect to Figure 2A and Figure 2B and the related descriptions for Tables 1-8.

[0110] According to some embodiments, the additional transformation information in the lookup table 80 may include transformation information for converting the projection model of a previously processed image to a different desired projection model by the processing application 90. For example, where an image having a projection model of a rectilinear cylinder is received, but the processing application 90 desires an orthogonal projection model (e.g., for a bird's eye lane detection application), in addition to the other transformations discussed above, the lookup table 80 may include transformation information configured to convert the projection model of the rectilinear cylinder to an orthogonal projection model. In other words, even where the image has not been directly provided by the camera (e.g., somewhere during software processing, the image has been converted and then subsequently provided in its converted format), the image may still be converted from its current projection model to the desired projection model.

[0111] Similarly, where an image with a pinhole, fisheye, etc. projection model directly from a camera is received, but the processing application 90 desires an orthographic projection, the lookup table 80 may provide the conversion information to make the desired conversion.

[0112] Such transformation information according to some examples may include a rotation vector configured to be applied to the rectilinear image information to produce a rectilinear cylindrical “normalized” image.

[0113] After the virtual viewpoint has been realized by applying the transformation information to the image data, the virtual viewpoint can be provided to a processing application for processing and final control operations (step 430). Figure 2BIn the case of the virtual viewpoint of the vehicle, the ECU 10 can evaluate the input from the sensor 20 and use, for example, the system controller 32 to cause the operation of various steering controls to maintain the vehicle within the identified lane line 210. Those skilled in the art will recognize that other controls such as, for example, the accelerator, brakes, etc. can also be operated by the system controller 32 as a result of the command from the ECU 10 to cause the modification of the vehicle's travel.

[0114] Based on the embodiments of the present disclosure, it becomes possible to implement a processing application across multiple vehicle designs, with various image capture devices installed at various locations, and providing various raw image data views of the vehicle's surroundings. This is at least because the first viewpoint parameters of the raw image data view can be converted from any vehicle, any installation location, and any type of camera into a virtual viewpoint with characteristic parameters desired by the processing application. Therefore, it becomes unnecessary to redevelop the processing application 90 for each vehicle design and camera arrangement.

[0115] Throughout the specification (including claims), unless otherwise stated, the term "comprising a" should be understood as being synonymous with "comprising at least one". In addition, unless otherwise stated, any ranges set forth in the specification (including claims) should be understood to include their end values. Specific values ​​for described elements should be understood to be within accepted manufacturing or industry tolerances known to those skilled in the art, and any use of the terms "substantially" and / or "approximately" and / or "generally" should be understood to mean falling within such accepted tolerances.

[0116] Where reference is made to any standard of a national, international, or other standards body (e.g., ISO, etc.), such reference is intended to refer to the standard as defined by the national or international standard as of the priority date of this specification. Any subsequent substantive changes to such standards are not intended to modify the scope and / or definition of the present disclosure and / or claims.

[0117] Although the present disclosure has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the present disclosure.

[0118] It is intended that the specification and examples be considered exemplary only, with the true scope of the disclosure being indicated by the following claims.

Claims

1. A system for generating a virtual image view for a vehicle, comprising: one or more image capture devices configured to capture image data about a vicinity of the vehicle and to provide identifiers identifying the one or more corresponding image capture devices, wherein the image data is defined at least in part by the first viewpoint parameter; a storage device configured to store a plurality of virtualization records containing transformation information related to virtualized viewpoints and a plurality of image capture devices and a plurality of processing applications, the processing applications including at least one of an object detection and tracking application, and a lane identifier application; and A processing device configured to: receiving captured image data; identifying a processing application in the plurality of processing applications with desired virtual viewpoint parameters based on the captured image data; selecting said processing application of desired virtual viewpoint parameters; converting the first viewpoint parameters of the captured image data to virtual viewpoint parameters of the selected processing application based on conversion information associated with a virtualization record stored by the storage device to generate the virtual image view, wherein the virtualization record is identified based on at least the identifier and the selected processing application; At least one driver assistance and / or automated driving function is performed based on the virtual image view.

2. The system for generating a virtual image view for a vehicle according to claim 1, wherein: The conversion information includes at least one of: distortion compensation information, image correction information, image refraction information, and rotation information.

3. The system for generating a virtual image view for a vehicle according to claim 1, wherein: Each virtualization viewpoint is defined by: At least one of resolution and field of view; and At least one of a pinhole model, a fisheye model, a cylindrical model, a spherical model, and a straight line model.

4. The system for generating a virtual image view for a vehicle according to claim 1, wherein: The storage means comprises means for identifying a processing application linked to a virtual image view.

5. The system for generating a virtual image view for a vehicle according to claim 4, wherein: The storage device includes a lookup table linking camera identifiers with transformation information for generating a virtual image view required by the processing application.

6. The system for generating a virtual image view for a vehicle according to claim 1, wherein: The conversion information includes calibration parameters for the identified image capture device, and wherein the processing device is configured to verify the calibration parameters for each of the one or more image capture devices.

7. A vehicle comprising the system according to claim 1.

8. A method for generating a virtual image view for a vehicle, comprising: receiving image data captured by at least one image capture device from a surrounding environment of the vehicle, the image data being defined at least in part by first viewpoint parameters; receiving identifiers identifying one or more image capture devices providing the image data; a processing application that identifies a desired viewpoint parameter among a plurality of processing applications including at least one of an object detection and tracking application, and a lane identifier application; selecting application of said processing to desired viewpoint parameters; determining a virtualization record in a virtualization record containing transformation information related to a virtualized viewpoint, a plurality of image capture devices, and the plurality of processing applications, wherein the virtualization record is determined based on at least the identifier and the selected processing application; converting the first viewpoint parameters of the captured image data to virtual viewpoint parameters of the selected processing application based on conversion information associated with the determined virtualization record and the selected processing application to generate the virtual image view; and At least one driver assistance and / or automated driving function is performed based on the virtual image view.

9. The method for generating a virtual image view for a vehicle according to claim 8, wherein: The conversion information includes at least one of: distortion compensation information, image correction information, image refraction information, and rotation information.

10. The method for generating a virtual image view for a vehicle according to claim 8, wherein: Each virtualization viewpoint is defined by: At least one of resolution and field of view; and At least one of a pinhole model, a fisheye model, a cylindrical model, a spherical model, and a straight line model.

11. The method for generating a virtual image view for a vehicle according to claim 8, wherein: The determining includes identifying a processing application linked to the virtual image view.

12. The method for generating a virtual image view for a vehicle according to claim 11, wherein: The determining includes searching a lookup table including information linking a camera identifier with transformation information for generating a virtual image view required by the processing application.

13. The method for generating a virtual image view for a vehicle according to claim 8, wherein: The conversion information includes calibration parameters for the identified image capture device, and the method further includes verifying the calibration parameters for each of the one or more image capture devices.

Images