Vehicle, control device, and method for assessing the calibration of one or more cameras mounted on the vehicle

Abstract

Various aspects relate to a vehicle, a method, and a control device, the control device including one or more processors (110) configured to receive a first longitudinal image showing the surroundings in front of or behind the vehicle at a first time point, receive a first lateral image showing the surroundings adjacent to the side of the vehicle at the first time point, receive a second lateral image showing the surroundings adjacent to the side of the vehicle at a second time point, use the first longitudinal image and the first lateral image to generate a first image showing a top view of the scene at the first time point, use the second lateral image to generate a second image showing the top view of the scene at the second time point, and determine whether the longitudinal cameras (102, 104) and the lateral cameras are correctly calibrated by comparing the first image with the second image.

Description

[Technical Field]

[0001] Various aspects of the present disclosure relate to vehicles, control devices for use in vehicles, and methods for assessing the calibration of one or more cameras mounted on a vehicle, including quantitatively assessing the quality of the camera calibration. [Background technology]

[0002] The following discussion of the background art is intended merely to facilitate understanding of the present disclosure. It should be understood that this discussion is not an acknowledgment or admission that any of the material referred to was published, known, or part of the common general knowledge of those skilled in the art in any jurisdiction as of the priority date of this disclosure.

[0003] A vehicle's on-board control system can provide various driving assistance features, such as surround view (e.g., top view), parking assistance, and lane departure warning systems, that use sensors, such as cameras, to detect the vehicle's surroundings. When a vehicle is moving (e.g., while driving), the vehicle body, and therefore the sensors (e.g., cameras), may change their orientation and / or position. Therefore, it may be necessary (e.g., to meet safety requirements) to recalibrate the sensors (e.g., cameras) online (i.e., while the vehicle is in use). For example, one or more sensor calibration parameters may be recalculated while driving (e.g., in real time) to compensate for the above-mentioned changes (e.g., using an online calibration algorithm). Here, it may be necessary and / or desirable to evaluate the sensor calibration (e.g., to determine whether the sensor is correctly calibrated or requires recalibration). Summary of the Invention

[0004] Various aspects relate to a vehicle, a control device for use in a vehicle, and a method for evaluating the calibration of each of one or more sensors (eg, a camera).

[0005] According to various aspects, a vehicle may include a longitudinal camera configured to capture longitudinal images of surroundings in front of the vehicle or behind the vehicle, and a lateral camera configured to capture lateral images of surroundings adjacent to the vehicle, the longitudinal camera and the lateral camera having overlapping fields of view such that a portion of the longitudinal image and a portion of the lateral image show the same area around the vehicle, and the vehicle receives from the longitudinal camera a first longitudinal image of surroundings in front of or behind the vehicle at a first time point, receives from the lateral camera a first lateral image of surroundings adjacent to the vehicle at the first time point, and receives from the lateral camera a second lateral image of surroundings adjacent to the vehicle at the first time point. The system may include one or more processors configured to receive a second lateral image showing the lateral surroundings adjacent to the vehicle at a second time point different from the first time point, a portion of the second lateral image showing the scene shown in the first longitudinal image and the first lateral image at the first time point, use the first longitudinal image and the first lateral image to generate a first image showing a top view of the scene at the first time point, use the second lateral image to generate a second image showing a top view of the scene at the second time point, and determine whether the longitudinal camera and the lateral camera are correctly calibrated by comparing the first image with the second image.

[0006] According to various aspects, a method is provided for evaluating calibration of one or more cameras mounted on a vehicle, the method including receiving, from a longitudinal camera, a first longitudinal image depicting a surrounding in front of or behind the vehicle at a first time point; receiving, from a lateral camera, a first lateral image depicting a surrounding adjacent to a side of the vehicle at the first time point; receiving, from the longitudinal camera, a second longitudinal image depicting a surrounding in front of or behind the vehicle at a second time point different from the first time point; and receiving, from the lateral camera, a second lateral image depicting a surrounding adjacent to a side of the vehicle at the second time point. The method may include receiving a first image showing a top view of the scene at a first time point, wherein a portion of the second horizontal image is not shown in the second vertical image and shows the scene shown in the first vertical image and the first horizontal image at a first time point; using the first vertical image and the first horizontal image to generate a first image showing a top view of the scene at the first time point; using the second horizontal image to generate a second image showing a top view of the scene at a second time point; and determining whether the portrait camera and the landscape camera are correctly calibrated by comparing the first image with the second image.

[0007] The present disclosure will be better understood with reference to the detailed description, when considered in conjunction with the non-limiting examples and the accompanying drawings. [Brief explanation of the drawings]

[0008] [Figure 1A] 1 illustrates a vehicle according to various aspects. [Figure 1B] 1 illustrates an exemplary top view of a vehicle and its surroundings in accordance with various aspects. [Figure 2A] 1 illustrates various aspects of collecting data for assessing the calibration of one or more cameras of a vehicle. [Figure 2B] 1 illustrates various aspects of collecting data for assessing the calibration of one or more cameras of a vehicle. [Figure 2C] 1 illustrates various aspects of collecting data for assessing the calibration of one or more cameras of a vehicle. [Figure 2D]1 illustrates various aspects of collecting data for assessing the calibration of one or more cameras of a vehicle. [Figure 3] 1 illustrates a flow diagram of a method for evaluating the calibration of at least one camera in accordance with various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following detailed description refers to the accompanying drawings, which show, by way of illustration, specific details and embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure. Other embodiments may be utilized, and structural and logical changes may be made without departing from the scope of the present disclosure. Various embodiments are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments.

[0010] An embodiment described in the context of a vehicle, one of the devices, or a method is equally valid for other vehicles, devices, systems, or methods. Similarly, an embodiment described in the context of a device is equally valid for a system (e.g., a vehicle) and / or method, and vice versa.

[0011] Features described in the context of one embodiment may be correspondingly applicable to identical or similar features in other embodiments. Features described in the context of one embodiment may be correspondingly applicable to other embodiments even if not explicitly described in those other embodiments. Furthermore, additions and / or combinations and / or alternatives described for certain features in the context of one embodiment may be correspondingly applicable to identical or similar features in other embodiments.

[0012] In the context of the various embodiments, the articles "a," "an," and "the" when used in reference to a feature or element include a reference to one or more of the feature or element.

[0013] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0014] Terms such as "first," "second," and the like may be used to describe various devices, such as cameras, but are not limited by the above terms. The above terms are used only to distinguish one device from another, and do not define the order and / or importance of the devices.

[0015] As used herein, the term "data" may be understood to include information in any suitable analog or digital format, for example, provided as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, etc. Furthermore, the term "data" may be used to mean a reference to information, for example, in the form of a pointer. However, the term "data" is not limited to the above examples and can take various forms and represent any information understood in the art. Any type of information, as described herein, may be processed, for example, via one or more processors, in any suitable manner, for example, as data.

[0016] For example, as used herein, the terms "processor" or "controller" may be understood as any type of entity that enables processing of data. The data may be processed according to one or more specific functions performed by the processor or controller. Furthermore, as used herein, a processor or controller may be understood as any type of circuit, such as any type of analog or digital circuit. Thus, a processor or controller may be or include an analog circuit, a digital circuit, a mixed-signal circuit, a logic circuit, a processor, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), an integrated circuit, an application-specific integrated circuit (ASIC), etc., or any combination thereof. Any other type of implementation of the respective functions described in more detail below may also be understood as a processor, controller, or logic circuit. It will be understood that any two (or more) processors, controllers, or logic circuits detailed herein may be implemented as a single entity having equivalent functionality, etc., and conversely, any single processor, controller, or logic circuit detailed herein may be implemented as two (or more) separate entities having equivalent functionality, etc.

[0017] The term "memory" as detailed herein may be understood to include any suitable type of memory or memory device, such as, for example, a hard disk drive (HDD), a solid state drive (SSD), flash memory, etc.

[0018] Various aspects relate to a controller for use in a vehicle. For example, the controller may be configured to control one or more functions (e.g., operation) of the vehicle. The controller may be capable of evaluating the calibration of one or more sensors of the vehicle. Thus, the controller may be part of a sensor control (e.g., monitoring) system and / or a driver assistance system and / or a safety system. Although various aspects refer to the use of a controller in the vehicle itself, it is understood that the processing described herein may be performed outside the vehicle (e.g., on a cloud computing device).

[0019] As an example, a vehicle may include four cameras (e.g., a front camera, a rear camera, a left side camera, and a right side camera) for capturing images of the surroundings around the vehicle, and a control device of the vehicle may be configured to determine whether the at least two cameras are correctly calibrated (or whether recalibration is required) based on the captured images of at least two of the four cameras. As described herein, the control device can evaluate the camera calibration without requiring any additional devices (besides the at least two (e.g., four) cameras).

[0020] As an illustrative example of how to evaluate the calibration of two cameras, a vehicle may include a forward camera for capturing forward images of the surroundings in front of the vehicle, a rearward camera for capturing images of the surroundings behind the vehicle, a leftward camera for capturing images of the surroundings to the left of the vehicle, and a rightward camera for capturing images of the surroundings to the right of the vehicle. The controller may continuously receive these images at a predetermined frame rate and, at various time instances, may be configured to generate respective top-view images showing a top view of the vehicle and its surroundings at a particular point in time (e.g., to assist with parking). By way of example, the forward and leftward cameras may have partially overlapping fields of view such that portions of their captured images show the same area around the vehicle (from different perspectives). Thus, the controller may generate portions of the top-view image by merging (e.g., stitching) information from both the forward and leftward images. When the vehicle is traveling forward on a substantially straight course, the vehicle may pass through a particular scene that is initially within this same area and later located to the left of the vehicle outside this same area. The control device can generate a first top-view image showing a scene within the merged portion and a second top-view image showing the same scene outside the merged portion. The control device can use the second top-view image as a ground truth image for the first top-view image to evaluate the calibration of the forward camera and the left side-view camera. For example, the control device can compare the first top-view image with the second top-view image and determine that the forward camera and the left side-view camera are correctly calibrated if the difference between them is less than a predetermined threshold.

[0021] FIG. 1A illustrates a vehicle 100 according to various embodiments. The vehicle 100 may include at least two sensors configured to detect the vehicle's 100 surroundings. The at least two sensors may have partially overlapping fields of view. The vehicle 100 may include one or more processors 110. For example, the vehicle 100 may include a control device, which may include one or more processors 110. The one or more processors 110 may be configured to receive respective sensor data from the at least two sensors at various time instances (e.g., continuously) and evaluate the calibration of the at least two sensors using the received sensor data. The one or more processors 110 may be configured to generate, at one or more (e.g., each) time instances, respective top-view images illustrating a top view of the vehicle's 100 surroundings (and optionally, a top view of the vehicle 100 itself). The one or more processors 110 may be configured to evaluate the calibration of the at least two sensors using the generated top-view images. Below, the at least two sensors are described as being respective cameras configured to capture images of the vehicle's 100 surroundings. However, while this serves to illustrate the principles of evaluating sensor calibration according to the present disclosure, it is understood that one or more of the at least two sensors may be other sensors (e.g., radio detection and ranging (radar) sensors, optical detection and ranging (lidar) sensors, etc.) so long as the constraints described herein are met (e.g., at least two sensors have overlapping fields of view).

[0022] 1A, vehicle 100 may include a forward camera 102 (which may also be referred to as a (first) longitudinal camera or forward-looking camera), a rearward camera 104 (which may also be referred to as a (second) longitudinal camera or rearward-looking camera), a left-side view camera 106 (which may also be referred to as a (first) lateral camera or left-side view camera), and a right-side view camera 108 (which may also be referred to as a (second) lateral camera or right-side view camera). While vehicle 100 is described as having four cameras, it is understood that the principles described herein can be used to evaluate the calibration of the cameras as long as there are at least two cameras (with overlapping fields of view).

[0023] The cameras described herein may be any type of (camera) device capable of capturing images of the surroundings of the vehicle 100. Accordingly, a camera may also be referred to as a visual sensor, an image sensor, or a camera device. It is understood that a camera may be configured to provide captured images to one or more processors 110 directly or indirectly (e.g., by storing them in a memory device). It is further understood that a camera may be configured to output unprocessed or preprocessed images. The cameras may be configured to capture respective images of the (respective) surroundings of the vehicle 100 at a predetermined frame rate. The cameras described herein may be wide-angle cameras (e.g., having an aperture angle of 120° or greater). The forward camera 102 may be configured to capture (e.g., at each instance depending on the frame rate of the camera) forward images showing the surroundings ahead of the vehicle 100 (e.g., ahead of the vehicle 100 relative to the forward direction 12). The rear camera 104 may be configured to capture (e.g., at each instance depending on the camera's frame rate) rearward images showing surroundings behind the vehicle 100 (e.g., in front of the vehicle 100 relative to the reverse direction 14 (opposite the forward direction)). The left side camera 106 may be configured to capture (e.g., at each instance depending on the camera's frame rate) leftward images showing surroundings to the left of the vehicle 100 (e.g., on the left side of the vehicle 100 relative to the forward direction 12). The right side camera 108 may be configured to capture (e.g., at each instance depending on the camera's frame rate) rightward images showing surroundings to the right of the vehicle 100 (e.g., on the left side of the vehicle 100 relative to the forward direction 12). The left side camera 106 may have a first partially overlapping field of view with the forward camera 102 and a second partially overlapping field of view (different from the first partially overlapping field of view) with the rear camera 104.Thus, a first portion of the field of view (fov) of the left side camera 106 may overlap with a portion of the fov of the front camera 102, a second portion of the fov of the left side camera 106 may overlap with a portion of the fov of the rear camera 104, and a third portion of the fov of the left side camera 106 may not overlap with either the front or rear camera fov. This is similarly true for the right side camera 108, which has a partially overlapping fov with the front camera 102 and a partially overlapping fov with the rear camera 104. Partially overlapping fields of view as described herein are understood to mean fovs that do not completely overlap.

[0024] It is understood that the nominal positions of each of the forward camera 102, rearward camera 104, left side view camera 106, and right side view camera 108 may be stored in a memory device (e.g., as part of computer-aided design (CAD) data for the vehicle 100), and the one or more processors 110 use the information on the memory device during processing. Additionally, the memory device may store various (e.g., specific) parameters of the cameras. The left side view camera 106 may be mounted on the left side mirror of the vehicle 100. The right side view camera 108 may be mounted on the right side mirror of the vehicle 100.

[0025] Vehicle 100 may be any type of vehicle that detects its surroundings using sensors with overlapping fields of view. The sensors (e.g., cameras) may be part of a surround view system (SVS) configured to provide several views of vehicle 100 (e.g., output via a display device and / or for further processing), such as a top view, a rear view, a panoramic view, etc.

[0026] As one example, vehicle 100 may be a non-autonomous vehicle that includes a display device for showing an overhead view image of the vehicle and its surroundings to the driver of the vehicle to assist in parking (e.g., to indicate and / or warn of obstacles around vehicle 100). As another example, vehicle 100 may be a semi-autonomous or autonomous vehicle with automatic parking capabilities (i.e., parking the vehicle without the need for steering and / or acceleration / deceleration by the driver) that use sensors (e.g., using an overhead view image) to detect the vehicle's surroundings. Automatic parking capabilities may include parking assistance when the driver is inside vehicle 100, remote parking that does not require the driver to be inside vehicle 100, and / or valet parking.

[0027] A "vehicle" may be a ground vehicle (e.g., a vehicle configured to travel on land (e.g., a road, railroad, roadway, etc.)), an air vehicle (e.g., a vehicle configured to move above the ground), or a water vehicle (e.g., a vehicle capable of moving on or below the surface of a liquid (e.g., water)).

[0028] It is understood that vehicle 100 can include various additional components (eg, engine, steering system, etc.) typical of each type of vehicle.

[0029] 1B illustrates an exemplary top view of a vehicle 100 and its surroundings according to various embodiments. As described herein, each of the front camera 102, rear camera 104, left side camera 106, and right side camera 108 may be configured to capture respective images at a (respective) predetermined frame rate. The frame rates may be synchronized with one another. One or more processors 110 receive the front image from the front camera 102, the rear image from the rear camera 104, the left side image from the left side camera, and the right side image from the right side camera 108, all relating to substantially the same time point t, and use the front image, rear image, left side image, and right side image to generate an upper perspective image O for this time point t. tFor example, FIG. 1B shows an exemplary top view image O for a particular time t. t The top view image O t represents the surroundings in front of the vehicle 100 and may have a front portion 112 that includes information captured by the front camera 102 and does not include information from the rear camera 104, left side camera 106, and right side camera 108. t represents the surroundings behind the vehicle 100 and may have a rear portion 114 that includes information captured by the rear camera 104 and does not include information from the front camera 102, left side camera 106, and right side camera 108. t represents the surroundings to the left of the vehicle 100 and may have a left side portion 116 that includes information captured by the left side camera 106 and does not include information from the front camera 102, the rear camera 104, and the right side camera 108. t may have a right side portion 118 that represents the surroundings to the right of the vehicle 100 and includes information captured by the right side camera 108, but does not include information from the front camera 102, the rear camera 104, and the left side camera 106. As described herein, each camera pair may have overlapping foVs. Thus, the one or more processors 110 may generate a top view image O t The camera pair may be configured to use information from both cameras when generating the top view image O. t may have a left front portion 120 containing information captured by the forward camera 102 and the left side view camera 106, a right front portion 122 containing information captured by the forward camera 102 and the right side view camera 108, a left rear portion 124 containing information captured by the rear camera 104 and the left side view camera 106, and a right rear portion 126 containing information captured by the rear camera 104 and the right side view camera 108.

[0030] Various aspects of evaluating the calibration of each of the front camera 102, rear camera 104, left side camera 106, and / or right side camera 108 are described below with reference to FIGS. 2A-2D.

[0031] 2A, vehicle 100 may travel ahead 12 (e.g., in a substantially straight course) through region 200. At a first time point (in some embodiments, a time point is referred to as a time instance) t=i, region 200 may be located to the left front of vehicle 100 and thus shown in a forward image captured by forward camera 102 and a left side image captured by left side camera 106 at first time point t=i. One or more processors 110 may generate a corresponding top view image O t=i At a second time t=i+F0, when traveling ahead 12, region 200 may be located to the left of vehicle 100 and therefore only shown in the left side view image captured by left side camera 106 at second time t=i+F0. One or more processors 110 may generate a corresponding top view image O t=i+F0 At a third time point t=i+F0+R0, region 200 may be located to the left rear of vehicle 100 and thus shown in the rear image captured by rear camera 104 and the left side image captured by left side camera 106 at the third time point t=i+F0+R0. One or more processors 110 may generate a corresponding top view image O t=i+F0+R0 Thus, the same area around the vehicle 100 can be captured in various top-view images O t It will be appreciated that this applies equally to areas located to the right of the vehicle 100. While this example describes the vehicle 100 traveling forward 12, the principles described herein apply equally to traveling backward 14. It will be appreciated that when traveling backward, the second time point t=i+F0 is earlier than the first time point t=i (i.e., F0<0), and the third time point t=i+F0+R0 is earlier than the second time point t=i+F0 (i.e., R0<0).

[0032] As mentioned above, at a first time point t=i, the region 200 may be located in the left front portion 120 (whether traveling forward or backward). Referring to FIG. 2B, the one or more processors 110 may generate a left front image FL(O) showing a first scene (e.g., within the region 200) at the first time point t=i. t=i ) (also called left front patch), and / or a right front image FR(O t=i The one or more processors 110 may be configured to generate (e.g., extract) a left image L(O) (also referred to as a right front patch) showing the first scene at a second time point t=i+F0. t=i+F0 ) (also called left ground truth patch), and / or a right image R(O t=i+F0 The one or more processors 110 may be configured to generate (e.g., extract) a left rear image RL(O) (also referred to as a right ground truth patch) that shows the first scene at a third time point t=i+F0+R0. t=i+F0+R0 ) (also called left rear patch), and / or a right rear image RR(O t=i+F0+R0 ) (also called the right rear patch).

[0033] According to various embodiments, the left image L(O t=i+F0 ) may be symmetrical with respect to the optical axis of the left side camera 106 and / or the right image R(O t=i+F0 ) may be symmetrical with respect to the optical axis of the right side camera 108.

[0034] According to various embodiments, the left image L(O t=i+F0 ) is generated only from the left side image (it is not stitched with the front or rear image), so the left image L(O t=i+F0) can be a left ground truth image that represents the ground truth of how the first scene should look. If the front camera 102 and the left side camera 106 are properly calibrated, the left front image FL(O t=i ) (also representing the first scene) is the left ground truth image, i.e., the left image L(O t=i+F0 ) is expected (e.g., for uncorrupted real-world data). Similarly, if the rear camera 104 and left side camera 106 are properly calibrated, then the left rear image FL(O t=i ) is the left ground truth image, i.e., the left image L(O t=i+F0 ) is expected to correspond substantially to the right-hand image FR(O t=i ) and right rear image RR(O t=i+F0+R0 ) is the right ground truth image, i.e., the right image R(O t=i+F0 ) is expected to correspond to

[0035] The one or more processors 100 may generate a left front image FL(O) to evaluate the calibration of the front camera 102, rear camera 104, left side camera 106, and / or right side camera 108. t=i ) and / or left rear image RL(O t=i ) to the (ground truth) left image L(O t=i+F0 ) and / or the right front image FR(O t=i ) and / or right rear image RR(O t=i ) to the (ground truth) right image R(O t=i+F0) may be configured to compare the images. According to some embodiments, the two images compared to each other may have the same size. According to other embodiments, the two images compared to each other may have different sizes, and the one or more processors 110 may be configured to pre-process the images to have the same size prior to comparison. If the vehicle 100 is not traveling in a straight course, the scene (e.g., the first scene or the second scene) shown in each image on the (left or right) side may vary slightly (e.g., rotate and / or shift). The one or more processors 110 may be configured to rotate and / or shift at least one of the two images (e.g., using rigid registration) prior to comparison. Optionally, the vehicle 100 may include an input device that allows a user (e.g., the driver) to (alternatively or additionally) manually shift and / or rotate the images. This allows the user (e.g., the driver) to manually verify the results of the evaluation.

[0036] The one or more processors 110 may be configured to compare the two images with each other using any suitable metric. For example, the one or more processors 110 may be configured to determine a comparison value that represents the difference between the two images. Similarity is understood to also represent dissimilarity. By way of example, the comparison value may be or represent a sum of absolute differences (SAD), a mean squared error (MSE), and / or a structural similarity index (SSIM).

[0037] According to various aspects, vehicle 100 may include or be connected to a display device. One or more processors 110 may be configured to determine a difference image representing the difference between two images compared to one another. One or more processors 110 may be configured to provide control instructions to a display device to display the difference image (e.g., to a user, such as a driver of vehicle 100), thereby providing a visual comparison to a user (e.g., the driver). Optionally, the difference image may include pseudocoloring to indicate deviations from the ground truth image.

[0038] The one or more processors 110 may be configured to determine that the two images match one another if the determined comparison value indicates that the difference between the first image and the second image is less than or equal to a predetermined difference threshold. The one or more processors 110 may be configured to determine whether the camera is properly calibrated (or miscalibrated) based on the one or more comparisons. Various exemplary cases are described below to illustrate how to evaluate the calibration of a camera. Left front image FL(O t=i ) is the left image L(O t=i+F0 ), the one or more processors 110 may determine that the front camera 102 and the left side camera 106 are correctly calibrated. Left rear image RL(O t=i+F0+R0 ) is the left image L(O t=i+F0 ), the one or more processors 110 may determine that the rear camera 104 and the left side camera 106 are correctly calibrated. Right front image FR(O t=i ) is the right image R(O t=i+F0 ), the one or more processors 110 may determine that the front camera 102 and the right side camera 108 are correctly calibrated. Right rear image RR(O t=i+F0+R0 ) is the right image R(O t=i+F0 ), the one or more processors 110 may determine that the rear camera 104 and the right side camera 108 are correctly calibrated. Left front image FL(O t=i ) is the left image L(O t=i+F0 ) and the left rear image RL(O t=i+F0+R0 ) is the left image L(O t=i+F0 ), the one or more processors 110 may determine that the front camera 102 and the left side camera 106 are correctly calibrated, but that the rear camera 104 is not correctly calibrated. Left front image FL(O t=i ) is the left image L(O t=i+F0) does not match the left rear image RL(O t=i+F0+R0 ) is the left image L(O t=i+F0 ), the one or more processors 110 may determine that the rear camera 104 and the left side camera 106 are correctly calibrated, and that the front camera 102 is not correctly calibrated. Right front image FR(O t=i ) is the right image R(O t=i+F0 ) and the right rear image RR(O t=i+F0+R0 ) is the right image R(O t=i+F0 ), the one or more processors 110 may determine that the front camera 102 and the right side camera 106 are correctly calibrated, but that the rear camera 104 is not correctly calibrated. Right front image FR(O t=i ) is the right image R(O t=i+F0 ) does not match, and the right rear image RR(O t=i+F0+R0 ) is the right image R(O t=i+F0 ), the one or more processors 110 may determine that the rear camera 104 and the right side camera 106 are correctly calibrated, and that the front camera 102 is not correctly calibrated. Right front image FR(O t=i ) is the right image R(O t=i+F0 ) and the right rear image RR(O t=i+F0+R0 ) is the right image R(O t=i+F0 ) and the left front image FL(O t=i ) and left rear image RL(O t=i+F0+R0 ) is the left image L(O t=i+F0 ), the one or more processors 110 may determine that the front camera 102, the rear camera 104, and the right side camera 108 are correctly calibrated, but that the left side camera 106 is not correctly calibrated. Left front image FL(O t=i ) is the left image L(O t=i+F0 ) and the left rear image RL(O t=i+F0+R0 ) is the left image L(O t=i+F0 ) and the right front image FR(O t=i ) and right rear image RR(O t=i+F0+R0 ) is the right image R(O t=i+F0), the one or more processors 110 may determine that the front camera 102, the rear camera 104, and the left side view camera 106 are correctly calibrated, but that the right side view camera 108 is not correctly calibrated.

[0039] It is understood that the above evaluations are examples and that there are further combinations of logic that allow for determining whether a camera is properly calibrated (or not properly calibrated).

[0040] For illustrative purposes, FIG. 2C shows a left-front image FL(O) having a horizontal size Px and a vertical size Py. t=i ) In various embodiments, the vertical size Py may correspond to the number of pixels shifted between two consecutive frames captured by the left side camera 106. In each frame, the respective top view image O t Therefore, in this case, the left front image FL(O t=i ) (e.g., the first scene or the second scene) is projected onto the left front image FL(O t=i ) and the subsequent top-view image O t+1 The same applies to the right-front image FR(O t=i ) is also true. This is exemplarily shown in FIG. 2D. As shown, F0 is the time when the scene and the further scene are reflected in the left image L(O t=i+F0 ) and right image R(O t=i+F0 Similarly, R0 may be the minimum number of top view images O that are generated before the scene and further scenes are shown in the rear region (left rear image RL(O)). t=i+F0+R0 ) and / or right rear image RR(O t=i+F0+R0 )))))))).

[0041] According to various embodiments, the images described herein (e.g., left front image, right front image, left rear image, and / or right rear image) may be generated using two or more top perspective images. For example, a left front image FL(O t=i~i+F0) is FL(O t=i )~FL(O t=i+F0 ) all left front images FL(O t ) (i.e., the left front image FL(O t=i ), FL(O t=i+1 ), FL(O t=i+2 ), ···, FL(O t=i+F0 The right front image FR(O t=i~i+F0 ) is FR(O t=i )~FR(O t=i+F0 ) all right front images FR(O t ) (i.e., right front image FR(O t=i ), FR(O t=i+1 ), FR(O t=i+2 ), ···, FR(O t=i+F0 )) Similarly, a left rear image may be generated by concatenating two or more left rear images, and / or a right rear image may be generated by concatenating two or more right rear images.

[0042] Correspondingly, the left ground truth image L(O t=i+F0 ) is L(O t=i )~L(O t=i+F0+F0 ) all left images L(O t=i to ) (i.e., the left image L(O t=i+F0 ), L(O t=i+F0+1 ), L(O t=i+Fo+2 ), ···, L(O t=i+F0+F0 )) so that the generated left ground truth image can show the same scene as the concatenated left front image.

[0043] According to various aspects, the left ground truth image can be compared to the concatenated left front image. Similarly, the right ground truth image R(O t=i+F0 ) is R(O t=i )~R(O t=i+F0+F0 ) all right images R(O t~ ) (i.e., the right image R(O t=i+F0 ), R(O t=i+F0+1 ), R(O t=i+Fo+2), ···, R(O t=i+F0+F0 ))

[0044] As described herein, an image (e.g., a left front image FL(O t=i ), right-front image, left-rear image, and / or right-rear image) can have a horizontal size Px and a vertical size Py. Py may be the vertical displacement of the vehicle, which corresponds to the number of pixels moved in successive frames. According to various aspects, the vertical size Py may correspond to the number of pixels moved in two successive frames captured by the respective cameras. When concatenating two or more images, the (left and / or right) ground truth images (optionally concatenated left-front image, concatenated right-front image, concatenated left-rear image, and / or concatenated right-rear image) may be generated by buffering two or more top-view images. The ground truth images may have a horizontal size X equal to Px, for example. g The ground truth image may have, for example, a vertical size Y g The vertical size Y of the ground-through image may be g may correspond to the product of the vertical size Py of each image and the number of images to be concatenated. For example, if F0 images are concatenated, Yg may correspond to Py*F0. According to some embodiments, R0 may be equal to F0. In this case, the following conditional relationship may exist: Px==X g )&&(F0*Py==Y g )&&(R0*Py==Y g ).

[0045] Various aspects relate to the evaluation of the calibration. The calibration may be an online calibration (e.g., to accommodate online modifications as described herein). As described herein, ground truth images can be generated using only data generated from a camera. This allows, for example, ground truth data to be generated with low equipment and computational costs. Other approaches, such as using reprojection errors, are not suitable for evaluating online calibration because using reprojection errors requires geometric knowledge of the scene (which may not be known and typically cannot be obtained online).

[0046] 3 illustrates a flow diagram of a method 300 for evaluating the calibration of at least one camera according to various embodiments. The method 300 may include receiving (302) a first longitudinal image from a longitudinal camera (e.g., the forward camera 102 or the rearward camera 104) illustrating an environment in front of or behind the vehicle at a first time point (e.g., at t=i or t=i+F0+R0). The method 300 may include receiving (304) a first lateral image from a lateral camera (e.g., the left side camera 106 or the right side camera 108) illustrating an environment adjacent to the vehicle at the first time point. The method 300 may include receiving (306) a second longitudinal image from the longitudinal camera illustrating an environment in front of or behind the vehicle at a second time point (e.g., at t=i+F0) different from the first time point. The method 300 may include receiving (308) a second lateral image from the lateral camera showing the lateral surroundings adjacent to the vehicle at a second time point. For example, a portion of the second lateral image shows a scene that is not shown in the second longitudinal image and that is shown in the first longitudinal image and the first lateral image at the first time point. The method 300 may include receiving (308) a first image (e.g., a left front image FL(O)) showing a top view of the scene at the first time point. t=i ), rear left image RL(O t=i+F0+R0 ), right front image FR(O t=i ), or right rear image RR(O t=i+F0+R0The method 300 may include generating (310) a second image (e.g., a left image L(O)) showing a top view of the scene at a second time point using the first portrait image and the first landscape image. t=i+F0 ) or right image R(O t=i+F0 )) using the second lateral image (312).

[0047] The method 300 may include determining whether the portrait and landscape cameras are properly calibrated by comparing the first image to the second image (314). Capturing the images, generating the top view image, generating the image from the top view image, comparing the two images to each other, etc. may be performed as described herein with reference to FIGS. 1A-2D.

[0048] Various aspects of the present disclosure are illustrated below. It should be noted that aspects described with respect to a control device or a vehicle may be correspondingly implemented in a method, and vice versa.

[0049] Example 1 is a vehicle including a longitudinal camera configured to capture a longitudinal image showing a surrounding area in front of the vehicle or behind the vehicle, and a lateral camera configured to capture a lateral image showing a surrounding area adjacent to the vehicle, wherein the longitudinal camera and the lateral camera have partially overlapping fields of view such that a portion of the longitudinal image and a portion of the lateral image show the same area around the vehicle, and the vehicle receives from the longitudinal camera a first longitudinal image showing a surrounding area in front of or behind the vehicle at a first time point, and receives from the lateral camera a first lateral image showing a surrounding area adjacent to the vehicle at the first time point, and The system includes one or more processors configured to receive a second lateral image showing the surroundings adjacent to the vehicle at a second time point different from the first time point, a portion of the second lateral image showing the scene shown in the first longitudinal image and the first lateral image at the first time point, use the first longitudinal image and the first lateral image to generate a first image showing a top view of the scene at the first time point, use the second lateral image to generate a second image showing the top view of the scene at the second time point, and determine whether the longitudinal camera and the lateral camera are correctly calibrated by comparing the first image with the second image.

[0050] This allows for the generation of test images (as test data) and ground truth images (as ground truth data) during a drive (i.e., online) that allow for an assessment of the calibration of the sensor, for example, to determine whether the sensor is correctly calibrated or whether it should be recalibrated.

[0051] In Example 2, the subject matter of Example 1 may optionally include the one or more processors being configured to: determine a comparison value representing a difference between the first image and the second image by comparing the first image with the second image; determine whether the comparison value represents a difference between the first image and the second image that is less than or equal to a predetermined difference threshold; and determine that the portrait camera and the landscape camera are correctly calibrated if it is determined that the comparison value represents a difference between the first image and the second image that is less than or equal to the predetermined difference threshold.

[0052] This example details how a first image and a second image can be compared to each other.

[0053] In Example 3, the subject matter of Example 1 or 2 may optionally include one or more processors configured to use the first longitudinal image and the first lateral image to generate a first upper perspective image showing a top view of the surroundings adjacent to the vehicle at least at a first time point and the surroundings in front of or behind the vehicle, and to generate the first image by extracting a portion of the first upper perspective image, and / or to use the second lateral image to generate a second upper perspective image showing a top view of the surroundings adjacent to the vehicle at least at the first time point and to generate the second image by extracting a portion of the second upper perspective image.

[0054] This allows for (eg, continuously) generating inspection images (as inspection data) and ground truth images (as ground truth data) from the top viewpoint images.

[0055] In Example 4, the subject matter of any one of Examples 1 to 3 can optionally include: the first image and the second image having the same image size.

[0056] This facilitates comparison between the first and second images.

[0057] In Example 5, the subject matter of any one of Examples 1 to 4 may optionally include: the vertical camera is configured to capture successive vertical images at a predetermined frame rate; and the first image and / or the second image have a vertical image size in a vertical direction perpendicular to the side substantially equal to the number of pixels shifted in two successive vertical images.

[0058] This allows inspection data images showing each scene to be generated consecutively without any overlap between two consecutive images.

[0059] In Example 6, the subject matter of any one of Examples 1 to 5 may optionally include: the one or more processors configured to receive, from the longitudinal camera, a second longitudinal image showing an environment in front of or behind the vehicle at a second time, wherein the second longitudinal image does not show a scene.

[0060] In Example 7, the subject matter of any one of Examples 1 to 6 is a vehicle-mounted ... the second time point is between the first and second time points in time; receiving from the lateral camera a third lateral image showing the surroundings adjacent to the vehicle at a third time point, the scene being shown in the first rear image and the third lateral image at the third time point; using the rear image and the third lateral image to generate a third image showing a top view of the scene at the third time point; and determining whether the rear camera and the lateral camera are correctly calibrated by comparing the third image with the second image.

[0061] This allows for the generation of further test images (as test data) for the same ground truth image as the previous test image. Thus, each ground truth image may be the ground truth for two different test images (one for the front region of the vehicle and the other for the rear region of the vehicle), thus increasing the efficiency of data generation. Furthermore, this allows for the determination of whether the camera generating the ground truth images is miscalibrated.

[0062] In Example 8, the subject matter of Example 7 can optionally include the one or more processors being configured to determine a further comparison value representing a difference between the third image and the second image by comparing the third image with the second image, determine whether the further comparison value represents a difference between the third image and the second image that is less than or equal to a further predetermined difference threshold, and determine that the rear camera and the lateral camera are correctly calibrated if it is determined that the further comparison value represents a difference between the third image and the second image that is less than or equal to the further predetermined difference threshold.

[0063] This example details how the third image and the second image can be compared to each other.

[0064] In Example 9, the subject matter of Examples 2 and 8 can optionally include the one or more processors being configured to determine that the front camera is not properly calibrated if the comparison value is determined to represent a difference between the first image and the second image that is greater than a predetermined difference threshold and the further comparison value is determined to represent a difference between the third image and the second image that is less than or equal to a further predetermined difference threshold, and / or determine that the rear camera is not properly calibrated if the comparison value is determined to represent a difference between the first image and the second image that is less than or equal to a predetermined difference threshold and the further comparison value is determined to represent a difference between the third image and the second image that is greater than a further predetermined difference threshold.

[0065] This details an example of how to determine which sensors are properly calibrated and which are not.

[0066] In Example 10, the subject matter of any one of Examples 1 to 9 in combination with Example 7 is the left side camera configured to capture a left side image as a lateral image showing surroundings to the left of the vehicle, the vehicle further includes a right side camera configured to capture a right side image showing surroundings to the right of the vehicle, the right side camera and the front camera have overlapping fields of view such that a further portion of the front image and a first portion of the right side image show the same area around the vehicle, the right side camera and the rear camera have overlapping fields of view such that a further portion of the rear image and a second portion of the right side image show the same area around the vehicle, and the one or more processors capture from the right side camera a first right side image showing surroundings to the right of the vehicle at a first time point, a second right side image showing surroundings to the right of the vehicle at a second time point, and a third right side image showing surroundings to the right of the vehicle at a third time point. and a third right lateral image showing a surrounding area around the vehicle, a portion of the second right lateral image showing a further scene as shown in the first portrait image and the first right lateral image at a first time point and as shown in the first rearward image and the third right lateral image at a third time point; using the first portrait image and the first right lateral image to generate a fourth image showing a top view of the further scene at the first time point; using the second right lateral image to generate a fifth image showing a top view of the further scene at the second time point; using the first rearward image and the third right lateral image to generate a sixth image showing a top view of the further scene at the third time point; comparing the fourth image with the fifth image; and comparing the sixth image with the fifth image to determine whether the forward camera, the rearward camera, and / or the right lateral camera are correctly calibrated.

[0067] This allows test and ground truth images to be consistently generated for both the left and right sides of the vehicle, thus further improving the efficiency of test and ground truth data generation.

[0068] In Example 11, the subject matter of Examples 3 and 10 can optionally include the one or more processors being configured to generate a first top perspective image using the first longitudinal image, the first rear image, the first lateral image, and the first right side image, such that the first top perspective image shows a top view of the surroundings all around the vehicle.

[0069] This further increases the efficiency of data generation, as the test images (left and right) can be generated from the same top-view image, and the ground truth images (left and right) can also be generated from the same top-view image.

[0070] In Example 12, the subject matter of Example 10 or 11 in combination with Example 8 or 9 further comprises the one or more processors: determining a first comparison value representing a difference between the fourth image and the fifth image by comparing the fourth image with the fifth image; determining whether the first comparison value represents a difference between the fourth image and the fifth image that is less than or equal to a first predetermined difference threshold; determining a second comparison value representing a difference between the sixth image and the fifth image by comparing the sixth image with the fifth image; determining whether the second comparison value represents a difference between the sixth image and the fifth image that is less than or equal to a second predetermined difference threshold; The method may optionally include determining that the front camera is not correctly calibrated when the first comparison value represents a difference between the fourth image and the fifth image and the further comparison value represents a difference between the third image and the second image that is less than a further predetermined difference threshold, and determining that the right side view camera is not correctly calibrated when the first comparison value represents a difference between the fourth image and the fifth image that is greater than a first predetermined difference threshold and the second comparison value represents a difference between the sixth image and the fifth image that is greater than a second predetermined difference threshold (optionally further determining that the rear camera, the front camera, and the left side view camera are correctly calibrated).

[0071] This example details how to determine if the right side camera that generates the ground truth image is miscalibrated.

[0072] In Example 13, the subject matter of any one of Examples 1 to 12 can optionally include, wherein the one or more processors are configured to generate a difference image representing a difference between the two images when comparing the two images to each other, and to provide control instructions to control the display device to display the difference image.

[0073] This allows the driver (or passenger) to be informed about the sensor calibration.

[0074] In Example 14, the subject matter of any one of Examples 1 to 13 can optionally include, wherein the one or more processors are configured to rotate and / or shift at least one of the two images (e.g., by rigid registration) before comparing the two images to one another.

[0075] This allows for use cases where the vehicle may be steered left or right rather than moving in a straight line.

[0076] In Example 15, the subject matter of any one of Examples 1 to 14 may optionally include, when it is determined that a camera (e.g., a front camera, a rear camera, a left side camera, and / or a right side camera) is not correctly calibrated, the one or more processors are configured to provide control instructions that control recalibration of the camera.

[0077] Example 16 is a method for evaluating calibration of one or more cameras mounted on a vehicle, the method including receiving, from a longitudinal camera, a first longitudinal image depicting a surrounding in front of the vehicle or a surrounding behind the vehicle at a first time point; receiving, from a lateral camera, a first lateral image depicting a surrounding adjacent to a side of the vehicle at the first time point; receiving, from the longitudinal camera, a second longitudinal image depicting a surrounding in front of or a surrounding behind the vehicle at a second time point different from the first time point; and receiving, from the lateral camera, a second lateral image depicting a surrounding adjacent to a side of the vehicle at the second time point. receiving a portion of the second horizontal image that is not shown in the second vertical image and that shows the scene shown in the first vertical image and the first horizontal image at a first time point; using the first vertical image and the first horizontal image to generate a first image that shows a top view of the scene at the first time point; using the second horizontal image to generate a second image that shows a top view of the scene at a second time point; and determining whether the portrait camera and the landscape camera are correctly calibrated by comparing the first image with the second image.

[0078] In Example 17, the method of Example 16 may optionally further include: determining a comparison value representing a difference between the first image and the second image by comparing the first image with the second image; determining whether the comparison value represents a difference between the first image and the second image that is less than or equal to a predetermined difference threshold; and determining that the portrait camera and the landscape camera are correctly calibrated if it is determined that the comparison value represents a difference between the first image and the second image that is less than or equal to the predetermined difference threshold.

[0079] In Example 18, the method of Example 16 or 17 may optionally further include using the first longitudinal image and the first lateral image to generate a first upper viewpoint image showing a top view of the lateral surroundings adjacent to the vehicle and the surroundings in front of or behind the vehicle at least at a first time point, and generating the first image by extracting a portion of the first upper viewpoint image, and / or using the second lateral image to generate a second upper viewpoint image showing a top view of the lateral surroundings adjacent to the vehicle at least at the first time point, and generating the second image by extracting a portion of the second upper viewpoint image.

[0080] In Example 19, the subject matter of any one of Examples 16 to 18 can optionally include the first image and the second image having the same image size.

[0081] In Example 20, the subject matter of any one of Examples 16 to 19 can optionally include the vertical camera being configured to capture successive vertical images at a predetermined frame rate, and the first image and / or the second image having a vertical image size in the vertical direction perpendicular to the side substantially equal to the number of pixels shifted in two successive vertical images.

[0082] In Example 21, the subject matter of any one of Examples 16 to 20 can optionally include the following: the longitudinal camera is a forward camera; and the method further includes receiving, from the rear camera, a first rear image showing surroundings rearward of the vehicle at a third time point, where the second time point is temporally between the first time point and the second time point; receiving, from the lateral camera, a third lateral image showing surroundings adjacent to a side of the vehicle at the third time point, where the scene is shown in the first rear image and the third lateral image at the third time point; using the rear image and the third lateral image to generate a third image showing a top view of the scene at the third time point; and determining whether the rear camera and the lateral camera are correctly calibrated by comparing the third image with the second image.

[0083] In Example 22, the method of Example 21 can optionally further include receiving a second rear image from the rear camera showing the surroundings behind the vehicle at a second time, where the second rear image does not show the scene.

[0084] In Example 23, the method of Examples 21 or 22 may optionally further include: determining a further comparison value representing a difference between the third image and the second image by comparing the third image with the second image; determining whether the further comparison value represents a difference between the third image and the second image that is less than or equal to a further predetermined difference threshold; and determining that the rear camera and the lateral camera are correctly calibrated if it is determined that the further comparison value represents a difference between the third image and the second image that is less than or equal to a further predetermined difference threshold.

[0085] In Example 24, the methods of Examples 17 and 23 may optionally further include determining that the front camera is not calibrated correctly if the comparison value is determined to represent a difference between the first image and the second image that is greater than a predetermined difference threshold and the further comparison value is determined to represent a difference between the third image and the second image that is less than or equal to a further predetermined difference threshold, and / or determining that the rear camera is not calibrated correctly if the comparison value is determined to represent a difference between the first image and the second image that is less than or equal to a predetermined difference threshold and the further comparison value is determined to represent a difference between the third image and the second image that is greater than a further predetermined difference threshold.

[0086] In Example 25, the subject matter of any one of Examples 16 to 24 in combination with Example 21 is that the lateral camera is a left side camera for capturing a left side image showing surroundings to the left of the vehicle, and the method includes receiving, from the right side camera, a first right side image showing surroundings to the right of the vehicle at a first time point, a second right side image showing surroundings to the right of the vehicle at a second time point, and a third right side image showing surroundings to the right of the vehicle at a third time point, wherein a portion of the second right side image is shown in the first longitudinal image, the first right side image, the first rearward image, and the third right side image, but is not shown in the second longitudinal image (and, in combination with Example 22, is not shown in the second rearward image). receiving a video signal showing a scene comprising: a first longitudinal image and a first right lateral image; generating a fourth image showing a top view of the further scene at a first time point using the first longitudinal image and the first right lateral image; generating a fifth image showing a top view of the further scene at a second time point using the second right lateral image; generating a sixth image showing a top view of the further scene at a third time point using the first rearward image and the third right lateral image; and determining whether the front camera, the rearward camera, and / or the right lateral camera are correctly calibrated by comparing the fourth image to the fifth image and comparing the sixth image to the fifth image.

[0087] In Example 26, the methods of Examples 18 and 25 can optionally further include generating a first top view image using the first longitudinal image, the first rear image, the first lateral image, and the first right side image, such that the first top view image shows a top view of the surroundings around the entire perimeter of the vehicle.

[0088] In Example 27, the method of Example 25 or 26 in combination with Example 23 or 24 includes: comparing the fourth image with the fifth image to determine a first comparison value representing a difference between the fourth image and the fifth image; and determining whether the first comparison value represents a difference between the fourth image and the fifth image that is less than or equal to a first predetermined difference threshold; comparing the sixth image with the fifth image to determine a second comparison value representing a difference between the sixth image and the fifth image; and determining whether the second comparison value represents a difference between the sixth image and the fifth image that is less than or equal to a second predetermined difference threshold; and determining whether the comparison value represents a difference between the sixth image and the fifth image that is less than or equal to the second predetermined difference threshold. The method can optionally further include determining that the front camera is not calibrated correctly if the first comparison value represents a difference between one image and a second image and the further comparison value is determined to represent a difference between a third image and the second image that is less than a further predetermined difference threshold, and determining that the right side camera is not calibrated correctly if the first comparison value represents a difference between a fourth image and a fifth image that is greater than a first predetermined difference threshold and the second comparison value is determined to represent a difference between a sixth image and the fifth image that is greater than a second predetermined difference threshold (optionally further determining that the rear camera, the front camera, and the left side camera are calibrated correctly).

[0089] In Example 28, the method of any one of Examples 16 to 27 can optionally further include generating a difference image representing a difference between the two images when the two images are compared to each other, and providing control instructions to control a display device to display the difference image.

[0090] In Example 29, the method of any one of Examples 1 to 28 can optionally further include rotating and / or shifting at least one of the two images (e.g., by rigid registration) before comparing the two images to each other.

[0091] In Example 30, the method of any one of Examples 16 to 29 can optionally further include, if a camera (e.g., a front camera, a rear camera, a left side camera, and / or a right side camera) is determined to be incorrectly calibrated, providing control instructions to control recalibration of the camera.

[0092] Example 31 is a non-transitory computer-readable medium having instructions recorded thereon, which, when executed by a processor of a vehicle, cause the processor to perform a method according to any one of Examples 16-30.

[0093] Example 32 is a control device for controlling a vehicle, the control device including one or more processors configured to receive, from a longitudinal camera of the vehicle, a first longitudinal image showing surroundings in front of or behind the vehicle at a first time point; receive, from a lateral camera of the vehicle, a first lateral image showing surroundings to a side adjacent to the vehicle at the first time point; receive, from the lateral camera of the vehicle, a second lateral image showing surroundings to a side adjacent to the vehicle at a second time point different from the first time point; a portion of the second lateral image showing a scene shown in the first longitudinal image and the first lateral image at the first time point; use the first longitudinal image and the first lateral image to generate a first image showing a top view of the scene at the first time point; use the second lateral image to generate a second image showing a top view of the scene at the second time point; and determine whether the longitudinal camera and the lateral camera are correctly calibrated by comparing the first image with the second image.

[0094] In Example 33, the subject matter of Example 32 may optionally include the one or more processors being configured to determine a comparison value representing a difference between the first image and the second image by comparing the first image with the second image, determine whether the comparison value represents a difference between the first image and the second image that is less than or equal to a predetermined difference threshold, and determine that the portrait camera and the landscape camera are correctly calibrated if it is determined that the comparison value represents a difference between the first image and the second image that is less than or equal to the predetermined difference threshold.

[0095] In Example 34, the subject matter of Example 32 or 33 may optionally include one or more processors configured to use the first longitudinal image and the first lateral image to generate a first upper perspective image showing a top view of the lateral surroundings adjacent to the vehicle and the surroundings in front of or behind the vehicle at least at a first time point, and to generate the first image by extracting a portion of the first upper perspective image, and / or to use the second lateral image to generate a second upper perspective image showing a top view of the lateral surroundings adjacent to the vehicle at least at the first time point, and to generate the second image by extracting a portion of the second upper perspective image.

[0096] In Example 35, the subject matter of any one of Examples 32 to 34 can optionally include the first image and the second image having the same image size.

[0097] In Example 36, the subject matter of any one of Examples 32 to 35 can optionally include the vertical camera being configured to capture successive vertical images at a predetermined frame rate, and the first image and / or the second image having a vertical image size in the vertical direction perpendicular to the side substantially equal to the number of pixels shifted in two successive vertical images.

[0098] In Example 37, the subject matter of any one of Examples 32 to 36 can optionally include the one or more processors being configured to receive, from the longitudinal camera, a second longitudinal image showing the surroundings in front of or behind the vehicle at a second time, wherein the second longitudinal image does not show a scene.

[0099] In Example 38, the subject matter of any one of Examples 32 to 37 can optionally include the following: the longitudinal camera is a forward camera configured to capture the forward image as a longitudinal image showing the surroundings in front of the vehicle; and the one or more processors are configured to receive, from a rear camera of the vehicle, a first rear image showing the surroundings behind the vehicle at a third time point, the second time point being temporally between the first time point and the second time point; receive, from the lateral camera, a third lateral image showing the surroundings to the side adjacent to the vehicle at the third time point; the scene is shown in the first rear image and the third lateral image at the third time point; use the rear image and the third lateral image to generate a third image showing a top view of the scene at the third time point; and determine whether the rear camera and the lateral camera are correctly calibrated by comparing the third image with the second image.

[0100] In Example 39, the subject matter of Example 38 may optionally include the one or more processors being configured to determine a further comparison value representing a difference between the third image and the second image by comparing the third image with the second image, determine whether the further comparison value represents a difference between the third image and the second image that is less than or equal to a further predetermined difference threshold, and determine that the rear camera and the lateral camera are correctly calibrated if it is determined that the further comparison value represents a difference between the third image and the second image that is less than or equal to a further predetermined difference threshold.

[0101] In Example 40, the subject matter of Examples 33 and 39 can optionally include the one or more processors being configured to determine that the front camera is not calibrated correctly if the comparison value is determined to represent a difference between the first image and the second image that is greater than a predetermined difference threshold and the further comparison value is determined to represent a difference between the third image and the second image that is less than or equal to a further predetermined difference threshold, and / or determine that the rear camera is not calibrated correctly if the comparison value is determined to represent a difference between the first image and the second image that is less than or equal to a predetermined difference threshold and the further comparison value is determined to represent a difference between the third image and the second image that is greater than a further predetermined difference threshold.

[0102] In Example 41, the subject matter of any one of Examples 32 to 40 in combination with Example 38 is a left side view camera configured to capture a left side view image as a lateral image showing surroundings to the left of the vehicle, and one or more processors receive from the right side view camera a first right side view image showing surroundings to the right of the vehicle at a first time point, a second right side view image showing surroundings to the right of the vehicle at a second time point, and a third right side view image showing surroundings to the right of the vehicle at a third time point, wherein a portion of the second right side view image is shown in the first longitudinal view image and the first right side view image at the first time point, and a portion of the second right side view image is shown in the first rear view image and the third right side view image at the third time point. the first longitudinal image and the first right lateral image to generate a fourth image showing a top view of the further scene at the first time point, the second right lateral image to generate a fifth image showing a top view of the further scene at the second time point, the first rearward image and the third right lateral image to generate a sixth image showing a top view of the further scene at the third time point, and determining whether the front camera, the rear camera, and / or the right lateral camera are correctly calibrated by comparing the fourth image with the fifth image and comparing the sixth image with the fifth image.

[0103] In Example 42, the subject matter of Examples 34 and 41 can optionally include the one or more processors being configured to generate a first top perspective image using the first longitudinal image, the first rear image, the first lateral image, and the first right side image, such that the first top perspective image shows a top view of the surroundings around the entire perimeter of the vehicle.

[0104] In Example 43, the subject matter of Example 41 or 42 in combination with Example 39 or 40 is further characterized in that the one or more processors: determine a first comparison value representing a difference between the fourth image and the fifth image by comparing the fourth image with the fifth image; and determine whether the first comparison value represents a difference between the fourth image and the fifth image that is less than or equal to a first predetermined difference threshold; determine a second comparison value representing a difference between the sixth image and the fifth image by comparing the sixth image with the fifth image; and determine whether the second comparison value represents a difference between the sixth image and the fifth image that is less than or equal to a second predetermined difference threshold; and determine whether the comparison value represents a difference between the first image and the fifth image that is less than or equal to a second predetermined difference threshold. and determining that the front camera is not correctly calibrated if the first comparison value represents a difference between the fourth image and the fifth image that is greater than the first predetermined difference threshold and the second comparison value represents a difference between the sixth image and the fifth image that is greater than a second predetermined difference threshold, and determining that the right side camera is not correctly calibrated (optionally further determining that the rear camera, the front camera, and the left side camera are correctly calibrated) if the first comparison value represents a difference between the fourth image and the fifth image that is greater than the first predetermined difference threshold and the second comparison value represents a difference between the sixth image and the fifth image that is greater than a second predetermined difference threshold.

[0105] In Example 44, the subject matter of any one of Examples 32 to 43 may optionally include the one or more processors being configured to generate a difference image representing the difference between the two images when comparing the two images with each other, and to provide control instructions to control a display device to display the difference image.

[0106] In Example 45, the subject matter of any one of Examples 32 to 44 can optionally include one or more processors configured to rotate and / or shift at least one of the two images (e.g., by rigid registration) before comparing the two images to each other.

[0107] In Example 46, the subject matter of any one of Examples 32 to 45 may optionally include one or more processors configured to provide control instructions to control recalibration of the cameras (e.g., a front camera, a rear camera, a left side camera, and / or a right side camera) when it is determined that the cameras are not correctly calibrated. [Explanation of symbols]

[0108] 12 forward 14 rear 16 left side 18 Right side 20 Upward View Direction 100 vehicles 102 Front camera 104 Rear camera 106 Left side camera 108 Right side camera 110 One or more processors 112 Front part 114 Rear part 116 Left side part 118 Right side part 120 Left front part 122 Right front part 124 Left rear part 126 Right rear part 200 areas 300 ways 302-314 Characteristics of the method

Claims

1. A control device for controlling a vehicle, receiving a first longitudinal image from a longitudinal camera (102, 104) of a vehicle (100) showing an environment in front of or behind the vehicle (100) at a first time point; receiving a first lateral image from a lateral camera (106, 108) of the vehicle (100) showing surroundings on a side (16, 18) adjacent to the vehicle (100) at the first time point; receiving, from the lateral cameras (106, 108), a second lateral image showing the surroundings on the side (16, 18) adjacent the vehicle (100) at a second time point different from the first time point, a portion of the second lateral image showing the scene shown in the first longitudinal image and the first lateral image at the first time point; generating a first image showing a top view of the scene at the first time point using the first portrait image and the first landscape image; using the second lateral image to generate a second image showing a top view of the scene at the second time point; By comparing the first image with the second image, it is determined whether the vertical cameras (102, 104) and the horizontal cameras (106, 108) are properly calibrated. one or more processors (110) configured to A control device comprising:

2. The one or more processors (110) comparing the first image with the second image to determine a comparison value representing a difference between the first image and the second image; determining whether the comparison value represents a difference between the first image and the second image that is less than or equal to a predetermined difference threshold; determining that the portrait camera (102, 104) and the landscape camera (106, 108) are properly calibrated when it is determined that the comparison value represents a difference between the first image and the second image that is less than or equal to the predetermined difference threshold; The control device according to claim 1 , configured to:

3. The one or more processors (110) generating a first top-view image using the first longitudinal image and the first lateral image, the first top-view image showing a top view of the surroundings adjacent to the side of the vehicle (100) and the surroundings in front of or behind the vehicle (100) at least at the first time point, and generating the first image by extracting a portion of the first top-view image; Using the second lateral image, a second top-view image is generated showing a top view of the surroundings to the side adjacent to the vehicle (100) at least at the first time point, and the second image is generated by extracting a portion of the second top-view image. The control device according to claim 1 or 2, configured to:

4. The control device according to any one of claims 1 to 3, wherein the first image and the second image have the same image size.

5. the longitudinal cameras (102, 104) are configured to capture successive longitudinal images at a predetermined frame rate; the first image and / or the second image have a vertical image size in a vertical direction perpendicular to the sides (16, 18) substantially equal to the number of pixels shifted between two successive vertical images; The control device according to any one of claims 1 to 4.

6. the longitudinal cameras (102, 104) are forward cameras configured to capture a forward image as the longitudinal image showing the surroundings in front of the vehicle (100); The one or more processors (110) receiving a first rear image from a rear camera (104) of the vehicle (100) showing the surroundings behind the vehicle (100) at a third point in time, the second point in time being between the first point in time and the third point in time; receiving a third lateral image from the lateral cameras (106, 108) showing the surroundings on the sides (16, 18) adjacent the vehicle (100) at the third time point, the scene being shown in the first rear image and the third lateral image at the third time point; generating a third image showing a top view of the scene at the third time using the rear image and the third lateral image; Comparing the third image with the second image determines whether the rear camera (104) and the lateral cameras (106, 108) are properly calibrated. The control device according to any one of claims 1 to 5, configured as follows:

7. The one or more processors (110) determining a further comparison value representing the difference between the third image and the second image by comparing the third image with the second image; determining whether the further comparison value represents a difference between the third image and the second image that is less than or equal to a further predetermined difference threshold; determining that the rear camera (104) and the lateral cameras (106, 108) are correctly calibrated if it is determined that the further comparison value represents a difference between the third image and the second image that is less than or equal to the further predetermined difference threshold. The control device according to claim 6 , configured to:

8. The one or more processors (110) determining that the forward camera is not correctly calibrated if the comparison value is determined to represent a difference between the first image and the second image that is greater than the predetermined difference threshold and the further comparison value is determined to represent a difference between the third image and the second image that is less than or equal to the further predetermined difference threshold; and / or determining that the rear camera (104) is not properly calibrated if the comparison value is determined to represent a difference between the first image and the second image that is less than or equal to the predetermined difference threshold, and if the further comparison value is determined to represent a difference between the third image and the second image that is greater than the further predetermined difference threshold. The control device according to claims 2 and 7, configured to:

9. the lateral cameras (106, 108) being a left side view camera (106) configured to capture a left side view image as the lateral image showing the surroundings to the left of the vehicle (100); The one or more processors (110) From the right side camera (108), a first right side image showing the surroundings to the right of the vehicle (100) at the first time point; a second right side image showing the surroundings to the right of the vehicle (100) at the second time point; a third right side image showing the surroundings to the right of the vehicle (100) at the third time point; and Receive, a portion of the second right side image showing a further scene that is shown in the first longitudinal image and the first right side image at the first time point and that is shown in the first rearward image and the third right side image at the third time point; generating a fourth image using the first portrait image and the first right-side image, the fourth image showing a top view of the further scene at the first time point; using the second right side image to generate a fifth image showing a top view of the further scene at the second time point; generating a sixth image using the first rear image and the third right side image, the sixth image showing a top view of the further scene at the third time point; Comparing the fourth image to the fifth image and comparing the sixth image to the fifth image to determine whether the front camera, the rear camera (104), and / or the right side camera (108) are properly calibrated. A control device according to any one of claims 1 to 8 in combination with claim 6, configured so that

10. The one or more processors (110) generating the first top-view image using the first longitudinal image, the first rear image, the first lateral image, and the first right-side image such that the first top-view image shows a top view of the surroundings all around the vehicle (100); 10. The control device according to claims 3 and 9, configured to:

11. The one or more processors (110) comparing the fourth image with the fifth image to determine a first comparison value representing a difference between the fourth image and the fifth image; determining whether the first comparison value represents a difference between the fourth image and the fifth image that is less than or equal to a first predetermined difference threshold; comparing the sixth image with the fifth image to determine a second comparison value representing a difference between the sixth image and the fifth image; determining whether the second comparison value represents a difference between the sixth image and the fifth image that is less than or equal to a second predetermined difference threshold; the comparison value represents a difference between the first image and the second image less than or equal to the predetermined difference threshold; determining that the forward camera is not properly calibrated when the further comparison value represents a difference between the third image and the second image that is less than or equal to the further predetermined difference threshold; The first comparison value represents a difference between the fourth image and the fifth image that is greater than the first predetermined difference threshold, and the second comparison value represents a difference between the sixth image and the fifth image that is greater than the second predetermined difference threshold. If it is determined that the right side camera (108) is not correctly calibrated, 11. A control device according to claim 9 or 10 in combination with claim 7 or 8, configured to:

12. The control device of any one of claims 1 to 11, wherein the one or more processors (110) are configured to generate a difference image representing the difference between two images when comparing the two images with each other, and to provide control instructions to control a display device to display the difference image.

13. The control device of any one of claims 1 to 12, wherein the one or more processors (110) are configured to rotate and / or shift at least one of the two images before comparing them with each other.

14. A vehicle (100) comprising a control device according to any one of claims 1 to 13.

15. A method (300) for assessing the calibration of one or more cameras mounted on a vehicle, comprising: receiving (302) a first longitudinal image from a longitudinal camera showing an environment in front of the vehicle or behind the vehicle at a first point in time; receiving (304) a first lateral image from a lateral camera showing a lateral surrounding adjacent to the vehicle at the first time point; receiving (306) a second longitudinal image from the longitudinal camera showing the surroundings in front of or behind the vehicle at a second point in time different from the first point in time; receiving (308) from the lateral camera a second lateral image showing the surroundings on the side adjacent the vehicle at the second time point, a portion of the second lateral image not shown in the second longitudinal image and showing a scene shown in the first longitudinal image and the first lateral image at the first time point; generating (310) a first image showing a top view of the scene at the first time point using the first portrait image and the first landscape image; generating (312) a second image showing a top view of the scene at the second time point using the second lateral image; determining whether the portrait and landscape cameras are properly calibrated by comparing the first image with the second image (314); A method (300) comprising:

Images