Vehicle, control device and method for evaluating the calibration of one or more cameras mounted to a vehicle

Abstract

Various aspects relate to a vehicle, a method, and a control device including one or more processors (110) configured to: receive a first longitudinal image showing the surrounding environment in front of or behind the vehicle at a first time point, a first lateral image showing the surrounding environment adjacent to the vehicle in a lateral direction at the first time point, and a second lateral image showing the surrounding environment adjacent to the vehicle in a lateral direction at a second time point; generate a first image using the first longitudinal image and the first lateral image, the first image showing a top view of the scene at the first time point; generate a second image using the second lateral image, the second image showing a top view of the scene at the second time point; and determine whether the longitudinal camera (102, 104) and the lateral camera are correctly calibrated by comparing the first image with the second image.

Description

Technical Field

[0001] Various aspects of this disclosure relate to a vehicle, a control device for use in the vehicle, and a method for evaluating the calibration of one or more cameras mounted on the vehicle (e.g., quantitatively evaluating the quality of camera calibration). Background Technology

[0002] The following discussion of background technology is intended only to facilitate understanding of this disclosure. It should be understood that this discussion does not constitute an admission or endorsement that any material mentioned is publicly available, known, or part of common general knowledge in any jurisdiction as of the priority date of this disclosure.

[0003] The onboard control system of a vehicle can provide various driver assistance functions, such as surround view (e.g., top view), parking assist, lane departure warning systems, etc. These driver assistance functions use sensors, such as cameras, to detect the vehicle's surrounding environment. When the vehicle is in motion (e.g., traveling), the vehicle body and therefore the sensors (e.g., cameras) can change their orientation and / or position. Therefore, it may be necessary (e.g., to meet safety requirements) to recalibrate the sensors (e.g., cameras) online (i.e., during vehicle use). For example, one or more sensor calibration parameters can be recalculated during travel (e.g., in real time) to compensate for the aforementioned changes (e.g., using online calibration algorithms). Here, it may be necessary and / or desirable to evaluate the sensor calibration (e.g., to determine whether the sensor is correctly calibrated or should be recalibrated). Summary of the Invention

[0004] The various aspects relate to a means of transport, a control device for use in the means of transport, and a method for evaluating the corresponding calibration of one or more sensors (e.g., a camera).

[0005] According to various aspects, a vehicle may include: a longitudinal camera configured to capture a longitudinal image showing the surrounding environment in front of or behind the vehicle; a lateral camera configured to capture a lateral image showing the surrounding environment adjacent to the vehicle in a lateral direction, wherein the longitudinal camera and the lateral camera have partially overlapping fields of view such that a portion of the longitudinal image and a portion of the lateral image show the same area in the surrounding environment of the vehicle; and one or more processors configured to: receive a first longitudinal image from the longitudinal camera showing the surrounding environment in front of or behind the vehicle at a first time point; and receive a first lateral image from the lateral camera, the first longitudinal image showing the surrounding environment in front of or behind the vehicle at a first time point; A lateral image shows the surrounding environment adjacent to the vehicle in the lateral direction at a first time point; a second lateral image is received from the lateral camera, showing the surrounding environment adjacent to the vehicle in the lateral direction at a second time point different from the first time point, wherein a portion of the second lateral image shows the scene shown in the first longitudinal image and the first lateral image at the first time point; a first image is generated using the first longitudinal image and the first lateral image, showing a top view of the scene at the first time point; a second image is generated using the second lateral image, showing a top view of the scene at the second time point; and the longitudinal camera and the lateral camera are determined to be correctly calibrated by comparing the first image with the second image.

[0006] According to various aspects, a method is provided for evaluating the calibration of one or more cameras mounted on a vehicle, the method comprising: receiving a first longitudinal image from a longitudinal camera, the first longitudinal image showing the surrounding environment in front of or behind the vehicle at a first time point; receiving a first lateral image from a lateral camera, the first lateral image showing the surrounding environment adjacent to the vehicle in a lateral direction at the first time point; receiving a second longitudinal image from the longitudinal camera, the second longitudinal image showing the surrounding environment in front of or behind the vehicle at a second time point different from the first time point; receiving the second lateral image from the lateral camera, the first ... The two lateral images show the surrounding environment adjacent to the vehicle in the lateral direction at the second time point, wherein a portion of the second lateral image shows the scene not shown in the second longitudinal image and shown in the first longitudinal image and the first lateral image at the first time point; a first image is generated using the first longitudinal image and the first lateral image, the first image showing a top view of the scene at the first time point; a second image is generated using the second lateral image, the second image showing a top view of the scene at the second time point; and the longitudinal camera and the lateral camera are determined to be correctly calibrated by comparing the first image with the second image. Attached Figure Description

[0007] This disclosure will be better understood in conjunction with the non-limiting examples and accompanying drawings, and with reference to the detailed description, in which:

[0008] Figure 1A The vehicles are shown according to various aspects;

[0009] Figure 1B An exemplary top view of a vehicle and its surroundings is shown, according to various aspects;

[0010] Figures 2A to 2D Each illustrates various aspects of collecting calibration data from one or more cameras used to evaluate vehicles; and

[0011] Figure 3 A flowchart is shown showing a method for evaluating the calibration of at least one camera based on various aspects. Detailed Implementation

[0012] The following detailed description refers to the accompanying drawings, which illustrate by way of illustration specific details and embodiments in which the present disclosure can be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure. Other embodiments may be utilized and structural and logical changes may be made without departing from the scope of the present disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments.

[0013] The embodiments described in the context of one of the means of transport, apparatus, or method are similarly effective for other means of transport, apparatus, systems, or methods. Similarly, the embodiments described in the context of an apparatus are similarly effective for a system (e.g., a means of transport) and / or method, and vice versa.

[0014] Features described in the context of one embodiment may be correspondingly applied to the same or similar features in other embodiments. Features described in the context of one embodiment may be correspondingly applied to other embodiments, even if not explicitly described in those other embodiments. Furthermore, additions and / or combinations and / or alternatives described for features in the context of one embodiment may be correspondingly applied to the same or similar features in other embodiments.

[0015] In the context of the various embodiments, the articles “a,” “an,” and “the” used with respect to a feature or element include references to one or more of the features or elements.

[0016] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0017] While terms such as "first" and "second" can be used to describe various devices, such as cameras, these devices are not limited by these terms. These terms are only used to distinguish one device from another and do not define the order and / or importance of the devices.

[0018] As used herein, the term "data" can be understood to include information in any suitable analog or digital form, such as information provided as a file, a portion of a file, a collection of files, a signal or stream, a portion of a signal or stream, a collection of signals or streams, etc. Furthermore, the term "data" can also be used to refer to a reference to information, for example, in the form of a pointer. However, the term "data" is not limited to the examples above and can take various forms and represent any information as understood in the art. As described herein, any type of information can be processed, for example, via one or more processors in a suitable manner (e.g., as data).

[0019] For example, the terms "processor" or "controller" as used herein can be understood as any kind of entity that allows the processing of data. Data can be processed according to one or more specific functions performed by the processor or controller. Further, a processor or controller as used herein can be understood as any kind of circuit, such as any kind of analog or digital circuit. Therefore, a processor or controller can be or includes analog circuits, digital circuits, mixed-signal circuits, logic circuits, processors, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), integrated circuits, application-specific integrated circuits (ASICs), and any combination thereof. Any other type of implementation of the corresponding functions, which will be described in further detail below, can also be understood as a processor, controller, or logic circuit. It should be understood that any two (or more) of the processors, controllers, or logic circuits detailed herein can be implemented as a single entity with equivalent functions, and conversely, any single processor, controller, or logic circuit detailed herein can be implemented as two (or more) separate entities with equivalent functions.

[0020] The term “memory” as used in this article can be understood to include any suitable type of memory or memory device, such as hard disk drives (HDDs), solid-state drives (SSDs), flash memory, etc.

[0021] Various aspects relate to a control device for use in a vehicle. For example, the control device may be configured to control one or more functions (e.g., operation) of the vehicle. The control device may be able to evaluate the calibration of one or more sensors of the vehicle. Therefore, the control device may be part of a sensor control (e.g., monitoring) system and / or a driver assistance system and / or a safety system. Although various aspects relate to the use of a control device within the vehicle itself, it should be understood that the processes described herein may also be performed outside the vehicle (e.g., in a cloud computing device).

[0022] As an example, a vehicle may include four cameras (e.g., a front camera, a rear camera, a left-facing camera, and a right-facing camera) for capturing images of the complete surrounding environment of the vehicle, and the vehicle's control unit may be configured to determine, based on images captured by at least two of the four cameras, whether those at least two cameras are correctly calibrated (or whether they need recalibration). As described herein, the control unit is capable of evaluating camera calibration without requiring any additional means (besides the at least two (e.g., four) cameras).

[0023] As an illustrative example of how to evaluate the calibration of two cameras: A vehicle may include a front camera for capturing a frontal image of the surrounding environment in front of the vehicle, a rear camera for capturing the surrounding environment behind the vehicle, a left-facing camera for capturing a left-facing image of the surrounding environment to the left of the vehicle, and a right-facing camera for capturing a right-facing image of the surrounding environment to the right of the vehicle. A control unit may continuously receive these images at a predefined frame rate and may be configured to generate corresponding top-down images at various times, showing a top-down view of the vehicle and its surrounding environment at a specific point in time (e.g., to assist in parking). As an example, the front camera and the left-facing camera may have partially overlapping fields of view, such that a portion of the images they capture (from different angles) shows the same area in the vehicle's surrounding environment. Therefore, the control unit can generate a portion of the top-down image by merging (e.g., stitching) information from both the front and left-facing images. In the case where the vehicle is traveling forward along a substantially straight route, the vehicle may pass through a specific scene that is initially located in the same area and later located outside that same area, immediately to the left of the vehicle. The control device can generate a first top-view image showing the scene within the merged section and a second top-view image showing the same scene outside the merged section. The control device can use the second top-view image as a reference ground truth image of the first top-view image to evaluate the calibration of the front and left-side cameras. For example, the control device compares the first and second top-view images and determines that the front and left-side cameras are correctly calibrated if the difference between them is less than a predefined threshold.

[0024] Figure 1AA vehicle 100 is illustrated according to various aspects. The vehicle 100 may include at least two sensors configured to detect the surrounding environment of the vehicle 100. The at least two sensors may have partially overlapping fields of view. The vehicle 100 may include one or more processors 110. For example, the vehicle 100 may include a control device, and the control device may include one or more processors 110. The one or more processors 110 may be configured to receive corresponding sensor data from the at least two sensors at various times (e.g., continuously) and use the received sensor data to evaluate the calibration of the at least two sensors. The one or more processors 110 may be configured to generate corresponding top-view images at one or more (e.g., each) times, showing a top view of the surrounding environment of the vehicle 100 (and optionally also a top view of the vehicle 100 itself). The one or more processors 110 may be configured to use the generated top-view images to evaluate the calibration of the at least two sensors. Hereinafter, the at least two sensors are described as corresponding cameras configured to capture images(s) of the surrounding environment(s) of the vehicle 100. However, it should be understood that this is used to illustrate the principles of calibrating the evaluation sensors according to this disclosure, and one or more of the at least two sensors may also be other sensors (e.g., radio detection and ranging (radar) sensors, light detection and ranging (LiDAR) sensors, etc.), as long as the constraints described herein are met (e.g., at least two sensors have partially overlapping fields of view).

[0025] like Figure 1A As shown, the vehicle 100 may include a front camera 102 (also referred to as a (first) longitudinal camera or front-view camera), a rear camera 104 (also referred to as a (second) longitudinal camera or rear-view camera), a left-facing camera 106 (also referred to as a (first) side camera or left-view camera), and a right-facing camera 108 (also referred to as a (second) side camera or right-view camera). Although the vehicle 100 is described as having four cameras, it should be understood that, using the principles described herein, camera calibration can be evaluated as long as at least two cameras (with partially overlapping fields of view) are present.

[0026] As described herein, the camera can be any kind of (camera) device capable of capturing images of the surrounding environment of vehicle 100. Therefore, the camera can also be referred to as a vision sensor, image sensor, or camera device. It should be understood that the camera can be configured to provide the captured images directly or indirectly (e.g., by storing them in a memory device) to one or more processors 110. Further, it should be understood that the camera can be configured to output unprocessed or preprocessed images. The camera can be configured to capture corresponding images of the (corresponding) surrounding environment of vehicle 100 at a predefined frame rate. The camera described herein can be a wide-angle camera (e.g., having an aperture angle equal to or greater than 120°). The forward camera 102 can be configured to capture (e.g., at each moment according to the camera's frame rate) a forward image showing the surrounding environment in front of vehicle 100 (e.g., in front of vehicle 100 relative to the forward travel direction 12). The rear camera 104 can be configured to capture (e.g., at each moment, depending on the camera's frame rate) a rearward image of the surrounding environment behind the vehicle 100 (e.g., in front of the vehicle 100 relative to the rearward direction 14 (opposite to the forward direction of travel)). The left-facing camera 106 can be configured to capture (e.g., at each moment, depending on the camera's frame rate) a left-facing image of the surrounding environment to the left of the vehicle 100 (e.g., to the left of the vehicle 100 relative to the forward direction of travel 12). The right-facing camera 108 can be configured to capture (e.g., at each moment, depending on the camera's frame rate) a right-facing image of the surrounding environment to the right of the vehicle 100 (e.g., to the left of the vehicle 100 relative to the forward direction of travel 12). The left-facing camera 106 can have a first partially overlapping field of view with the front camera 102 and a second partially overlapping field of view with the rear camera 104 (different from the first partially overlapping field of view). Therefore, the first portion of the field of view (fov) of the left-facing camera 106 may overlap with a portion of the fov of the front camera 102, the second portion of the fov of the left-facing camera 106 may overlap with a portion of the fov of the rear camera 104, and the third portion of the fov of the left-facing camera 106 may not overlap with either the fov of the front camera or the fov of the rear camera. This similarly applies to the right-facing camera 108, such that the right-facing camera 108 has a partially overlapping fov with the front camera 102 and a partially overlapping fov with the rear camera 104. As described herein, partially overlapping fields of view should be understood to mean fovs that do not completely overlap.

[0027] It should be understood that the corresponding nominal positions of the front camera 102, rear camera 104, left-facing camera 106, and right-facing camera 108 can be stored (e.g., as part of computer-aided design (CAD) data associated with vehicle 100) in a memory device, and one or more processors 110 use the information on the memory device during processing. Furthermore, the memory device can store various (e.g., internal) parameters of the cameras. The left-facing camera 106 can be mounted to the left-side rearview mirror of vehicle 100. The right-facing camera 108 can be mounted to the right-side rearview mirror of vehicle 100.

[0028] Vehicle 100 can be any type of vehicle that uses sensors with partially overlapping fields of view to detect the environment around the vehicle. The sensors (e.g., cameras) can be part of a surround view system (SVS) configured to provide (e.g., output via a display device and / or for further processing) several views of vehicle 100 (e.g., top view, rear view, panoramic view, etc.).

[0029] As an example, vehicle 100 may be a non-autonomous vehicle including a display device to show the driver a top-down view of the vehicle and its surroundings to assist in parking (e.g., showing and / or warning of obstacles in the surrounding environment of vehicle 100). As another example, vehicle 100 may be a semi-autonomous vehicle or an autonomous vehicle with an automatic parking function (i.e., parking the vehicle without requiring the driver to steer and / or accelerate / decelerate), which uses sensors to detect the surrounding environment of the vehicle (e.g., using a top-down view). The automatic parking function may include assisted parking when the driver is inside vehicle 100, remote parking without the driver being inside vehicle 100, and / or valet parking.

[0030] "Vehicle" can be a ground vehicle (e.g., a vehicle configured to travel on the ground (e.g., on roads, tracks, streets, etc.), an air vehicle (e.g., a vehicle configured to be maneuvered above the ground), or a water vehicle (e.g., a vehicle capable of being maneuvered on or below the surface of a liquid (e.g., water)).

[0031] It should be understood that the vehicle 100 may include various additional components common to vehicles of the corresponding type (e.g., engine, steering system, etc.).

[0032] Figure 1BAn exemplary top view of the vehicle 100 and its surroundings is shown according to various aspects. As described herein, each of the front camera 102, rear camera 104, left-facing camera 106, and right-facing camera 108 can be configured to capture a corresponding image at a (corresponding) predefined frame rate. The frame rates can be synchronized with each other. One or more processors 110 can be configured to receive a front image from the front camera 102, a rear image from the rear camera 104, a left-facing image from the left-facing camera, and a right-facing image from the right-facing camera 108 (all of these images are associated with substantially the same time point t), and generate a top view image O associated with that time point t using the front image, rear image, left-facing image, and right-facing image. t .For example, Figure 1B An exemplary top-view image O can be represented as being associated with a specific time point t. t Top view image O t It may have a front portion 112, which represents the surrounding environment in front of the vehicle 100, and this front portion includes information captured by the front camera 102 but does not include information from the rear camera 104, the left-side camera 106, and the right-side camera 108. Top view image O t It may have a rear portion 114, which represents the surrounding environment behind the vehicle 100, and this rear portion includes information captured by the rear camera 104 but does not include information from the front camera 102, the left-side camera 106, and the right-side camera 108. Top view image O t It may have a left-facing portion 116, which represents the surrounding environment to the left of the vehicle 100, and this left-facing portion includes information captured by the left-facing camera 106 but does not include information from the front camera 102, the rear camera 104, and the right-facing camera 108. Top view image O t It may have a right-facing portion 118, which represents the surrounding environment to the right of the vehicle 100, and this right-facing portion includes information captured by the right-facing camera 108 but does not include information from the front camera 102, the rear camera 104, and the left-facing camera 106. As described herein, each camera pair may have partially overlapping fovs. Therefore, one or more processors 110 may be configured to generate a top-view image O t Information from both cameras in the camera pair is used. Therefore, the top-view image O tIt may have: a left front portion 120, which includes information captured by the front camera 102 and the left-facing camera 106; a right front portion 122, which includes information captured by the front camera 102 and the right-facing camera 108; a left rear portion 124, which includes information captured by the rear camera 104 and the left-facing camera 106; and a right rear portion 126, which includes information captured by the rear camera 104 and the right-facing camera 108.

[0033] In the following text, see references Figures 2A to 2D Various aspects of evaluating the corresponding calibrations of the front camera 102, rear camera 104, left-facing camera 106, and / or right-facing camera 108 are described.

[0034] like Figure 2A As shown, vehicle 100 can travel in a forward direction 12 (e.g., along a substantially straight route) through area 200. At a first time point (in some respects, this time point is referred to as moment t=i), area 200 can be located to the left front of vehicle 100, and is therefore shown in the forward image captured by forward camera 102 and the left-facing image captured by left-facing camera 106 at the first time point t=i. One or more processors 110 can generate the corresponding top-view image O. t＝i At the second time point t = i + F0, when traveling in the forward direction 12, region 200 may be located to the left of vehicle 100 and therefore only shown in the left-facing image captured by camera 106 from the left at the second time point t = i + F0. One or more processors 110 may generate the corresponding top-view image O. t＝i+F0 At the third time point t = i + F0 + R0, region 200 can be located to the left rear of vehicle 100 and is therefore shown in the rear image captured by rear camera 104 and the left-facing image captured by left-facing camera 106 at the third time point t = i + F0 + R0. One or more processors 110 can generate the corresponding top-view image O. t＝i+F0+R0 Therefore, the same area in the surrounding environment of the vehicle 100 can be represented by the various top-view images O. t This indicates that the same principle applies to the area to the right of vehicle 100. Although this example is described with reference to the case where vehicle 100 is traveling in the forward direction 12, the principles described herein are similarly applicable to travel in the backward direction 14. As understood, in the case of backward travel, the second time point t = i + F0 is before the first time point t = i (i.e., F0 < 0), and the third time point t = i + F0 + R0 is before the second time point t = i + F0 (i.e., R0 < 0).

[0035] As described, at the first time point t=i, region 200 can be located in the left front portion 120 (independent of travel in the forward or backward direction). (Reference) Figure 2B One or more processors 110 can be configured to generate (e.g., cut out) a left-front image FL(O) showing a first scene (e.g., in region 200) at a first time point t=i. t＝i (Also referred to as the left anterior patch) and / or the right anterior image FR(O) showing the second scene (different from the first scene) at the first time point t=i. t ＝i (Also referred to as the right front tile). One or more processors 110 can be configured to generate (e.g., cut out) a left-side image L(O) of the first scene shown at the second time point t = i + F0. t＝i+F0 (Also referred to as the left-hand baseline truth plot) and / or the right-hand image R(O) showing the second scene at the second time point t = i + F0. t＝i+F0 (Also referred to as the right-side baseline truth map). One or more processors 110 can be configured to generate (e.g., cut out) a left rear image RL(O) showing the first scene at the third time point t = i + F0 + R0. t＝i+F0+R0 (Also referred to as the left rear plot) and / or the right rear plot RR(O) of the second scene at the third time point t = i + F0 + R0. t＝i+F0+R0 (This can also be referred to as the right rear panel).

[0036] Based on various aspects, the left-hand image L(O) t＝i+F0 The image R(O) can be symmetrical about the optical axis of camera 106 on the left and / or on the right. t＝i+F0 It can be symmetrical about the optical axis of camera 108 on the right side.

[0037] Based on various aspects, due to the left image L(O) t＝i+F0 The left image L(O) is generated solely from the left-side image (and is not stitched with the front or back images), therefore the left-side image L(O) is... t＝i+F0 The left-hand ground truth image (FL) can represent how the first scene should look. Assuming the front camera 102 and the left-hand camera 106 are correctly calibrated, the expected (e.g., assuming the real-world data is not corrupted) left-hand front image FL(O) is... t＝i (It also shows the first scenario) basically corresponds to the left-hand reference truth image, i.e., the left-hand image L(O) t＝i+F0 Similarly, assuming the rear camera 104 and the left-facing camera 106 are correctly calibrated, the expected left rear image FL(O) t＝i This basically corresponds to the left-hand reference truth image, i.e., the left-hand image L(O) t＝i+F0This similarly applies to the right side, such that, assuming the camera is correctly calibrated, the expected right front image FR(O) t＝i ) and right rear image RR(O t＝i+F0+R0 ) corresponds to the right-hand reference truth image, i.e., the right-hand image R(O) t＝i+F0 ).

[0038] One or more processors 100 can be configured to process the left front image FL(O) t＝i ) and / or left rear image RL(O t＝i The left image L(O) of the (benchmark truth) t＝i+F0 ) compare and / or compare the right anterior image FR(O) t＝i ) and / or right rear image RR (O t ＝i The image R(O) to the right of the (benchmark truth) t＝i+F0 The images are compared to evaluate the calibration of the front camera 102, rear camera 104, left-facing camera 106, and / or right-facing camera 108. According to some aspects, the two images being compared may be the same size. According to others, the two images being compared may be different sizes, and one or more processors 110 may be configured to preprocess these images to make them the same size before comparison. If the vehicle 100 is not traveling in a straight line, the scene shown in the corresponding image on one side (left or right) (e.g., a first scene or a second scene) may be slightly varied (e.g., rotated and / or shifted). One or more processors 110 may be configured to rotate and / or shift at least one of the two images (e.g., using rigid registration) before comparison. Optionally, the vehicle 100 may include an input device that allows a user (e.g., a driver) to manually (alternatively or additionally) shift (also referred to as translation) and / or rotate the images. This allows the user (e.g., the driver) to manually verify the evaluation results.

[0039] One or more processors 110 can be configured to compare two images with each other using any suitable metric. For example, one or more processors 110 can be configured to determine a comparison value representing the difference between two images. It should be understood that similarity also represents difference. As an example, the comparison value can or may represent the sum of absolute differences (SAD), mean squared error (MSE), and / or structural similarity index (SSIM).

[0040] Depending on various aspects, vehicle 100 may include or be coupled to a display device. One or more processors 110 may be configured to determine a difference image representing the difference between two images being compared to each other. One or more processors 110 may be configured to provide control instructions to the display device to display the difference image (e.g., to a user of vehicle 100, such as a driver). This provides a visual comparison to the user (e.g., the driver). Optionally, the difference image may include pseudo-coloring to indicate deviations from a reference ground truth image.

[0041] One or more processors 110 may be configured to determine that two images match each other if a determined comparison value representing the difference between a first image and a second image is equal to or less than a predefined difference threshold. One or more processors 110 may be configured to determine whether a camera is correctly calibrated based on one or more comparisons. Various exemplary cases are described below to illustrate how camera calibration can be evaluated:

[0042] -In the left front image FL(O) t＝i ) and the left image L(O) t＝i+F0 In the case of a match, one or more processors 110 can determine that the front camera 102 and the left-facing camera 106 are correctly calibrated.

[0043] - In the left rear image RL(O) t＝i+F0+R0 ) and the left image L(O) t＝i+F0 In the case of a match, one or more processors 110 can determine that the rear camera 104 and the left-facing camera 106 are correctly calibrated.

[0044] - In the right front image FR(O) t＝i ) and the right-hand image R(O t＝i+F0 In the case of a match, one or more processors 110 can determine that the front camera 102 and the right-facing camera 108 are correctly calibrated.

[0045] - In the right rear image RR(O) t＝i+F0+R0 ) and the right-hand image R(O t＝i+F0 In the case of a match, one or more processors 110 can determine that the rear camera 104 and the right-facing camera 108 are correctly calibrated.

[0046] -In the left front image FL(O) t＝i ) and the left image L(O) t＝i+F0 Match and left rear image RL(O) t＝i+F0+R0 ) and the left image L(O) t＝i+F0 In the event of a mismatch, one or more processors 110 can determine that the front camera 102 and the left-facing camera 106 are correctly calibrated and the rear camera 104 is not correctly calibrated.

[0047] -In the left front image FL(O) t＝i ) and the left image L(O) t＝i+F0 ) mismatch and left rear image RL(O) t＝i+F0+R0 ) and the left image L(O) t＝i+F0 In the case of a match, one or more processors 110 can determine that the rear camera 104 and the left-facing camera 106 are correctly calibrated and the front camera 102 is not correctly calibrated.

[0048] - In the right front image FR(O) t＝i ) and the right-hand image R(O t＝i+F0 Match and right rear image RR(O) t＝i+F0+R0 ) and the right-hand image R(O t＝i+F0 In the event of a mismatch, one or more processors 110 can determine that the front camera 102 and the right-facing camera 106 are correctly calibrated and the rear camera 104 is not correctly calibrated.

[0049] - In the right front image FR(O) t＝i ) and the right-hand image R(O t＝i+F0 ) mismatch and right rear image RR(O) t＝i+F0+R0 ) and the right-hand image R(O t＝i+F0 In the case of a match, one or more processors 110 can determine that the rear camera 104 and the right-facing camera 106 are correctly calibrated and the front camera 102 is not correctly calibrated.

[0050] - In the right front image FR(O) t＝i ) and the right-hand image R(O t＝i+F0 Matching, right rear image RR(O) t＝i+F0+R0 ) and the right-hand image R(O t＝i+F0 Match and left front image FL(O) t＝i ) and left rear image RL(O t＝i+F0+R0 ) and the left image L(O) t＝i+F0 In the event of a mismatch, one or more processors 110 can determine that the front camera 102, the rear camera 104, and the right-facing camera 108 are correctly calibrated, while the left-facing camera 106 is not correctly calibrated.

[0051] -In the left front image FL(O) t＝i ) and the left image L(O) t＝i+F0 Matching, left rear image RL(O) t＝i+F0+R0 ) and the left image L(O) t＝i+F0 Match and right front image FR(O) t＝i ) and right rear image RR(O t＝i+F0+R0 ) and the right-hand image R(O t＝i+F0In the event of a mismatch, one or more processors 110 can determine that the front camera 102, the rear camera 104, and the left-facing camera 106 are correctly calibrated, while the right-facing camera 108 is not correctly calibrated.

[0052] It should be understood that the above assessment is an example, and there are other logical combinations that can allow determination of whether the camera has been correctly calibrated.

[0053] To illustrate, Figure 2C The left anterior image FL(O) with lateral size Px and longitudinal size Py is shown. t＝i Depending on various aspects, the vertical size Py can correspond to the number of pixels moved in two consecutive frames captured by camera 106 from the left. At each frame, a corresponding top-view image O can be generated. t Therefore, in this case, in the left anterior image FL(O) t＝i The scene shown in the image (e.g., the first scene or the second scene) can be displayed immediately following the left front image FL (O). t＝i The next subsequent top-view image O t+1 As shown in the diagram, and so on. This can also be applied to the right anterior image FR(O). t＝i This is in Figure 2D This is illustrated exemplarily. As shown, F0 can be in the left image L(O) respectively. t＝i+F0 ) and the right-hand image R(O t＝i+F0 The minimum number of top-view images O to be generated before another scene is shown in the diagram. Similarly, R0 can be the number of images in the rear region (represented by the left rear image RL(O)). t＝i+F0+R0 ) and / or right rear image RR (O t＝i+F0+R0 The minimum number of top-view images O to be generated between the scene and another scene is shown in the diagram.

[0054] Depending on various aspects, the images described herein (e.g., left front image, right front image, left rear image, and / or right rear image) can be generated using more than one top-view image. For example, the left front image FL(O t＝i至i+F0 ) can be obtained from FL(O) t＝i ) to FL(O t＝i+F0 All left front images FL(O) t (That is, the left front image FL(O)) t＝i ), FL(O t＝i+1 ), FL(O t ＝i+2 ), ..., FL(O t＝i+F0 )) cascaded to generate. Right front image FR(O t＝i至i+F0 ) can be obtained from FR(O t＝i ) to FR(O t＝i+F0 All right anterior images FR(O) t (That is, the right anterior image FR(O)) t＝i ), FR(O t＝i+1 ), FR(O t＝i+2 ), ..., FR(O t＝i+F0 The left rear image can be generated by concatenating two or more left rear images, and / or the right rear image can be generated by concatenating two or more right rear images.

[0055] Correspondingly, the left-side reference ground truth image L(O) t＝i+F0 ) can be obtained from L(O t＝i ) to L(O t＝i+F0+F0 All left-hand images L(O) t＝i至 (That is, the left image L(O)) t＝i+F0 ), L(O t＝i+F0+1 ), L(O t＝i+Fo+2 ), ..., L(O t＝i+F0+F0 The images are generated by cascading. Therefore, the left-side reference ground truth image generated in this way can show the same scene as the cascaded left-side front image.

[0056] Based on various aspects, the left-side reference ground truth image can be compared with the cascaded left-front image. Similarly, the right-side reference ground truth image R(O) t＝i+F0 ) can be obtained from R(O) t＝i ) to R(O t＝i+F0+F0 All right-hand images R(O) t＝i至 (That is, the right-hand image R(O)) t＝i+F0 ), R(O t＝i+F0+1 ), R(O t＝i+Fo+2 ), ..., R(O) t＝i+F0+F0 )) Cascade to generate.

[0057] As described in this article, the image (e.g., the left anterior image FL(O)) t＝i The front right image, rear left image, and / or rear right image can have a lateral size of Px and a longitudinal size of Py. Py can be the longitudinal displacement of a vehicle, equivalent to the number of pixels moved in consecutive frames. Depending on various aspects, the longitudinal size Py can correspond to the number of pixels moved in two consecutive frames captured by the respective camera. In the case of concatenating two or more images, the (left and / or right) baseline ground truth image (and optionally the concatenated front left image, concatenated front right image, concatenated rear left image, and / or concatenated rear right image) can be generated by buffering two or more top view images. The baseline ground truth image can, for example, have a lateral size X equal to Px. g A baseline ground truth image can, for example, have a vertical dimension Y. gThe vertical size Y of the baseline ground truth image g This can correspond to the product of the vertical size Py of the corresponding image and the number of images concatenated. For example, in the case of concatenating F0 images, Yg can correspond to Py*F0. Depending on some aspects, R0 can be equal to F0. In this case, the following conditional relationship can exist: (Px == X) g )&&(F0*Py==Y g )&&(R0*Py==Y g ).

[0058] Various aspects relate to the evaluation of calibration. This calibration can be online calibration (e.g., to accommodate online variations as described herein). As described herein, ground truth images(s) can be generated using only data generated by the camera. For example, this allows for the generation of ground truth data with reduced device and computational costs. Other methods (such as using reprojection errors) are not suitable for evaluating online calibration because using reprojection errors requires geometric knowledge of the scene (which may be unknown and often not available online).

[0059] Figure 3 A flowchart is shown of a method 300 for evaluating the calibration of at least one camera according to various aspects. Method 300 may include receiving a first longitudinal image from a longitudinal camera (e.g., front camera 102 or rear camera 104) showing the surrounding environment in front of or behind a vehicle at a first time point (e.g., at t = i or t 0i + F0 + R0) (in 302). Method 300 may include receiving a first lateral image from a lateral camera (e.g., left-side camera 106 or right-side camera 108) showing the surrounding environment immediately adjacent to the vehicle in the lateral direction at the first time point (in 304). Method 300 may include receiving a second longitudinal image from the longitudinal camera showing the surrounding environment in front of or behind the vehicle at a second time point different from the first time point (e.g., at t = i + F0) (in 306). Method 300 may include receiving a second lateral image from a lateral camera, the second lateral image showing the surrounding environment (in 308) in the lateral direction immediately adjacent to the vehicle at a second time point. For example, a portion of the second lateral image shows a scene not shown in the second longitudinal image and shown in both the first longitudinal and first lateral images at a first time point. Method 300 may include generating a first image (e.g., a left front image FL(O)) using the first longitudinal and first lateral images. t＝i ), left rear image RL(O) t＝i+F0+R0 ), right anterior image FR(O) t＝i ) or right rear image RR(O t＝i+F0+R0The first image shows a top-down view of the scene at a first time point (in 310). Method 300 may include generating a second image using a second lateral image (e.g., a left-side image L(O)). t＝i+F0 ) or the right-hand image R(O t＝i+F0 The second image shows a top view of the scene at a second point in time (in 312).

[0060] Method 300 may include determining whether the longitudinal and lateral cameras are correctly calibrated by comparing a first image with a second image (in 314). Capturing images, generating top-view images, generating images from top-view images, comparing two images to each other, etc., can be as described herein. Figures 1A to 2D Perform it as described.

[0061] The various aspects of this disclosure will be described below. It should be noted that aspects described with reference to a control device or vehicle can be implemented in the method accordingly, and vice versa.

[0062] Example 1 is a vehicle comprising: a longitudinal camera configured to capture a longitudinal image showing the surrounding environment in front of or behind the vehicle; a lateral camera configured to capture a lateral image showing the surrounding environment adjacent to the vehicle in a lateral direction, wherein the longitudinal camera and the lateral camera have partially overlapping fields of view such that a portion of the longitudinal image and a portion of the lateral image show the same area in the surrounding environment of the vehicle; and one or more processors configured to: receive a first longitudinal image from the longitudinal camera showing the surrounding environment in front of or behind the vehicle at a first time point; and receive a first lateral image from the lateral camera, the first longitudinal image showing the surrounding environment in front of or behind the vehicle at a first time point; and receive a first lateral image from the lateral camera, the first longitudinal image showing the surrounding environment in front of or behind the vehicle at a first time point; and receive a first lateral image from the lateral camera showing the lateral ... A lateral image shows the surrounding environment adjacent to the vehicle in the lateral direction at a first time point; a second lateral image is received from the lateral camera, showing the surrounding environment adjacent to the vehicle in the lateral direction at a second time point different from the first time point, wherein a portion of the second lateral image shows the scene shown in the first longitudinal image and the first lateral image at the first time point; a first image is generated using the first longitudinal image and the first lateral image, showing a top view of the scene at the first time point; a second image is generated using the second lateral image, showing a top view of the scene at the second time point; and the longitudinal camera and the lateral camera are determined to be correctly calibrated by comparing the first image with the second image.

[0063] This allows for the generation of test images (as test data) and baseline ground truth images (as baseline ground truth data) during operation (i.e., online), enabling the evaluation of sensor calibration. For example, this allows determining whether a sensor is correctly calibrated or whether it should be recalibrated.

[0064] In Example 2, the subject matter as described in Example 1 may optionally include: the one or more processors being configured to: determine a comparison value representing the difference between the first image and the second image by comparing the first image with the second image; determine whether the comparison value representing the difference between the first image and the second image is equal to or less than a predefined difference threshold; and, if it is determined that the comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold, determine that the longitudinal camera and the lateral camera are correctly calibrated.

[0065] This example details how the first and second images can be compared to each other.

[0066] In Example 3, the subject matter as described in Example 1 or 2 may optionally include: the one or more processors being configured to: generate a first top-view image using the first longitudinal image and the first lateral image, the first top-view image showing a top view of the surrounding environment adjacent to the vehicle in the lateral direction and the surrounding environment in front of or behind the vehicle at least at the first time point, and to generate the first image by extracting a portion of the first top-view image; and / or generate a second top-view image using the second lateral image, the second top-view image showing a top view of the surrounding environment adjacent to the vehicle in the lateral direction at least at the first time point, and to generate the second image by extracting a portion of the second top-view image.

[0067] This allows (e.g., continuously) the generation of test images (as test data) and baseline ground truth images (as baseline ground truth data) from top-view images.

[0068] In Example 4, the subject matter as described in any one of Examples 1 to 3 may optionally include: the first image and the second image having the same image size.

[0069] This helps in comparing the first image with the second image.

[0070] In Example 5, the subject matter as described in any one of Examples 1 to 4 may optionally include: the longitudinal camera is configured to capture consecutive longitudinal images at a predefined frame rate; wherein the size of the first image and / or the second image in the longitudinal direction perpendicular to the lateral direction is substantially equal to the number of pixels moved in the two consecutive longitudinal images.

[0071] This allows for the continuous generation of test data images that illustrate the corresponding scenarios, with no overlap between two consecutive images.

[0072] In Example 6, the subject as described in any one of Examples 1 to 5 may optionally include: the one or more processors being configured to receive a second longitudinal image from the longitudinal camera, the second longitudinal image showing the surrounding environment in front of or behind the vehicle at the second time point, wherein the scene is not shown in the second longitudinal image.

[0073] In Example 7, the subject matter as described in any one of Examples 1 to 6 may optionally include: the longitudinal camera is a front camera configured to capture a frontal image as the longitudinal image showing the surrounding environment in front of the vehicle; wherein the vehicle further includes a rear camera configured to capture a rear image showing the surrounding environment behind the vehicle, wherein the rear camera and the lateral camera have partially overlapping fields of view such that a portion of the rear image and another portion of the lateral image show the same area in the surrounding environment of the vehicle (wherein, the other portion of the lateral image is different from the portion of the lateral image); wherein the one or more processors are configured to: from the rear A first rearward image is received by a lateral camera, showing the surrounding environment behind the vehicle at a third time point, wherein the second time point is time-between the first and second time points; a third lateral image is received from a lateral camera, showing the surrounding environment adjacent to the vehicle in the lateral direction at the third time point, wherein the scene is shown in the first rearward image and the third lateral image at the third time point; a third image is generated using the rearward image and the third lateral image, showing a top-down view of the scene at the third time point; the rearward camera and the lateral camera are compared with the second image to determine whether they are correctly calibrated.

[0074] This allows for the generation of another test image (as test data) associated with the same ground truth image as the previous test image. Therefore, each ground truth image can be a ground truth for two different test images (one associated with the area in front of the vehicle, and the other with the area behind the vehicle), thus improving the efficiency of data generation. This further allows for the determination of whether the camera generating the ground truth image has been incorrectly calibrated.

[0075] In Example 8, the subject matter as described in Example 7 may optionally include: the one or more processors being configured to: determine another comparison value representing the difference between the third image and the second image by comparing the third image with the second image; and determine whether the other comparison value representing the difference between the third image and the second image is equal to or less than another predefined difference threshold; and if it is determined that the other comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold, determine that the rear camera and the side camera are correctly calibrated.

[0076] This example details how the third and second images can be compared to each other.

[0077] In Example 9, the subject matter as described in Examples 2 and 8 may optionally include: the one or more processors being configured to: determine that the front camera is not properly calibrated if a comparison value representing the difference between the first image and the second image is greater than a predefined difference threshold and another comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold; and / or determine that the rear camera is not properly calibrated if a comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold and another comparison value representing the difference between the third image and the second image is greater than the other predefined difference threshold.

[0078] This details an example of how to determine which sensor is correctly calibrated and which is not.

[0079] In Example 10, the subject matter described in any of Examples 1 to 9, in conjunction with Example 7, may optionally further include: the lateral camera is a left-facing camera configured to capture a left-facing image as the lateral image showing the surrounding environment on the left side of the vehicle; wherein the vehicle further includes a right-facing camera configured to capture a right-facing image showing the surrounding environment on the right side of the vehicle, wherein the right-facing camera and the front camera have partially overlapping fields of view such that another portion of the front image and a first portion of the right-facing image show the same area in the surrounding environment of the vehicle, and wherein the right-facing camera and the rear camera have partially overlapping fields of view such that another portion of the rear image and a second portion of the right-facing image show the same area in the surrounding environment of the vehicle; wherein the one or more processors are configured to: receive from the right-facing camera: a first right-facing image showing the surrounding environment on the right side of the vehicle at the first time point, and a right-facing image showing the surrounding environment on the right side of the vehicle at the first time point. A second right-facing image showing the surrounding environment to the right of the vehicle at a second time point, and a third right-facing image showing the surrounding environment to the right of the vehicle at a third time point, wherein a portion of the second right-facing image shows another scene shown in the first longitudinal image and the first right-facing image at the first time point and in the first rear image and the third right-facing image at the third time point; and a fourth image is generated using the first longitudinal image and the first right-facing image, showing a top view of the other scene at the first time point; a fifth image is generated using the second right-facing image, showing a top view of the other scene at the second time point; a sixth image is generated using the first rear image and the third right-facing image, showing a top view of the other scene at the third time point; and it is determined whether the front camera, the rear camera, and / or the right-facing camera are correctly calibrated by comparing the fourth image with the fifth image and comparing the sixth image with the fifth image.

[0080] This allows for the stable generation of test images and benchmark ground truth images on both sides (left and right) of the vehicle, thereby further improving the efficiency of generating test data and benchmark ground truth data.

[0081] In Example 11, the subject matter as described in Examples 3 and 10 may optionally include: the one or more processors being configured to generate the first top-view image using the first longitudinal image, the first rear image, the first lateral image, and the first right-side image, such that the first top-view image shows a top-view view of the surrounding environment completely around the vehicle.

[0082] This further improves the efficiency of data generation, because the test images (left and right) can be generated from the same top-view image, and the baseline ground truth images (left and right) can also be generated from the same top-view image.

[0083] In Example 12, the subject matter described in conjunction with Examples 8 or 9, as in Examples 10 or 11, may optionally include: the one or more processors being configured to: determine a first comparison value representing the difference between the fourth image and the fifth image by comparing the fourth image with the fifth image; determine whether the first comparison value representing the difference between the fourth image and the fifth image is equal to or less than a first predefined difference threshold; determine a second comparison value representing the difference between the sixth image and the fifth image by comparing the sixth image with the fifth image; determine whether the second comparison value representing the difference between the sixth image and the fifth image is equal to or less than a second predefined difference threshold; and If the following conditions are met: the comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold; another comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold, it is determined that the front camera is not correctly calibrated; and if the first comparison value representing the difference between the fourth image and the fifth image is greater than the first predefined difference threshold and the second comparison value representing the difference between the sixth image and the fifth image is greater than the second predefined difference threshold, it is determined that the right-facing camera is not correctly calibrated (and optionally, it is further determined that the rear camera, the front camera, and the left-facing camera are correctly calibrated).

[0084] This example details how to determine if the right-facing camera used to generate the baseline ground truth image has been incorrectly calibrated.

[0085] In Example 13, the subject matter as described in any one of Examples 1 to 12 may optionally include: the one or more processors being configured to generate a difference image representing the difference between the two images when comparing them to each other, and to provide control instructions to control a display device to display the difference image.

[0086] This allows the driver (or co-driver) to be notified of sensor calibration.

[0087] In Example 14, the subject matter as described in any one of Examples 1 to 13 may optionally include: the one or more processors being configured to rotate and / or shift at least one of the two images (e.g., via rigid registration) before comparing the two images with each other.

[0088] This allows for the consideration of use cases where vehicles do not move in a straight line but may turn left or right.

[0089] In Example 15, the subject matter as described in any one of Examples 1 to 14 may optionally include: the one or more processors being configured to provide control instructions for recalibrating the camera if it is determined that the camera (e.g., the front camera, the rear camera, the left-facing camera, and / or the right-facing camera) is not properly calibrated.

[0090] Example 16 is a method for evaluating the calibration of one or more cameras mounted on a vehicle, the method comprising: receiving a first longitudinal image from a longitudinal camera, the first longitudinal image showing the surrounding environment in front of or behind the vehicle at a first time point; receiving a first lateral image from a lateral camera, the first lateral image showing the surrounding environment adjacent to the vehicle in a lateral direction at the first time point; receiving a second longitudinal image from the longitudinal camera, the second longitudinal image showing the surrounding environment in front of or behind the vehicle at a second time point different from the first time point; and receiving a second lateral image from the lateral camera, the second lateral image... The image shows the surrounding environment adjacent to the vehicle in the lateral direction at the second time point, wherein a portion of the second lateral image shows the scene not shown in the second longitudinal image and shown in the first longitudinal image and the first lateral image at the first time point; a first image is generated using the first longitudinal image and the first lateral image, the first image showing a top view of the scene at the first time point; a second image is generated using the second lateral image, the second image showing a top view of the scene at the second time point; and the longitudinal camera and the lateral camera are determined to be correctly calibrated by comparing the first image with the second image.

[0091] In Example 17, the method described in Example 16 may optionally further include: determining a comparison value representing the difference between the first image and the second image by comparing the first image with the second image; determining whether the comparison value representing the difference between the first image and the second image is equal to or less than a predefined difference threshold; and determining that the longitudinal camera and the lateral camera are correctly calibrated if the comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold.

[0092] In Example 18, the method described in Example 16 or 17 may optionally further include: generating a first top-view image using the first longitudinal image and the first lateral image, the first top-view image showing a top view of the surrounding environment adjacent to the vehicle in the lateral direction at least at the first time point and the surrounding environment in front of or behind the vehicle, and generating the first image by extracting a portion of the first top-view image; and / or generating a second top-view image using the second lateral image, the second top-view image showing a top view of the surrounding environment adjacent to the vehicle in the lateral direction at least at the first time point, and generating the second image by extracting a portion of the second top-view image.

[0093] In Example 19, the subject matter as described in any one of Examples 16 to 18 may optionally include: the first image and the second image having the same image size.

[0094] In Example 20, the subject matter as described in any one of Examples 16 to 19 may optionally include: the longitudinal camera is configured to capture consecutive longitudinal images at a predefined frame rate; wherein the size of the first image and / or the second image in the longitudinal direction perpendicular to the lateral direction is substantially equal to the number of pixels moved in the two consecutive longitudinal images.

[0095] In Example 21, the subject matter as described in any one of Examples 16 to 20 may optionally include: the longitudinal camera is a front camera; wherein the method further includes: receiving a first rear image from a rear camera, the first rear image showing the surrounding environment behind the vehicle at a third time point, wherein the second time point is temporally between the first time point and the second time point; receiving a third lateral image from the lateral camera, the third lateral image showing the surrounding environment adjacent to the vehicle in the lateral direction at the third time point, wherein the scene is shown in the first rear image and the third lateral image at the third time point; generating a third image using the rear image and the third lateral image, the third image showing a top view of the scene at the third time point; and determining whether the rear camera and the lateral camera are correctly calibrated by comparing the third image with the second image.

[0096] In Example 22, the method described in Example 21 may optionally further include: receiving a second rear image from the rear camera, the second rear image showing the surrounding environment behind the vehicle at the second time point, wherein the scene is not shown in the second rear image.

[0097] In Example 23, the method described in Examples 21 or 22 may optionally further include: determining another comparison value representing the difference between the third image and the second image by comparing the third image with the second image; determining whether the other comparison value representing the difference between the third image and the second image is equal to or less than another predefined difference threshold; and determining that the rear camera and the side camera are correctly calibrated if the other comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold.

[0098] In Example 24, the method described in Examples 17 and 23 may optionally further include: determining that the front camera is not properly calibrated if the comparison value representing the difference between the first image and the second image is greater than the predefined difference threshold and another comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold; and / or determining that the rear camera is not properly calibrated if the comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold and another comparison value representing the difference between the third image and the second image is greater than the other predefined difference threshold.

[0099] In Example 25, the subject matter described in any of Examples 16 to 24, in conjunction with the conditions of Example 21, may optionally further include: the lateral camera is a left-facing camera for capturing a left-facing image showing the surrounding environment to the left of the vehicle, and the method further includes: receiving from a right-facing camera: a first right-facing image showing the surrounding environment to the right of the vehicle at a first time point, a second right-facing image showing the surrounding environment to the right of the vehicle at a second time point, and a third right-facing image showing the surrounding environment to the right of the vehicle at a third time point, wherein a portion of the second right-facing image is shown in the first longitudinal image, the first right-facing image, the first rear image, and the third right-facing image and is shown in the... Another scene not shown in the second longitudinal image (and, in conjunction with Example 22, not shown in the second rear image); and a fourth image generated using the first longitudinal image and the first right-facing image, the fourth image showing a top view of the other scene at the first time point; a fifth image generated using the second right-facing image, the fifth image showing a top view of the other scene at the second time point; a sixth image generated using the first rear image and the third right-facing image, the sixth image showing a top view of the other scene at the third time point; and determining whether the front camera, the rear camera, and / or the right-facing camera are correctly calibrated by comparing the fourth image with the fifth image and comparing the sixth image with the fifth image.

[0100] In Example 26, the method described in Examples 18 and 25 may optionally further include generating the first top-view image using the first longitudinal image, the first rear image, the first lateral image, and the first right-side image, such that the first top-view image shows a top view of the surrounding environment completely around the vehicle.

[0101] In Example 27, the method described in conjunction with Examples 23 or 24, as in Examples 25 or 26, may optionally further include: determining a first comparison value representing the difference between the fourth image and the fifth image by comparing the fourth image with the fifth image; determining whether the first comparison value representing the difference between the fourth image and the fifth image is equal to or less than a first predefined difference threshold; determining a second comparison value representing the difference between the sixth image and the fifth image by comparing the sixth image with the fifth image; determining whether the second comparison value representing the difference between the sixth image and the fifth image is equal to or less than a second predefined difference threshold; and determining the following In each of the following cases: if the comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold; if another comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold, it is determined that the front camera is not correctly calibrated; and if the first comparison value representing the difference between the fourth image and the fifth image is greater than the first predefined difference threshold and the second comparison value representing the difference between the sixth image and the fifth image is greater than the second predefined difference threshold, it is determined that the right-facing camera is not correctly calibrated (and optionally further determined that the rear camera, the front camera, and the left-facing camera are correctly calibrated).

[0102] In Example 28, the method as described in any one of Examples 16 to 27 may optionally further include: when two images are compared to each other, generating a difference image representing the difference between the two images, and providing control instructions to control a display device to display the difference image.

[0103] In Example 29, the method as described in any one of Examples 1 to 28 may optionally further include rotating and / or shifting at least one of the two images (e.g., via rigid registration) before comparing the two images with each other.

[0104] In Example 30, the method as described in any one of Examples 16 to 29 may optionally further include: providing control instructions for recalibrating the camera if it is determined that the camera (e.g., the front camera, the rear camera, the left-facing camera, and / or the right-facing camera) is not properly calibrated.

[0105] Example 31 is a non-transitory computer-readable medium having instructions recorded thereon that, when executed by a processor of a vehicle, cause the processor to perform the method according to any one of Examples 16 to 30.

[0106] Example 32 is a control device for controlling a vehicle, the control device comprising: one or more processors configured to: receive a first longitudinal image from a longitudinal camera of the vehicle, the first longitudinal image showing the surrounding environment in front of or behind the vehicle at a first time point; receive a first lateral image from a lateral camera of the vehicle, the first lateral image showing the surrounding environment adjacent to the vehicle in a lateral direction at the first time point; receive a second lateral image from the lateral camera, the second lateral image showing the surrounding environment adjacent to the vehicle in a lateral direction at a second time point different from the first time point, wherein a portion of the second lateral image shows the scene shown in the first longitudinal image and the first lateral image at the first time point; generate a first image using the first longitudinal image and the first lateral image, the first image showing a top view of the scene at the first time point; generate a second image using the second lateral image, the second image showing a top view of the scene at the second time point; and determine whether the longitudinal camera and the lateral camera are correctly calibrated by comparing the first image with the second image.

[0107] In Example 33, the subject matter as described in Example 32 may optionally include: the one or more processors being configured to: determine a comparison value representing the difference between the first image and the second image by comparing the first image with the second image; determine whether the comparison value representing the difference between the first image and the second image is equal to or less than a predefined difference threshold; and, if it is determined that the comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold, determine that the longitudinal camera and the lateral camera are correctly calibrated.

[0108] In Example 34, the subject matter as described in Example 32 or 33 may optionally include: the one or more processors being configured to: generate a first top-view image using the first longitudinal image and the first lateral image, the first top-view image showing a top view of the surrounding environment adjacent to the vehicle in the lateral direction at least at the first time point and the surrounding environment in front of or behind the vehicle, and to generate the first image by extracting a portion of the first top-view image; and / or generate a second top-view image using the second lateral image, the second top-view image showing a top view of the surrounding environment adjacent to the vehicle in the lateral direction at least at the first time point, and to generate the second image by extracting a portion of the second top-view image.

[0109] In Example 35, the subject matter as described in any one of Examples 32 to 34 may optionally include: the first image and the second image having the same image size.

[0110] In Example 36, the subject matter as described in any one of Examples 32 to 35 may optionally include: the longitudinal camera is configured to capture consecutive longitudinal images at a predefined frame rate; wherein the size of the first image and / or the second image in the longitudinal direction perpendicular to the lateral direction is substantially equal to the number of pixels moved in the two consecutive longitudinal images.

[0111] In Example 37, the subject as described in any one of Examples 32 to 36 may optionally include: the one or more processors being configured to receive a second longitudinal image from the longitudinal camera, the second longitudinal image showing the surrounding environment in front of or behind the vehicle at the second time point, wherein the scene is not shown in the second longitudinal image.

[0112] In Example 38, the subject matter as described in any one of Examples 32 to 37 may optionally include: the longitudinal camera is a front camera configured to capture a frontal image as the longitudinal image showing the surrounding environment in front of the vehicle; wherein the one or more processors are configured to: receive a first rear image from a rear camera of the vehicle, the first rear image showing the surrounding environment behind the vehicle at a third time point, wherein the second time point is temporally between the first time point and the second time point; receive a third lateral image from a lateral camera, the third lateral image showing the surrounding environment adjacent to the vehicle in the lateral direction at the third time point, wherein the scene is shown in the first rear image and the third lateral image at the third time point; generate a third image using the rear image and the third lateral image, the third image showing a top view of the scene at the third time point; and determine whether the rear camera and the lateral camera are correctly calibrated by comparing the third image with the second image.

[0113] In Example 39, the subject matter as described in Example 38 may optionally include: the one or more processors being configured to: determine another comparison value representing the difference between the third image and the second image by comparing the third image with the second image; and determine whether the other comparison value representing the difference between the third image and the second image is equal to or less than another predefined difference threshold; and if it is determined that the other comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold, determine that the rear camera and the side camera are correctly calibrated.

[0114] In Example 40, the subject matter as described in Examples 33 and 39 may optionally include: the one or more processors being configured to: determine that the front camera is not properly calibrated if a comparison value representing the difference between the first image and the second image is greater than a predefined difference threshold and another comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold; and / or determine that the rear camera is not properly calibrated if a comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold and another comparison value representing the difference between the third image and the second image is greater than the other predefined difference threshold.

[0115] In Example 41, the subject matter described in any of Examples 32 to 40, in conjunction with Example 38, may optionally include: the lateral camera is a left-facing camera, configured to capture a left-facing image as the lateral image showing the surrounding environment to the left of the vehicle; and the one or more processors are configured to: receive from the right-facing camera: a first right-facing image showing the surrounding environment to the right of the vehicle at a first time point, a second right-facing image showing the surrounding environment to the right of the vehicle at a second time point, and a third right-facing image showing the surrounding environment to the right of the vehicle at a third time point, wherein a portion of the second right-facing image shows the first longitudinal image and the first right-facing image at the first time point. Another scene is shown in the side view image and in the first rear view image and the third right view image at the third time point; and a fourth image is generated using the first longitudinal image and the first right view image, which shows a top view of the other scene at the first time point; a fifth image is generated using the second right view image, which shows a top view of the other scene at the second time point; a sixth image is generated using the first rear view image and the third right view image, which shows a top view of the other scene at the third time point; and it is determined whether the front camera, the rear camera, and / or the right view camera are correctly calibrated by comparing the fourth image with the fifth image and comparing the sixth image with the fifth image.

[0116] In Example 42, the subject matter as described in Examples 34 and 41 may optionally include: the one or more processors being configured to generate the first top-view image using the first longitudinal image, the first rear image, the first lateral image, and the first right-side image, such that the first top-view image shows a top-view view of the surrounding environment completely around the vehicle.

[0117] In Example 43, the subject matter described in conjunction with Examples 41 or 42 and Examples 39 or 40 may optionally further include: the one or more processors being configured to: determine a first comparison value representing the difference between the fourth image and the fifth image by comparing the fourth image with the fifth image; determine whether the first comparison value representing the difference between the fourth image and the fifth image is equal to or less than a first predefined difference threshold; determine a second comparison value representing the difference between the sixth image and the fifth image by comparing the sixth image with the fifth image; and determine whether the second comparison value representing the difference between the sixth image and the fifth image is equal to or less than a second predefined difference threshold; Furthermore, if the following conditions are met: the comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold; another comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold, it is determined that the front camera is not correctly calibrated; and the first comparison value representing the difference between the fourth image and the fifth image is greater than the first predefined difference threshold and the second comparison value representing the difference between the sixth image and the fifth image is greater than the second predefined difference threshold, it is determined that the right-facing camera is not correctly calibrated (and optionally, it is further determined that the rear camera, the front camera, and the left-facing camera are correctly calibrated).

[0118] In Example 44, the subject matter as described in any one of Examples 32 to 43 may optionally include: the one or more processors being configured to generate a difference image representing the difference between the two images when comparing them to each other, and providing control instructions to control a display device to display the difference image.

[0119] In Example 45, the subject matter as described in any one of Examples 32 to 44 may optionally include: the one or more processors being configured to rotate and / or shift at least one of the two images (e.g., via rigid registration) before comparing the two images with each other.

[0120] In Example 46, the subject matter as described in any one of Examples 32 to 45 may optionally include: the one or more processors being configured to provide control instructions for recalibrating the camera if it is determined that the camera (e.g., the front camera, the rear camera, the left-facing camera, and / or the right-facing camera) is not properly calibrated.

[0121] Figure Labels

[0122] 12: Forward direction

[0123] 14: Backward direction

[0124] 16: Left-hand direction

[0125] 18: Right-hand direction

[0126] 20: View from above

[0127] 100: Transportation

[0128] 102: Front camera

[0129] 104: Rear Camera

[0130] 106: Left-facing camera

[0131] 108: Camera facing right

[0132] 110: One or more processors

[0133] 112: Foreground section

[0134] 114: Rear section

[0135] 116: Left-side portion

[0136] 118: Right-side portion

[0137] 120: Front left part

[0138] 122: Right front part

[0139] 124: Left rear part

[0140] 126: Right rear part

[0141] 200: Area

[0142] 300: Method

[0143] 302-314: Methodological Features

Claims

1. A control device for controlling a vehicle, the control device comprising: • One or more processors (110), which are configured to: ○ Receive a first longitudinal image from a longitudinal camera (102, 104) of a vehicle (100), the first longitudinal image showing the surrounding environment in front of or behind the vehicle (100) at a first time point; ○ Receive a first lateral image from the side camera (106, 108) of the vehicle (100), the first lateral image showing the surrounding environment of the vehicle (100) in the lateral direction (16, 18) at the first time point; ○ Receive a second lateral image from the lateral camera (106, 108), the second lateral image showing the surrounding environment of the vehicle (100) in the lateral direction (16, 18) at a second time point different from the first time point, wherein a portion of the second lateral image shows the scene shown in the first longitudinal image and the first lateral image at the first time point; A first image is generated using the first longitudinal image and the first lateral image, the first image showing a top view of the scene at the first time point; Using this second lateral image, a second image is generated, which shows a top-down view of the scene at that second point in time; and ○ The first image is compared with the second image to determine whether the longitudinal camera (102, 104) and the lateral camera (106, 108) are correctly calibrated.

2. The control device according to claim 1, in, The one or more processors (110) are configured to: • A comparison value representing the difference between the first image and the second image is determined by comparing the first image with the second image; • Determine whether the comparison value representing the difference between the first image and the second image is equal to or less than a predefined difference threshold; and • If the comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold, it is determined that the longitudinal camera (102, 104) and the lateral camera (106, 108) are correctly calibrated.

3. The control device according to claim 1 or 2, in, The one or more processors (110) are configured to: • A first top-view image is generated using the first longitudinal image and the first lateral image. The first top-view image shows a top view of the surrounding environment immediately adjacent to the vehicle (100) in the lateral direction (16, 18) and the surrounding environment in front of or behind the vehicle (100) at least at the first time point, and the first image is generated by extracting a portion of the first top-view image; and / or • A second top-view image is generated using the second lateral image, which shows a top view of the surrounding environment of the vehicle (100) in the lateral direction (16, 18) at least at the first time point, and the second image is generated by extracting a portion of the second top-view image.

4. The control device according to any one of claims 1 to 3, in, The first image and the second image have the same image size.

5. The control device according to any one of claims 1 to 4, ·in, The longitudinal camera (102, 104) is configured to capture consecutive longitudinal images at a predefined frame rate; • Wherein, the size of the first image and / or the second image in the longitudinal direction perpendicular to the lateral direction (16, 18) is substantially equal to the number of pixels moved in the two consecutive longitudinal images.

6. The control device according to any one of claims 1 to 5, ·in, The longitudinal camera (102, 104) is a front camera, which is configured to capture a frontal image as a longitudinal image showing the surrounding environment in front of the vehicle (100); • Wherein, the one or more processors (110) are configured as follows: ○ Receive a first rear image from the rear camera (104) of the vehicle (100), the first rear image showing the surrounding environment behind the vehicle (100) at a third time point, wherein the second time point is time-between the first time point and the second time point; ○ A third lateral image is received from the lateral camera (106, 108), which shows the surrounding environment of the vehicle (100) in the lateral direction (16, 18) at the third time point, wherein the scene is shown in the first rear image and the third lateral image at the third time point; Using the rear image and the third lateral image, a third image is generated, which shows a top view of the scene at that third point in time; ○ The rear camera (104) and the side camera (106, 108) are correctly calibrated by comparing the third image with the second image.

7. The control device according to claim 6, in, The one or more processors (110) are configured to: • Another comparison value representing the difference between the third image and the second image is determined by comparing the third image with the second image; and • Determine whether another comparison value representing the difference between the third image and the second image is equal to or less than another predefined difference threshold; and • If the other comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold, it is determined that the rear camera (104) and the side camera (106, 108) are correctly calibrated.

8. The control device according to claims 2 and 7, in, The one or more processors (110) are configured to: If a comparison value representing the difference between the first image and the second image is greater than a predefined difference threshold, and another comparison value representing the difference between the third image and the second image is equal to or less than the other predefined difference threshold, it is determined that the front camera is not properly calibrated; and / or • If the comparison value representing the difference between the first image and the second image is equal to or less than the predefined difference threshold and another comparison value representing the difference between the third image and the second image is greater than the other predefined difference threshold, it is determined that the rear camera (104) is not properly calibrated.

9. The control device according to any one of claims 1 to 8, in conjunction with claim 6. ·in, The side camera (106, 108) is a left-facing camera (106) configured to capture a left-facing image as a side image showing the surrounding environment to the left of the vehicle (100); • Wherein, the one or more processors (110) are configured as follows: ○Received from the right side to the camera (108): ■ First right-facing image, which shows the surrounding environment to the right of the vehicle (100) at the first time point. ■ A second right-side view image, which shows the surrounding environment to the right of the vehicle (100) at the second time point, and ■ The third right-side view shows the surrounding environment to the right of the vehicle (100) at the third time point. ■ In this second right-facing image, a portion shows another scene, which is shown in the first longitudinal image and the first right-facing image at the first time point and in the first rear image and the third right-facing image at the third time point; and Using the first longitudinal image and the first right-facing image, a fourth image is generated, which shows a top view of the other scene at the first time point; Using the second right-facing image, a fifth image is generated, which shows a top-down view of the other scene at that second point in time; Using the first rear view and the third right-facing view, a sixth image is generated, which shows a top-down view of the other scene at the third time point; and ○ By comparing the fourth image with the fifth image and the sixth image with the fifth image, it is determined whether the front camera, the rear camera (104), and / or the right-facing camera (108) are correctly calibrated.

10. The control device according to claims 3 and 9, in, The one or more processors (110) are configured to: • The first top view image is generated using the first longitudinal image, the first rear image, the first lateral image, and the first right-side image, such that the first top view image shows a top view of the surrounding environment completely around the vehicle (100).

11. The control device according to claim 9 or 10 in conjunction with claim 7 or 8, in, The one or more processors (110) are configured to: • A first comparison value representing the difference between the fourth image and the fifth image is determined by comparing the fourth image with the fifth image; • Determine whether the first comparison value, representing the difference between the fourth image and the fifth image, is equal to or less than a first predefined difference threshold; • A second comparison value representing the difference between the sixth image and the fifth image is determined by comparing the sixth image with the fifth image; • Determine whether the second comparison value, representing the difference between the sixth image and the fifth image, is equal to or less than a second predefined difference threshold; and • Under the following conditions: ○ indicates that the comparison value of the difference between the first image and the second image is equal to or less than the predefined difference threshold. ○ indicates that if the difference between the third image and the second image is equal to or less than another predefined difference threshold, it is determined that the front camera has not been properly calibrated. ○ indicates that the first comparison value of the difference between the fourth image and the fifth image is greater than the first predefined difference threshold, and indicates that the second comparison value of the difference between the sixth image and the fifth image is greater than the second predefined difference threshold. It was determined that the right-facing camera (108) was not properly calibrated.

12. The control device according to any one of claims 1 to 11, in, The one or more processors (110) are configured to generate a difference image representing the difference between two images when comparing them, and to provide control instructions to control a display device to display the difference image.

13. The control device according to any one of claims 1 to 12, in, The one or more processors (110) are configured to rotate and / or shift at least one of the two images before comparing the two images with each other.

14. A means of transport (100) comprising a control device according to any one of claims 1 to 13.

15. A method (300) for evaluating the calibration of one or more cameras mounted on a vehicle, the method comprising: • Receive a first longitudinal image from a longitudinal camera, the first longitudinal image showing the surrounding environment in front of or behind the vehicle at a first time point (302); • Receive a first lateral image from a lateral camera, the first lateral image showing the surrounding environment adjacent to the vehicle in the lateral direction at the first time point (304); • Receive a second longitudinal image from the longitudinal camera, the second longitudinal image showing the surrounding environment in front of or behind the vehicle at a second time point different from the first time point (306); • Receive a second lateral image from the lateral camera, the second lateral image showing the surrounding environment adjacent to the vehicle in the lateral direction at the second time point, wherein a portion of the second lateral image shows a scene not shown in the second longitudinal image and shown in the first longitudinal image and the first lateral image at the first time point (308); • A first image is generated using the first longitudinal image and the first lateral image, the first image showing a top view of the scene at the first time point (310); • Using the second lateral image, a second image is generated, which shows a top-down view of the scene at that second time point (312); and • Determine whether the longitudinal camera and the lateral camera are correctly calibrated by comparing the first image with the second image (314).

Images