Method for determining optical flux in camera images and vehicle

Abstract

The invention relates to a method for determining the optical flux in camera images by means of an in-vehicle processing unit, wherein the camera images are recorded with at least one camera (2) mounted on a vehicle (1) and oriented towards the environment. The method according to the invention is characterized in that the in-vehicle processing unit determines the current speed of movement of the vehicle (1) and compares it with a defined threshold value, wherein, as soon as the speed of movement falls below the threshold value, the processing unit controls an actuator for moving the camera (2) relative to the vehicle structure and / or an optomechanical image stabilizer associated with the image sensor of the camera (2) for moving the image sensor.

Description

[0001] The invention relates to a method for determining the optical flux in camera images by a vehicle-internal computing unit of the type defined in more detail in the preamble of claim 1, and to a vehicle for carrying out the method.

[0002] Driver assistance systems enhance comfort and safety for the driver. For example, a parking assistant can support the driver when maneuvering a vehicle during a parking maneuver. This can involve determining the distance between the vehicle and surrounding objects, for instance, using ultrasonic sensors, and alerting the driver with beeping sounds when approaching obstacles. Furthermore, the vehicle may be equipped with one or more cameras, allowing the driver to see areas that are otherwise difficult to see, such as the area behind the vehicle. Modern vehicles are also capable of automatically controlling the vehicle's longitudinal and / or lateral movement during a parking maneuver.

[0003] Situations can arise where the detection accuracy of the sensor systems used is limited or reduced. Therefore, it is desirable to use additional sensor systems to detect the vehicle's position in its surroundings within the context of a parking assistant, in order to provide redundancy. In particular, the most accurate possible distance detection is desired. While radar and LiDAR sensors can determine distances and relative speeds relatively accurately, such sensor systems are often not used at close range. These sensor systems are expensive and often only installed in conjunction with other ADAS systems. Furthermore, these sensor systems primarily serve to detect objects in the environment while driving and, due to their specific design, have a field of view and a vehicle arrangement that provides insufficient coverage for detecting objects while parked.

[0004] It is therefore desirable to specify means by which the relative position of the vehicle to its surroundings can be reliably determined in a simple and cost-effective manner during a parking maneuver.

[0005] It is known to calculate the motion of a camera using the so-called optical flow. For this purpose, an image sequence is recorded with the moving camera, and the optical flow in this image sequence is determined. The optical flow is the vector field of the velocity of visible points in the camera images underlying the image sequence, projected onto the image plane.

[0006] Determining the rotational speed of a helicopter using optical flow is known, for example, from: G. Sabiron, P. Chavent, T. Raharijaona, P. Fabiani and F. Ruffier, “Low-speed optic-flow sensor onboard an unmanned helicopter flying outside over fields,” 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, Germany, 2013, pp. 1742-1749, doi:10.1109 / ICRA.2013.6630806.

[0007] The detection of parking spaces using a system based on optical flow analysis and integrated into a vehicle is also known from: Free Parking Space Detection Using Optical Flow-based Euclidean 3D Reconstruction, Suhr, Jae & Bae, Kwanghyuk & Kim, Jaihie & Jung, Ho Gi. (2007). 563-566. MVA2007 IAPR Conference on Machine Vision Applications, May 16-18, 2007, Tokyo, Japan.

[0008] Furthermore, DE 10 2017 200 278 A1 discloses a method for determining the direction of movement of a camera. The camera is fixed to a vehicle. To validate the optical flow present in an image sequence recorded by the camera, the method additionally considers the vehicle's speed of movement. If movement is detected based on the optical flow and the vehicle is stationary, this indicates that it is not the vehicle, but rather an observed environmental object, such as a roller door, that is moving.

[0009] DE 10 2016 208 199 A1 discloses a method for a driver assistance system of a vehicle, wherein a distance of at least one camera, which is arranged in a lateral area of ​​the vehicle and serves to capture image data from a lateral and / or rear vehicle environment, is changed by means of an electronic control unit depending on the operation of the vehicle with a trailer relative to the vehicle.

[0010] German patent application DE10 2018 114 229 A1 describes a method for determining the motion state of an object in the environment of a motor vehicle using a driver assistance system. The environment is detected by a sensor of the driver assistance system, and an image of the detected environment is generated by an electronic processing unit of the driver assistance system. The image is then evaluated with respect to its external shape using a first neural network. An optical flow generated by the electronic processing unit is evaluated with respect to the relative motion of the object to the sensor using a second neural network.

[0011] DE 10 2022 212 273 A1 relates to a method for determining the distance of a stationary vehicle from at least one object in the vicinity of the vehicle, wherein the vehicle has a camera system comprising at least one camera.

[0012] US patent 2016 / 0 140 403 A1 discloses a vehicle with a side camera that can be moved between two opposing positions. If it is determined that the current viewing direction of the side camera does not match the current direction of vehicle movement, the viewing position of the side camera can be adjusted.

[0013] DE 10 2020 208 308 A1 describes a method for detecting objects in the immediate vicinity of a vehicle, comprising the following steps: -Capture of a live image of the immediate area using an imaging sensor and detection of objects located in the immediate area by comparing the live image with a previously stored reference image. An object is inferred to be in the immediate area if a structure in the reference image is not fully visible but partially obscured by the object. Furthermore, at least two live images are captured sequentially using the imaging sensor, and moving objects in the immediate area are detected by comparing the successively captured images using a difference method of digital image processing. As an alternative or additional step for object detection within the vehicle, the imaging sensor is moved between at least two positions while the vehicle is stationary, and at least one image is captured at each position.By comparing the images captured at the different positions of the imaging sensor using a difference method or preferably using a structure-from-motion analysis, an object in the immediate vicinity of the vehicle can be detected.

[0014] German patent DE 10 2020 106 301 A1 discloses a method for determining object information about an object in the vicinity of a vehicle using at least one camera. The method enables a spatial view of the vehicle's surroundings even when the vehicle is stationary or traveling at very low speeds, using only one camera. The method comprises the following steps: -Capturing the environment with a camera from a first viewpoint and creating a first image consisting of first pixels; - Changing the camera's position and capturing the surroundings from a second position, creating a second image consisting of two pixels. Then the process continues. - Determining object information about an object in the captured environment by: - Selecting at least one first pixel in the first image and at least one second pixel in the second image, where the first and second pixels are assigned to the same object point of the object, and - Determining object coordinates of the assigned object point from first image coordinates of the first image point and second image coordinates of the second image point by triangulation, assuming a base length between the two camera positions.

[0015] To accurately determine the speed or distance of the camera system relative to surrounding objects, a sufficiently pronounced optical flow—that is, sufficiently long vectors in the underlying vector field—is required. However, this is not always the case due to the low speed of movement during parking maneuvers. Because of the vehicle's slow motion, the points tracked in successive camera images also move only slightly, especially if the objects themselves are stationary and have no intrinsic motion. This makes accurate speed or distance determination difficult or even impossible using classical methods.

[0016] The present invention is based on the objective of providing an improved method for determining the optical flow in camera images by means of a vehicle-internal computing unit, which is easy and cost-effective to implement and allows for improved speed determination or distance determination compared to the prior art.

[0017] According to the invention, this problem is solved by a method for determining the optical flux in camera images using a vehicle-integrated computing unit with the features of claim 1. Advantageous embodiments and further developments, as well as a vehicle for carrying out the method, are described in the dependent claims.

[0018] A generic method for determining the optical flux in camera images by a vehicle-integrated computing unit, wherein the camera images are recorded with at least one camera mounted on a vehicle and oriented towards the environment, provides that the vehicle-integrated computing unit determines the current speed of movement of the vehicle and compares it with a defined threshold value, wherein, as soon as the speed of movement falls below the threshold value, the computing unit controls an actuator to move the camera relative to the vehicle structure and / or an optomechanical image stabilizer associated with the camera's image sensor to move the image sensor.

[0019] As mentioned earlier, reliable speed and distance determination based on optical flow analysis requires sufficient relative motion between the camera and the observed object or environment. This is usually not the case during slow-speed driving, especially during parking maneuvers. The method analyzes the vehicle's speed to identify situations where reliable speed and distance determination based on optical flow is not possible. When such situations occur, movements are actively imposed on the vehicle-mounted camera or its image sensor. This increases the camera's relative motion to the environment, thereby increasing the velocity vectors of the tracked points in the vector field representing the optical flow.

[0020] This increases the accuracy in determining the camera speed, which also allows for a more precise determination of the camera's distance to the observed objects. Since the camera's mounting position on the vehicle is known, the relative speed between the vehicle and surrounding objects, as well as the vehicle's distance to those objects, can also be determined more reliably.

[0021] The process can be executed simultaneously for multiple camera systems on the vehicle. This allows different cameras to be directed towards different regions of the vehicle, for example to enable a 360° panoramic view.

[0022] Depending on the camera system used and its mounting position on the vehicle, developers can define a suitable threshold for the vehicle's speed. Below this threshold, active movements are imposed on the camera or image sensor to improve the accuracy of the optical flow analysis. All common actuators can be used to move the camera, such as electric motors, piezoelectric actuators, or hydraulic or pneumatic deflection. The actuator can be connected to the camera indirectly via a gearbox. Multiple actuators can also be used to impose the movements. Additionally or alternatively, the camera's image sensor can be moved by actively controlling the optomechanical image stabilizer associated with the image sensor. Such an optomechanical image stabilizer is primarily designed to compensate for slight movements or...To compensate for vibrations during image capture and obtain a shake-free camera image, the image sensor is deliberately deflected in a controlled manner, resulting in a more pronounced vector field of the optical flow.

[0023] The movement imprinted on the camera or image sensor is known. This means that the direction and magnitude of the displacement are defined. Since this information is known, its influence on the optical flux can be taken into account. Furthermore, the frequency and timing of image recording can be set to optimally utilize the image sensor's displacement, for example, by capturing two consecutive image recordings with the highest possible degree of inherent movement or a reproducible inherent movement in the form of displacement per unit time of the image sensor. In particular, the image recording frequency should be significantly higher than the frequency of, for example, a cyclic movement of the image sensor. The optical flux analysis can otherwise be performed in the established manner. For example, homogeneous surfaces with low optical flux can be neglected. This allows the ground or...For example, the wall of a garage can be filtered out. Furthermore, moving objects can be identified. This is achieved by recognizing vector fields in the optical flow with high similarity. This means that in a bounded, connected vector field region, the direction and magnitude of the optical flow are similar.

[0024] Behavior such as distance, speed, and direction of movement can be tracked and predicted, allowing for the differentiation of various road users, such as pedestrians, dogs, cyclists, and the like. When the method is used during a parking maneuver, it specifically identifies objects the vehicle is approaching. Such approach is detectable by divergent flow. This means that a corresponding vector field exists, whose velocity vectors move from the inside to the outside as the vehicle approaches the object. The relative speed and distance of the camera or vehicle to such an object are then determined.

[0025] The method according to the invention provides that the image stabilizer excites the image sensor with periodic oscillations to impose a movement in at least one spatial direction, in particular in the form of a sinusoidal oscillation, a sawtooth oscillation, or a chirp. The influence of periodic oscillations of the image sensor on the optical flux can be determined particularly easily and reliably. This, in turn, facilitates the determination of relative velocities and relative distances. Sinusoidal oscillations, sawtooth oscillations, and chirps have proven particularly effective in this regard.

[0026] According to a further advantageous embodiment of the method according to the invention, the movement comprises a rotation about an axis extending transversely to the direction of travel and / or an axis parallel to the vehicle's vertical axis. The movements imposed on the camera are thus oriented according to the vehicle's orientation or the direction of travel. This further simplifies the determination of the movement's influence on the optical flow, which, as already mentioned, allows for a more reliable determination of the aforementioned relative distances and / or relative velocities.

[0027] A further advantageous embodiment of the method according to the invention provides that the processing unit calculates a relative velocity between the vehicle and an object contained in the camera images, a distance between the vehicle and the object, and / or a time until impact with the object from the optical flow and provides these quantities as input to a downstream assistance system, in particular in the form of a parking assistant. The method according to the invention therefore does not necessarily have to be used in the context of a parking assistant, but can also be used, for example, in traffic jams or stop-and-go traffic or in other driving situations. The camera images generated by the camera can be evaluated using machine vision methods. This makes it possible to recognize and classify surrounding objects. Objects recognized in this way can be correlated with the vectors or vector fields present in the optical flow.Additionally, the time until impact with an object can be calculated. This time is also referred to as "time to collision." Corresponding relative speeds, distances, and the time to impact can then be processed by downstream assistance systems. This allows for an increase in the reliability of assistance functions such as an emergency braking assistant. As is known from the prior art, the method and system used can also be employed to detect available parking spaces.

[0028] Preferably, the assistance system, in response to at least one of the aforementioned parameters, issues a warning to the driver and / or automatically intervenes in the vehicle's longitudinal and / or lateral control. By analyzing the relative speeds between the camera / vehicle and surrounding objects, as well as distances and / or the time until impact, the vehicle's internal processing unit can detect hazardous situations. For example, if the relative speed exceeds a defined threshold, the distance falls below a defined threshold, or the time until impact falls below a defined threshold, appropriate emergency measures can be initiated.

[0029] In a first embodiment, warning messages can be issued to the driver. This alerts the driver to the hazardous situation, enabling them to initiate appropriate manual countermeasures, such as braking. These warning messages can be delivered via various transmission methods, including audible, visual, and / or haptic signals. Additionally or alternatively, automated intervention in the vehicle's control behavior is possible. This allows the vehicle to brake automatically, come to a complete stop, or perform steering maneuvers to avoid the aforementioned objects. This enhances vehicle safety.

[0030] A further advantageous embodiment of the method according to the invention provides that at least one of the following cameras is used to record the camera images: a reversing camera and / or a side mirror camera. Modern vehicles usually already have such reversing cameras and side mirror cameras. A reversing camera is a camera that is installed at the rear of the vehicle and captures the area behind the vehicle, particularly when tilted relative to the ground. Such a reversing camera can be integrated into a vehicle as part of a proven parking system or parking assistant. A side mirror camera is a camera that is integrated into one of the vehicle's side mirrors. In particular, side mirror cameras are provided in both side mirrors. Such a side mirror camera can be used to capture the left or right side of the vehicle.The use of cameras that are usually already available facilitates the simple and cost-effective implementation and execution of the method according to the invention. In general, it is also conceivable to use other cameras, such as a camera facing the front of the vehicle, for example a so-called "multipurpose camera", a rear camera, a side camera, or the like.

[0031] Preferably, to imprint the movement onto the camera, the reversing camera is moved between a retracted and an extended position in an installation recess integrated into the vehicle's outer skin, and / or the side mirror is folded in and out. As already mentioned, modern vehicles often have a reversing camera as well as side mirror cameras. These reversing cameras are usually retractable. Accordingly, the system used for extending and retracting the reversing camera can be used to imprint the aforementioned movements onto the camera. Similarly, the side mirrors of vehicles can usually be folded in and out automatically, for example, when locking the vehicle. A corresponding actuator for moving the side mirrors can be used in the inventive method to imprint the movement onto the side mirror cameras.Therefore, no additional actuators are needed to impose the movements, which further facilitates the simple and cost-effective implementation of the process.

[0032] A further advantageous embodiment of the method according to the invention provides that the optical flow is evaluated by a machine learning model, wherein the machine learning model is trained to calculate, based on the patterns contained in the optical flow, a relative velocity between the camera and an object contained in the camera images, a distance between the camera and the object, and / or a position in space of the object in the coordinate system of the vehicle or a global coordinate system, and / or a time until impact on the object, wherein, in particular, the machine learning model determines the optical flow itself from a camera image sequence recorded by the at least one camera. Patterns can be reliably recognized with the aid of machine learning models, especially in the form of artificial neural networks.This further improves the accuracy in determining the aforementioned relative speeds, distances, and impact times. The machine learning model can be trained using proven methods, particularly supervised learning. During the training process, raw images from the camera system, enriched with so-called "labels," can be fed to the machine learning model as input data. Such a label marks relevant environmental objects and includes information describing the relative distance between the camera and the object for the respective raw image. These labels can be generated using established sensor systems for capturing depth information, such as LiDAR. In a reference system, all raised objects, or objects that cannot be driven over by the vehicle, are marked, for example, by exceeding a defined geodetic height above ground.Such a label can comprise a bounding box or a freeform shape. Depending on how the determined relative distances between successive raw images change, the relative velocity and the impact time can be deduced. This information can be included as additional labels in the training dataset. These labels thus comprise four pieces of information: a marker for raised objects, the relative velocity of the camera to the objects, the position of the objects in space, and the distance to the objects. A correspondingly trained machine learning model is then integrated into the vehicle's onboard processing unit for processing the camera images. In particular, the machine learning model is trained to independently determine the optical flow from a given camera image sequence.However, the optical flow can also be determined in a preliminary step using ordinary algorithms, taking into account the movements actively imposed on the camera or image sensor.

[0033] According to a further advantageous embodiment of the method according to the invention, the processing unit is further provided that the extent of the movement is configured as a function of at least one of the following parameters: the speed of the vehicle, the presence of at least one object in at least one camera image, and / or an object property. "Extent" here refers in particular to the frequency, magnitude, stroke, and positioning speed of the movement applied to the camera. In particular, if the speed of the vehicle decreases, the frequency, magnitude, stroke, or positioning speed can be increased to counteract, compensate for, or even overcompensate for a reduction in vector lengths in the optical flow.

[0034] The degree of movement can also be changed depending on the presence of a detected object and / or an object property. Object properties include, in particular, its dimensions, structure, optical properties, and the distance between the camera or vehicle and the object.

[0035] A vehicle of this type, comprising a computing unit and at least one camera movable by means of an actuator, wherein an optomechanical image stabilizer is additionally or alternatively assigned to an image sensor of the camera, is further developed according to the invention in that the at least one camera and the computing unit are configured to carry out a method described above. Movements can thus be imposed on the camera or the image sensor by the actuator and / or the optomechanical image stabilizer. Corresponding cameras can therefore also be rigidly connected to the vehicle structure, with movements being imposed exclusively by means of the optomechanical image stabilizer. However, combining both deflection mechanisms is particularly advantageous.

[0036] The vehicle in question can be any road vehicle such as a car, truck, van, bus, or similar. Generally speaking, it is also conceivable that it could be a rail vehicle, watercraft, or aircraft.

[0037] Further advantageous embodiments of the inventive method for determining the optical flux and of the inventive vehicle also result from the exemplary embodiments, which are described in more detail below with reference to the figures.

[0038] This shows: Fig. 1 a schematic side view of a vehicle according to the invention during a parking maneuver; and Fig. 2 A schematic top view of the vehicle according to the invention during the performance of a further parking operation.

[0039] Fig. Figure 1 shows a highly schematic representation of a vehicle 1 according to the invention, which travels backwards in a direction R towards an object 3. The vehicle 1 has one or more cameras 2 for environmental sensing. In the Fig. In the embodiment shown in Figure 1, at least one reversing camera 2.1 is provided. The vehicle 1 can be equipped with a proven parking assistant that allows distance detection between vehicle 1 and object 3 using an ultrasonic sensor system. Camera images generated by the reversing camera 2.1 can be displayed on a display device (not shown) inside the vehicle.

[0040] According to the invention, the vehicle 1 performs a method according to the invention for determining the optical flux in camera images generated by the camera 2. For this purpose, camera images or a sequence of camera images generated by the camera 2 are processed by a vehicle-integrated processing unit. The vehicle-integrated processing unit has read access to a computer-readable storage medium containing machine-interpretable instructions that enable a processor to execute the method according to the invention.

[0041] Due to the slow movement speeds typically encountered during a parking maneuver, it is not reliably possible to determine relative velocities or distances from the optical flow using established methods. The method according to the invention provides for actively moving the camera 2 or the image sensor of the camera 2. For this purpose, suitable translational and / or rotational movements can be applied to both the camera 2 and the image sensor. Fig. Figure 1 shows the reversing camera 2.1 being moved between its retracted and extended positions within a recess in the vehicle's outer skin. This is indicated by the differently hatched viewing areas 4.1 and 4.2.

[0042] Moving camera 2 and / or the image sensor increases the relative motion between camera 2 or image sensor and the environment. This makes the optical flow, or the underlying vector field, more pronounced. This facilitates the reliable determination of distances, relative velocities, or even the time until impact with object 3. This is indicated in the figures by a divergent optical flow 5 in the form of outward-pointing arrows.

[0043] Fig.Figure 2 shows a further embodiment of the inventive method or the inventive vehicle 1, in which a side mirror camera 2.2 is used as the camera 2. The inventive method is advantageously carried out simultaneously for several camera systems of the vehicle, so that the determination of distances, relative velocities and / or impact times is performed in several spatial directions. Here, for example, this also applies to the left side mirror camera.

Claims

[1] Method for determining the optical flux in camera images by a vehicle-integrated computing unit, wherein the camera images are recorded with at least one camera (2) mounted on a vehicle (1) and directed towards the environment, wherein the vehicle-integrated computing unit determines the current speed of movement of the vehicle (1) and compares it with a defined threshold value, wherein, as soon as the speed of movement falls below the threshold value, the computing unit controls an actuator for moving the camera (2) relative to the vehicle structure and / or an optomechanical image stabilizer associated with the image sensor of the camera (2) for moving the image sensor, characterized by , that the image stabilizer excites the image sensor with periodic vibrations to impose a movement in at least one spatial direction, in particular in the form of a sinusoidal oscillation, a sawtooth oscillation or a chirp. [2] Method according to claim 1, characterized by that the movement includes a rotation about an axis running perpendicular to the direction of travel and / or an axis running parallel to the vehicle's vertical axis. [3] Method according to any one of claims 1 to 2, characterized by , that the computing unit calculates from the optical flow a relative velocity between the vehicle (1) and an object (3) contained in the camera images, a distance between the vehicle (1) and the object (3) and / or a position of the object (3) and / or a time until impact on the object (3) and provides these quantities as input to a downstream assistance system, in particular in the form of a parking assistant. [4] Method according to claim 3, characterized bythat the assistance system, in response to at least one of the variables, issues a warning to a vehicle operator and / or performs an automated intervention in the vehicle's longitudinal and / or lateral guidance. [5] Method according to any one of claims 1 to 4, characterized by , that at least one of the following cameras (2) is used to record the camera images: a reversing camera (2.1) and / or a side mirror camera (2.2). [6] Method according to claim 5, for imprinting the movement onto the camera (2), the reversing camera (2.1) is moved between a retracted and an extended position in an installation recess provided in the vehicle outer skin and / or the side mirror is folded in and out. [7] Method according to any one of claims 1 to 6, characterized by, that the optical flow is evaluated by a machine learning model, wherein the machine learning model is trained to calculate, based on the patterns contained in the optical flow, a relative velocity between camera (2) and an object (3) contained in the camera images, a distance between the camera (2) and the object (3), a position of the object (3) in space and / or a time until impact on the object (3), wherein in particular the machine learning model determines the optical flow itself from a camera image sequence recorded by the at least one camera (2). [8] Method according to any one of claims 1 to 7, characterized by , that the computing unit designs the extent of the movement depending on at least one of the following quantities: the level of the vehicle's speed of travel (1), the presence of at least one object (3) in at least one camera image and / or an object property. [9] Method according to any one of claims 1 to 8, characterized by , that the processing unit is designed to determine the timing and / or frequency of the recording of the camera images as a function of at least one of the following quantities: position, speed, frequency and / or spatial / temporal distance to the last image capture of the camera (2) and / or the image sensor. [10] Vehicle (1) comprising a computing unit and at least one camera (2) movable by means of an actuator, wherein an optomechanical image stabilizer is additionally or alternatively assigned to an image sensor of the camera (2), characterized by , that the at least one camera (2) and the computing unit are configured to perform a method according to one of claims 1 to 9.

Images