Camera system with variable image plane

Abstract

The invention relates to a sensor assembly (1) comprising a computing unit (6) for evaluating sensor data and a camera sensor (2) for detecting surroundings which comprises at least one image sensor (4), a lens arrangement (3), a housing and at least one means (5) for varying the image plane, wherein the image plane is moved by the at least one means (5) for varying the image plane, wherein during the varying of the image plane a sequence of camera images is recorded.

Description

[0001] 202407677

[0002] 1

[0003] Camera system with variable image plane

[0004] The invention relates to a camera system with a variable image plane for performing a through focus scan.

[0005] Among vehicle sensors, the camera, as with humans, is the dominant sensor for environmental perception. Object detection presents a challenging combination of requirements for the optical design of camera-based Advanced Driver Assistance Systems (ADAS). A few examples include: objects must be detected over a wide range of distances, typically from two to several hundred meters. Furthermore, the camera must exhibit the highest possible light sensitivity and dynamic range to deliver usable images even in adverse environmental conditions and at night. Currently, driven by cost and reliability considerations, only a fixed-focus system is used in vehicles. This necessitates a comparably complex optical design with a large depth of field. The requirement for a large depth of field is further exacerbated by manufacturing tolerances.

[0006] A problem with current systems is their complex optical design. Furthermore, a large depth of field and tight manufacturing tolerances are required.

[0007] It is therefore an object of the present invention to provide a camera system with a computer program product which has a reduced occurrence of image distance errors as well as reduced requirements for the depth of field range.

[0008] The invention is based on the technological possibility of using a sensor capable of shifting its image plane and thus performing a through-focus scan. This significantly reduces the required depth of field and simultaneously enables better object recognition via AI-based, real-time software. 202407677

[0009] 2

[0010] This problem is solved by independent claim 1 and independent claim 6. Further advantageous embodiments are the subject of the dependent claims.

[0011] According to the invention, a sensor assembly is proposed comprising a computing unit for sensor data evaluation and a camera sensor for environmental detection, comprising at least one image sensor, a lens arrangement, a housing and at least one means for changing the image plane, wherein the image plane is moved by means of the at least one means for changing the image plane, and wherein a sequence of individual images or image segments is recorded during the changing of the image plane.

[0012] According to the invention, a sensor capable of shifting its image plane and thus performing a through-focus scan is proposed. This significantly reduces the required depth of field and simultaneously enables better object detection via AI-based real-time software. The shift of the image plane in the proposed sensor typically occurs periodically across the entire depth of field, but its amplitude and frequency can be adjusted or even completely stopped depending on the situation. During this scan movement, a rapid image sequence is captured and processed by low-level software. For example, the software can combine the image sequence into a single image or adaptively adjust the focus to detect critical objects.

[0013] A comparable method in optics is called through focus and is used, among other things, in 3D microscopy. Here, the image plane is shifted using the focus or the process of the sample stage, and a 3D image is generated by processing the image sequence. The innovation of the sensor proposed here lies in continuously shifting the image plane internally and using an intelligent system to select the focus area. In addition to the previously described advantages in design and manufacturing, this enables a significant increase in functionality, as well as in image quality.

[0014] 3

[0015] Dynamics. The sensor proposed here can determine the distance to objects using through-focus measurements. This functionality would otherwise only be possible with a stereo camera, which, however, requires almost twice the material and costs of a mono camera. Furthermore, the exposure is adjusted region-dependently between images in the sequence, thus expanding the dynamic range of the composite image. The proposed sensor is also capable of compensating for age-related focus shifts. This is an essential function, because with increasing vehicle autonomy, the demands on functional safety also rise throughout the vehicle's entire lifespan.

[0016] In a preferred embodiment, the means for changing the image plane is selected from:

[0017] - a means of moving the image sensor;

[0018] - a geometrically variable element in the lens arrangement;

[0019] - a means for the variable adjustment of optical elements relative to each other

[0020] - or

[0021] - a means of changing the optical properties of individual elements of the sensor assembly.

[0022] In another preferred embodiment, the means for moving the image sensor is a MEMS actuator. Accordingly, the imager is placed on a MEMS actuator to change the distance between the imager and the lens. A MEMS actuator is a miniaturized actuator in which the application of an electrical voltage causes a change in position.

[0023] A liquid lens is particularly preferred as the geometrically variable element in the lens arrangement. Liquid lenses allow the geometry of the lens to be manipulated by applying an external electrical voltage, thus changing the focus. 202407677

[0024] 4

[0025] Furthermore, the means for modifying optical properties is particularly preferred if it includes a means for generating an electric field, thereby changing the refractive index. According to the Kerr effect and the Pockels effect, materials change their refractive index in an electric field. The major advantage over the other two methods is that it is based purely on an opto-electrical effect and therefore does not cause any mechanical wear.

[0026] Furthermore, according to the invention, a computer-implemented method for improved object detection in a vehicle is proposed, comprising the following steps:

[0027] - Recording a sequence of environmental images of the vehicle's surroundings using at least one camera sensor, whereby the image plane is changed during recording; decomposing the individual environmental images of the sequence into sections with preferred properties;

[0028] - Merging the sections into an optimized overall image for object recognition.

[0029] Alternatively, the focus can be adjusted after evaluating the image:

[0030] - Kl Software performs “instance” or “semantic segmentation”

[0031] - For image areas and objects where there is high uncertainty, the focus is adjusted to improve sharpness.

[0032] Algorithms are also possible that record a stack of images (number n>1) with different focus settings and pass them to the AI ​​software. These images are then evaluated together. Simple solutions include, for example, feature extraction from a larger stack of images; that is, when n>2, the object recognition software receives not just 3 images (red, green, yellow) but 6 images.

[0033] The advantages of the sensor and method proposed here are that they enable controlled focusing of objects, extraction of depth information, and reduced manufacturing tolerances (202407677).

[0034] 5

[0035] Changing the image plane enables higher light sensitivity without loss of sharpness and an expansion of the dynamic range.

[0036] In the case of AI-controlled focusing of objects, for example, objects that move out of the focus area, e.g., when overtaking a corresponding object via AI, can be refocused if necessary by shifting the mean value of the Through Focus Sequence (TFS) accordingly.

[0037] Depth information is extracted by measuring the sharpness, or a similar quality indicator, of an object in the individual images of the TFS. By determining the point of maximum sharpness, the distance to the object can be measured.

[0038] By changing the image plane on the sensor side, aging effects can be compensated for, meaning the camera's sharpness does not degrade. Simultaneously, manufacturing tolerances are reduced, thus increasing the yield. The optical module changes over its lifespan. This is primarily due to cost-driven design compromises that have become established as state of the art. For example, the adhesive used to fix the lens shrinks.

[0039] Secondly, the circuit board on which the imager is mounted deforms due to thermo-mechanical cycling. The resulting change in image distance must be compensated for. Changing the image plane eliminates the need for adjusting the imager tilt and, ideally, also the need for active alignment, although this depends on the image plane displacement range. Furthermore, this allows for compensation of manufacturing-related variations in image distance. For a given object distance, the image distance should be constant across different object angles, but this is not achievable in practice. Minimizing variations in image distance is primarily achieved through tight tolerances and thus directly impacts the cost of the lens.

[0040] Furthermore, the integration of a larger aperture becomes possible because the requirements for a wide depth of field decrease significantly. This means a higher 202407677

[0041] 6

[0042] Light sensitivity without loss of sharpness in the reduced working range is particularly beneficial for detection at night and in poor visibility conditions, as it allows for shorter exposure times and thus minimizes motion blur. This results in improved sharpness in the near field and an extended depth of field of less than 1 meter. The image distance increases rapidly with short object distances. This means that two different types of cameras are typically used in vehicles: cameras for distant objects and cameras for near-field detection. The latter are optimized for a depth of field ranging from dozens of centimeters to a few meters. Extending the depth of field of the object detection camera into the near field would allow for the integration of optical modules. Object detection is also improved for close objects and at steep angles.

[0043] The expansion of the dynamic range is made possible by targeted exposure control for the selected image section or image within the Through Focus sequence. For this purpose, a KL component would derive exposure settings from the previous image in the sequence and change the corresponding settings on the imager via the hardware interface.

[0044] Furthermore, the proposed sensor arrangement and method can be used for the following applications:

[0045] Neutralization of object angle-dependent image distance errors. Minimizing these is a key optimization goal in optical design and usually means a more complex lens.

[0046] Compensation for temperature-induced image distance error. The camera operates under a wide range of thermal conditions. These are caused by ambient temperature, solar radiation, and significant self-heating. The latter is due to the computationally intensive image processing. This leads to deformations in the optical module due to differing thermal expansion. The resulting image distance error must be compensated. 202407677

[0047] 7

[0048] Minimizing object-distance-dependent image distance errors. Due to the angle of incidence, image distances are dependent on the object distance. This dependency results in a significant difference in sharpness between objects in the near and far fields. Combined with the resulting loss of sharpness and the fact that distant objects are projected smaller onto the imager, distant objects pose a major challenge for recognition. This is because not only is less information contained in the pixels, but fewer pixels also cover an object. For this reason, established optical designs aim to position the image plane close to the image distance for an infinitely distant object during manufacturing. This means that approximately half of the depth of field is lost, and objects in the near field are rendered less sharply.The goal is to be able to image distant and near objects with equal sharpness and to reduce the requirements for the optical design to account for object distance-dependent image distance errors.

[0049] Reducing the manufacturing requirements for the image plane position. Positioning the image plane at an image distance suitable for infinitely distant objects is desirable in practice, but is generally not achieved due to manufacturing tolerances. Besides machine capability, the shrinkage of the adhesive during curing in the optical module manufacturing process is primarily responsible. This means that a corresponding depth-of-field reserve must be incorporated into the optical design, and a correspondingly complex active aliment process has become established.

[0050] Further advantageous embodiments of the invention are shown in the drawings. These show:

[0051] Fig. 1: a schematic representation of a through focus sequence of a series of camera images according to an embodiment of the invention;

[0052] Fig. 2: a schematic representation of an adaptation of the Through Focus sequence according to an embodiment of the invention; 202407677

[0053] 8

[0054] Fig. 3: a schematic flowchart of a method according to one embodiment of the invention;

[0055] Fig. 4: a schematic representation of a sensor assembly according to one embodiment of the invention.

[0056] Fig. 1 shows a schematic representation of a through focus sequence of a series of camera images according to an embodiment of the invention.

[0057] The operating principle of the proposed sensor involves the continuous, periodic shifting of the image plane. At different positions along this shift, an image or image section is read out by the imager. The respective time of image capture is illustrated by a colored dot. The resulting sequence of multiple images is referred to as the Through Focus Sequence (TFS). The targeted selection of partial sections from the individual images within the TFS forms the foundation upon which the new functionalities of the proposed sensor are built. Thus, the selection of the respective sharp image sections allows for the generation of a sharp overall image. Figure 1 demonstrates this using a TFS of five images. The origin of the illustratively selected partial sections, within the merged image, is highlighted in color and shown in relation to the position of the image plane. Much more complex merges are conceivable.The software that selects the section and image from the sequence can, in successive development stages, take into account increasingly complex causes for local differences in sharpness. A method is required to manipulate the image plane. The corresponding methods are already sufficiently described in the specifications and requirements.

[0058] Figure 2 shows a schematic representation of an adaptation of the Through Focus sequence according to an embodiment of the invention. This representation illustrates the function of a controlled focusing of objects during movement.

[0059] Figure 9 illustrates the nature of an object through the image. Typically, an object continuously changes its angle and distance, which leads to a decrease in sharpness as angles and / or distances increase. As an example, a vehicle or object being overtaken is shown here. The vehicle is initially in the center of the image and, as it approaches and passes overhead, moves into the near field with a high angle, where it then leaves the field of view. It is conceivable that the vehicle being overtaken might obscure another vehicle, which would be poorly resolved in established camera systems. An intelligence integrated into the proposed sensor recognizes such scenarios and focuses on potentially obscured areas, as seen here during the overtaking maneuver, in order to detect possible obscured objects with a higher confidence level and reliably trigger a vehicle reaction. This is demonstrated in Figure 2.Here, the mean TFS value is shifted to the near field for a short period (solid line). If a distant object is detected with low confidence during this time, this object can be immediately brought into focus by reopening the TFS area (dashed line).

[0060] Figure 3 shows a schematic flowchart of a method according to an embodiment of the invention. In step S1, a sequence of environmental images of the vehicle's surroundings is recorded using at least one camera sensor, whereby an image plane of the camera sensor is changed during recording. In step S2, the individual environmental images of the sequence are divided into sections with preferred properties. In a further step S3, the sections are combined to form an optimized overall image for object recognition.

[0061] Figure 4 shows a schematic representation of a sensor assembly according to an embodiment of the invention. The sensor assembly 1 comprises a camera sensor 2 with a lens arrangement 3 and an image sensor 4. Furthermore, the camera sensor 2 includes a housing and at least one means 5 for changing the image plane. In addition, a processing unit 202407677 is integrated into the sensor assembly 1.

[0062] 10

[0063] The computing unit 6 is designed to evaluate and process the sensor data from the camera sensor 2. Furthermore, the computing unit 6 is configured to perform at least steps S2 and S3 of the procedure. The camera sensor 2 is connected to the computing unit 6 via a data connection D. This data connection D can be wired or wireless, e.g., WLAN, Bluetooth, or similar.

[0064] 202407677

[0065] 11

[0066] Reference symbol list

[0067] 1 sensor assembly

[0068] 2 camera sensors 3 lens arrangement

[0069] 4 image sensors

[0070] 5 methods for changing the image plane

[0071] 6 Computer equipment

[0072] D Data connection S1-S3 Procedure steps

Claims

202407677 12 Patent claims 1. Sensor assembly (1) comprising a computing unit (6) for sensor data evaluation and a camera sensor (2) for environmental detection, comprising at least one image sensor (4), a lens arrangement (3), a housing and at least one means (5) for changing the image plane, characterized in that the image plane is moved by means of the at least one means (5) for changing the image plane, wherein a sequence of camera images is recorded during the changing of the image plane.

2. Sensor assembly (1) according to claim 1, characterized in that the means for changing the image plane is selected from: - a means for moving the image sensor (4); - a geometrically variable element in the lens arrangement (3); or - a means for modifying the optical properties of individual elements of the sensor assembly (1 ).

3. Sensor assembly (1 ) according to claim 2, characterized in that the means for moving the image sensor (4) is a MEMS actuator.

4. Sensor assembly (1 ) according to claim 3, characterized in that the geometrically variable element in the lens arrangement (3) is a liquid lens.

5. Sensor assembly (1) according to claim 3, characterized in that 202407677 13 the means for changing optical properties includes a means for generating an electric field, whereby the refractive index can be changed.

6. Computer-implemented method for improved object detection in a vehicle comprising the following steps: - Recording (S1 ) a sequence of images of the vehicle's surroundings using at least one camera sensor (2), wherein an image plane of the camera sensor (2) is changed during recording; Decomposition (S2) of the individual environment images of the sequence of environment images into sections with preferred properties; - Merging (S3) the sections into an optimized overall image for object recognition. - Processing of image stacks with n images (n>1 ) with varying depth of field using existing (e.g., YOLO) or new frameworks - Adaptive focusing on areas of high detection uncertainty

Images