Camera system with variable image plane

Abstract

The invention relates to a sensor assembly (1) comprising a computing unit (6) for sensor data evaluation and a camera sensor (2) for environmental detection, a at least one image sensor (4), a lens arrangement (3), a housing and at least one means (5) for changing the image plane, wherein the image plane is moved by means of the at least one means (5) for changing the image plane, and wherein a sequence of camera images is recorded during the changing of the image plane.

Description

The invention relates to a camera system with a variable image plane for performing a through focus scan. Among vehicle sensors, the camera, as with humans, is the dominant sensor for environmental perception. Object detection presents a challenging combination of requirements for the optical design of camera-based Advanced Driver Assistance Systems (ADAS). A few examples include: objects must be detected over a wide range of distances, typically from two to several hundred meters. Furthermore, the camera must exhibit the highest possible light sensitivity and dynamic range to deliver usable images even in adverse environmental conditions and at night. Currently, driven by cost and reliability considerations, only a fixed-focus system is used in vehicles. This necessitates a comparably complex optical design with a large depth of field. The requirement for a large depth of field is further exacerbated by manufacturing tolerances. A problem with current systems is their complex optical design. Furthermore, a large depth of field and tight manufacturing tolerances are required. It is therefore an object of the present invention to provide a camera system with a computer program product which has a reduced occurrence of image distance errors as well as reduced requirements for the depth of field range. The invention is based on the technological possibility of using a sensor capable of shifting its image plane and thus performing a through-focus scan. This significantly reduces the required depth of field and simultaneously enables better object recognition via AI-based, real-time software. This problem is solved by independent claim 1 and independent claim 6. Further advantageous embodiments are the subject of the dependent claims. According to the invention, a sensor assembly is proposed comprising a computing unit for sensor data evaluation and a camera sensor for environmental detection, comprising at least one image sensor, a lens arrangement, a housing and at least one means for changing the image plane, wherein the image plane is moved by means of the at least one means for changing the image plane, and wherein a sequence of individual images or image segments is recorded during the changing of the image plane. According to the invention, a sensor capable of shifting its image plane and thus performing a through-focus scan is proposed. This significantly reduces the required depth of field and simultaneously enables better object detection via AI-based, real-time software. The shift of the image plane in the proposed sensor typically occurs periodically across the entire depth of field, but its amplitude and frequency can be adjusted or even completely stopped depending on the situation. During this scan movement, a rapid sequence of images is captured and processed by low-level software. For example, the software can combine the image sequence into a single image or adaptively adjust the focus to detect critical objects. A comparable method in optics is known as through-focus and is used, among other things, in 3D microscopy. Here, the image plane is shifted using the focus or the process of the sample stage, and a 3D image is generated by processing the image sequence. The innovation of the sensor proposed here lies in its ability to continuously shift the image plane internally and to select the focus area with the aid of an intelligent system. In addition to the previously described advantages in design and manufacturing, this enables a significant increase in functionality, as well as in image quality and dynamic range. For example, the sensor proposed here can determine the distance to objects using through-focus measurements. This functionality would otherwise only be possible with a stereo camera, which, however, requires almost twice the material and costs of a monocular camera.Furthermore, the exposure is adjusted region-dependently between images in the sequence, which expands the dynamic range of the composite image. The proposed sensor is also capable of compensating for age-related focus shifts. This is an essential function, because with increasing vehicle autonomy, the demands on functional safety also rise throughout the vehicle's entire lifespan. In a preferred embodiment, the means for changing the image plane is selected from: - a means for moving the image sensor; - a geometrically variable element in the lens arrangement; - a means for variably adjusting optical elements relative to each other; or - a means for changing optical properties of individual elements of the sensor assembly. In another preferred embodiment, the means for moving the image sensor is a MEMS actuator. Accordingly, the imager is placed on a MEMS actuator to change the distance between the imager and the lens. A MEMS actuator is a miniaturized actuator in which the application of an electrical voltage causes a change in position. A liquid lens is particularly preferred as the geometrically variable element in the lens arrangement. Liquid lenses allow the geometry of the lens to be manipulated by applying an external electrical voltage, thus changing the focus. Furthermore, the means for modifying optical properties is particularly preferred if it includes a means for generating an electric field, thereby changing the refractive index. According to the Kerr effect and the Pockels effect, materials change their refractive index in an electric field. The major advantage over the other two methods is that it is based purely on an opto-electrical effect and therefore does not cause any mechanical wear. Furthermore, according to the invention, a computer-implemented method for improved object detection in a vehicle is proposed, comprising the following steps: - Recording a sequence of environmental images of the vehicle's surroundings using at least one camera sensor, wherein the image plane is changed during recording; - Decomposing the individual environmental images of the sequence of environmental images into sections with preferred properties; - Combining the sections to form an optimized overall image for object detection. Alternatively, the focus can be adjusted after evaluating the image: - AI software performs "instance" or "semantic segmentation" - For image areas and objects where there is high uncertainty, the focus is adjusted to improve sharpness. Algorithms are also possible that record a stack of images (number n>1) with different focus settings and pass them to the AI ​​software. These images are then evaluated together. Simple solutions here include, for example, feature extraction from a larger stack of images; that is, when n>2, the object recognition software is provided with six images instead of just three (red, green, yellow). The advantages of the sensor and method proposed here are that they enable AI-controlled focusing of objects, extraction of depth information, reduced manufacturing tolerance by changing the image plane, higher light sensitivity without loss of sharpness, and an extension of the dynamic range. In AI-controlled object focusing, for example, objects that move out of the focus area, e.g., when overtaking a corresponding object, can be refocused via AI if necessary by shifting the mean value of the Through Focus Sequence (TFS) accordingly. Depth information is extracted by measuring the sharpness, or a similar quality indicator, of an object in the individual images of the TFS. By determining the point of maximum sharpness, the distance to the object can be measured. By changing the image plane on the sensor side, aging effects can be compensated for, meaning the camera's sharpness does not degrade. Simultaneously, manufacturing tolerances are reduced, thereby increasing the yield. The optical module changes over its lifespan. This is primarily due to cost-driven design compromises that have become state-of-the-art. For example, the adhesive used to fix the lens shrinks. Additionally, the circuit board on which the imager is mounted deforms due to thermo-mechanical cycling. The resulting change in image distance must be compensated for. Changing the image plane eliminates the need to adjust the imager tilt and, ideally, also the need for active alignment, although this depends on the image plane shift range. Furthermore, this allows for compensation of manufacturing-related variations in image distance.For a given object distance, the image distance should be constant across different object angles, but this is not achievable in practice. Minimizing variations in image distance is primarily achieved through tight tolerances and thus directly impacts the cost of the lens. Furthermore, the integration of a larger aperture becomes possible because the requirements for a wide depth of field decrease significantly. This means higher light sensitivity without loss of sharpness in the reduced working range, which has a particularly positive effect on detection at night and in poor visibility conditions, as shorter exposure times are possible and thus motion blur is minimized. Accordingly, improved sharpness in the near field and an extension of the depth of field to less than 1 meter become possible. The image distance increases rapidly with short object distances. This means that two different types of cameras are typically used in vehicles: cameras for distant objects and cameras for near-field perception. The latter are optimized for a depth of field ranging from dozens of centimeters to a few meters.Extending the depth of field of object recognition cameras into the near field would allow for the integration of optical modules. This would also improve object recognition for close objects and at high angles. The expansion of the dynamic range is made possible by targeted exposure control for the selected image section or image within the Through Focus sequence. For this purpose, an AI component would derive exposure settings from the previous image in the sequence and change the corresponding settings on the imager via the hardware interface. Furthermore, the proposed sensor arrangement and method can be used for the following applications: Neutralizing object angle-dependent image distance errors. Minimizing these errors is a key optimization goal in optical design and usually requires a more complex lens. Compensation for temperature-induced image distance error. The camera operates under a wide range of thermal conditions. These include ambient temperature, solar radiation, and significant self-heating, the latter resulting from computationally intensive image processing. This leads to deformations in the optical module due to differing thermal expansion. The resulting image distance error must be compensated. Minimizing object distance-dependent image distance errors. Due to the angle of incidence, image distances depend on the object distance. This dependency results in a significant difference in sharpness between objects in the near and far fields. Combined with the resulting loss of sharpness and the fact that distant objects appear smaller on the imager, distant objects pose a significant challenge for recognition. This is because not only is less information contained in the pixels, but fewer pixels also cover an object. For this reason, established optical designs aim to position the image plane close to the image distance for an infinitely distant object during manufacturing. This means that approximately half of the depth of field is lost, and objects in the near field are rendered less sharply.The goal is to be able to image distant and near objects with equal sharpness and to reduce the requirements for the optical design to account for object distance-dependent image distance error. Reducing the manufacturing requirements for the image plane position. Positioning the image plane at an image distance suitable for infinitely distant objects is desirable in practice, but is generally not achieved due to manufacturing tolerances. Besides machine capability, the shrinkage of the adhesive during curing in the optical module manufacturing process is primarily responsible. This means that a corresponding depth-of-field reserve must be incorporated into the optical design, and a correspondingly complex active aliment process has become established. Further advantageous embodiments of the invention are shown in the drawings. These show: Fig. 1: a schematic representation of a through-focus sequence of a series of camera images according to one embodiment of the invention; Fig. 2: a schematic representation of an adaptation of the through-focus sequence according to one embodiment of the invention; Fig. 3: a schematic flowchart of a method according to one embodiment of the invention; Fig. 4: a schematic representation of a sensor assembly according to one embodiment of the invention. Fig. 1 shows a schematic representation of a through-focus sequence of a series of camera images according to one embodiment of the invention. The operating principle of the sensor proposed here involves the continuous periodic shifting of the image plane. At different positions of the shift, an image or image section is read out by the imager. The respective time of image capture is illustrated by a colored dot. The resulting sequence of several images is referred to below as a through-focus sequence (TFS). Targeted selection of partial sections of the individual images from the TFS is the foundation upon which the new functionalities of the proposed sensor are built. Thus, the selection of the respective sharp image sections allows the generation of a sharp overall image. This is demonstrated in Fig. 1 using a TFS of five images.The origin of the illustratively selected partial sections in the merged image is highlighted in color and shown in relation to the position of the image plane. Much more complex merges are conceivable. The software, which selects the section and the image from the sequence, can, in successive development stages, take into account increasingly complex causes for local differences in sharpness. A method is required to manipulate the image plane. The corresponding methods are already sufficiently described in the specifications and requirements. Fig. 2 shows a schematic representation of an adaptation of the through focus sequence according to one embodiment of the invention. This illustration is intended to clarify the function of AI-controlled focusing of objects during their movement across the image. Typically, an object continuously changes its angle and distance, which leads to a decrease in sharpness as the angle increases and / or the distance decreases. As an example, a vehicle or object being overtaken is shown here. The vehicle is initially in the center of the image and, as it approaches and passes overhead, moves into the near field with a high angle, where it then leaves the field of view. It is conceivable that the vehicle being overtaken might obscure another participant that would be poorly resolved in established camera systems.An integrated intelligence within the proposed sensor recognizes such scenarios and focuses on potentially obscured areas, as seen here during an overtaking maneuver, in order to detect possible obscured objects with a higher confidence level and trigger a more reliable vehicle reaction. This is demonstrated in Fig. 2. Here, the mean TFS value is shifted to the near field for a short duration (solid line). Should a distant object be detected with low confidence during this time, this object can be immediately brought into focus by reopening the TFS area (dashed line). Fig. 3 shows a schematic flowchart of a method according to an embodiment of the invention. In step S1, a sequence of environmental images of the vehicle's surroundings is recorded using at least one camera sensor, whereby one image plane of the camera sensor is changed during recording. In step S2, the individual environmental images of the sequence are divided into sections with preferred properties. In a further step S3, the sections are combined to form an optimized overall image for object recognition. Fig. 4 shows a schematic representation of a sensor assembly according to an embodiment of the invention. The sensor assembly 1 comprises a camera sensor 2 with a lens arrangement 3 and an image sensor 4. The camera sensor 2 further comprises a housing and at least one means 5 for changing the image plane. A processing unit 6 is also provided in the sensor assembly 1, by means of which the sensor data from the camera sensor 2 can be evaluated and processed. The processing unit 6 is further configured to perform at least steps S2 and S3 of the method. The camera sensor 2 is connected to the processing unit 6 via a data connection D. The data connection D can be wired or wireless, e.g., WLAN, Bluetooth, or similar. Reference symbol list 1 Sensor assembly 2 Camera sensor 3 Lens assembly 4 Image sensor 5 Means for changing the image plane 6 Computing unit D Data connection S1-S3 Process steps

Claims

Sensor assembly (1) comprising a computing unit (6) for sensor data evaluation and a camera sensor (2) for environmental detection comprising at least one image sensor (4), a lens arrangement (3), a housing and at least one means (5) for changing the image plane, characterized in that the image plane is moved by means of the at least one means (5) for changing the image plane, wherein a sequence of camera images is recorded during the changing of the image plane. Sensor assembly (1) according to claim 1, characterized in that the means for changing the image plane is selected from: - a means for moving the image sensor (4); - a geometrically variable element in the lens arrangement (3); or - a means for changing optical properties of individual elements of the sensor assembly (1). Sensor assembly (1) according to claim 2, characterized in that the means for moving the image sensor (4) is a MEMS actuator. Sensor assembly (1) according to claim 3, characterized in that the geometrically variable element in the lens arrangement (3) is a liquid lens. Sensor assembly (1) according to claim 3, characterized in that the means for changing optical properties comprises a means for generating an electric field, whereby the refractive index can be changed. A computer-implemented method for improved object detection in a vehicle, comprising the following steps: - Recording (S1) a sequence of environmental images of the vehicle's surroundings using at least one camera sensor (2), wherein an image plane of the camera sensor (2) is changed during recording; - Decomposing (S2) the individual environmental images of the sequence into sections with preferred properties; - Combining (S3) the sections into an optimized overall image for object detection; - Processing image stacks with n images (n>1) with changing depth of field using existing (e.g., YOLO) or new frameworks; - Adaptive focusing on areas of high detection uncertainty

Images