Image acquisition device

Abstract

An image acquisition device has an optoelectronic image sensor, a position sensor, a graphical user output interface and an evaluation unit. The image sensor comprises an image data set representing an imaging of a scene located in front of the image sensor on an image plane of the image sensor. The position sensor detects a spatial position of the image plane relative to a reference direction and provides position data indicating a rotation angle and a tilt angle. The evaluation unit determines a projection transformation using the position data, which maps the image data set onto a projection plane tilted relative to the image plane according to the rotation and the tilt. The evaluation unit determines an image portion of the image data set in the projection plane mapped onto the projection plane by the projection transformation and displays the image portion and an area of the imaged scene located at least within the image portion in the graphical user output interface.

Description

Technical Field

[0001] The present invention relates to an image acquisition device including a photoelectric image sensor and a method for operating the image acquisition device including the photoelectric image sensor. Background Technology

[0002] Image acquisition devices (such as cameras or camcorders) including photoelectric image sensors image a scene located in front of the image acquisition device onto the image area of ​​the image sensor, generating image data representing the scene imaged onto the image area. During the imaging of the scene, if the image area is tilted relative to a reference area of ​​the scene (e.g., a vertical plane relative to the facade of a building), the reference area of ​​the scene may not be parallel to the image area of ​​the image sensor. Therefore, different regions within the reference area of ​​the scene have different spacing from the image area of ​​the image sensor, and these different regions are imaged onto the image area at different magnifications. This results in perspective distortion of the scene imaged onto the image area. As a result, for example, straight lines extending parallel to each other within the reference plane can be imaged as so-called converging lines, which can occur particularly when imaging tall buildings from a lower viewpoint.

[0003] Perspective distortion caused by the tilt of the image acquisition device can be transformed during subsequent image data processing through an inverse transform that compensates for the perspective distortion that occurred during image data acquisition. In this regard, it is common practice to identify parallel lines extending from each other in the original scene within the image data, and then correct the image data through an inverse transform so that the identified lines are also aligned parallel to each other in the acquired scene.

[0004] However, in the inverse transform, the edges of the image data are also distorted, so the transformed image data must be cropped to avoid edge tilting. This reduces the usable portion of the image. This may result in image regions that existed in the original acquisition and are important for image synthesis having to be cropped and becoming unusable during subsequent processing of the acquired image data. Summary of the Invention

[0005] The purpose of this invention is to enable users of image acquisition devices to easily generate image data, wherein the image data, after correcting for perspective distortion that occurred during acquisition, still includes all image regions related to the synthetic acquisition.

[0006] The image acquisition device and the method of operating the image acquisition device according to the independent claim achieve this objective. Further limitations are specified in the dependent claims.

[0007] The image acquisition device includes a photoelectric image sensor, a position sensor, a graphical user interface, and an evaluation unit. The image sensor is configured to acquire an image dataset representing the image of a scene located in front of the image sensor on an image area of ​​the image sensor. Furthermore, the position sensor is configured to detect the spatial position of the image area relative to a reference direction and provide position data indicating both a rotation angle and a tilt angle. The rotation angle is the angle by which the image area rotates about the optical axis of the image sensor when the image dataset is acquired, particularly the angle of rotation relative to the perpendicular projection of the reference direction onto the image area. The tilt angle is the angle by which the image area tilts about a horizontal axis when the image dataset is acquired, particularly the angle of tilt relative to the reference direction. Here, the horizontal axis is oriented perpendicular to both the optical axis and the reference direction. The evaluation unit is configured to determine a projection transformation based on the position data, which maps the image dataset onto a projection plane according to both the rotation and tilt of the image area. The projection plane is tilted relative to the image area according to the tilt angle and intersects the image area along an intersection line that rotates within the image area relative to the central axis of the image area according to the rotation angle. The evaluation unit is also configured to determine an image portion of the image dataset mapped onto the projection plane by means of a projection transformation, and to display, in the graphical user output interface, a region within that image portion of the scene at least imaged onto the image region, together with the image portion.

[0008] Within the framework of this invention, simple and, most importantly, accurate perspective correction of perspective distortion is achieved by automatically detecting the position of the image sensor relative to the acquired scene using a position sensor, and calculating a projection transformation for compensating for perspective distortion based on the position data of the position sensor. Since the projection transformation considers not only the tilt of the image sensor relative to the horizontal axis (i.e., the tilt of the image plane of the image sensor relative to a reference plane aligned parallel to the reference direction of the scene) but also the rotation of the image sensor relative to the optical axis, accurate perspective correction of the imaged scene is possible, especially when the image sensor is rotated.

[0009] Then, the image portion determined by the evaluation unit and the scene imaged onto the image area are displayed together in the user output interface of the image acquisition device. This allows the user to orient the image sensor during scene acquisition, ensuring that all image areas crucial for the composite acquisition are within the image portion, even if the image data is subsequently cropped. Based on this, the user of the image acquisition device receives direct feedback during acquisition regarding the image portion that can still be used after perspective distortion correction, allowing the user to select all relevant acquisition parameters while considering the image portion that will be available later.

[0010] Specifically, the reference direction used to determine the projection transformation can be independent of the rotation and tilt of the image sensor, determined by the position sensor. Therefore, the absolute spatial position of the projection plane is also independent of the rotation of the image sensor about its optical axis and the tilt of the image sensor about the horizontal axis. Since the direction of the horizontal axis is perpendicular to both the optical axis and the reference direction, after applying the projection transformation, the optical axis is also positioned in a vertical plane that is traversed by the reference direction and the optical axis and is perpendicular to the horizontal axis. Based on this, after applying the projection transformation, the viewing direction of the imaged scene will not shift from the vertical plane. This is advantageous for image composition during scene acquisition compared to transformations where, for example, the projection plane always has the same orientation relative to the image sensor regardless of how the image sensor rotates about its optical axis.

[0011] In particular, the tilt of the projection plane relative to the image region can depend on the tilt angle, such that the tilt of the projection plane relative to the image region corresponds to the tilt angle or the tilt angle minus the residual angle. The tilt of the projection plane relative to the image region can especially depend on the tilt angle, such that the projection plane is tilted relative to the image region with a non-zero tilt angle and is aligned parallel to the image region with a zero tilt angle.

[0012] Similarly, the rotation of the intersection line between the image region and the projection plane relative to the central axis of the image region can depend on a rotation angle, such that the rotation of the intersection line relative to the central axis corresponds to the rotation angle. The rotation of the intersection line relative to the central axis can, in particular, depend on the rotation angle, such that the intersection line does not extend parallel to the central axis with a non-zero rotation angle, but rather extends parallel to the central axis of the image region with a zero rotation angle. Then, in a rectangular image region, the intersection line also does not extend parallel to the edge of the image region with a non-zero rotation angle.

[0013] The position sensor can be configured as a triaxial sensor, which determines the spatial position of three mutually orthogonal axes of the image sensor. The position sensor can be fixedly arranged within a common housing with the image sensor. For example, the position sensor can be configured as a gravity sensor or a gyroscope sensor. Alternatively, a reference orientation can be predefined by user input, for example, in response to user input, by aligning the central axis of the image sensor along a desired reference orientation and calibrating the position determination to the corresponding spatial position.

[0014] When determining the spatial location of an image sensor, the reference direction used can be a fixed, predefined direction, such as the direction of gravitational acceleration. In the case of gravitational acceleration, the horizontal axis is specifically aligned parallel to the horizon of the scene being captured, such that the tilt angle indicates the degree of tilt of the image region relative to the vertical direction of that horizontal axis.

[0015] To ensure the horizontal axis position is clearly defined when the reference direction is aligned parallel to the optical axis, the evaluation unit can be configured to determine the horizontal axis as perpendicular to both the reference direction and the optical axis only when the tilt angle corresponds to at most a predefined limit value. This limit value can be less than 90 degrees, for example, less than 45 degrees. For instance, the limit value could be 1 degree, 3 degrees, 5 degrees, or 10 degrees less than 90 degrees or 45 degrees. Furthermore, the evaluation unit can be configured to determine the projection transformation and image portion only for cases where the tilt angle corresponds to at most the predefined limit value. Optionally, the evaluation unit can also be configured to align the horizontal axis parallel to a predefined central axis of the image region when the tilt angle exceeds the predefined limit value.

[0016] Image acquisition devices can be configured as cameras or video cameras. In particular, they can be configured as mirrorless cameras. To image a scene onto the image area of ​​an image sensor, the image acquisition device can have imaging optics, such as a lens. During imaging, the lens is fixedly connected to the image sensor and a position sensor, for example, to a housing comprising the image sensor and the position sensor. The imaging optics can, in particular, be interchangeable lenses. The optical axis of the imaging optics can coincide with the optical axis of the image sensor.

[0017] The user output interface can be an electronic or optical viewfinder or screen of an image acquisition device. If it is an optical viewfinder, the image portion can be superimposed on an area of ​​the imaged scene displayed within the viewfinder, so that the image portion and at least the area within the imaged scene are simultaneously displayed in the user output interface, which serves as the viewfinder. The screen and / or viewfinder can also be arranged on a housing including an image sensor and a position sensor. Optionally, at least the screen can be arranged remotely from such a housing, for example, within a mobile device such as a smartphone or tablet, which is connected to the image sensor and position sensor via a data link.

[0018] The evaluation unit may include at least one programmable logic unit, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a microchip, etc. The evaluation unit may be housed within a common housing shared with the image sensor and the position sensor. Optionally, at least some components of the evaluation unit may be located outside the housing, for example, in an external processing unit of the image acquisition device, such as a smartphone or tablet, which is connected to the housing via a data link.

[0019] Image sensors can be configured, for example, as charge-coupled device (CCD) sensors or complementary metal-oxide-semiconductor (CMOS) sensors. The image region of an image sensor is formed by the area within the sensor that detects incident electromagnetic radiation and converts it into electronic signals to create an image dataset. The image region can be rectangular.

[0020] For example, an image portion can be defined as a rectangular image portion. To determine the image portion, the evaluation unit can image at least all corner points of the image region onto a projection plane using a projection transformation. Specifically, the evaluation unit can determine the image portion by mapping only the corner points of the image region onto the projection plane, rather than mapping each pixel of the image sensor onto the projection plane. After determining the coordinates of the image portion within the projection plane, the evaluation unit can image the coordinates of the image portion back to the image region of the image sensor by applying the inverse transformation of the projection transformation. This allows the image portion within the untransformed scene to be displayed in the graphical user interface. Thus, in particular, it is possible to simultaneously display the image portion and all image data acquired from the image region of the image sensor.

[0021] Alternatively, the untransformed portion of the image can also be displayed directly in the user output interface along with the area in the imaging scene that is also mapped onto the projection plane and lies within the image portion. This allows the portion of the image that is available after perspective distortion correction to be displayed over the entire area within the user output interface, and thus displayed at a higher magnification.

[0022] In a further development, the reference direction corresponds to the direction of gravitational acceleration. This allows the spatial position of the image sensor to be easily determined automatically. Perspective distortion typically occurs when the imaging scene includes tall, vertical objects (e.g., buildings) captured from a low or high viewpoint. Especially in cases of perspective distortion, the projection transformation can be determined without user input. In such a further development, the position sensor can, for example, be configured as an acceleration sensor for detecting the Earth's gravitational field.

[0023] In further development, the projection plane is aligned parallel to the horizontal axis and / or a reference direction. If the projection plane is parallel to the horizontal axis, rotation of the image region about the optical axis is fully corrected. If the projection plane is parallel to the reference direction, tilt of the image region relative to the reference direction, such as tilt relative to the vertical direction determined by gravitational acceleration, is fully compensated.

[0024] Optionally, there may be a residual angle between the projection plane and the horizontal axis, which is not equal to zero and is smaller than the rotation angle, such that the rotation about the optical axis is not fully compensated. Optionally or additionally, there may be another residual angle between the projection plane and the reference direction, which is not equal to zero and is smaller than the tilt angle. Thus, the tilt of the image region about the horizontal axis is not fully compensated. Especially in the case of undercorrection, i.e., when the tilt of the image region relative to the horizontal axis is not fully compensated, the imaged scene can look more natural after applying perspective correction than in the case of fully corrected tilt.

[0025] In further development, the horizontal axis intersects the optical axis at the center of the image region. Therefore, the projection plane is also located at the center of the image region and intersects the image region along the horizontal axis.

[0026] In further developments, the projection center of the projection transformation is positioned on the optical axis. This allows for perspective correction and compensation for perspective distortion that occurs during imaging.

[0027] In a further development, the distance from the projection center to the image region corresponds to the focal length of the imaging optics of the image acquisition device that images the scene onto the image sensor, and this focal length is normalized to the diagonal of the image region. This allows for complete correction of perspective distortion that occurs during imaging. If the image acquisition device is configured to accommodate different imaging optics as interchangeable lenses, the evaluation unit can be configured to determine the distance between the projection centers based on the focal length of the imaging optics used during acquisition. The focal length can be provided, for example, by the imaging optics themselves, such as by the electronic unit of the imaging optics, and can be detected by the evaluation unit. If the imaging optics are configured to image the acquired scene with an adjustable focal length, for example, in the case where the imaging optics are configured as a zoom lens, the evaluation unit can be configured to detect the focal length set in each case.

[0028] In a further development, the evaluation unit was configured to determine image portions using only the corner points of the image region projected onto the projection plane. This allows for particularly fast and simple determination of image portions. In this further development of the image acquisition device, the determination of image portions can be performed, in particular, without projecting other pixels of the image sensor, especially without projecting all pixels of the image sensor onto the projection plane. Therefore, image portions can be determined particularly quickly with very little computation.

[0029] In a further development, the evaluation unit is configured to determine image portions based on a predefined aspect ratio. By predefining the aspect ratio, image portions can then be determined automatically. For example, the predefined aspect ratio may correspond to the aspect ratio of an image region and / or the aspect ratio of a user output interface. For instance, the aspect ratio may be immutably stored in the evaluation unit, or it may be predefined by the user of the image acquisition device through a user input interface.

[0030] In further development, a predefined aspect ratio is used, for example, different from the aspect ratio of the image region and / or the aspect ratio of the user output interface, and the evaluation unit is configured to receive user input via the user input interface to specify the predefined aspect ratio. Therefore, the image portion can be automated in a particularly flexible manner.

[0031] In a further development, the evaluation unit is configured to define the image portion as, for example, a rectangular portion, such that in the projection plane, a first central axis of the image portion extends parallel to the horizontal axis, and a second central axis of the image portion, perpendicular to the first central axis, extends parallel to the reference direction. Thus, the orientation of the image portion defined in the projection plane completely corrects for the rotation of the image sensor about its optical axis.

[0032] In a further development, the evaluation unit is configured to determine an image portion such that, while maintaining a predefined aspect ratio, at least two corners of the image portion lie on the edge of the image region, which is mapped onto the projection plane via a projection transformation. Thus, the image portion can be determined to be particularly large, specifically as the maximum usable image portion, while maintaining the predefined aspect ratio.

[0033] In a further development, the evaluation unit is configured to determine the image portion independently of the image sensor's position, such that the center of the image region projected onto the projection plane via projection transformation lies on the central axis of the image portion. This facilitates symmetrical imaging of the scene. Furthermore, in such image portion determination, the region within the image portion in the imaging scene exhibits minimal image error (e.g., distortion), because image errors caused by imaging optics typically increase with increasing distance from the optical axis, and therefore this error increases with increasing distance from the center of the image region.

[0034] In a further development, the evaluation unit was configured to use the same algorithm to determine the image portions for both positive and negative tilt angles, regardless of whether only a positive or negative tilt angle exists. This algorithm was then applied to the projected image region flipped along a reference direction, rather than the image region projected onto the projection plane. Therefore, only a small number of cases need to be distinguished to achieve correction for both positive and negative tilt angles.

[0035] To apply the algorithm to the tilt angles of the flipped projected image region—that is, all positive tilt angles or all negative tilt angles—the relevant points (e.g., corner points) of the image region can first be mapped onto the projection plane before flipping. Then, after applying the algorithm, the determined image portion can also be flipped along the reference direction to obtain the position of the image portion within the original, unflipped projected image region.

[0036] In a further development, the evaluation unit is configured to: at least when the tilt angle is zero, determine the angle of the image portion in the projection plane as the intersection of the diagonal of the image portion and the edge of the image region projected onto the projection plane, the diagonal being predefined by the aspect ratio. Through such graphical determination of the image portion, it can be determined particularly easily and with low computational cost. When the tilt angle is zero, the image portion determined in this way also corresponds to the largest possible image portion while maintaining the aspect ratio. At this point, the diagonal of the image portion can be determined starting from the center of the image portion, where this center can specifically correspond to the center of the image region transformed onto the projection plane.

[0037] In a further development, the evaluation unit is configured to: determine the angle of the image portion in the projection plane as the intersection of the diagonal of half the image portion and the edge of the image region projected onto the projection plane, at least when the tilt angle differs from zero by at least one threshold and the rotation angle is equal to zero. This diagonal is predefined by the aspect ratio, wherein, in the projection plane, the diagonal extends through the center of another edge of the image portion aligned parallel to the horizontal axis. This also enables simple graphical determination of the image portion with low computational cost.

[0038] Specifically, the center of another edge of the image portion (aligned parallel to the horizontal axis) can be arranged on the central axis of the image portion, which is aligned parallel to the reference direction and extends through the center of the image area transformed onto the projection plane. The image portion can be adjacent to the other edge of the image portion, which is aligned parallel to the horizontal axis. This other edge can particularly be the longest edge of the image portion aligned parallel to the horizontal axis.

[0039] The evaluation unit can be configured to: determine the image portion as the intersection of the diagonal of the image portion and the projected edge of the image region only when the tilt angle is equal to zero or less than a threshold, the diagonal being predefined by the aspect ratio; and determine the image portion as the intersection of the diagonal of half of the image portion and the projected edge of the image region only when the tilt angle is not equal to zero or differs from zero by at least a threshold and the rotation angle is equal to zero or differs from zero by at least another threshold, the diagonal being predefined by the aspect ratio.

[0040] In a further development, the evaluation unit is configured to display regions within the image portion of the scene imaged onto the image region in the image user interface as regions of the image dataset transformed onto the projection plane by means of projection transformation, and is configured to display the image portion by means of the cropped image dataset (e.g., by means of the image dataset cropped by means of the edge of the user output interface). This directly displays the cropped and perspective-corrected region of the imaged scene to the user.

[0041] In a further development, the evaluation unit is configured to fully display the scene imaged onto the image area in the graphical user interface without applying projection transformation. The evaluation unit is configured to display the image portion by overlaying a box onto the imaged scene. Thus, the scene imaged onto the image area of ​​the image sensor can be fully displayed to the user of the image acquisition device. In particular, when preparing for acquisition, the user can also consider scene areas located outside the image portion.

[0042] If the scene projected onto the image plane and displayed in the graphical user interface is untransformed (i.e., no projection transformation has been applied), then the inverse projection transformation can be applied to a defined portion of the image on the projection plane to display it in the user interface. Specifically, a bounding box surrounding the image portion can be displayed by applying the inverse transformation. For example, the inverse transformation can serve as a further projection transformation that projects the image portion from the projection plane onto an image plane that includes the image region, where the projection center of the projection plane lies on the optical axis.

[0043] The box can be displayed in the user interface as a separate line. Alternatively, variations of the visual display can include a gray area surrounding the box, or a color representation of the area outside the box that differs from the color representation inside the box. Furthermore, the box can often be displayed using overlaid grid lines, which can be aligned parallel and / or orthogonally to the edges of the box.

[0044] Such grid lines can be aligned parallel and / or orthogonal to the edges of the frame, especially in the projection plane. When the image scene is displayed untransformed in the user output interface, both the frame and grid lines can be transformed onto the image plane for display via inverse transformation.

[0045] In a further development, the evaluation unit is configured to display the locations of measurement points used to determine the acquisition parameters of the image acquisition device in the user output interface, wherein the locations of the measurement points are displayed relative to a complete and untransformed scene imaged onto the image area. Therefore, measurement points located outside the image portion can also be specifically displayed and used to determine the acquisition parameters. In particular, the image acquisition device can be configured to consider measurement data from measurement points even outside the image portion.

[0046] Measurement points can be, for example, focus measurement points, exposure measurement points, color temperature measurement points used to determine the white balance of an image dataset, etc. For example, measurement points can be set via user input. User input can be detected via mechanical operating elements of the image acquisition device, such as buttons, switches, or joysticks. Alternatively or additionally, user input can also be detected via electronic control elements (such as touchscreens or touch-sensitive screens).

[0047] In a further development, the image acquisition device includes a combined user interface comprising a user output interface and a superimposed position input interface for defining the position of a measurement point relative to an untransformed scene. The combined user interface is configured, for example, to detect the position of the measurement point as the position within the untransformed scene detected by the actuation of the superimposed position input interface. This allows for particularly easy and intuitive determination of the measurement point's position. For example, the combined user interface may be a touch-sensitive screen, wherein, in particular, the untransformed scene can be displayed on the screen.

[0048] In a further development, the reference direction lies within the projection plane. Optionally, the projection plane can be tilted relative to the reference direction by a residual angle, wherein the residual angle is not equal to zero and is less than the tilt angle. For example, the residual angle can reach a maximum of 10% of the tilt angle, such as a maximum of 5%, a maximum of 2%, or a maximum of 1%. In particular, the residual angle can reach 2% of the tilt angle. Simultaneously, the projection plane can be aligned parallel to the horizontal axis. Because the tilt is only corrected to a non-zero residual angle, an undercorrection of the tilt is caused, which can make the corrected image dataset appear more natural than a fully corrected one, especially when the tilt is large.

[0049] In a further development, the image sensor is configured to acquire a sequence of image datasets representing a scene imaged onto an image region at consecutive time points, and the position sensor is configured to detect the corresponding spatial position of the image sensor for each image dataset and provide corresponding position data. Furthermore, the evaluation unit is configured to determine a corresponding image portion projected onto a corresponding projection plane for each image dataset by means of a projection transformation determined based on the corresponding position data, and is configured to continuously display each image portion and at least the region within the corresponding image portion of the scene imaged onto the image region in a graphical user interface.

[0050] This allows the user of the image acquisition device to define the scene imaged onto the image sensor based on the display in the user output interface. Specifically, in this further development, the user output interface can be configured as an electronic viewfinder or a screen arranged in the housing of the image acquisition device. The image acquisition device can, for example, be configured to perform the acquisition of a corresponding image dataset and the determination and / or display of corresponding image portions at a repetition rate of at least 5 images per second, such as at least 5, 10, 20, 40, 50, 60, or 120 images per second. In this case, position data can also be provided by a position sensor at a frequency different from the repetition rate, for example, at a frequency lower than the repetition rate. In this case, image portions can be determined and / or displayed consecutively multiple times based on the same position data.

[0051] A method for operating an image acquisition device is also proposed, wherein the method includes:

[0052] - Use the image sensor of the image acquisition device to acquire an image dataset, wherein the image dataset represents the image of the scene located in front of the image sensor on the image area of ​​the image sensor;

[0053] - Detect the spatial position of the image region relative to the reference direction;

[0054] - Provides positional data indicating both rotation angle and tilt angle, wherein the tilt angle is the angle by which the image region rotates around the optical axis of the image sensor when acquiring the image dataset, particularly the angle of rotation relative to the vertical projection of the image region onto the reference direction; the tilt angle is the angle by which the image region tilts around the horizontal axis when acquiring the image dataset, particularly the angle of tilt relative to the reference direction, wherein the direction of the horizontal axis is perpendicular to both the optical axis and the reference direction;

[0055] - A projection transformation is determined based on position data, wherein the projection transformation maps the image dataset onto a projection plane according to the rotation and tilt of the image region. The projection plane is tilted relative to the image region about a horizontal axis according to a tilt angle, and intersects the image region along an intersection line that rotates within the image region relative to its central axis according to a rotation angle.

[0056] - Identify the portion of the image dataset that is mapped onto the projection plane via a projection transformation within the projection plane;

[0057] - Display both the image portion and at least the area within the image portion of the scene that is imaged onto the image area in the graphical user interface of the image acquisition device.

[0058] In particular, this method can be performed by a specified image acquisition device. In this regard, all further developments and technical effects disclosed in conjunction with the image acquisition device are also relevant to the specified method, and vice versa. Attached Figure Description

[0059] The invention will now be described in detail with reference to the accompanying drawings. Schematic diagrams are shown in each case.

[0060] Figure 1 The image acquisition device and the scene in front of the image sensor of the image acquisition device are shown.

[0061] Figure 2 The first, second, third, and fourth spatial positions of the image acquisition device are shown.

[0062] Figure 3 An image acquisition device is shown displaying a scene in the user output interface of the image acquisition device;

[0063] Figure 4 The scene captured and projected onto the projection plane is shown;

[0064] Figure 5 An image portion of the scene defined in the projection plane is shown;

[0065] Figure 6 A first display showing a user output interface with an image portion and a region of the scene located within that image portion is shown;

[0066] Figure 7 A second display showing a user output interface with an image portion and a region of the scene located within that image portion is shown;

[0067] Figure 8 This illustrates the projection transformation from the image region of the image sensor to the projection plane;

[0068] Figure 9 The image region and projection plane of the image sensor are shown;

[0069] Figure 10 A method for displaying a portion of an image in the user output interface of an image acquisition device is shown;

[0070] Figure 11 An image region, which is projected onto a projection plane and has a first image portion, a second image portion, and a third image portion, is shown.

[0071] Figure 12 The method for determining the first image portion is shown;

[0072] Figure 13The first image portion and the projected image region are shown when the rotation angle and tilt angle are not equal to zero.

[0073] Figure 14 The first image portion is shown when the tilt angle is zero;

[0074] Figure 15 The method for determining the second image portion is shown;

[0075] Figure 16 The second image portion and the projected image region are shown when the rotation angle and tilt angle are not equal to zero.

[0076] Figure 17 The second image portion is shown when the tilt angle is zero;

[0077] Figure 18 The diagram illustrates the method for determining the third image portion;

[0078] Figure 19 Another method for determining the third image portion is shown;

[0079] Figure 20 The determination is shown Figure 18 The method for finding the intersection points shown;

[0080] Figure 21 This shows a projected image region with more intersections;

[0081] Figure 22 The determination is shown Figure 21 The method for finding more intersections is shown;

[0082] Figure 23 The projected image region with more intersections is shown; and

[0083] Figure 24 Another method for determining image portions is shown. Detailed Implementation

[0084] Figure 1 An image acquisition device 10 and a scene 1 located in front of the image acquisition device 10 are shown. The image acquisition device is configured as a mirrorless system camera. The image acquisition device 10 includes a photoelectric image sensor 12 and an imaging optics 18, which is configured with an interchangeable lens to image scene 1, including the building 2 arranged in scene 1, onto an image area 30 of the image sensor 12 facing the imaging optics 18. Here, the optical axis 43 of the image sensor 30 coincides with the optical axis of the imaging optics 18.

[0085] The image acquisition device 10 also includes an evaluation unit 20 connected to the image sensor 12, a position sensor 14 of the image acquisition device 10, a first user interface 16 of the image acquisition device 10 configured as an electronic viewfinder, a second user interface 17 of the image acquisition device configured as a rear screen, and an electronic storage unit 22. The various components of the image acquisition device 10 are arranged in or within a common housing 11.

[0086] Specifically, the photoelectric image sensor 12 and the position sensor 14 are arranged within the housing 11 in a fixed manner relative to each other. The position sensor 14 is configured as a triaxial accelerometer that determines the spatial position of the position sensor 14 and the image sensor 12, which is fixedly connected to the position sensor 14, relative to a reference direction 50 determined by gravitational acceleration 15, and sends position data representing the spatial position to the evaluation unit 20.

[0087] exist Figure 1 In the spatial position of the image acquisition device 10 shown, the image sensor 30 is tilted at an angle 54 around a horizontal axis 121 that is perpendicular to the reference direction 50 and oriented perpendicular to the optical axis 43, such that the optical axis 43 intersects the building 2 at an angle other than a right angle. As a result, the optical axis 43 deviates from the vertical axis 123 that is perpendicular to the horizontal axis 121 and oriented perpendicular to the reference direction 50.

[0088] In addition, Figure 1 In the spatial position of the image acquisition device 10 shown, the image region 30 is not rotated about the optical axis 43 relative to the reference direction 50. Therefore, the first central axis 41 of the image region 30 coincides with the horizontal axis 121, and the second central axis 42 of the image region 30 is oriented perpendicular to the first central axis 41 and arranged in a vertical plane traversed by the reference direction 50 and the optical axis 43. Here, the first central axis 41 and the second central axis 42 are respectively oriented perpendicular to the optical axis 43. The central axis 41, the central axis 42, and the optical axis 43 together form a Cartesian sensor coordinate system. Similarly, the horizontal axis 121, the vertical axis 122 extending parallel to the reference direction 50, and the longitudinal axis 123 form a Cartesian coordinate system.

[0089] Figure 2 Image acquisition device 10 is shown in four spatial positions: a first spatial position 115, a second spatial position 116, a third spatial position 117, and a fourth spatial position 118. In each case, the orientation of the sensor coordinate system is selected at each of the different positions 115, 116, 117, and 118 such that the angle between the second central axis 42 and the reference direction 50 is smaller than the angle between the first central axis 41 and the reference direction. Furthermore, the orientation of the sensor coordinate system is selected such that the coordinates projected onto the second central axis 42 by the reference vector defining the reference direction 50 are positive.

[0090] As a result, in the first position 115, the lower side 111 of the image acquisition device 10 faces the reference direction 50, the first central axis 41 faces the right side 113, and the second central axis faces the lower side 111 of the image acquisition device 10. In the second position 116, the upper side 112 of the image acquisition device 10 faces the reference direction 50, the first central axis 41 faces the left side 114 of the image acquisition device 10, and the second central axis 42 faces the upper side 112. In the third position 117, the left side 114 of the image acquisition device 10 faces the reference direction 50, therefore the first central axis 41 faces the lower side 111 of the image acquisition device 10, and the second central axis 42 faces the left side 114 of the image acquisition device 10. In the fourth position 118, the right side 113 of the image acquisition device 10 faces the reference direction 50, therefore the first central axis 41 faces the upper side 112 of the image acquisition device 10, and the second central axis 42 faces the right side 113 of the image acquisition device 10.

[0091] exist Figure 1 During the acquisition scenario 1 shown, the image sensor 12 acquires a sequence of image datasets, each image dataset representing a scene imaged onto the image region 30 at consecutive time points. Simultaneously, each of the individual image datasets in the sequence is displayed regionally in at least the user output interfaces 16 and 17, allowing the user of the image acquisition device 10 to select the region of scenario 1 to be acquired.

[0092] Figure 3 Screen 17 with such an image dataset 100 is shown as an example. The image dataset 100 is displayed on screen 17 without transformation, such that the parallel edges 105 of building 2 are reproduced as converging lines due to the tilt of image sensor 12 about horizontal axis 121.

[0093] The evaluation unit 20 is configured to perform perspective correction on the image dataset 100 in each case by projection transformation, taking into account the tilt angle 54 and the rotation angle of the image sensor 12 about the optical axis 43, where the rotation angle corresponds to the angle between the horizontal axis 121 and the first central axis 41. Here, both the tilt angle 54 and the rotation angle are determined based on position data provided by the position sensor 14.

[0094] Figure 4 It shows Figure 3 The result of this projection transformation of the image dataset 100 shown. After applying the projection transformation, the edges 105 of building 2 extend parallel to each other, and are parallel to the vertical axis 122 and therefore parallel to the reference direction 50. From Figure 4 It can be seen that the edges of the image dataset 100 corresponding to the edges of image region 30 are distorted after projection transformation, so that the image dataset 100 no longer has rectangular edges.

[0095] To correct the distortion at the edges of image region 30 when applying projection transformation, such as Figure 5 As shown, the evaluation unit 20 is configured to determine the rectangular image portion 60. Thus, the image dataset 100 is completely determined. Figure 5 The image portion 60 is shown. In an alternative embodiment, the image portion 60 may also include a region outside the image dataset 100. The evaluation unit 20 is also configured to display the image portion 60 and the region in scene 1 that is at least located within the image portion 60 in the user output interfaces 16, 17.

[0096] Figure 6 A first display of the image portion 60 in the second user output interface 17 and the area within the image portion 60 in scene 1 is shown. Here, the area within the image portion 60 is displayed without transformation. In addition to the area within the image portion 60 in scene 1, the remaining portion of scene 1 imaged onto the image area 30 is also displayed without transformation. The image portion 60 is reproduced as a frame superimposed on the illustrated scene 1.

[0097] exist Figure 6 The image also shows a first measurement point 91, which is displayed superimposed on the image dataset 100 in the user output interface 17. This first measurement point defines the focus measurement point for the autofocus function of the imaging optics 18. The focus measurement point is located outside the image portion 60, causing the autofocus function to focus on the area of ​​scene 1 located outside the image portion 60. Furthermore, a second measurement point 92, defining the exposure measurement point, is also displayed superimposed on the image dataset 100. The second measurement point 92 is located within the image portion 60, causing the area of ​​scene 1 located within the image portion 60 to be exposed.

[0098] The second user output interface 17 is part of a combined user interface configured as a touch-sensitive screen and includes an overlaid touch-sensitive position input interface in addition to the user output interface 17. The positions of measurement points 91 and 92 can be defined by the position input interface. Specifically, the first measurement point can be placed within the image portion 60 and / or the second measurement point 92 can be placed outside the image portion 60.

[0099] Figure 7 A second display shows the image portion 60 and the area within the image portion 60 of scene 1 imaged onto the image sensor 12. In this respect, the area within the image portion 60 of scene 1 is as follows: Figure 4 and Figure 5 The image is transformed and reproduced, and is defined by the edges of the user output interface 17. Here, the image portion 60 is displayed by showing only the area within the image portion 60 of scene 1 imaged onto the image sensor 12.

[0100] In an alternative embodiment, the area within the image portion 60 in scene 1 can also be displayed by applying a projection transformation, and the image portion 60 can be displayed as a box. Here, all areas in scene 1 outside the image portion 60 can also be additionally displayed, for example... Figure 5 The method shown is transformed and reproduced. In this display, measurement points 91 and 92 can also be set outside and inside the image portion.

[0101] Figure 8 This illustrates the process performed by the evaluation unit 20 to determine the image portion 60, targeting... Figure 1 The spatial position of the image acquisition device 10 shown is projected. In the projection transformation, the image area 30 of the image sensor 12 is projected onto the projection plane 120 through the projection center 125 arranged on the optical axis 43. The projection plane 120 and the image plane 40 defined by the image area 30 have an inclination angle of 54.

[0102] The projection plane 120 is oriented perpendicular to a vertical plane traversed by the optical axis 43 and the reference direction 50, and extends through the center 39 of the image region 30. Furthermore, the reference direction 50 lies within the projection plane 120. Therefore, Figure 1 The horizontal axis 121 and the vertical axis 122 shown are also located in the projection plane 120. The projection plane 120 intersects the image region 30 along the horizontal axis 121, such that the horizontal axis 121 forms an intersection line between the projection plane 120 and the image region 30.

[0103] exist Figure 1 In the spatial position of the image region 30 shown, the image region 30 does not rotate about the optical axis 43. Therefore, the horizontal axis 121 of the image region 30 coincides with the first central axis 41, and the second central axis 42 of the image region 30 is tilted at an angle 54 relative to the vertical axis 122.

[0104] Figure 9 The positions of the image region 30 and the projection plane 120 are shown for a rotation angle 52 that is not equal to zero. The projection plane 120 intersects the image region 30 along a horizontal axis 121, wherein the horizontal axis 121 is rotated by a rotation angle 52 relative to a first central axis 41 of the image region 30 about an optical axis 43. Furthermore, the central axis 41 of the image region 30 is rotated by a rotation angle 52 relative to a vertical projection 51 onto the image region 30 or the image plane 40 relative to a reference direction 50.

[0105] Similarly, Figure 9As shown, there is a spacing 126 between the projection center 125 and the center 39 of the image region 30. The spacing 126 is determined by the focal length of the imaging optics 18, which is normalized to the diagonal of the image region 30, and corresponds to:

[0106]

[0107] Among them, f 35mm This refers to the 35mm equivalent focal length of the imaging optics 18.

[0108] The tilt angle 54 specifies the tilt of the image region 30 from the projection plane 120 about the horizontal axis 121, wherein the tilt angle 54 from the projection plane 120 is defined as positive with respect to a right-hand rotation about the horizontal axis 121 and negative with respect to a left-hand rotation about the horizontal axis 121. Figure 9 In the tilt shown, the tilt angle 54 is negative.

[0109] Rotation angle 52 specifies the rotation of the normal plane of image region 30 from a vertical plane traversed by reference direction 50 and optical axis 43 about optical axis 43. This normal plane is defined by the second central axis 42 of image region 30 and optical axis 43. The rotation angle 52 from the vertical plane is defined as positive relative to right-hand rotation about optical axis 43 and negative relative to left-hand rotation about optical axis 43. Figure 9 In the rotation shown, the rotation angle 52 is negative.

[0110] Figure 10 A method 300, performed by evaluation unit 20, for displaying image portion 60 in user output interfaces 16 and 17, is illustrated. Method 300 first includes acquiring image dataset 100 305 via image sensor 12. Subsequently, method 300 includes detecting the spatial position of image sensor 12 via position sensor 14, and then providing position data 315 via position sensor 14. The provided position data is then sent to evaluation unit 12.

[0111] Evaluation unit 12 then determines (320) a projection transformation, through which image region 30 is transformed to Figure 8 On the projection plane 120 shown. In particular, the evaluation unit 12 determines the spatial position of the projection plane 120 based on the tilt angle 54 and the rotation angle 52.

[0112] In the following description, a sensor coordinate system is used, in which the X-axis corresponds to the first central axis 41, the Y-axis corresponds to the second central axis 42, and the Z-axis corresponds to the optical axis 43. Furthermore, a reference coordinate system is used, in which the X-axis corresponds to the horizontal axis 121, the Y-axis corresponds to the vertical axis 122, and the Z-axis corresponds to the longitudinal axis 123. Homogeneous coordinates are also used.

[0113] In the sensor coordinate system, the pixels within image region 30 are then given in homogeneous coordinates as follows:

[0114] p 3d图像点 =(x,y,0,1) T

[0115] Specifically, the corner points of image region 30 are given by the following formula:

[0116]

[0117] Where x TL y TL This refers to the coordinates of the first corner point in the third quadrant of the XY plane of the sensor coordinate system, which has negative X and negative Y coordinates. TR y TR The x-coordinate refers to the coordinates of the second corner point located in the second quadrant of the XY plane of the sensor coordinate system, possessing a positive X-coordinate and a negative Y-coordinate. BR y BR This refers to the coordinates of a point that has positive X and Y coordinates and is located in the third corner of the first quadrant of the XY plane of the sensor coordinate system, where x... BL y BL It refers to the coordinates of the fourth corner point in the fourth quadrant of the XY plane of the sensor coordinate system, which has a negative X coordinate and a positive Y coordinate.

[0118] exist Figure 10 In method 300 shown, these corner points are first transformed (325) to the reference coordinate system by the following matrix transformation:

[0119] M 3d图像→3d世界 =R(φ,θ)=R x (φ)·R z (θ)

[0120] The matrix transformation consists of a first rotation matrix and a second rotation matrix applied subsequently.

[0121] The first rotation matrix is:

[0122]

[0123] The second rotation matrix is:

[0124]

[0125] The first rotation matrix represents the rotation of image region 30 about optical axis 43, where θ corresponds to rotation angle 52. The second rotation matrix represents the tilt of image region 30 about horizontal axis 121, where φ corresponds to tilt angle 54.

[0126] Then, Figure 8 The projection transformation 330 shown, from the corner points of image region 30 to the projection plane 120, is given by the XY plane of the reference coordinate system, and takes into account tilt about tilt angle 54 and rotation about rotation angle 52. The projection transformation on the XY plane of the corresponding coordinate system can be expressed in matrix notation as follows:

[0127] q = -d z ;d=(d x d y d z ) T , .

[0128] Where d refers to the projection center 125 located on the optical axis 43. In the sensor coordinate system, d is given by the following formula:

[0129] d sens = (0, 0, -f) normalized ,1) T

[0130] In the reference coordinate system, d is given by the following equation:

[0131] d ref =R(φ, θ)·(0, 0, -f) 归一化的 ,1) T .

[0132] After applying the projection matrix, the projection transformation P is performed. xy-plane The determined coordinates still need to be normalized using the fourth vector component according to the following formula:

[0133] (x p y p ,0,w p ) T =P xy-面 ·(x, y, z, 1) T

[0134]

[0135] Subsequently, the image portion 60 in the projection plane 120 is determined by defining the four corners of the image portion 60.

[0136] Then, these angles of image portion 60 are transformed back to the sensor coordinate system from the reference coordinate system by the following transformation (345):

[0137]

[0138] This transformation and M 3d图像→3d世界 Reverse.

[0139] If the image portion 60 is to be displayed together with the untransformed scene 1 imaged onto the image area 30, then the corner points are subsequently projected 350 onto the image plane 40 corresponding to the XY plane of the sensor coordinate system via the projection center 125 in the sensor coordinate system. Then, the corner points of the image portion 60 transformed in this way are displayed together with the untransformed scene 1 imaged onto the image area 30 (390).

[0140] Figure 11 An image region 30 projected onto the projection plane 120 is shown, wherein a first corner point 35 is in the third quadrant of the reference coordinate system, a second corner point 36 is in the second quadrant of the reference coordinate system, a third corner point 37 is in the first quadrant of the reference coordinate system, and a fourth corner point 38 is in the fourth quadrant of the reference coordinate system. Here, the projected image region 30 is shown with a positive tilt angle 54 and a positive rotation angle 52.

[0141] In addition to image region 30 Figure 11 A first image portion 81, a second image portion 82, and a third image portion 83 are shown, wherein each image portion 81, 82, and 83 is entirely located within the image region 30 and forms an image portion that can be determined by the evaluation unit 20 as image portion 60. Each image portion 81, 82, and 83 is rectangular and has a corresponding first edge 61, a second edge 62, a third edge 63, and a fourth edge 64. Each first edge 61 and each third edge 63 extends parallel to the horizontal axis 121, and each second edge 62 and each fourth edge 64 extends parallel to the vertical axis 122. Furthermore, image portions 81, 82, and 83 have a corresponding first central axis 75 aligned parallel to the horizontal axis 121 and a corresponding second central axis 76 aligned parallel to the vertical axis 122.

[0142] Each of the image portions 81, 82, and 83 is determined while maintaining a predefined aspect ratio. Furthermore, each image portion 81, 82, and 83 is determined such that their respective second central axis 76 extends through the center 39 of the image region 30 projected onto the projection plane 120.

[0143] Furthermore, the first image portion 81 is defined such that its center 85 corresponds to the center 39 of the image region 30. The second image portion 82 is defined such that the distance from one of its edges 61 and 63, which extend parallel to the horizontal axis 121, to the horizontal axis 121 is the same as the distance from one of the corner points 35, 36, 37, and 38 of the image region 30 to the horizontal axis 121. For the positive tilt angle 54, this is the corner point closest to the horizontal axis 121 and with a negative Y-coordinate, i.e. Figure 11The second corner point 36 in the projection is shown. For the negative tilt angle 54, this is the corner point located closest to the horizontal axis 121 and having a positive Y coordinate. Finally, the third image portion 83 is determined such that at least two of its corners lie on the edge of the image region 30, while maximizing the area of ​​the third image portion 83.

[0144] Figure 12 and Figure 13 The method 400 performed during the determination of image portion 60 is illustrated when image portion 60 is determined as first image portion 81. Here, starting from the center 39 of image region 30, a first diagonal 77 and a second diagonal 78 of the first image portion 81 are determined (405). The first diagonal 77 has a negative gradient in the XY plane of the reference coordinate system, the magnitude of which corresponds to the reciprocal of a predefined aspect ratio for determining image portion 60, and the second diagonal 78 has a positive gradient in the XY plane, the magnitude of which also corresponds to the reciprocal of a predefined aspect ratio. The predefined aspect ratio is always defined as the range of image portion 60 along the horizontal axis 121 to the range of image portion 60 along the vertical axis 122.

[0145] Subsequently, calculation 410 is performed on the eight intersection points 46 of diagonals 77 and 78, where a straight line extends through the first corner point 35 and the second corner point 36 and the first edge 31 of image region 30, a straight line extends through the second corner point 36 and the third corner point 37 and the second edge 32 of image region 30, a straight line extends through the third corner point 37 and the fourth corner point 38 and the third edge 33 of image region 30, and a straight line extends through the fourth corner point 38 and the first corner point 35 and the fourth edge 34 of image region 30. Figure 13 The intersection of the second diagonal 78 and the straight line extending through the first edge 31 is not shown.

[0146] Then, method 400 includes defining 415 a first image portion 81. Here, an intersection point 46 with a minimum spacing 47 to the center 39 of the image region 30 is determined. Subsequently, during defining 415 the first image portion 81, the angles of the first image portion 81 are determined such that these angles lie on diagonals 77, 78, and the respective spacings of these angles to the center 39 of the image region 30 correspond to the minimum spacing 47.

[0147] like Figure 14As shown, when the tilt angle 54 is zero, the first image portion 81 determined by method 400 corresponds to the third image portion 83. This means that when the tilt angle 54 is zero, the first image portion 81 is also determined by method 400 such that the second central axis 76 of the first image portion 81 extends through the center 39 of the image region 30, and when maintaining a predefined aspect ratio, the first image portion 81 includes the largest area within the image region 30. Here, the two corners of the first image portion 81 are located on the edges 31, 32, 33, and 34 of the image region 30.

[0148] exist Figure 15 and Figure 16 The diagram illustrates a method 500 for determining the second image portion 82. Here, 505 first checks 505 for the presence of a negative tilt angle 54. If a negative tilt angle 54 exists, the image region 30 projected onto the projection plane 120 along the vertical axis 122 is first flipped 510. Therefore, the first corner point 35 is interchanged with the fourth corner point 38, the second corner point 36 is interchanged with the third corner point 37, and it is ensured that the extent of the image region 30 extending through the edges 31, 33 of the second and third quadrants of the reference coordinate system along the central axis 121 is greater than the extent of the edges 31, 33 of the image region 30 extending through the first and fourth quadrants.

[0149] Subsequently, the center 74 of the first edge 61 of the second image portion 82, aligned parallel to the horizontal axis 121, is determined (515) such that the height of this center on the vertical axis 122 is the same as the height of the corner points 35, 36 of the image region 30 in the second and third quadrants of the reference coordinate system closest to the central axis 121. Figure 16 This is the second corner point 36 in the spatial location of the image sensor 12 shown.

[0150] Subsequently, method 500 includes determining 520 the intersection point 46 between the first diagonal 72 and the straight lines extending through the second edge 32, third edge 33, and fourth edge 34 of the image region 30, respectively, and the intersection point 46 between the second diagonal 73 and the straight lines extending through the second edge 32, third edge 33, and fourth edge 34 of the image region 30, respectively. In an alternative embodiment, it is also possible to determine only the intersection point 46 between the first diagonal 72 and the straight lines extending through the third edge 33 and fourth edge 34 of the image region 30 and / or only the intersection point 46 between the second diagonal 73 and the straight lines extending through the second edge 32 and third edge 33 of the image region 30.

[0151] Diagonals 72 and 73 are the diagonals of the two halves 70 of the second image portion 82, separated by the second central axis 76, and both diagonals 72 and 73 extend through the center 74. The first diagonal 72 has a negative gradient, and the second diagonal 73 has a positive gradient, wherein the magnitude of each gradient corresponds to twice the inverse of a predefined aspect ratio.

[0152] Subsequently, the second image portion 82 is defined 525 by determining the intersection point 46 with the center 74 having a minimum spacing 47, and by calculating the angular position of the second image portion 82 based on the minimum spacing 47 and a predefined aspect ratio. Then, if a negative tilt angle 54 has been determined, the defined image portion 82 is flipped 590 along the vertical axis 122 to compensate for the flipping 510 of the projected image region 30 performed at the beginning of method 500.

[0153] Therefore, in method 600, the same algorithm used to determine the second image portion 82 is applied to the positive tilt angle 54 and the negative tilt angle 54, wherein the algorithm includes method steps 515, 520, and 525 performed between the initial flip 510 and the final flip 590. In an alternative embodiment, when the positive tilt angle 54 is determined, flips 510 and 590 may also be performed, followed by method steps 515, 520, and 525, such that the center 74 is located on the third edge 63 of the second image portion 82, which extends through the first and fourth quadrants of the reference coordinate system.

[0154] like Figure 17 As shown, when the rotation angle 52 is zero, the second image portion 82 determined by method 500 corresponds to the third image portion 83. This means that, when the rotation angle 52 is zero, like the third image portion 83, the second image portion 82 determined by method 500 is determined such that its second central axis 76 extends through the center 39 of the image region 30, and when maintaining a predefined aspect ratio, the second image portion 82 includes the largest area within the image region 30. Thus, the two corners of the second image portion 82 are located on the edges 31, 32, 33, and 34 of the image region 30. However, unlike the first image portion 81, the center 85 of the second image portion 82 does not correspond to the center 39 of the image region 30.

[0155] Figure 18 The illustration shows a method for determining the third image portion 83. The method first includes calculating the intersection point A of the first edge 31 of the image region 30 projected onto the projection plane 120 and the vertical axis 122, and calculating the intersection point E of the third edge 33 of the projected image region 30 and the vertical axis 122. Subsequently, the evaluation range for determining the third image portion 83 is narrowed down to the Y-coordinate values ​​between intersection points A and E.

[0156] Subsequently, the boundary function f is constrained. 顶 (y), f 右 (y), f 底 (y) and f 左 (y), where f 顶 (y) specifies the dependency of the X coordinate of the first edge 31 on the Y coordinate, f 右 (y) specifies the dependency of the X coordinate of the second edge 32 on the Y coordinate, f 底 (y) specifies the dependency of the X coordinate of the third edge 33 on the Y coordinate, and f 左 (y) specifies the dependence of the X coordinate on the Y coordinate of the fourth edge 34 of image region 30. Then, for further calculations, the function f is used. 顶 (y), f 右 (y), f 底 (y) and f 左 The absolute value of (y), which corresponds to the vertical axis 122-based mirror image of the segments 31, 32, 33, 34 of the extended image region 30 in the third and fourth quadrants of the reference coordinate system, and is defined using the piecewise constraint function h(y) = min(|f 顶 (y)|,|f 右 (y)|,|f 底 (y)|,|f 左 (y)|).

[0157] In the spacing [0, x maX In the evaluation range along the X-axis, where x maX =max(h(y)), then determine the x-coordinate value x of the line extending parallel to the vertical axis 122 by applying the following formula:

[0158]

[0159] Where SV refers to the predefined aspect ratio, and L refers to the length of the line segment extending parallel to the vertical axis 122 under the function h(y).

[0160] Figure 19 Another method 600 for determining the third image portion 83 is shown. In method 600, it is first checked (605) whether a negative tilt angle 54 exists. If it exists, then a combination is first performed. Figure 15 The projected image region 30 along the vertical axis 122 is flipped 510. Then, intersection points A and E are determined as described above. If there is no negative tilt angle 54, method 600 starts directly from determining intersection points A and E 615.

[0161] Subsequently, it was determined that (620) Figure 18 The other intersection points B, C, and D are shown. (Use...) Figure 20The method shown is used for determination. Here, firstly, it is checked whether the Y coordinate of the first corner point 35 of the image region 30 (626) is less than the Y coordinate of the second corner point 36. If it is less, then the intersection point B is determined (628) as the intersection point between the first edge 31 and the fourth edge 34 of the projected image region 30, which is a mirror image based on the vertical axis 122, as shown. Figure 18 As shown. Otherwise, intersection point B is determined (629) as the intersection between the first edge 31 mirrored based on the vertical axis 122 and the second edge 32 of the projected image region 30.

[0162] Subsequently, method 620 includes determining intersection point C (630) as the intersection of the second edge 32 with the horizontal axis 121 or the intersection of the fourth edge 34 with the horizontal axis 121 as a mirror image of the vertical axis 122. Afterwards, it is checked (632) whether the Y-coordinate of the fourth corner point 38 of the projected image region 30 is less than the Y-coordinate of the third corner point 37 of the projected image region 30. If it is less, then intersection point D (634) is determined as the intersection of the third edge 33 of the projected image region 30 with the second edge 32 as a mirror image of the vertical axis 122, such as... Figure 18 As shown. Otherwise, the intersection point D (635) is determined as the intersection between the third edge 33 of the projection and the fourth edge 34 of the image region 30 based on the mirror image of the vertical axis 122.

[0163] Then, as Figure 19 Another method 600 for determining the third image portion 83, as shown, includes defining 650. Figure 21 The first intersection point P1 shown and also in Figure 21 The second intersection point P2 shown, the first intersection point P1 and the second intersection point P2 define the line segments of the edges 31, 32, 33, 34 of the projected image region 30 that intersect with the first triangle 67 of the third image segment 83 to be determined.

[0164] Limited to 650 Figure 22 As shown, this includes: firstly, determining the intersection point F of the first edge 31 of the image region 30 projected by 652 and the straight line extending parallel to the vertical line 122 through the intersection point C, as shown. Figure 21 As shown. Subsequently, defining 650 includes determining the intersection point G of the first edge 31 and the straight line extending through intersection point D parallel to the vertical line 122, also as... Figure 21 As shown.

[0165] Then the connecting line CF is compared with a predefined aspect ratio (656). If the ratio of twice the X-coordinate value of intersection point F to the length of connecting line CF is not greater than the predefined aspect ratio, then the first intersection point P1 is defined as intersection point B, and the second intersection point P2 is defined as intersection point C (657). Otherwise, the connecting line DG is compared with a predefined aspect ratio (658). If the ratio of twice the X-coordinate value of intersection point G to the length of connecting line DG is greater than the predefined aspect ratio, then the first intersection point P1 is defined as intersection point C, and the second intersection point P2 is defined as intersection point D (660), as follows. Figure 21 As shown. Otherwise, define the first intersection point P1 as intersection point D, and the second intersection point P2 as intersection point E (661).

[0166] Subsequently, as Figure 23 As shown, determine the intersection point K between the line extending through intersection points A and B (662) and the line extending through intersection points P1 and P2, and the intersection point L between the line extending through intersection points P1 and P2 and the vertical axis 122. Wherein, Figure 23 The relative length ratio represented in the middle is different from that in the middle. Figures 18 to 21 The length ratio described.

[0167] Then, defining 650 includes: determining the dimensions of the triangle defined by intersection points A, K, and L, where, Figure 23 The lengths a, b, and c shown are determined by the coordinates of the intersection points A, K, and L. Length a corresponds to the difference in the Y-coordinates of points A and K, length b corresponds to the difference in the Y-coordinates of points K and L, and length c corresponds to the distance from point K to the vertical axis 122 that forms the Y-axis.

[0168] Subsequently, angles γ and δ were determined as follows:

[0169] tan(γ)=a / c

[0170] tan(δ)=b / c

[0171] Then, method 600 includes determining the position and size of the third image portion 83. This is achieved by first calculating the X-coordinate value g of the second edge 62 of the third image portion 83. The predefined aspect ratio is given by the following formula:

[0172]

[0173] Where d=f·tan(γ), e=f·tan(δ) and f=cg are further applicable, the X coordinate value g can be calculated as:

[0174]

[0175] Then, the total width w of the third image portion 83 along the horizontal axis 121 is w = 2·g, and the total height h along the vertical axis 122 is h = d + e. In addition, the Y coordinate of the center 85 of the third image portion 83 is K.y + (ed) / 2, the Y coordinate of the intersection point K is Ky, and the X coordinate of the center 85 of the third image portion 83 is zero.

[0176] like Figure 19 As shown, (685) is then calculated based on the position and size of the third image portion 83. Figure 18 The corners 65, 66, 67, and 68 of the third image portion 83 are shown. Finally, in the presence of a negative tilt angle 54, the positions of the determined corners 65, 66, 67, and 68 of the image portion 83 are flipped (590) along the vertical axis 122 to compensate for the flipping 510 of the corners 35, 36, 37, and 38 of the image region 30 performed at the beginning of method 600.

[0177] Therefore, in method 600, the same algorithm used to determine the third image portion 83 is applied to both the positive tilt angle 54 and the negative tilt angle 54, wherein the algorithm includes method steps 615, 620, 650, and 685 performed between the initial flip 510 and the final flip 590. In an alternative embodiment, flips 510 and 590 may also be performed when the positive tilt angle 54 is determined, wherein method steps 615, 620, 650, and 685 must be adjusted accordingly.

[0178] Figure 24 Another method 700 is shown that can be used to perform the determination 340 of image portion 60. Here, as in method 600, image portion 60 is determined to have a predefined aspect ratio, the second central axis 76 of which extends through the projection center 39 of image region 30, and has a maximum area within image region 30 projected onto projection surface 120. In method 700, image portion 60 is also specifically determined such that at least two of angles 65, 66, 67, 68 are located on edges 31, 32, 33, 34 of the projected image region 30.

[0179] Another method 700 first includes checking 705 the tilt angle 54. If the tilt angle 54 deviates from zero by at most one threshold, especially if the tilt angle 54 is equal to zero, then by combining... Figures 12 to 14 The described method 400 determines the image portion 60. Therefore, compared to other methods 500 and 600, the image portion 60 can be determined particularly easily, wherein the image portion 60 is also determined as the largest image portion having a predefined aspect ratio and a second central axis 76 extending through the center 39 of the image region 30, such as... Figure 14As shown. Otherwise, if a negative tilt angle 54 exists, the projected image region 30 is flipped 510 along the vertical axis 122. For a positive tilt angle 54, the flipping 510 is omitted.

[0180] Then check the rotation angle 52. If the rotation angle 52 differs from zero by at most another threshold, in particular, if the rotation angle 52 is equal to zero, then by combining... Figures 15 to 17 Method steps 515, 520, and 525 of method 500 are used to calculate image portion 60. The computational complexity for determining image portion 60 is also less than that in method 600. Figure 17 As shown, in this case, the determined image portion 60 also corresponds to a third image portion 83 with the largest area that can be determined by method 600.

[0181] If the check 710 of rotation angle 52 indicates that rotation angle 52 deviates from zero by more than another threshold, then image portion 60 is calculated by method steps 620, 650, and 685 of method 600. Finally, in the case where tilt angle 54 deviates from zero by more than a threshold and a negative tilt angle 52 exists, the determined image portion 60 is flipped 590.

[0182] In alternative embodiments of the image acquisition device 10, the image portion 60 can also be determined semi-automatically based on its center 85, which can be predefined by the user of the image acquisition device 10. Alternatively or additionally, the predetermined aspect ratio can also be predefined by user input. Furthermore, in alternative embodiments of the image acquisition device 10, the image portion 60 can also be determined to include the area outside the projected image area 30 arranged in the projection plane 120. In these cases, the evaluation unit 20 can be configured to fill the area outside the projected image area 30 with calculated image information (e.g., interpolated image information).

[0183] Evaluation unit 20 is also configured to store reference data for perspective correction of the acquired image dataset 100 in storage unit 22. In this regard, the reference data may be stored separately from the image dataset 100 in a separate file, or it may be stored together with the image dataset 100 in a common file, for example, as metadata. The reference data may include, for example, tilt angle 54 and / or rotation angle 52 and / or the focal length of the imaging optics 18, such as a normalized focal length, for example, a focal length normalized to 35mm format. Alternatively or additionally, the reference data may include image portion data defining the position and size of the image portion 60.

[0184] Such image portion data may in particular include the positions of angles 65, 66, 67, and 68 of image portion 60 and / or a predefined aspect ratio and / or the relative size of the corrected image relative to image dataset 100. Here, the positions of angles 65, 66, 67, and 68 may be stored, for example, in normalized form as values ​​between 0 and 1, the predefined aspect ratio may be stored as the ratio of width to height, and the relative size may be stored as the ratio of the height of image dataset 100 to the height of the corrected image. Furthermore, reference data may include information specifying whether the stored image dataset 100 has been corrected through a projection transformation.

[0185] List of reference numerals

[0186] 1. Scene

[0187] 2 buildings

[0188] 10 Image acquisition equipment

[0189] 11. Shell

[0190] 12 Image Sensors

[0191] 14 Position Sensors

[0192] 15. Gravitational acceleration

[0193] 16 First User Output Interface

[0194] 17 Second User Output Interface

[0195] 18 Imaging Optics

[0196] 20 Evaluation Units

[0197] 22 storage units

[0198] 30 Image Regions

[0199] 31 First Edge

[0200] 32 Second Edge

[0201] 33 Third Edge

[0202] 34 Fourth Edge

[0203] 35 First Corner Point

[0204] 36 Second Corner Point

[0205] 37 Third Corner Point

[0206] 38. Fourth corner point

[0207] 39 Center

[0208] 40 Image plane

[0209] 41 First central axis

[0210] 42 Second central axis

[0211] 43 optical axis

[0212] 46 intersections

[0213] 47 Spacing

[0214] 50 Reference Directions

[0215] 51 Projection

[0216] 52 rotation angle

[0217] 54 Inclination Angle

[0218] 60 Image Section

[0219] 61 First Edge

[0220] 62 Second Edge

[0221] 63 Third Edge

[0222] 64 Fourth Edge

[0223] 65 First Corner

[0224] 66 Second Corner

[0225] 67 Third Corner

[0226] 68. The fourth corner

[0227] 69 diagonals

[0228] 70 Half

[0229] 72 First diagonal

[0230] 73 Second diagonal

[0231] 74 Center

[0232] 75 First central axis

[0233] 76 Second Central Axis

[0234] 77 First diagonal

[0235] 78 Second diagonal

[0236] 81 First Image Section

[0237] 82 Second Image Section

[0238] 83 Third Image Section

[0239] 85 Center

[0240] 86 First intersection

[0241] 87 Second intersection point

[0242] 91 First Measurement Point

[0243] 92 Second Measurement Point

[0244] 100 Image Dataset

[0245] 105 Edge

[0246] 111 upper side

[0247] 112 Lower side

[0248] 113 Right side

[0249] 114 Left side

[0250] 115 First position

[0251] 116 Second position

[0252] 117 Third position

[0253] 118 Fourth position

[0254] 120 Projection Plane

[0255] 121 Horizontal axis

[0256] 122 Vertical axis

[0257] 123 Vertical axis

[0258] 125 Projection Center

[0259] 126 spacing

[0260] 127 Vertical plane

[0261] 130 Mirror Image Area

[0262] 300 methods

[0263] 305 Obtaining Image Datasets

[0264] 310 Detecting spatial position

[0265] 315 provides location data

[0266] 320 Determine the projection transformation

[0267] 325 Transforms the image angular coordinates to reference coordinates

[0268] 330 projects the image angle onto the projection plane.

[0269] 340 Determine image portion

[0270] 345. Transform the coordinates of the angles in the image portion back to the reference coordinates.

[0271] 350° projects a portion of the angle back onto the image plane.

[0272] 390 displays the image dataset and image portions.

[0273] 400 Method for determining the first image portion

[0274] 405 Determine the diagonal

[0275] 410 Calculate the intersection point

[0276] 415 Limiting the first image portion

[0277] 500 Method for Determining the Second Image Part

[0278] 505 Check negative tilt angle

[0279] 510 Flip the projected image area

[0280] 515 Determine the Center

[0281] 520 Determine the intersection point

[0282] 525 Limitation to the second image portion

[0283] 590 Flip the image portion

[0284] 600 Method for determining the third image portion

[0285] 605 Check negative tilt angle

[0286] 615 Determine the Center

[0287] 620. Determine the intersection points B, C, and D.

[0288] 626 Compare the Y coordinates of the first and second corner points

[0289] 628 Determine intersection point B

[0290] 629 Determine intersection point B

[0291] 630 Determine the intersection point C

[0292] 632 Compare the Y coordinates of the third and fourth corner points

[0293] 634 Determine the intersection point D

[0294] 635 Determine the intersection point D

[0295] 650 limits the intersection points P1 and P2

[0296] 652 Determine the intersection point F

[0297] 654 Determine the intersection point G

[0298] 656 compares the CF cable with a predefined aspect ratio.

[0299] 657 defines the first intersection point P1 as intersection point B, and the second intersection point P2 as intersection point C.

[0300] 658 compares the connecting cable DG with a predefined aspect ratio.

[0301] 660 defines the first intersection point P1 as intersection point C and the second intersection point P2 as intersection point D.

[0302] 661. Define the first intersection point P1 as intersection point D, and the second intersection point P2 as intersection point E.

[0303] 662 Determine the intersection points K and L

[0304] 664 Determine the dimensions of the triangle

[0305] 666 Determine the position and size of the third image portion

[0306] 685 Calculate the angle of a portion of the image.

[0307] 700 Methods for determining image portions

[0308] 705 Check tilt angle

[0309] 710 Check rotation angle

Claims

1. An image acquisition device (10), the image acquisition device comprising a photoelectric image sensor (12), a position sensor (14), a graphical user interface (16, 17), and an evaluation unit (20). in, The image sensor (12) is configured to acquire an image dataset (100) representing an image of a scene (1) located in front of the image sensor (12) on an image region (30) of the image sensor (12). The position sensor (14) is configured to detect the spatial position of the image region (30) relative to a reference direction (50) and provide position data indicating both a rotation angle (52) and a tilt angle (54). The rotation angle is the angle by which the image region (30) rotates around the optical axis (43) of the image sensor (12) when the image dataset (100) is acquired, and the tilt angle is the angle by which the image region (30) tilts around the horizontal axis (121) when the image dataset (100) is acquired. The horizontal axis (121) is oriented perpendicular to the optical axis (43) and perpendicular to the reference direction (50). The evaluation unit (20) is configured to determine a projection transformation based on the position data, the projection transformation mapping the image dataset (100) onto the projection plane (120) based on both the rotation and tilt of the image region (30). The projection plane (120) is tilted relative to the image region (30) according to the tilt angle (54) and intersects the image region (30) along an intersection line, which rotates within the image region (30) relative to the central axis (41, 42) of the image region (30) according to the rotation angle (52). The evaluation unit (20) is configured to determine the image portions (60, 81, 82, 83) in the projection plane (120) for the image dataset (100) mapped onto the projection plane (120) by the projection transformation. The evaluation unit (20) is also configured to simultaneously display, in the graphical user output interface (16, 17), the area of ​​the scene (1) imaged onto the image region (30) located within the image region (60, 81, 82, 83) and the image region (60, 81, 82, 83).

2. The image acquisition device (10) according to claim 1, in, The evaluation unit (20) is configured to define the image portions (60, 81, 82, 83) as rectangular portions such that, in the projection plane (120), a first central axis (75) of the image portions (60, 81, 82, 83) extends parallel to the horizontal axis (121), and a second central axis (76) of the image portions (60, 81, 82, 83) perpendicular to the first central axis (75) extends parallel to the reference direction (50).

3. The image acquisition device (10) according to claim 1 or 2, in, The evaluation unit (20) is configured to determine the image portions (60, 81, 82, 83) such that, while maintaining a predefined aspect ratio, at least two corners (65, 66, 67, 68) of the image portions (60, 81, 82, 83) lie on the edges (31, 32, 33, 34) of the image region (30), which are mapped onto the projection plane (120) by the projection transformation.

4. The image acquisition device (10) according to claim 1 or 2, in, The evaluation unit (20) is configured to determine the image portions (60, 81, 82, 83) independently of the position of the image sensor (12), such that the center (39) of the image region (30) projected onto the projection plane (120) by the projection transformation is located on the central axis (76) of the image portions (60, 81, 82, 83).

5. The image acquisition device (10) according to claim 1 or 2, in, The evaluation unit (20) is configured to, at least when the tilt angle (54) is zero, determine the angle (65, 66, 67, 68) of the image portion (60, 81, 82, 83) in the projection plane (120) as the intersection of the diagonal (69) of the image portion (60, 81, 82, 83) and the edge (31, 32, 33, 34) of the image region (30) projected onto the projection plane (120), the diagonal being predefined by the aspect ratio.

6. The image acquisition device (10) according to claim 1 or 2, in, The evaluation unit (20) is configured to: at least when the tilt angle (54) differs from zero by at least a threshold and the rotation angle (52) is equal to zero, determine the angle (65, 66, 67, 68) of the image portion (60, 81, 82, 83) in the projection plane (120) as the intersection of the diagonal (72, 73) of half (70) of the image portion (60, 81, 82, 83) and the edge (31, 32, 33, 34) of the image region (30) projected onto the projection plane (120), the diagonal being predefined by the aspect ratio. In the projection plane (120), the diagonal (72, 73) extends through the center (74) of another edge (61, 62, 63, 64) of the image portion (60, 81, 82, 83) that is aligned parallel to the horizontal axis (121).

7. The image acquisition device (10) according to claim 1 or 2, in, The projection center (125) of the projection transformation is arranged on the optical axis (43); The distance (126) from the image region (30) to the projection center (125) corresponds to the focal length of the imaging optics (18) of the image acquisition device (10) that images the scene (1) onto the image sensor (12), and the focal length is normalized to the diagonal of the image region (30).

8. The image acquisition device (10) according to claim 1 or 2, in, The evaluation unit (20) is configured to determine the image portion (60, 81, 82, 83) using only the corner points (35, 36, 37, 38) of the image region (30) projected onto the projection plane (120).

9. The image acquisition device (10) according to claim 1 or 2, in, The evaluation unit (20) is configured to determine the image portions (60, 81, 82, 83) based on a predefined aspect ratio. The predefined aspect ratio is different from the aspect ratio of the image sensor (12) and / or the aspect ratio of the user output interface (16, 17), and the evaluation unit (20) is configured to receive user input via the user input interface (17) to specify the predefined aspect ratio.

10. The image acquisition device (10) according to claim 1 or 2, in, The evaluation unit (20) is configured to display, in the graphical user interface (16, 17), a region of the scene imaged onto the image region (30) that is located within the image portion as a region of the image dataset (100) transformed into the projection plane (120) by means of the projection transformation. The evaluation unit (20) is configured to display the image portions (60, 81, 82, 83) using the cropped and transformed image dataset (100).

11. The image acquisition device (10) according to claim 1 or 2, in, The evaluation unit (20) is configured to display the scene (1) imaged onto the image region (30) in its entirety in the graphical user interface (16, 17) without using the projection transformation. The evaluation unit (20) is configured to display the image portions (60, 81, 82, 83) by means of boxes superimposed on the imaged scene (1).

12. The image acquisition device (10) according to claim 11, in, The evaluation unit (20) is configured to display the positions of measurement points (91, 92) used to determine the acquisition parameters of the image acquisition device (10) in the user output interface (16, 17). The positions of the measurement points (91, 92) are displayed relative to the complete and untransformed scene (1) imaged onto the image region (30). The image acquisition device (10) includes a combined user interface (17), which includes the user output interface and a superimposed position input interface for defining the position of the measurement point (91, 92) relative to the untransformed scene (1). The combined user interface is configured to detect the position of the measurement point (91, 92) as the position of the actuation of the superimposed position input interface within the untransformed scene (1).

13. The image acquisition device (10) according to claim 1 or 2, in, The reference direction (50) is located in the projection plane (120), or The projection plane (120) is tilted by a residual angle relative to the reference direction (50), and the residual angle is not equal to zero and is less than the tilt angle (54).

14. The image acquisition device (10) according to claim 1 or 2, in, The image sensor (12) is configured to acquire a sequence of image datasets representing the scene (1) imaged onto the image region (30) at consecutive time points. The position sensor (14) is configured to detect the corresponding spatial position of the image sensor (12) for each image dataset (100) and provide corresponding position data. The evaluation unit (20) is configured to determine, by means of a projection transformation determined based on the corresponding position data, the corresponding image portions (60, 81, 82, 83) projected onto the corresponding projection plane (120) for each image dataset (100). The evaluation unit (20) is also configured to continuously display the corresponding image portions (60, 81, 82, 83) in the user output interface (16, 17) together with the regions located within the corresponding image portions (60, 81, 82, 83) in the scene (1) imaged onto the image region (30).

15. The image acquisition device (10) according to claim 10, in, The image portions (60, 81, 82, 83) are displayed using the image dataset (100) after edge cropping transformation via the user output interface (16, 17).

16. A method (300) for operating an image acquisition device (10), characterized in that, The method (300) includes: - Use the image sensor of the image acquisition device to acquire (305) an image dataset (100), The image dataset (100) represents the image of the scene (1) located in front of the image sensor (12) on the image region (30) of the image sensor (12); - Detect (310) the spatial position of the image region (30) relative to the reference direction (50); - Provides positional data indicating both rotation angle (52) and tilt angle (54), wherein the rotation angle is the angle by which the image region (30) rotates around the optical axis (43) of the image sensor (12) when the image dataset (100) is acquired (305), and the tilt angle is the angle by which the image region (30) tilts around the horizontal axis (121) when the image dataset (100) is acquired (305); The horizontal axis (121) is oriented to be perpendicular to the optical axis (43) and perpendicular to the reference direction (50). - A projection transformation (320) is determined based on the position data, wherein the projection transformation maps the image dataset (100) onto the projection plane (120) based on both the rotation and tilt of the image region (30). The projection plane (120) is tilted relative to the image region (30) about the horizontal axis (121) according to the tilt angle (54) and intersects the image region (30) along the intersection line, which is rotated in the image region (30) relative to the central axis (41) of the image region (30) according to the rotation angle (52); - Determine (340) the image portions (60, 81, 82, 83) of the image dataset (100) mapped onto the projection plane (120) by the projection transformation in the projection plane (120). - The image portions (60, 81, 82, 83) and the area within the image portions (60, 81, 82, 83) of the scene (1) at least imaged onto the image area (30) are displayed in the graphical user output interface (16, 17) of the image acquisition device (10).

Images