Method and device for adjusting an indoor camera for outside-in tracking of smart glasses

Abstract

The invention relates to a method for providing a camera pose indication, which specifies a camera pose of an adjustable interior camera, comprising the following steps: - Recording a camera image of the interior camera; - Providing CAD data of a vehicle interior; - Performing an optimization method by generating interior views based on the CAD data for various assumed camera pose indications, wherein a measure of agreement between the camera image and the interior view is maximized or a deviation between the camera image and the interior view is minimized by comparison in order to obtain an optimized camera pose indication that specifies the actual camera pose;

Description

The invention relates to the use of smart glasses in vehicles, and in particular for the contact-analogous display of display objects. The invention further relates to measures for providing a spectacle pose by an external system for carrying out a sensor fusion in the smart glasses.Smart glasses, also referred to as head-mounted displays, are known, which can display an image on one or two display surfaces in the field of view of the wearer of the smart glasses with the aid of a display device. The display surfaces usually correspond to reflection surfaces which direct images into the eye of the wearer of the smart glasses. The viewing openings of the smart glasses are transparent, so that the real environment can be perceived in the usual manner by the smart glasses. The display areas are located in the viewing openings so that information to be displayed, such as text, symbols, graphics, video displays and the like, can be displayed superimposed on the perception of the environment.The information is generally presented contact-analog to the wearer of the smart glasses, i.e. presented such that it is superimposed as object information on a specific assigned object in the real environment or oriented at the same or that the object information to be displayed is displayed in a specific orientation of the smart glasses or their wearer. Furthermore, the contact-analogous object information can be represented in such a way that it appears perspectively correct with respect to the object in the real environment, i.e. the conclusion arises that the object of the real environment has actually been supplemented by the additional feature of the visual object information.In order to display the object information in a contact-analogous manner on the display surfaces of the smart glasses, it is necessary to know the position of the object in the environment and the position of the user's eyes and the viewing direction thereof. The viewing direction of the user is permanently assigned to his eye glasses pose when wearing the smart glasses, i.e. to the 3D position as well as to the 3D orientation of the smart glasses with respect to a reference coordinate system, such as the vehicle coordinate system (vehicle-fixed) or the surrounding coordinate system (world-fixed).In particular when data glasses are used in a vehicle, a pose recognition unit can be provided in the data glasses for determining the spectacle pose information about the spectacle pose of the data glasses. The pose detection unit generally has a spectacle movement sensor system and a computing device, for example in the form of a microprocessor. With the aid of glasses movement information acquired at high frequency, in particular in the form of 3D acceleration information and / or 3D angular velocity information, the current glasses pose of the smart glasses in the vehicle interior can be tracked by temporal integration of the relative glasses movement information in order to update the glasses pose indication. In this case, the current spectacle pose is initially updated and updated with respect to the world environment. While a simple temporal integration is sufficient for updating the absolute orientation of the smart glasses, twice the temporal integration of the translational 3D acceleration information is required for updating the absolute position of the smart glasses.When determining the absolute spectacle pose indication by taking into account the integration of the spectacle movement information, an integration error generally occurs, which without correction leads to an increasing deviation of the absolute spectacle pose indication provided in the smart glasses and the actual current spectacle pose. In particular, due to the double integration, the integration error is considerable when determining the absolute position of the smart glasses.For ascertaining the current spectacle pose information with respect to a vehicle coordinate system, movement information of the smart glasses captured in the vehicle can be transmitted to the smart glasses, so that an absolute spectacle pose information relative to the vehicle (with respect to a vehicle coordinate system) can be determined.The current spectacle pose can be determined by an external pose recognition unit (outside-in-tracking method), in which a camera in the interior of the vehicle captures the head of the wearer of the smart glasses and either the pose of the head is determined by evaluating the camera image with the aid of vision tracking methods and the absolute spectacle pose indication is derived therefrom or the absolute spectacle pose indication is directly determined. In these so-called outside-in tracking systems, a vehicle-related absolute spectacle pose indication is determined, i.e. an absolute spectacle pose indication with respect to a vehicle coordinate system fixed to the vehicle.The absolute spectacle pose information can be transmitted periodically (with time stamps) to the smart glasses and the updated absolute spectacle pose information determined by integration methods can be corrected there in the smart glasses by sensor fusion in order to compensate for the integration error.For outside-in tracking of smart glasses, it may be provided that a corresponding interior camera, which may be used for ascertaining an absolute spectacle pose fixed to the vehicle, is arranged in an interior mirror unit. Depending on the driver size and seat position, however, the interior mirror unit is usually set at the start of the trip, so that the driver sees the desired section of the rear vehicle environment from his angle of view via the mirror surface. Since the interior camera is arranged in the interior mirror unit, the interior camera located therein obtains a new orientation, i.e. a new camera pose, with each readjustment of the interior mirror unit. However, for determining the absolute spectacle pose, the precise knowledge of the orientation or even of the pose of the interior camera with respect to a vehicle coordinate system is necessary.It is therefore an object of the present invention to provide a method for determining an exact camera pose indication of an interior camera with respect to the vehicle coordinate system.This object is achieved by the method for determining the absolute camera pose indication of an interior camera for the determination of an absolute spectacle pose in a vehicle according to claim 1, and by the device and a vehicle system according to the subordinate claims.Further embodiments are given in the dependent claims.According to a first aspect, a method for providing a camera pose indication which indicates a camera pose of an adjustable interior camera is provided, having the following steps:recording a camera image of the interior camera;providing CAD data of a vehicle interior;performing an optimization method by generating interior views based on the CAD data for different assumed camera pose data, wherein iteratively by comparing a match measure of a match between the camera image and the interior view is maximized or a deviation between the camera image and the interior view is minimized in order to obtain an optimized camera pose data indicating the actual camera pose.In particular, an absolute spectacle pose of smart glasses can be determined in the camera image as a function of the optimized camera pose indication by a vision tracking method and the absolute spectacle pose can be transmitted to the smart glasses.For contact-analogous display of display objects in augmented reality smart glasses, knowledge of an absolute spectacle pose with respect to a reference system is necessary. If smart glasses are used in a vehicle, the reference system corresponds to a vehicle reference system in order to display contact-analogous display objects with respect to vehicle-mounted elements. That is, display objects are displayed superimposed on a specific position or element belonging to the vehicle.To use smart glasses for the contact-analogous display of display objects, it is necessary to know an exact current smart glasses pose indication of the smart glasses. The current spectacle pose specification corresponds to an absolute spectacle pose specification in the smart glasses, which is intended to specify the actual spectacle pose as accurately as possible and is used for contact-analogous display of display objects.The spectacle pose comprises a spatial absolute position (3D position) and the 3D orientation in three spatial directions. For determining pose, a spectacle motion sensor system in the form of an IMU unit (inertial measurement unit) is usually provided in smart glasses, which has acceleration sensors and gyroscopes in order to determine translatory accelerations, angular accelerations or angular velocities as spectacle motion information. These can be used starting from an absolute spectacle pose indication at a specific point in time by single and / or double time integration in order to update the spectacle pose indication in time by accumulation of incremental changes in the position and the orientation of the smart glasses.In order to determine the change in the orientation of the smart glasses, the angular speeds of the smart glasses are simply integrated over time. Thus, the incremental change in orientation may be determined to update the absolute eye glass orientation. Compensation with the vehicle movement detected in the vehicle is generally not necessary, since the orientation of the vehicle changes only at a very low frequency.However, determining the change in the position of the smart glasses as part of the spectacle pose is technically very demanding, since the difference between the acceleration information of the spectacle motion sensor system and a translatory acceleration information recorded by a vehicle-mounted vehicle sensor system must be taken into account for updating the current absolute position of the current smart glasses pose indication. In addition, kinematic terms must be taken into account in the moving vehicle, which include, for example, centripetal acceleration, Coriolis acceleration and the like. These terms are not directly measurable and must be estimated by a tracking filter. In order to obtain the current change in position, the differential acceleration including taken into account kinematic terms must first be simply integrated over time in order to determine the speed of the smart glasses and then the speed re-integrated.As a result of the integration, the drift in the updating of the absolute spectacle pose information increases quadratically over time, which can lead to large errors in the pose determination even after short periods of time. Therefore, a regular correction of the spectacle pose specification determined by integration is necessary in order to compensate for this drift in the position determination and to be able to provide the current spectacle position with sufficient accuracy.The regular transmission of an absolute spectacle pose indication from the spectacle-external vehicle system (with respect to a vehicle coordinate system) is thus necessary for the provision of the current spectacle pose indication in the smart glasses. This requires that the interior camera, which records the camera image, with which a pair of smart glasses can be identified in the vehicle interior with the aid of a vision tracking method and the spectacle pose of which can be determined, must be known exactly with respect to its position and orientation in the vehicle coordinate system. A camera image captured by the interior camera thus shows a section of the interior of the vehicle that is well-defined by the orientation and position, i.e. the camera pose, and the position of the smart glasses that is recognized in the camera image can thus be assigned exactly to a glasses position and orientation with respect to a vehicle coordinate system.One problem is that the orientation of the interior mirror can be changed by the user at any time, and thus the camera pose of the interior camera can be indeterminate. It is thus provided that, according to the above method, a camera pose indication of the interior camera is ascertained with respect to a vehicle coordinate system by carrying out an optimization method which maximizes, by image comparison, a correspondence between the camera image which shows the section of the interior at the current camera pose and an interior view reconstructed on the basis of CAD data for a varying camera pose of the interior camera. The maximizing is effected by varying the camera pose of the interior view reconstructed on the basis of CAD data, so that image deviations between the camera image, which shows the section of the interior at the current camera pose, and the interior view reconstructed on the basis of CAD data are reduced.The optimization method can be carried out at regular predetermined time intervals or after each newly captured camera image.The reconstruction of the interior view based on CAD data can be carried out in a manner known per se with the aid of generally available graphics engines and suitable algorithms, such as, for example, "differentiable rendering", or can optionally already be carried out by the common construction software programs, such as, for example, AUTOCAD and the like.It can be provided that the interior view is created with masked areas, so that the comparison with the camera image takes place only with those areas of the interior view that are not masked.In particular, the masked areas may correspond to those areas of the interior view that represent movable elements in the vehicle interior and / or that may be occupied by vehicle occupants.This interior view can be masked in order to potentially ignore regions in which movable elements can be located and regions in which vehicle occupants can stay in the image comparison. Thus, it may be sufficient to render the interior view only with the stationary elements of the vehicle interior that are visible in any seat occupancy.By comparing the images of the optimization method, an interior view rendered with respect to an assumed camera pose can be compared with the current camera image, in particular within the non-masked image regions, and by varying the assumed camera pose, a deviation between the non-masked regions of the rendered interior view and the camera image can be minimized or the non-masked regions of the rendered interior view and the camera image are brought into agreement.For optimization, suitable methods such as a gradient descent method and the like can be used.By varying the orientation of the viewing angle from which the reconstructed image is produced, it is thus possible to approximate the orientation of the interior camera and to determine, as the orientation of the interior camera, that orientation for the reconstructed image at which the deviation is the smallest. In this way, a determination of an orientation of the interior camera can be made regularly based on a vehicle interior model, which can be provided in the form of a 3D model using CAD data and the like.According to a further aspect, a device, in particular a control unit, is provided for providing a camera pose indication which indicates a camera pose of an adjustable interior camera, in particular on an adjustable interior mirror, wherein the device is designed for:recording a camera image of the interior camera;providing CAD data of a vehicle interior;performing an optimization method by generating interior views based on the CAD data for different assumed camera pose indications, wherein iteratively by comparing a match measure of a match between the camera image and the interior view is maximized or a deviation between the camera image and the interior view is minimized in order to obtain the camera pose indication indicating the actual camera pose.According to a further aspect, an arrangement for operating a vehicle system is provided, comprising:an adjustable interior mirror having an interior camera disposed thereon / therein;the above control unit;a data glasses configured to display at least one display object depending on a glasses pose indication;wherein the control unit is further configured to determine the absolute spectacle pose indication of the smart glasses by a vision tracking method in the camera image depending on the optimized camera pose indication and to transmit the absolute spectacle pose indication to the smart glasses.Embodiments are explained in more detail below with reference to the attached drawings. The following are shown: FIG. 1 is a schematic top view of a vehicle system including a display system; FIG. 2 shows a schematic illustration of a display system with smart glasses in a motor vehicle; and FIG. 3 shows a flow chart for illustrating a method for determining an orientation of an interior camera.FIG. 1 schematically shows a plan view of a vehicle system having a vehicle 1, in which a driver or a vehicle occupant N wears smart glasses 3. The smart glasses 3 are connected either by cable or wirelessly to a vehicle-mounted pose detection system 2 via a communication connection 4.The communication connection 4 is designed as a data transmission channel, e.g. in the form of a wireless communication connection or a wired communication connection. The communication link 4 is capable of transmitting any type of data and information between the assistance system 2 and the smart glasses 3, for example based on a packet-bound data transmission. The communication connection 4 can be based, for example, on Wi-Fi, Bluetooth, Bluetooth Low Energy or a comparable standardized radio protocol.FIG. 2 shows a schematic illustration of the vehicle system.The assistance system 2 can be part of a vehicle assistance system and in particular be provided in a stationary manner in the vehicle 1. The assistance system 2 can be equipped with a communication unit 23 that enables the communication connection 4 between smart glasses 3 and the assistance system 2.The assistance system 2 can furthermore be connected to the camera system 21, which has an interior camera 22 directed into the interior of the vehicle. The interior camera 22 is fastened to a movable interior mirror, so that when the interior mirror is adjusted or adjusted, for example, by a driver, the camera pose of the interior camera is adjusted. The camera system 21 can capture the vehicle occupants N as a camera image. The interior camera 22 may include, for example, an RGB, IR, Fisheye camera, a dynamic vision sensor, and the like.The assistance system 2 can have a control unit 24 in order to evaluate the camera image with the aid of a pattern recognition method and to determine an absolute spectacle pose specification of the one or more smart glasses 3 in the vehicle from this. The absolute spectacle pose specification specifies a spectacle pose with respect to a vehicle coordinate system, i.e. an absolute spectacle position and an absolute spectacle orientation (in each case preferably in all spatial directions).The assistance system 2 furthermore comprises a vehicle movement sensor system (IMU sensor system) 25, which can be designed, for example, in the form of a 6-DOF inertial sensor. This provides movement information of a movement of the vehicle 1 in the form of translatory accelerations or angular accelerations or angular speeds.The smart glasses 3 comprise two transparent viewing panes 32, which are enclosed in a frame 31 in a manner known per se. The frame 31 is provided with eyeglasses frames 33, for example, so that the smart glasses 3 can be worn on the head of a user.One or both viewing panes 32 (spectacle lenses) are furthermore provided with a transparent display surface 35 through which a display image for the display of virtual display objects can be projected into the eye of the wearer of the smart glasses 3 by a suitable device, such as a display device 36 arranged on the frame 31. The display device 36 can have a microprocessor or a comparable arithmetic unit and a display unit, such as a projection device or the like. The display unit can be designed to direct the electronically generated display image onto the display surface 35 and to image / display it there.Due to the transparent configuration of the display surface 35, the electronically generated image can overlap the real environment perceptible by the display surface 35. Using the display device 36, a virtual display object such as a text, a symbol, video information, a graphic, or the like may be displayed on one or both display areas 35.The smart glasses 3 can be worn on the head of the user like a typical vision aid, wherein the smart glasses 3 can rest with the frame 31 on the nose of the user and the brackets 33 can rest laterally on the head of the user. The viewing direction of the user in the straight line orientation is then effected through the transparent display surfaces 35 of the viewing panes 32, so that the viewing direction of the user, which is predetermined by an eye position and an optical viewing axis (eye axis), has a fixed relationship to the orientation of the smart glasses 3. This reference depends individually on the wearer of the smart glasses 3 and is indicated by calibration information.For displaying display objects, corresponding object information in the form of object data is transmitted from the assistance system 2 to the smart glasses 3 or is already provided in the smart glasses 3. The object data indicate the type of display object, such as a text object, an icon or another identification of a display area, the viewing angle area or the viewing angle areas in which the display object is / is to be displayed on the display area 35 and the object position in the vicinity of the smart glasses 3. The object position can be specified with respect to a spatially fixed environment coordinate system or a vehicle-fixed vehicle coordinate system.A spectacle motion sensor system 38 (IMU sensor system, inertial sensor system) can be provided, which is designed, for example, in the form of a 6-DOF inertial sensor (acceleration sensor and gyroscope). This provides spectacle movement information of a movement of the smart glasses 3 in the form of translatory accelerations and angular accelerations or angular speeds, which can be converted in particular by a respective (inter alia double) integration into a change in position and orientation, i.e. a differential specification of spectacle pose, such that a specification of a spectacle pose with high frequency can be updated by applying a time integration method.The position and orientation changes (differential spectacle pose) determined via the spectacle motion sensor system 38 can be used in a manner known per se for determining a world-fixed current spectacle pose indication for a current spectacle pose of the smart glasses 3. In order to determine a piece of spectacle pose information fixed to the vehicle, the piece of spectacle movement information must be corrected as a function of the piece of vehicle movement information, so that only the movement of the smart glasses 3 in the vehicle can be evaluated. In this case, the difference in movement between the movement of the smart glasses and the movement of the vehicle is determined and evaluated to determine the current specification of the eye glasses pose.By means of a control unit 37, object information can be received by the assistance system 2 or provided and processed in some other way via a communication device 39, for example, such that said object information is displayed in a contact-analogous manner depending on the current spectacle pose indication with respect to the surrounding coordinate system in the respective viewing angle range into which the user of the smart glasses 3 is looking and can be perceived in the surrounding object. That is, the display object indicated by the object information is displayed on the display surface 35 when its surrounding position is in the viewpoint range.The user N of the smart glasses 3 is located as an occupant in the vehicle 5, and the eye pose of the smart glasses 3 can change due to head movements and a movement of the vehicle. For the contact-analogous display of a display object with respect to a surrounding position, i.e. the corresponding object information is provided with a surrounding position, on the display surface of the smart glasses 3, a current smart glasses pose indication with respect to the surrounding coordinate system U must be present in the smart glasses 3. Accordingly, for the contact-analogous display of a display object with respect to a vehicle-fixed position, i.e. the corresponding object information is provided with a position relative to the vehicle (vehicle-fixed), an absolute piece of spectacle pose with respect to the vehicle coordinate system F must be present on the display surface of the smart glasses 3.In the display system, a method is carried out, as is described in more detail, for example, with reference to the flow chart of FIG. 3.In step S 1, a camera image is captured using the interior camera 22.The camera image shows a section of the vehicle interior including a vehicle occupant who optionally wears smart glasses 3.Since the orientation of the interior camera 22 may have changed by adjusting the interior mirror, an optimization method is correspondingly started in step S 2. The optimization method provides for the determination of a measure of agreement that provides a deviation between the camera image provided by the interior camera and a rendered interior view. The rendered interior view corresponds to a simulated image determined from vehicle interior CAD data. The rendered interior view is generated with respect to an assumed camera pose indication of the interior camera 22. By maximizing the degree of agreement or minimizing the image deviation between the camera image and the interior view, the assumed camera pose indication is adapted to the actual camera pose, so that after ending the optimization method, the optimized camera pose indication indicates the actual camera pose.For the optimization, regions of the interior view are masked and not taken into account for the calculation of the measure of agreement or the deviation. The masked areas of the interior view may correspond to the areas of dynamically adjustable elements in the vehicle interior and / or areas that can be occupied by vehicle occupants.The optimization method is preferably carried out iteratively and thus enables the determination of the camera pose indication of the interior camera 22 as the actual camera pose. In particular, the approach to finding the optimal camera pose indication can be carried out with the aid of a gradient descent method. If the optimal assumed camera pose indication is found for the rendered interior image, this is assumed as a camera pose indication that corresponds to the actual camera pose of the interior camera 22.With the aid of the camera pose indication and a vision tracking method being carried out with the captured camera image, the absolute spectacle pose indication of the smart glasses 3 can be determined in step S 3 by simple vector math with respect to the vehicle coordinate system.This absolute piece of spectacle pose information can now be transmitted to the smart glasses 3 in step S 4, so that the absolute piece of spectacle pose information updated by integration and the transmitted absolute pieces of spectacle pose information can be provided in the smart glasses 3 there in step S 5 with the aid of a data fusion. This can be used for contact-analogous, i.e. position-wide display of display objects.The method described above can be carried out regularly, i.e. with the recording of each new camera image, or only after an adjustment of the interior mirror has been detected with the aid of other means.List of reference characters1 Vehicle 2 Assistance system 3 Smart glasses 4 Communication connection 21 Camera system 22 Interior camera 23 Communication unit 24 Control unit 25 Vehicle movement sensor system 32 Viewing panes 31 Frame 33 Eye glass bracket 35 Display surface 36 Display device 38 Eye glass movement sensor system 37 Control unit N User

Claims

Method for providing a camera pose indication which indicates a camera pose of an adjustable interior camera (22), in particular on an adjustable interior mirror, having the following steps: - recording (S1) a camera image of the interior camera (22); - providing CAD data of a vehicle interior; - carrying out (S2) an optimization method in that interior views are generated on the basis of the CAD data for different assumed camera pose indications, wherein iteratively by comparing a match measure of a match between the camera image and the interior view is maximized or a deviation between the camera image and the interior view is minimized in order to obtain an optimized camera pose indication which indicates the actual camera pose.Method according to Claim 1, having the further steps: - determining (S3) an absolute spectacle pose of data spectacles (3) by a vision tracking method in the camera image as a function of the optimized camera pose indication; - transmitting (S4) the absolute spectacle pose to the data spectacles (3).Method according to claim 1 or 2, wherein the optimization method is carried out at regular time intervals or after each newly acquired camera image.Method according to one of Claims 1 to 3, wherein the interior view is created with masked regions, such that the comparison with the camera image takes place only with those regions of the interior view which are not masked.Method according to Claim 4, wherein the masked regions correspond to those regions of the interior view which represent movable elements in the vehicle interior, in particular adjustable vehicle seats and adjustable sun visors, and / or which can be occupied by vehicle occupants.Method according to one of Claims 1 to 5, wherein the camera pose indication is ascertained on the basis of a newly captured camera image regularly, at predefined times or at a time if it is ascertained that an inner mirror has been adjusted.Device, in particular a control unit (24), for providing a camera pose indication which indicates a camera pose of an adjustable interior camera (22), in particular on an adjustable interior mirror, wherein the device is configured to: - record a camera image of the interior camera (22); - provide CAD data of a vehicle interior; - perform an optimization method in which interior views are generated based on the CAD data for different assumed camera pose indications, wherein iteratively by comparing a measure of correspondence of a correspondence between the camera image and the interior view is maximized or a deviation between the camera image and the interior view is minimized in order to obtain the camera pose indication which indicates the actual camera pose.Arrangement for operating a vehicle system, comprising: - an adjustable inner mirror having an interior camera (22) arranged thereon / therein; - a control unit (24) according to claim 7; - smart glasses (3) which are designed to display at least one display object as a function of a smart glasses pose indication; - wherein the control unit is furthermore designed to determine the absolute smart glasses pose indication of the smart glasses (3) by a vision tracking method in the camera image as a function of the optimized camera pose indication and to transmit the absolute smart glasses pose indication to the smart glasses (3).

Images