Method and apparatus for controlling a mobile input device

Abstract

The invention relates to a method for controlling a mobile input device (2) having a screen (1) on which information can be displayed in a first pixel grid formed by image elements (9) and in which a planar optical fingerprint reader, which covers at least half of the screen surface, is integrated, the optical fingerprint reader having a second pixel grid formed by light-sensitive sensor elements (10). The invention also relates to a device for carrying out the method. In the method, for at least one finger (3) placed on the screen in an activated application, the fingerprint of the finger and a plurality of fingerprint characteristics are determined by means of the fingerprint reader in the following manner, the fingerprint characteristics comprising the position of the finger (3) on the screen (1) and, if necessary, the movement pattern of the finger (3), i.e., the light intensity incident on the sensor element (10) is recorded once at a predetermined time in an image or the light intensity incident on the sensor element is recorded by the sensor element several times during a predetermined time period in an image sequence. The recorded intensity values ​​are pieced together by an image processing algorithm to form a static pattern of at least one fingerprint and, if necessary, a movement pattern of at least one finger (3). The combination of static patterns, fingerprint characteristics and, if necessary, motion patterns is compared with a database. If the combination is stored in the database, a check is performed to determine whether a subsequently executed predetermined action is associated with the combination for the activated application, or if no action is associated with the combination, and then a first standard action is performed. If the combination is not stored in the database, a second standard action is performed. According to the invention, the orientation angle (α) of at least one finger (3) on the screen (1), the rotation angle (γ) of at least one finger (3) on the screen (1) and / or the angle of attack (β) are determined as additional fingerprint characteristics. In this way, additional and intuitive input possibilities are achieved.

Description

Technical Field

[0001] The invention relates to a method for controlling a mobile input device having a screen on which information can be displayed in a first pixel grid consisting of image elements, and in which a planar optical fingerprint reader is integrated, which comprises at least half of the screen surface, but preferably comprises a larger surface part, in particular the entire screen surface, and has a second pixel grid consisting of light-sensitive sensor elements. A mobile input device is understood here to be, for example, a smartphone, a tablet or a notebook, but also other special input devices, such as a remote control or an input device, for example, for operating "Smart Home" features or for controlling devices integrated in a "Smart Home", such as door locks, blinds, household appliances, etc., wherein a smartphone can also be used for these inputs.

[0002] In this method, in an activated application (also referred to as an application or program), the fingerprint and a plurality of fingerprint characteristics of at least one finger placed on the screen are determined with the aid of a fingerprint recorder. If the finger moves continuously above the screen in contact with the screen, the position of the finger on the screen and, if necessary, the movement pattern of the finger are regarded as fingerprint characteristics, for example. For this purpose, the light intensity incident on the sensor element should first be recorded by the sensor element once in an image at a predetermined time point or (if the movement pattern is to be determined) by the sensor element multiple times in a sequence of images during a predetermined time period. The recorded intensity values ​​are then pieced together by an image processing algorithm to form a static pattern of at least one fingerprint and, if necessary, the movement pattern of the at least one finger. In this case, the static pattern corresponds to a fragment of the fingerprint or the fingerprint itself. Since the fingerprint recorder is a planar element, the position of the finger and the movement pattern can be easily determined. Background Art

[0003] The recognition of fingerprints or their characteristics is clearly described in the prior art, and reference is made in particular to the "Handbook of Fingerprint Recognition" by D. Maltoni et al., Springer-Verlag, 2003, the disclosure of which is expressly incorporated herein. For example, singular points (e.g., bow-shaped, dustpan-shaped, and bucket-shaped), different shapes of detail features, that is, the distinctive conditions of the lines of the finger lines (e.g., the intersection of two lines or the bifurcation, termination, or isolation of the lines), and sweat pores are considered as features that can distinguish different personalized fingerprints from each other. The singular point that usually appears in the majority and in the center of the fingerprint is also called the core singular point. If the core singular point is contained in the part of the fingerprint that is applied, the center point of the core singular point can be used to locate the finger on the display screen. The motion pattern can be obtained, for example, by tracking the center point of the core singular point in the image sequence.

[0004] After the fingerprint image has been captured, the features of the fingerprint must be extracted from the image data. For this purpose, an orientation image is generated, for example, by dividing the captured image of the finger into individual blocks, which are then analyzed one by one. Then, in each block that is larger than a line, the local orientation of the line is determined, that is, the angle between the tangent of the line and the predetermined coordinate system axis. A comparison with the results of other blocks is then performed in order to determine the reliability of the analysis. Another way to extract features from the fingerprint image is to generate a frequency image, which indicates how many lines of a specific orientation are present in a smaller local image area.

[0005] In principle, the specific and unambiguous properties of the fingerprint, namely the positions of the detail features, can then be determined from the orientation image using known mathematical methods. In addition, the orientation of the ridges of the finger lines can also be determined in each case in a smaller local environment (so-called blocks) and used as a property of the fingerprint. By evaluating the adjacent local environments / blocks, the above-mentioned singular points (e.g., bows, scoops and buckets) can be determined from these local orientations, which describe the shape of the ridges in an image area consisting of a plurality of blocks.

[0006] Finally, a comparison is also performed between the pattern recorded and analyzed during operation as described above and a plurality of patterns in the database. The so-called "matching methods" used for this purpose can be, for example, based on correlation (in which the images are compared pixel by pixel and the degree of consistency is determined) or based on detail features (in which the position and orientation of the detail features are compared), the latter being determined by the position in a two-dimensional coordinate system and the orientation therein. Methods based on texture recognition can also be used, in which the comparison is performed in combination with the frequency, orientation and shape of the texture.

[0007] After determining the static pattern and the fingerprint characteristics, the combination of at least one static pattern, the fingerprint characteristics and, if necessary, at least one motion pattern is compared with a database. In the database, for example, typically multiple fingerprints are stored for users of mobile phones, ideally also with different fingerprint characteristics and motion patterns. Afterwards, the situation is differentiated depending on whether the combination is stored or not stored in the database. If the combination is stored in the database, it is checked whether a predetermined action is associated with the combination for the enabled application, and then the predetermined action is performed. If no action is associated with the combination, a first standard action is performed. If the combination is not stored in the database, a second standard action is performed. Here, the first and second standard actions can also be the same, for example, it can be displayed to the user that no action is associated with his input, a sound signal can be emitted, or a vibration signal can be triggered.

[0008] Optical fingerprint recorders are already known in the prior art, and planar fingerprint recorders are also known, as described, for example, in DE 10 2015 115 381 A1 and the unpublished German patent No. 102017 119 983. The combination of a fingerprint recorder and a screen is also known in the prior art, for example, in US 2010 / 0231356 A1 and US 8,745,490 B2, mobile input devices are described, which combine a touch-sensitive sensor with a fingerprint recorder and a display screen, and the corresponding sensors or LCD modules that implement the screen are arranged in the aforementioned order from the perspective of the observer. The method described in B2 of US 8,745,490 also discloses the possibility of recording multiple fingerprints at the same time and triggering actions depending on the fingerprints, wherein the fingerprints are basically used for user authentication, and also discloses the function equivalent to a touch-sensitive sensor, that is, the corresponding touch-sensitive element is displayed on the screen, when the finger is authenticated, the action is associated with it, and then the action is performed. Movements can also be recorded, as is also described in US 8,745,490 B2. US 2018 / 0173867 A1 also describes the combination of fingerprints with different gestures, such as the rotation of a finger on the screen around an axis of rotation perpendicular to the plane of the screen.

[0009] However, the applications known in the prior art, which combine fingerprint sensors with screen displays, are limited to using the fingerprint for authentication. If the fingerprint is authenticated, further processing is performed corresponding to the working method in conventional touch-sensitive screens, also selected from US 2018 / 0173867A1, in which the fingerprint is not used to determine the rotation of the finger on the display, but an additional signal of the acceleration sensor is used.

[0010] By recognizing the fingerprint, only additional security features are added compared to conventional touch-sensitive sensors and gesture recognition. Thus, the functionality is only slightly expanded compared to known touch-sensitive screens.

[0011] However, there are numerous applications in which it is desirable to assign more functions to the fingers, in order in particular to also enable intuitive operation. In freehand drawing or calligraphic symbol applications, it is already possible to control the thickness of the line with the help of a pressure-sensitive sensor when drawing with the finger (the greater the pressure applied to the sensor from above, the thicker the stroke. In reality, however, the stroke shape can be controlled not only by the contact force, but also, for example, by the angle between the pen tip and the drawing plane (i.e. the screen plane) or by the angle between the pen longitudinal axis and the direction of movement. Summary of the invention

[0012] The object of the present invention is therefore to expand the fingerprint recognition in such a way that a variable, detailed and as intuitive as possible control of the application is possible based on the fingerprint recognition.

[0013] In a method of the aforementioned type, this object is achieved in that the orientation of at least one finger on the screen (with respect to the orientation angle α), the rotation angle γ of at least one finger on the screen and / or the angle of attack β are determined as further fingerprint characteristics of the finger, that is to say as at least one of the fingerprint characteristics. Together, these three angles or characteristics clearly define the position of the finger relative to the display plane.

[0014] The angle of attack β is understood to be the angle between the first knuckle and the plane of the screen. At an angle β = 0°, the finger is recorded lying flat, and at an angle β = 90°, only the tip of the fingertip is recorded. The smaller the angle from 90°, the more the fingertip is seen, until finally the entire finger is recorded lying flat. Based on the area of ​​placement, it can be determined by comparing it with the fingerprint pattern stored in the database at which the first knuckle is maintained. The orientation of the finger on the screen can also be represented by the angle on the screen relative to a predetermined axis, i.e. the orientation angle α.

[0015] Another fingerprint characteristic is the rotation angle γ. A finger lying flat on the screen with its lower side corresponds to a virtual plane of the palm, which is parallel to the plane of the screen, i.e. forms an angle of 0° with the plane of the screen. The rotation angle in this case is γ=0°. In order to change the rotation angle, the flatly lying finger is rotated around the longitudinal axis of the finger as described above. This can achieve an angle of up to approximately γ=+90° (e.g. clockwise) or approximately γ=-90° (e.g. counterclockwise). These two extreme angles correspond to the back of the hand standing vertically relative to the plane of the screen. However, the rotation angle also plays a role when the finger is not lying flat on the plane of the screen. When the finger is bent, the rotation angle always describes the longitudinal axis of the finger about which the finger extends centrally through the finger, at least the longitudinal axis of the first knuckle.

[0016] The third fingerprint property to be determined is the orientation of at least one finger on the screen. This can be determined in conjunction with a static pattern (as part of the entire fingerprint) even when not the entire finger is placed flat, but for example only the fingertip. The orientation, i.e. the orientation angle α, can then already be determined in conjunction with the entire fingerprint stored in the database.

[0017] When registering authorized users, in an initial step, an almost complete skin texture pattern of the underside of the finger, the side of the finger and the fingertip can be recorded by rolling the finger, for example, about its longitudinal axis on the sensor or display screen and also "rolling" the fingertip over the sensor. These image data can then be pieced together to form a coherent skin texture of the relevant area within the scope of the method. During operation, the finger parts, the local skin texture (static pattern) recorded within the operating range of the device can then be compared with the pieced-together skin texture; in this way, it can be determined which part of the finger is resting on the display screen. From this, further fingerprint properties can be determined, namely the orientation of the finger relative to the screen plane, that is, the orientation with an orientation angle α on the screen plane, the angle of attack β relative to the screen plane and the rotation angle γ for lateral rotation about the finger axis, for example an orientation with α=20°, an angle of attack β=45° and a rotation angle γ=+30°.

[0018] Alternatively or additionally, patterns for different angles of attack and rotation angles are also stored in the database, so that the rotation angle and angle of attack can be determined by interpolation if necessary. However, this is more time-consuming and occupies more storage space than the above-mentioned comparison, because overlapping areas must exist when the patterns are stored, so that parts of the fingerprint pattern are stored redundantly and more calculation operations or comparisons must be performed when comparing the stored patterns with the captured pattern pieces.

[0019] In this way, it is possible to relate not only the function to the finger itself, but also the additional function to the posture of the finger. Here, the change of the posture can be used to control the control parameters of these related functions.

[0020] Particularly preferably, in the activated application, the fingerprint and fingerprint characteristics, and optionally also the movement pattern, of at least two fingers placed on the screen are determined simultaneously by means of the fingerprint recorder. The combination of at least two static patterns, fingerprint characteristics and optionally the movement pattern of at least two fingers is then compared with a database. In this way, the range of possible functions is further increased.

[0021] Here, the orientation of the finger by means of the orientation angle α, the angle of attack β and / or the angle of rotation γ are advantageously determined in the following manner, that is, firstly a static pattern of the fingerprint is determined and then a check is made as to whether the fingerprint is stored in a database. If a fingerprint is stored, the rotation angle and / or the angle of attack are calculated based on a comparison of the static pattern with the stored fingerprint or the fingerprint characteristics of the stored static pattern. The static pattern measured and extracted from the image represents only a fragment of the entire recorded finger or skin texture, which is determined by the rolling of the finger around the rotation axis when the finger and the fingertip are placed, so that the key to this is which fragment of the stored fingerprint corresponds to the static pattern, that is, the measured partial texture of the finger. If no fingerprint is stored, the orientation angle α, the rotation angle γ and / or the angle of attack β are calculated based on the fingerprint characteristics with the static pattern stored in the database. Therefore, if no fingerprint is stored, it is still possible to estimate how large the orientation angle α, the rotation angle γ and the angle of attack β are based on the comparison with the so-called pattern texture. Ideally, for this purpose, corresponding data columns are stored in the database for different finger sizes in order to keep the possible error probability as small as possible. Furthermore, the user can also add to the database the stored fingerprint patterns of his own finger (coherent skin texture) and the texture characteristics corresponding thereto. The creation of such a coherent skin texture can be carried out with the aid of a corresponding query procedure, as is generally known in the prior art, for example by means of a method for rolling the finger around its longitudinal axis and rolling the fingertip or by piecing together the individual recorded parts of the fingerprint, which are known in the prior art, for example, from DE 10 2016 114 188 A1, which describes the creation of a rolled texture, in particular in the case of a device with a superimposed combined sensor and display, or from US 9,996,728 B2, which describes a method for piecing together a fingerprint from smaller individual recorded partial textures. This method or a combination of similar methods can also be used, for example, to generate a coherent skin texture, which includes the relevant finger area within the scope of the method described here.

[0022] Different possibilities are provided to make the design of the screen suitable for executing the method. In a feasible design scheme, the second pixel grid with sensor elements is located above the first pixel grid composed of image elements. In order not to significantly affect the brightness of the screen element, the second pixel grid of the sensor element is preferably at least partially transparent. Of course, the second pixel grid with sensor elements can also be arranged below the first pixel grid of the image element from the perspective of the screen observer, and the latter is at least partially transparent. The advantage is that the brightness of the screen is not affected. The first pixel grid of the image element can be constructed as an OLED, for example. The third possibility is that the first and second pixel grids are arranged in a plane, and the assumed screen and sensor still obtain the required resolution, which should be at least 250ppi (pixels per inch) for the sensor, so that the texture of the child's finger can still be captured so that the finger lines can be distinguished. For official applications, the resolution of the sensor should be at least 500ppi, because this can significantly better detect the features used for identity authentication. In order to improve the resolution of the captured fingerprint and also reduce the influence of ambient light, in a preferred design scheme, a shielding layer is arranged above the photosensitive sensor element of the second pixel grid to limit the detection angle range. In this case, the photosensitive sensor elements preferably overlap a larger part of the baffle in each case, so that only angles from the side are captured. A further improvement in resolution is achieved when the angular range captured from the side is further limited by the baffle, so that only light that is totally reflected on the placement surface reaches the photosensitive element. This can be achieved, for example, when the angular range captured is limited by the baffle to an angle greater than the limiting angle of total reflection in the case of a transition from a protective glass layer to the surrounding air. The limiting angle of total reflection depends on the material combination and is, for example, between 40° and 43° for a transition from glass to air, depending on the type of glass, and between 33° and 36° for a transition from ceramic to air, depending on the ceramic. The angular range can be limited in the horizontal and vertical direction by means of the baffle. In this way, the influence of the horizontal and vertical divergence of the illumination light is reduced, so that a thicker top layer can be used while the resolution remains the same, for example to improve screen protection.

[0023] In the simplest case, an image element is used for illumination when recording a fingerprint, which can be achieved in particular if a first pixel grid of the image element is arranged below a second pixel grid of the light-sensitive sensor element. In this way, the number of required components is kept small, so that the display can be designed relatively thinly and the production costs are kept low.

[0024] In another design that enables more accurate fingerprint recording, light is coupled into a large-area sheet-shaped optical waveguide arranged above or below the second pixel grid and having a light coupling element arranged or constructed thereon for illumination during fingerprint recording. The typical thickness of the sheet-shaped optical waveguide is between 25 μm and 5 mm, wherein a preferred thickness for mobile input devices is 0.05 mm. Here, the light is coupled in at such an angle that, in a purely sheet-shaped design of the optical waveguide without a light coupling element, the light propagates in the optical waveguide only by total reflection, so that it can only leave the optical waveguide at the end side. With the aid of the light coupling element, which can be designed, for example, in a prismatic shape, the light is coupled out of the optical waveguide, undergoes a change in direction, and leaves the optical waveguide in the direction of the placement surface at an angle within a predetermined range relative to the normal to the screen surface, for example, ±71°. The illumination light passes not only through the transparent area of ​​the sensor layer, but also through the top layer, the two layers being connected by an adhesive medium layer that is transparent to the light used. Here, total reflection occurs again at the boundary surface between the top layer and the air, where the light is incident at an angle of approximately 60°, and the illumination light is reflected back in the direction of the sensor layer and detected there by the photosensitive elements of the sensor layer. If the skin is located on the contact surface with skin protrusions and skin depressions, interfering total internal reflection (TIR) ​​occurs at the location of the contacted skin area and the illumination light is coupled out. In this way, the contrast between the skin depressions and the skin protrusions is imaged, because in the area of ​​the skin depressions the light is further totally reflected, which makes the image brighter. By using illumination as described above, which is precisely defined with regard to the coupling-out angle from the optical waveguide, it is possible to use a very narrow angular range that is limited overall by total reflection. This results in very high-resolution and high-contrast fingerprint recordings even when the sensor surface and the placement surface may be at a greater distance from each other (for example due to an additional protective layer between the sensor and the finger) than with diffuse illumination, because with diffuse illumination and without restrictions on the direction of the recorded light, a sharp image can only be produced if the sensor plane and the finger are at a smaller distance from each other than the distance between the midpoints of adjacent sensor pixels, wherein the image sharpness increases at smaller distances.

[0025] In another design, the placement of a finger is recorded using a touch-sensitive touch sensor layer, and the screen is preferably equipped with a touch-sensitive touch sensor layer. The recording is then triggered with the help of a fingerprint recorder. In this way, energy can be saved, especially when the fingerprint recorder is operated with an additional light source to record fingerprints and fingerprint characteristics, as described above. In addition, the use of a touch sensor layer also provides another advantage: the touch sensor already knows the position of the finger, that is, the position and area of ​​the finger placed on the screen. The fingerprint can then be detected and / or the recorded signal evaluated by the sensor layer limited to this two-dimensional area. This limitation of the evaluation makes it possible, on the one hand, to reduce the computational costs when evaluating the sensor image, and on the other hand, to make the detection of fingerprints more stable.

[0026] Advantageous embodiments of the method are described below, which in part also relate to specific applications.

[0027] Here, the basic function is used for authentication input. In a preferred design of the method, the screen is used as an input device or the application of the application or the selected function in the application is released only when a combination of a static pattern, a motion pattern and at least a part of the fingerprint characteristics is stored in a database and linked to the release function. In this design, it is necessary to place at least two fingers on the screen. Security is increased by additionally asking for at least one other, i.e. at least a second fingerprint and the corresponding fingerprint characteristics. For example, before releasing the use of a banking application, it can be required that the left thumb must be placed at an angle of attack of 0° and the right index finger must be placed at an angle of attack of 90° on a specific position on the screen that can be displayed. Here, the display is designed so that it only illuminates the area, without telling the user how to place the finger on the screen, which can be specified by the user in the corresponding application in advance.

[0028] A prerequisite for this authentication is that the combination of a static pattern and at least part of the fingerprint characteristics is stored in a database. For this purpose, the mobile input device must be initialized or learned, that is, the fingerprint of the user to be authorized must be stored in the database as completely as possible as a skin texture. This is done using the above-mentioned method, in which, for example, the finger is placed against the user and then rolled as far as possible to the sides at a rotation angle, and the fingertip is also rolled.

[0029] The fingerprint characteristics required for the release are preferably stored directly by the user, but can also be extracted from the fingerprint stored during initialization given the angular position required for the release and compared with the recorded static pattern.

[0030] In another preferred design scheme in which at least two fingers are also placed on the screen at the same time, for a combination of a static pattern, a fingerprint characteristic and, if necessary, a motion pattern, before checking whether the combination is associated with a predetermined action, a check is made as to whether the first static pattern of the at least two static patterns corresponds to a predetermined activation pattern for activating a multi-fingerprint record. Only if this is the case, a further check is performed and an action is triggered if necessary. For example, it can be predetermined for a left-handed person that the right thumb is predetermined as an activation pattern at an angle of attack of approximately 45° (wherein an error tolerance can be given here). This type of multi-fingerprint record avoids accidental operation inputs after activation on the one hand and, on the other hand, enables the execution of actions of high or low complexity but frequently used actions to be implemented with only a few fingers or finger gestures.

[0031] The application can be, for example, a text input function. The first static pattern of at least two static patterns is equivalent to an activation pattern, and depending on the second static pattern, for example, lowercase, uppercase, numeric mode or special symbol mode is set as the writing mode. Each mode can be associated with a specific finger of the hand, but the switch between uppercase and lowercase can also be combined with different angles of attack of one and the same finger, which can be checked during input, so that, for example, the relevant letters are capitalized at an angle of attack greater than 70°, and lowercase at an angle of attack less than this value.

[0032] In another example, a first static pattern of at least two static patterns corresponds to an activation pattern, and depending on the second static pattern and optionally the further static patterns, the functions of a computer mouse are emulated, wherein preferably one finger is respectively associated with a button or a scroll element of the computer mouse. Depending on the emulated element, further fingerprint characteristics or movement patterns can also be associated with specific functions. For example, the finger to which the scroll wheel function is assigned can also assume a scroll function (e.g., associated with a search function) in the following manner, that is, for example, a scrolling movement in one direction is assigned to a sliding movement of the finger upwards on the screen and a scrolling movement in the other direction is assigned to a sliding movement of the finger downwards on the screen.

[0033] For a large number of applications (when the first of at least two static patterns corresponds to the activation pattern), the zoom and mark functions can be implemented depending on the second static pattern. In a web browser, for example, the zoom function can be implemented by a corresponding movement pattern of a finger, such as the middle finger: when another finger, such as the index finger, slides over the relevant text position, the text can be marked, as long as the first activation pattern is recorded once or continuously.

[0034] In another design, the first static pattern of at least two static patterns is equivalent to the activation pattern, and the repeated occupation of the input element is achieved by different second static patterns, wherein each second static pattern corresponds to another finger. The input element is understood here as an element shown on the screen, such as an icon of a folder or other application. So far, this is usually achieved by touching for different lengths of time or applying different fixed pressures to the input element, but this often leads to unsatisfactory results in practice, because on the one hand, it either brings a longer duration or on the other hand, the pressure difference makes it inaccurate to use. Compared with the measures used so far, this type, text or image element copying is faster, more reliable and more comfortable. In the measures used so far, in order to mark, it is necessary to press on the touch screen for about 1 second, after which the element touched by the finger is marked, and then the boundary of the marked area can be changed forward or backward, so as to mark more or less content. Different functions can be quickly and accurately distinguished by the difference in fingerprints of fingers, which helps to provide greater benefits when users input or interact with users. In this case, repeated occupation of the input element can be achieved not only by different second static patterns but also by a unique second static pattern corresponding to a fingerprint, wherein different occupations correspond to different orientations, angles of attack and / or angles of rotation within the fingerprint characteristic.

[0035] The aforementioned process can also be used for particularly reliable authentication. If a first static pattern of at least two static patterns corresponds to an activation pattern, the authentication process is only started depending on at least one second static pattern. Preferably, the authentication process is only started if the fingerprint characteristics and / or the motion pattern of at least one second static pattern correspond to a predetermined value.

[0036] In a more simplified approach to authentication using a finger, the query for the activation pattern is omitted and only a check is made for the static pattern to see whether the fingerprint characteristics and / or the motion pattern correspond to predetermined values.

[0037] Furthermore, a large number of applications can be implemented in which the checking of the activation pattern can be omitted. Therefore, a switch can be provided in the basic settings of the mobile device, which allows multiple finger recording, as long as this has been enabled in the settings for the respective application. It is also possible to enter a first static pattern, the activation pattern, in advance in order to start the multi-fingerprint recording mode, which runs until the first of the two static patterns is recorded again, if necessary at a position provided for this purpose.

[0038] Furthermore, many applications are conceivable in which the separate activation of the multi-finger recording mode can be omitted and in which additional functions result from the fact that in addition to the fingerprint and possibly the movement pattern, further fingerprint properties, namely the orientation, the rotation angle or the angle of attack, are recorded.

[0039] In an enabled application, a change in the angle of attack or the angle of rotation of at least one finger can, for example, change the volume or the size of a segment consisting of a two-dimensional or three-dimensional graphic. This can be used, for example, in more complex composition applications, in which different controllers control different effects, each of which can be assigned to a finger, and the angle of attack of the finger can, for example, set the volume of an effect (e.g. of a specific instrument), and the change in volume is achieved by changing the angle of attack, wherein a flat finger can, for example, correspond to a mute switch.

[0040] Changing the size of a segment consisting of a two-dimensional or three-dimensional image by means of a change in the angle of attack or the angle of rotation can be particularly useful, for example, in navigation applications. A change in the angle of attack of at least one finger can also be used to switch back and forth between a two-dimensional image and a three-dimensional image of an object. The finger used for this can be another finger than the one used to implement the zoom function.

[0041] In a particularly preferred embodiment, in an activated application, a change in the angle of attack of at least one finger changes the angle of attack of the actual or emulated writing tool for the screen. In this case, the angle of attack of the actual writing tool, for example a stylus, is determined by the angle of attack of the finger, and it does not matter at which angle the stylus is held relative to the screen surface. In this way, the function of the stylus, which can usually only change the line thickness by pressing the screen, can be improved. This is better achieved when the finger itself is used to emulate the writing tool.

[0042] In an enabled application, a change in the orientation of at least one finger can be taken into account for control, for example to rotate an object displayed on the screen, or in another application to change the orientation of a writing tool for actual emulation of the screen. If, for example, a pen is emulated with a finger, the direction of the pen can be defined in this way in addition to its angle of attack, thereby improving the user convenience of, for example, calligraphy applications.

[0043] Finally, the authentication process can also be performed more reliably with the finger by initializing the authentication process only when the static patterns determined by means of the fingerprint reader each have at least two predetermined fingerprint characteristics (e.g. angle of attack and orientation) and / or when two static patterns determined in a predetermined time interval each reflect a change in a predetermined manner of at least two fingerprint characteristics (e.g. finger rolling or finger angle of attack changes in a predetermined manner, wherein changes in direction can also be taken into account). Alternatively or additionally, the authentication process is initialized only when the movement pattern determined by means of the fingerprint reader corresponds to the predetermined movement pattern.

[0044] It is understood that the features mentioned above and still to be explained below cannot be used in the combination given but can also be used in other combinations or alone, without departing from the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The invention is further explained below by way of example with reference to the accompanying drawings which disclose the essential features of the invention. Among them:

[0046] Figure 1a , b is a flowchart showing a method for controlling a mobile input device;

[0047] Figure 2a , b shows the parameter setting of finger posture and fingerprint pattern;

[0048] Figure 3 Shows the comparison of static pattern and fingerprint;

[0049] Figure 4a -c shows different designs of the screen of the mobile input device;

[0050] Figure 5a -d shows another design of this screen;

[0051] Figure 6 details showing alternative designs of the screen; and

[0052] Figure 7-9 showing different operating possibilities;

[0053] Fig.10 A detail of another design of the screen is shown. DETAILED DESCRIPTION

[0054] FIG1 first shows the flow of a method for controlling a mobile input device having a screen, on which information can be displayed in a first pixel grid of a picture element, and in which a planar optical fingerprint reader is integrated, which covers at least half of the screen surface and has a second pixel grid composed of light-sensitive sensor elements. The structural design of the mobile input device is explained in detail below in conjunction with FIGS. 4-6.

[0055] For the flow of the method it is necessary that first of all an application is started either by the user or by the operating system, or that the application is already running. Applications are understood not only to be programs that must be started by the user, but also applications that are started by the operating system. E-mail programs, text processing programs, games, etc. are considered to be the former, while screen management (usually started by touch), such as the appearance of different icons, so-called icons, on the screen of a mobile input device, is considered to be the latter, applications started by the operating system. This prerequisite for an application to be started or already running is defined in Figure 1a Identified by A). In the next step B), one or more fingers are placed on the screen. In step C), fingerprint recording is enabled and images or image sequences are recorded with the help of a fingerprint recorder: in the enabled application, the fingerprint and multiple fingerprint characteristics of the finger are determined for at least one finger placed on the screen with the help of a fingerprint recorder. The positioning of the finger on the screen, the orientation of at least one finger on the screen, the rotation angle and / or the angle of attack of at least one finger on the screen are all regarded as fingerprint characteristics. If at least one finger moves, the motion pattern is additionally recorded. The recording is performed in the following way, that is, the light intensity incident on the sensor element is recorded by the sensor element either once in an image at a predetermined time point or multiple times in an image sequence during a predetermined time period. The center of gravity of the static pattern of the finger area placed on the display screen can usually be known as the location of the finger, or alternatively, as described above, the center point of the core singularity is known. The motion pattern is obtained by tracking the location of the finger through a series of consecutive images.

[0056] In step D), the recorded intensity values ​​are pieced together by an image processing algorithm to form at least one static pattern of the fingerprint, which corresponds to a fragment of the entire fingerprint itself. If fingerprint characteristics cannot be determined during the recording with the fingerprint reader, such as in the case of positioning, further fingerprint characteristics, such as the angle of attack or the angle of rotation, are determined within the scope of image processing. If necessary, the motion pattern is also determined.

[0057] Finally, in step E), the combination of the static pattern of at least one finger (ie a fragment of its entire fingerprint), fingerprint characteristics and, if necessary, the motion pattern is compared with the database. Figure 1a1 ). Once the mentioned combination is saved in the database, step G) is executed and a check is made for the enabled application whether a predetermined action is associated with the combination, which is then executed, which is equivalent to step I). If the mentioned combination is saved in the database but there is no associated action for the enabled application, a first standard action is executed (step J) in FIG. 1 ). If the combination itself is not saved in the database at all, a second standard action is executed (step H) in FIG. 1 ). The first and second standard actions can also be identical and, for example, an error message is displayed in the following manner, i.e., no input is associated with the action.

[0058] Figure 1b Elaborated Figure 1a Step D). Step i) shows the initial situation, which includes a finger 3 (here two fingers 3.1 and 3.2) placed on the screen 1 of the mobile input device 2. The fingerprint recorder first captures a grayscale image, without distinguishing according to different colors, because the key to this is the contrast between the skin convexities and the skin concavities of the fingerprint. Based on the grayscale image shown in step ii), a binary mask image is generated, which is shown in step iii). The mask is used for filtering, that is, only the intensity values ​​recorded within the white area of ​​the binary mask image shown in step iii) are taken into account, and the cleaned grayscale image shown in step iv) is obtained. Then, based on the cleaned grayscale image, in another process, as shown Figure 1b As shown in step v) and as described above, a static pattern of the finger, i.e. a segment of the entire fingerprint, and further fingerprint properties, such as position, orientation or orientation angle α, angle of attack β and / or rotation angle γ, are determined and stored in a corresponding data structure for further processing.

[0059] exist Figure 1bIn the example shown in , in the enabled application, the fingerprint or the static pattern of these fingerprints, the fingerprint characteristics and the possible motion pattern are determined by means of a fingerprint recorder for at least two fingers placed on the screen at the same time, and then they are compared with the database. However, in order to perform the method and expand the control possibilities compared with traditional mobile input devices, it is sufficient to determine the fingerprint characteristics of one finger. The positioning and orientation (when the finger is placed flat) of one or more fingers are easily derived from the captured image, while the angle of attack β and the angle of rotation γ are not the case. In order to determine the angle of attack and the angle of rotation, the static pattern of the finger is first determined and it is checked whether the fingerprint has been saved in the database. In the case of a saved fingerprint, the orientation, the angle of rotation γ and / or the angle of attack β are determined by calculation in combination with the comparison of the static pattern with the saved fingerprint pattern and thus in combination with the fingerprint characteristics (which can be derived from it) belonging to the saved fingerprint. This can be done in the following way, that is, for example, in advance for initialization to capture the finger images under different rotation angles and / or angles of attack, but it is better to do it in the following way, that is, as mentioned above, for example, by rolling the finger on the screen and by rolling the fingertip on the screen to capture the entire fingerprint. If the fingerprint is not saved, the rotation angle and angle of attack can still be calculated in conjunction with a comparison to the static, averaged, and stored fingerprint characteristics in the database.

[0060] This approach should be combined with Figure 2a , 2b and Figure 3 Elaborate in detail. Figure 2a A finger 3 is shown placed against the screen. The screen, in which the fingerprint reader is integrated, lies in a plane spanned by the x and y axes of the Cartesian coordinate system; the direction z corresponds to the normal of the screen. The position of the finger on the screen can be described by the orientation, angle of attack and angle of rotation of the finger. The orientation is specified relative to a predetermined axis in the plane of the screen, in Figure 2a In the example shown in , this orientation is described by the orientation angle α in the screen plane (measured about the axis y' parallel to the y-axis). The angle of attack β indicates how much the first joint of the finger 3 is inclined relative to the screen plane and is measured relative to this screen plane. Finally, the angle of rotation γ describes the rotation of the finger 3 about its longitudinal axis. If this rotation takes place on the screen, it also means that the finger 3 "rolls". All three angles together clearly describe the posture (also called gesture) of the relevant finger relative to the screen, and in principle each 3-tuple value can be assigned a function, wherein it should be taken into account that in particular the angle of attack β and the angle of rotation γ can only be determined with relatively large tolerances, because the contact pressure can affect the size of the static pattern. If the contact pressure is additionally recorded, these tolerances can be reduced.

[0061] In addition, the orientation of the ridges of the fingertips can also be determined in smaller local areas (so-called patches) and used as fingerprint characteristics; this is Figure 2b According to these local orientations (in Figure 2b The above singularities (e.g. bucket, dustpan and bow) can be calculated by evaluating the adjacent local environment / blocks as follows: Figure 2b , which illustrate the shape of the texture in an image area composed of a plurality of blocks.

[0062] Figure 3 The comparison of a static pattern M with a complete fingerprint F stored in a database is shown. The complete fingerprint (usually only the first section of the finger) is determined in the above-described manner by rolling a flat finger on a fingerprint recorder and based on the rolling of the fingertip, so that the complete fingerprint is a pattern with a relatively large area, which is usually approximately rectangular. The static pattern M is recorded by the fingerprint recorder and represents the part of the complete fingerprint that is currently placed on the display. The determination of the fingerprint and its properties or features is carried out as is known in the prior art and has been explained at the beginning. First, the static pattern is compared to see which of the stored complete fingerprints belongs to the static pattern. If the complete fingerprint F is Figure 3 If the example shown in is present and found, then the orientation angle α, the rotation angle γ and the attack angle β are calculated in combination with the pattern comparison, that is, the segment of the entire fingerprint F corresponding to the static pattern M is found. Here, the comparison can include correlation, feature details, orientation image, frequency image or the shape of the texture, which is also known as described above and from the prior art described there. The position of the center of gravity of the static pattern M in the entire fingerprint F, for example, determines the attack angle β and the rotation angle γ, and the rotation of the static pattern M relative to the entire fingerprint F determines the orientation angle α. In the example shown, the position of the center of the circular static pattern M in the coordinate system developed by the rotation angle γ and the attack angle β corresponds to the numerical value that can be assigned to the static pattern M. Here, the tolerance is larger for the attack angle β, because the determination of the attack angle may be inaccurate due to the correlation with the placement pressure. In order to determine the attack angle β and the rotation angle γ, it is also advantageous to consider the shape and size of the static pattern M at the same time, wherein, for example, when the attack angle is more likely to be small, the pattern M is more likely to be large and have an elongated shape. However, due to individual anatomical reasons, the size and shape of the pattern M cannot be taken into account for all fingers or persons.

[0063] In conjunction with Figures 4-6, different possibilities for a mobile input device in the form of a screen in a layer stack should now be described. Figure 4a-4cIn the following, the general structure of such a layer stack is first briefly described. The layer stack is delimited toward the user by a placement surface 4. This placement surface is in turn part of a protective layer 5 for protecting the pixel grid of the image element and the sensor elements. The sensor layer 6, which contains a second pixel grid composed of light-sensitive sensor elements, is disposed on the surface of the sensor layer. Figure 4a In the embodiment, the first pixel grid of the picture element is arranged below the protective layer 5, but above the display layer 7, into which the first pixel grid of the picture element is integrated. Figure 4b In the embodiment, the sensor layer 6 is arranged below the display layer 7. Figure 4c In the embodiment, the sensor layer 6 and the display layer 7 are identical, wherein the sensor elements and the image elements are arranged next to each other. The illumination for recording the fingerprint is Figure 4a-4c In the embodiment shown, this is done with the aid of a screen element of a display layer 7, which can be, for example, an OLED layer. If the display layer 7 is arranged above the sensor layer, the image element must be light-transmissive so that part of the light that is returned from the finger to the protective layer 5 can be incident on the sensor element. Here, placing a finger on the display or screen is always specifically understood to mean placing the finger on the uppermost, accessible layer of the layer stack, that is, on the placement surface 4.

[0064] These layers are connected to one another by a transparent adhesive medium (not shown). Furthermore, a shielding layer (also not shown here) is arranged to limit the detection angle range of the sensor element beyond the second pixel grid.

[0065] In another preferred design that can also be combined with a shielding layer, light is coupled into a sheet-like optical waveguide having a large-area arranged or constructed light coupling-out element arranged above or below the second pixel grid for illumination when recording a fingerprint. If the fingertip of the finger 3 is not located on the placement surface 4, the light is coupled out via the light coupling-out element in the direction of the placement surface 4 for the finger 3 and is totally reflected there. Figure 5a-5d A configuration of this type is shown in , wherein a plate-like optical waveguide is symbolized by a luminous layer 8 , into which light is coupled in from the narrow side. Figure 5a It is shown in FIG. 1 that the luminescent layer 8 is arranged below the sensor layer 6 (which comprises the second pixel grid with the sensor elements) but above the display layer 7 (which comprises the first pixel grid with the picture elements). Figure 5b A design is shown in which the luminous layer 8 is arranged between the sensor layer 6 and the display layer 7, wherein the sensor layer 6 is arranged below the optical waveguide layer 8 when viewed from the observer. Since the light is totally reflected within the luminous layer 8 and can only leave via a light coupling-out element (not shown here), the luminous layer can also be arranged above the sensor layer 6 without interfering with the sensitivity of the light-sensitive sensor element and without causing disadvantageous brightening. Figure 5cA further embodiment is shown in which two pixel grids are combined to form a common layer consisting of sensor elements and image elements, and in which the luminous layer 8 is arranged below this layer. Figure 5d The common sensor display layers 6, 7 and the luminous layer 8 are shown relative to Figure 5c In reverse order, the luminous layer is located here above the combined sensor and display layer 6 , 7 .

[0066] Figure 6 Show according to Figure 5c Detail of the view in which the sensor layer 6 and the display layer 7 are arranged directly one above the other to form a common layer. The transparent screen elements 9 are arranged above the sensor elements 10, but they themselves do not necessarily have to be transparent. The resolution of the second pixel grid formed by the sensor elements 10 is smaller than the resolution of the first pixel grid formed by the image elements 9, and the light from the luminous layer 8 can be coupled out and then travel upward through the gaps between the sensor elements 10. In addition, it is also possible that, unlike in this case, Figure 6 In the embodiment shown in FIG. , sensor element 10 and screen element 9 are arranged in one plane.

[0067] Furthermore, the screen formed by the layer stack is additionally equipped with a touch-sensitive touch sensor layer (not shown here), which registers the contact of a finger and triggers the recording by means of a fingerprint reader.

[0068] The recording of additional fingerprint properties opens up a wide range of different application possibilities, including the expansion of existing programs. Figure 7-9 In FIG. 4 , different application programs are shown by way of example, all of which apply the above-mentioned method. Figure 7 For example, an application is shown in which four fingers 3.1, 3.2, 3.3 and 3.4 of the right hand are placed on the screen and the application is started or the permission for an operation (e.g. online payment) performed by the application is released only when the fingerprint (i.e. static pattern) and possibly other fingerprint characteristics (such as position, rotation angle and angle of attack) correspond to the pattern combination stored in the database. For example, it can be provided that the application using the screen as an input device or the application or the selected function in the application is released only when the combination of the static pattern and at least part of the fingerprint characteristics is stored in the database and is associated with the release function.

[0069] In another embodiment of the method, for a combination of a static pattern, a fingerprint characteristic and possibly a motion pattern, before checking whether a predetermined action is associated with the combination, it is checked whether the first of at least two static patterns corresponds to a predetermined activation pattern for activating multiple fingerprint recording, and only if this is the case, a further check is performed and an action may be triggered. This is for example in Figure 8 In this case, the thumb 3.5 of the left hand and the thumb 3.5 of the right hand are placed on the screen 1 of the mobile input device 2. In the case of right-handedness, for example, the fingerprint of the left thumb can be predetermined as the activation pattern, if desired also with the position, angle of attack and / or angle of rotation and orientation. In this case, a multi-fingerprint record is switched on, and depending on which finger of the right hand and possibly where on the screen the finger is placed (wherein multiple fingers can also be placed), a specific function can be realized. If Figure 8 If the enabled application in is, for example, a text processing program and the text input function has been enabled, the fingers of the right hand can be respectively assigned to the writing mode, for example, the thumb can be assigned to lowercase, the index finger to uppercase, another finger to the digital input pattern, and the fourth finger to the special symbol mode, so that when one of these fingers is placed, the corresponding writing mode belonging to the finger is set in addition to the left thumb. For easy identification, the corresponding keys are preferably shown on the screen, wherein each key will be repeatedly occupied in the described manner. The respectively enabled modes can be highlighted. For simplification, the method can be performed as follows, that is, the activation pattern only needs to be identified once and until the end, the left thumb no longer needs to be placed on the screen 1 in this example, but only the right hand finger is used to change the writing mode without the left thumb being placed. In the same way, the fingers are also assigned to the functions of the computer mouse; the fingers are respectively assigned to the buttons or scroll elements of the computer mouse, for example, the middle finger is assigned to the scroll wheel, the ring finger is assigned to the right mouse button and the index finger is assigned to the left mouse button. In a text processing program or on a web page, the right thumb or other fingers of the right hand can also be assigned different zoom or marking functions that are then carried out, while the left thumb that confirms the activation pattern is not considered for this action, that is, it is not assigned such a function.

[0070] The method can also be used to achieve repeated occupation of the input element by different second static patterns, each of which corresponds to a different finger. Fig. 9. Here, four different applications 11-14 are shown on the screen 1, namely the music application 11, the e-mail application 12, the folder application 13 and the phone application 14. If, for example, the index finger 3.1 is positioned on the application 11, the application can be opened and presented with the option of playing tracks. And if the middle finger 3.2 is positioned on the music application 11, the finger can be used, for example, to associate the switch between the "play" and "pause" functions. These functions can also be implemented without the need for activation by the recognition of the activation pattern, for example, by overall settings or application-related settings. Repeated occupation in this case is achieved by associating different functions depending on the fingerprint for the music application 11 (if it is displayed as an icon on the screen 1). Repeated occupation of the input element can also be achieved alternatively or supplementarily by different orientations, angles of attack and / or rotation angles within the fingerprint characteristics. In this way, the "fast forward" and "reverse" functions can be implemented, for example, by different rotation angles of the finger, preferably the index finger, wherein the rotation angle should have different signs. Fast forward corresponds to a positive rotation angle, ie, tilting the index finger to the right, while reverse is triggered by rolling the index finger to the left.

[0071] The change of the angle of attack of the index finger can be used to set the volume, for example. Here, of course, the volume setting can also be assigned to another finger (possibly additionally).

[0072] In mixing applications, in which music and audio signals are mixed or harmonized, the display of a slider and a rotary knob is implemented on the screen in the usual manner. The setting of the slider can be performed, for example, by setting the angle of attack of the finger associated with the slider, which can be associated with the orientation of the respective finger in the case of a rotary knob. Different sliders can be used to tune the treble, the midrange, the bass / bass, or also the sound superposition of an audio file. Through multi-fingerprint-based input, as described above, for example, the input category, such as the volume, can be specified using the fingers of one hand, and the volume of the individual channels can be adjusted using the fingers of the other hand, wherein advantageously, it is not necessary to look at the screen, because the setting range or the set value can also be intuitively recognized by the placement of the fingers. For example, a flat finger can always follow the minimum volume, and a finger with a pointed end can follow the maximum volume.

[0073] The multi-fingerprint input mode can also be used for navigation applications. In order to avoid unexpected actions, the prerequisite here can be that the multi-fingerprint input mode is first activated by placing a specific finger, that is, at least one of the static patterns is equivalent to the activation pattern. The setting of the angle of attack can be used to implement a zoom function in the map, for example, by first clicking on the relevant position of the map, and then implementing the map zoom at this location by changing the angle of attack. Alternatively, switching between two-dimensional and three-dimensional images can also be performed. The map can be rotated by rotating the finger.

[0074] In a CAD program, for example, by rotating the finger, the displayed component can be rotated, by tilting the finger, the component can be enlarged or brought closer to the screen surface, and vice versa, for example, by rolling the finger about an axis parallel to the screen plane and parallel to the flat finger, the marked component can be rotated relative to the screen plane.

[0075] With regard to authentication, it is particularly meaningful to establish a stronger security mechanism when the fingerprint sensor covers the entire screen. Therefore, when authentication only requires the placement of a fingerprint as is common so far, it may happen that the user unintentionally grants permission because he accidentally selects something else on the screen with the finger required for authentication and the authentication request appears at this moment. Therefore, it is meaningful for devices with comprehensive fingerprint sensors to additionally connect authentication with unusual security gestures in order to distinguish this process from normal, non-security-related device input processes. For example, the following inputs are used as such security gestures, which, on the one hand, authenticate the person via the fingerprint of one or more fingers used for the gesture, and on the other hand, manipulate security-related actions via the placement method (static patterns also contain fingerprint characteristics) or via movements that interpret the operator's intentions in combination with motion patterns. Static additional security gestures can, for example, consist in placing the fingertips of multiple fingers on the screen at the same time, where different fingerprint characteristics can also be additionally considered.

[0076] In summary, recording or determining additional fingerprint characteristics, namely the orientation of the finger on the screen, the angle of rotation on the screen and / or the angle of attack, allows the provision of additional degrees of freedom, so that each finger has multiple degrees of freedom, which are suitable for associating additional functions. In particular, the analysis of the angle of attack and the angle of rotation can be used to carry out graded input with the finger (without its movement on the screen), for example, similar to control devices such as rotary switches or scroll bar regulators. In contrast, in the prior art, mainly only binary input can be carried out with the finger, in which a symbol is pressed or not pressed; the graded input is carried out by moving the operating element on the screen, in which the finger must always be moved. Especially in the case of small screens, such as mobile phones, this is accompanied by higher inaccuracies, while the angle of attack can vary independently of the screen size, just like the angle of rotation. The evaluation of the pressure intensity of the finger on the touch-sensitive screen is also not very accurate when determining the pressure of the contact, for settings such as can be performed with a scroll bar regulator in a music application, and is too inconvenient for the user when controlling during the contact duration.

[0077] Finally, in Fig.10 1 shows a schematic diagram of an embodiment of a screen 1 having a first pixel grid consisting of image elements 9 and an integrated second pixel grid consisting of photosensitive sensor elements 10 for imaging a static finger line pattern (also referred to as finger pad structure) of at least one finger 3, which consists of finger pad lines P L and the concave part of the fingertip P T The screen can be used in a device for executing the method.

[0078] The screen 1 is a transparent or translucent body 101 of one or more layers, which has a placement surface 4, a first pixel grid 102 composed of image elements 9 integrated in a display plane 102.0, and a second pixel grid composed of photosensitive sensor elements 10 integrated in a sensor plane 103.0, wherein the sensor plane includes a large number of such photosensitive sensor elements 10 and a large number of regularly arranged angular shielding members 104 with shielding openings integrated in a shielding plane 104.0. Here, the placement surface 4 composed of the outer surface of the transparent or translucent body 101 and at least the sensor plane 103.0 and the shielding plane 104.0 are oriented parallel to each other. The thickness of the screen 1 is usually between 0.1 mm and 10 mm, preferably between 0.5 mm and 2 mm. The pixel-shaped image element 9 includes a plurality of sub-pixels R, G, B of the primary colors red, green and blue, respectively. The second pixel grid formed by the photosensitive sensor elements 10 comprises a plurality of regularly arranged photosensitive sensor elements 10, each of which is assigned one of the angle shields 104. The angle shields 104 are implemented as surface elements and each have a shielding opening, which is arranged and dimensioned relative to the sensor elements 10, or more precisely relative to their photosensitive surfaces, in such a way that only light reflected on the placement surface 4 or reflected on at least one placement finger 3 from a limited reflection angle range can pass through the shielding opening, that is, only light reflected on the parting surface 4.1 of the placement surface 4, preferably totally reflected light, can be incident on the sensor elements 10 obliquely. For this purpose, the shielding openings are each arranged upstream of the sensor elements 10 at a distance a in the vertical direction from the viewing direction to the placement surface 4 and in a parallel direction (offset direction R V The shielding opening is arranged parallel to the placement surface 4 and offset relative to the associated sensor element 10 in such a way that the shielding opening is arranged next to the sensor element 10 when viewed from a vertical direction toward the placement surface 4, so that the sensor element is completely covered. In order to control and read out the recording of the second pixel grid formed by the light-sensitive sensor elements 10 and to analyze and calculate it as necessary, these light-sensitive sensor elements or the second pixel grid are connected to a control unit and a computing unit.

[0079] exist Fig.10 In the embodiment shown in , the display plane 102.0 and the sensor plane 103.0 fall together, so that the shielding plane 104.0 is arranged in front of the display plane 102.0 in the direction of the light reflected on the placement surface 4. As a result, no scattered light or no light directly from the image elements 9 of the first pixel grid formed by the image elements 9 is incident on the photosensitive sensor elements 10, which are surrounded by scattered light shields 106 via a spacing a, and these scattered light shields are orthogonal to the angle shield 104.

[0080] The transparent or translucent body 101 can be implemented by layers of different materials with different refractive indices so that the light can be matched with the appearance of the image of at least one placed finger 3 in terms of wavelength range, reflection angle range or polarization direction, and / or in the case of an additional light source, the light used to illuminate the placement surface 4 is guided to the placement surface 4 as far as possible at an incident angle greater than the limit angle for total reflection.

[0081] The image element 9, which displays a primary light signal or a graphic representation to the user of the screen 1, can simultaneously serve as an illumination unit for the placement surface 4 and thus the placed object, in particular the finger 3. Alternatively or additionally, a background illumination 108 can be present and used as an illumination unit, which advantageously directs light in the invisible wavelength range through a first pixel grid formed by the image element 9, which is interlaced with a second pixel grid formed by the photosensitive sensor element 10, to the placement surface 4. Instead of the background illumination 108, there can be an additional illumination 109, which, for example, also directs light in the invisible wavelength range to the placement surface 4 only at angles greater than the limit angle for total reflection. If the totally reflected light is to be used for recording, it is important that the illumination unit emits at least part of the light that is incident on the placement surface 4 at an angle greater than / equal to the limit angle.

[0082] The light-sensitive sensor element 10 can advantageously have an electronic control unit as a shutter pixel for controlling the exposure time, for example as a rolling shutter or a global shutter, as disclosed in DE 10 2015 116 026 A1. The second pixel grid formed by the light-sensitive sensor element 10 is a shutter pixel sensor. This allows the exposure time and thus the insertion time to be adapted to different brightnesses of the display screen, which vary in user application scenarios or due to different ambient conditions.

[0083] The shielding openings preferably have a common shape. They can be, for example, circular, oval or preferably slot-shaped. The shielding openings are arranged in the offset direction R. V has a maximum width, which in the preferred case of a slot-shaped shielding opening (slot stop) is the slot width b. V In the vertical direction of the slotted plate, it has a maximum length, which in the preferred case of the slotted plate is the slot length l, which is perpendicular to Fig.10 extends across the drawing plane and is therefore not shown.

[0084] In a first alternative embodiment of the first embodiment of the screen 1 according to the invention or of the device according to the invention, the maximum length or preferably the gap length l is smaller than the maximum width or preferably the gap width b. It is so small that an isotropic contrast is obtained in the image acquisition with all the photosensitive sensor elements 10 of the pixel grid formed by the photosensitive sensor elements 10. Fig.10 In the embodiment shown in FIG. , all are staggered in the same direction R. V The offset relative to the light-sensitive sensor element 10 is however not mandatory. Two advantageous developments for shielding the opening are described below, both of which can be included in Fig.10 In the example shown, the figure shows the shielding opening only in cross section. According to a first advantageous embodiment, the shielding opening in the angle shield 104 is a rectangular slit 105 open toward the edge of the angle shield 104, which is bounded by an adjacent scattered light shield 106. In a second advantageous embodiment, the shielding opening is egg-shaped or elliptical. The smaller the maximum length or the slit length l in the preferred special case of the embodiment of the shielding opening as a slit 105, the smaller the (not shown) acceptance angle The smaller it is towards the length direction. In this direction, the gap length l can be further reduced by arranging the gap 105 (which in the case of a generally square light-sensitive sensor element 10 is arranged parallel to the side edges of the light-sensitive sensor element 10 in the direction of its gap length l) in the direction of the diagonal line. This can be particularly advantageous when a further reduction in the gap length l is not technically feasible or would imply significantly increased costs.

[0085] According to the sensor element 10 with the angular shield 104 arranged upstream, reference will be made to Fig.10 Explain the detected reflection angle range and the acceptance angle with respect to the gap width b The acceptance angle corresponds to the opening angle of the sensor element 10 with the baffle 104 .

[0086] An angular shield 104 with a slot 105 whose slot width b lies within the drawing plane is arranged upstream of the first sensor element 10 shown on the left. The reflection angle range, which is determined by the relative position of the slot 105 with respect to the sensor element 10 and by the slot width b and the dimensions of the sensor element 10 (here always meaning its photosensitive surface), is limited by the edge beam and forms a first angle τ (preferably greater than the limit angle of total reflection) and a second angle τ with respect to the perpendicular to the placement surface 4. To clamp, among which, is the acceptance angle in the direction of the gap width b, which can be set via the gap width b and the spacing a when the size of the sensor element 10 is predetermined. An angle shield 104 with a gap 105 can also be arranged upstream of another sensor element 10 (not shown in the figure), the gap length l of which is located in the drawing. By predetermining the gap length l, the acceptance angle can be set in the direction of the gap length l when the size of the sensor element 10 is predetermined.

[0087] Acceptance angle in the direction of the gap width b and the acceptance angle towards the gap length l In conjunction with the thickness of the transparent body 101 above the shielding plane 104 . 0 , the surface width and the surface length of the parting surface 4 . 1 , from which the light impinges on the respective sensor element 10 , are determined.

[0088] The basic structure of the display screen is independent of the geometrical embodiment of the sensor element 10, so that the sensor element advantageously has a regular equilateral polygonal shape. The slits 105 of the angle shields 104 arranged upstream are oriented in groups with a slit length l in each case parallel to one side of the sensor element 10. The sensor element 10 can also be designed, for example, as a circle or an ellipse.

[0089] Overall, the method presented here and the device presented here allow for an intuitive control of a greater number of applications than has been possible to date.

[0090] Reference numerals list

[0091] 1 Screen

[0092] 2 Mobile Input Devices

[0093] 3.3.1-3.5 fingers

[0094] 4 Placement surface

[0095] 4.1 Parting surface

[0096] 5. Protective layer

[0097] 6 Sensor Layer

[0098] 7 Display Layer

[0099] 8 Lighting Layer

[0100] 9 Graphics components

[0101] 10 Sensor element

[0102] 11-14 Application

[0103] 101 Subject

[0104] 102.0 Display plane

[0105] 102.2 Transparent Areas

[0106] 103.0 Sensor plane

[0107] 104 Angle shield

[0108] 104.0 Masking plane

[0109] 105 Gap

[0110] 106 Scattered light shielding parts

[0111] 108 Background lighting

[0112] 109 Additional lighting

[0113] α Orientation angle

[0114] β Angle of attack

[0115] γ Rotation angle

[0116] τ Angle

[0117] Acceptance Angle

[0118] a Spacing

[0119] b Gap width

[0120] x, y, z coordinate system

[0121] F Whole fingerprint

[0122] M Static Pattern

[0123] R, G, B sub-pixels

[0124] P L Finger line

[0125] P T Recessed part of finger pad

[0126] R V Staggered direction

Claims

1. A method for controlling a mobile input device (2) having a screen (1), on which information can be displayed in a first pixel grid formed by image elements (9), and in which a planar optical fingerprint reader is integrated, which covers at least half of the screen surface, and has a second pixel grid formed by light-sensitive sensor elements (10), wherein: - in an activated application, for at least one finger (3) placed on the screen (1), the fingerprint and a plurality of fingerprint characteristics of the finger are determined by means of the fingerprint recorder in the following manner, wherein the fingerprint characteristics include the positioning of the finger (3) on the screen (1) and the movement pattern of the finger (3), i.e., the light intensity incident on the sensor element (10) is recorded once in an image at a predetermined point in time, or the light intensity incident on the sensor element is recorded multiple times in a sequence of images during a predetermined period of time by the sensor element, - piecing together the recorded intensity values ​​into a static pattern (M) of at least one fingerprint and a motion pattern of said at least one finger by an image processing algorithm, - comparing the combination of at least one static pattern (M), the fingerprint characteristics and the motion pattern with a database, and, - If the combination is stored in the database, check: for enabled applications a. whether there is a subsequent scheduled action associated with the combination, or b. No action is associated with the combination, then the first standard action is performed, or - If the combination is not stored in the database, a second standard action is performed, characterized in that - determining as further fingerprint properties the orientation of at least one finger (3) on the screen (1) by means of an orientation angle (α), a rotation angle (γ) and / or an angle of attack (β) of at least one finger (3) on the screen (1).

2. The method according to claim 1, characterized in that In an enabled application, the fingerprint recorder is used to simultaneously determine fingerprints, fingerprint characteristics and motion patterns of at least two fingers placed on the screen (1), and the combination of at least two static patterns (M), fingerprint characteristics and motion patterns is compared with a database.

3. The method according to claim 1 or 2, characterized in that: The orientation angle (α), the angle of attack (β) and the angle of rotation (γ) of the finger (3) are determined in the following manner, i.e., firstly, a static pattern (M) of the fingerprint is determined and the following check is performed, i.e., whether the fingerprint is stored in the database, and when the fingerprint is stored, the orientation angle (α), the rotation angle (γ) and / or the angle of attack (β) are calculated based on the comparison between the static pattern (M) and the stored fingerprint, and when the fingerprint is not stored, the rotation angle (γ) and / or the angle of attack (β) are calculated based on the comparison with the static fingerprint characteristics stored in the database.

4. The method according to claim 1 or 2, characterized in that: A first pixel grid consisting of image elements (9) is at least partially transparent, and a second pixel grid having sensor elements (10) is located below the first pixel grid from the perspective of a screen observer, or the first and second pixel grids are located in one plane.

5. The method according to claim 1 or 2, characterized in that: A shielding layer is arranged above the sensor element (10) of the second pixel grid, for limiting the detection angle range of the sensor element (10).

6. The method according to claim 1 or 2, characterized in that: The image element (9) is used for illumination when recording a fingerprint.

7. The method according to claim 1 or 2, characterized in that: In order to provide illumination when recording fingerprints, light is coupled into a large-area sheet-like optical waveguide arranged above or below the second pixel grid and having a light coupling-out element arranged or constructed thereon. If the tip of a finger is not placed on the placement surface for the finger, the light is coupled out via the light coupling-out element in the direction of the placement surface and is totally reflected there.

8. The method according to claim 1 or 2, characterized in that: The screen is equipped with a touch-sensitive touch sensor layer which registers the placement of a finger and triggers the recording with the aid of the fingerprint reader.

9. The method according to claim 1 or 2, characterized in that: The screen (1) is used as an input device or the application of an application or a selected function in an application is released only when a combination of a static pattern (M) and at least a part of the fingerprint characteristics is stored in the database and associated with a released function.

10. The method according to claim 1 or 2, wherein: For a combination of a static pattern (M), a fingerprint characteristic and a motion pattern, before checking whether the combination is associated with a predetermined action, it is checked whether the first static pattern of at least two static patterns (M) corresponds to a predetermined activation pattern for enabling multi-fingerprint recording, and only if this is the case, further checks are performed and the action is triggered.

11. The method according to claim 10, wherein: The application is a text input function and the first static pattern of the at least two static patterns (M) is equivalent to an activation pattern, characterized in that, depending on the second static pattern (M), either lowercase, uppercase, numeric mode or special symbol mode is set as the writing mode.

12. The method according to claim 10, wherein: The first static pattern of the at least two static patterns (M) corresponds to an activation pattern, characterized in that it emulates the functionality of a computer mouse depending on the second static pattern (M) therein.

13. The method according to claim 10, wherein: The first static pattern of the at least two static patterns (M) corresponds to an activation pattern, characterized in that a zooming and marking function is implemented depending on a second static pattern (M) therein.

14. The method according to claim 10, wherein: A first static pattern of the at least two static patterns (M) corresponds to an activation pattern, characterized in that repeated occupation of the input element is achieved by different second static patterns (M), wherein each second static pattern corresponds to a different finger (3).

15. The method according to claim 10, wherein: The first of the at least two static patterns (M) corresponds to an activation pattern, characterized in that repeated occupancy of the input element is achieved by the second static pattern (M) therein, wherein different occupancy corresponds to different orientations and / or angles of attack and / or rotation angles (γ) in the fingerprint characteristics.

16. The method according to claim 10, wherein: A first static pattern of the at least two static patterns (M) corresponds to an activation pattern, characterized in that the authentication process is started dependent on at least one second static pattern (M).

17. The method according to claim 16, characterized in that The authentication process is initiated only when the fingerprint characteristics and / or the motion pattern of at least one second static pattern (M) correspond to predetermined values.

18. The method according to claim 1 or 2, characterized in that: In an enabled application, a change in the angle of attack (β) or the angle of rotation (γ) of at least one finger (3) changes the volume or adjusts the size of a segment consisting of a two-dimensional graphic or a three-dimensional graphic.

19. The method according to claim 1 or 2, characterized in that: In an enabled application, a change in the angle of attack (β) of at least one finger (3) switches between a two-dimensional view and a three-dimensional view of an object, or changes the angle of attack (β) of an actual or emulated writing instrument applied to the screen (1).

20. The method according to claim 1 or 2, characterized in that: In an enabled application, a change in the orientation of at least one finger (3) rotates an object shown on the screen (1), or changes the orientation of an actual or emulated writing instrument applied to the screen (1).

21. The method according to claim 1 or 2, characterized in that: The authentication process is only initialized when the static pattern (M) determined by means of the fingerprint recorder has at least two predetermined fingerprint characteristics and / or the two static patterns (M) determined within a predetermined time interval respectively reflect that at least two fingerprint characteristics change in a predetermined manner, and / or the motion pattern determined by means of the fingerprint recorder corresponds to a predetermined motion pattern.

22. The method according to claim 12, wherein: A finger (3) is respectively assigned to a button or a scroll element of a computer mouse.

23. An apparatus for implementing a method for controlling a mobile input device having a screen (1), the screen having a first pixel grid consisting of image elements (9) and an integrated second pixel grid consisting of photosensitive sensor elements (10) for optically recording a static finger line pattern of at least one finger (3), the screen being formed by one or more transparent bodies (101), the body having a placement surface (4), a first pixel grid consisting of image elements (9) integrated in a display plane (102.0), a second pixel grid integrated in a sensor plane (103.0), the second pixel grid comprising a plurality of photosensitive sensor elements (10) and an integrated angle shield (104), the angle shield having shielding openings in a shielding plane (104.0), the shielding openings being respectively associated with the sensor elements (10), characterized in that - the shielding plane (104.0) is located between the placement surface (4) and the sensor plane (103.0), and the angle shield (104) is arranged to completely cover the photosensitive sensor element (10) in an orthogonal direction relative to the placement surface (4), - the shielding opening is arranged offset relative to the photosensitive sensor element (10) along an offset direction (Rv) such that only the light reflected on the parting surface (4.1) of the placement surface (4) is incident on the associated photosensitive sensor element (10) obliquely from a limited angle range through the shielding opening, and The shielding opening has a maximum width in the staggered direction (Rv) and a maximum length in a direction orthogonal to the staggered direction (Rv), wherein the maximum length is smaller than the maximum width.

24. The device according to claim 23, characterized in that The display plane (102.0) and the sensor plane (103.0) coincide and the angle shield (104) is a separate surface element, each of which is assigned a scattered light shield (106), which surrounds the photosensitive sensor element (10) and adjoins the angle shield (104) orthogonally, so that no scattered light or no light directly originating from the image elements (9) of a first pixel grid formed by the image elements (9) can be incident on the photosensitive sensor element (10).

25. The device according to claim 23, characterized in that The display plane (102.0) is located between the placement surface (4) and the sensor plane (103.0), and a first pixel grid formed by image elements (9) has a transparent area, through which light reflected on the placement surface (4) is incident on an angle shield (104) arranged below the display plane (102.0).

26. The device according to any one of claims 23 to 25, characterized in that The restricted angular range is selected in such a way that only totally reflected light is incident on the sensor element (10).

Images