Determining the manual dexterity of a terminal user

Abstract

A method for determining the manual dexterity of a user of a mobile terminal (T) equipped with a camera (C), characterized in that it autonomously implements the following at the level of a manual dexterity determination device (MDD): - receiving (E201) from said terminal (T) a request to determine the manual dexterity of the user of said terminal, - receiving (E202) at least one image (Im) of the user from the camera (C), - determining (E203) from said at least one image (Im) a relative position (PR) of the terminal (T) with respect to the sagittal plane (PS) of the user, - determining (E204) the manual dexterity (Man) of the user from the determined relative position, - sending (E205) to the terminal (T) information relating to the manual dexterity (Man) of the user. Figure for the abstract: Figure 2

Description

Title of the invention: Determining the manuality of a user of a terminal Field of invention

[0001] This invention relates to the field of ergonomics and configuration of interfaces of portable terminals, such as for example mobile communication terminals of the smartphone type ("smart phone" in English), tablets, etc.

[0002] More specifically, the invention relates to the determination of the manual preference (manuality) of a user of a portable terminal of the aforementioned type. Prior art

[0003] Each person has a hand preference (also called manuality): for fine motor actions, each person will prefer to use one hand rather than the other. People are mostly right-handed (-90% of the population), but there are also left-handed and ambidextrous people.

[0004] The use of touch interfaces on a portable terminal (for example a smartphone or a tablet) requires a set of fine motor actions: pressing a button, moving a cursor, scrolling the screen, zooming with two fingers, etc. Therefore, users will have a strong preference for using their dominant hand to interact with this type of terminal (this is not absolute, however, and in certain situations, the user may use their non-dominant hand).

[0005] Due to the overrepresentation of right-handed people in the population, the interfaces of portable terminals are generally designed to be adapted to right-handed users. This design choice may be assumed, in order to guarantee the best ergonomics of a single interface for the majority of users, but may also be unconscious and stem from the manual skills of the designers / developers / testers (who are mostly right-handed). A left-handed user will therefore probably have a poorer user experience with these interfaces, because the means of interaction will not be adapted to their manual skills. For example, the majority of buttons will be placed on the right of the screen or in other cases, in the case of interfaces including buttons designed to be used with the thumb, buttons will be placed at the top right and therefore difficult to access with the thumb of the left hand when the terminal is held in that same hand.

[0006] Automatic adaptation of portable terminal interfaces according to the user's manuality addresses part of this problem. Such a method is for example described in patent US8922515. However, this type of method assumes that the user terminal is capable of recognizing the manuality of the user.

[0007] A known technique for recognizing the user's manuality consists of asking the user to manually select his manuality in the terminal settings. This solution, however, has the disadvantage of requiring actions on the part of the user. However, the latter is not necessarily aware of the existence of such a setting.

[0008] Ideally, it is therefore necessary to be able to automatically detect the user's manuality. Most existing approaches for automatically detecting manuality consist of analyzing the traces of use of the touch screen, which are different depending on the hand holding the terminal. One of these approaches is notably addressed in the publication “Recognizing the operating hand and the hand-changing process for user interface adjustment on smartphones. H. Guo, H. Huang, L. Huang, Y. Sun. In Sensors, 2016”.

[0009] However, methods based on the analysis of tactile interactions can lead to errors because they depend on the way in which the person holds the portable terminal (one-handed, two-handed, one hand and the other interacting, etc.), which is not a priori known to the terminal.

[0010] The present invention aims to propose a method for automatically detecting the manuality of a user of a portable touch terminal, which is more robust and does not suffer from the same limitations as the approaches of the state of the art. This invention can also be combined with other existing approaches (such as the analysis of the aforementioned touch interactions) to improve the accuracy of determining the user's manuality. Subject matter and summary of the invention

[0011] One of the aims of the invention is to remedy at least one of the drawbacks highlighted by the aforementioned state of the art by proposing a method for determining the manuality of a user of a portable terminal, in accordance with the invention which will be described below.

[0012] To this end, an object of the present invention relates to a method for determining the manuality of a user of a portable terminal equipped with a camera, characterized in that it implements the following autonomously at the level of a device for determining manuality: - receive from said terminal a request to determine the manuality of the user of said terminal, - receive at least one image of the user from the camera, - determine from said at least one image a relative position of the terminal with respect to the sagittal plane of the user, - determine the user's manuality from the determined relative position, - send information to the terminal relating to the user's manuality.

[0013] The invention offers the advantage of making it possible to detect the user's manuality without the user needing to enter his or her manuality on the terminal. In the case of a terminal shared between several people, this method also makes it possible to detect the manuality of the active user at a given time on the terminal.

[0014] According to a particular embodiment, the determination of said relative position of the terminal with respect to the sagittal plane of the user is preceded by a reception of data from a module integrated into the terminal making it possible to measure the orientation and acceleration of the terminal, said data being used in the determination of the relative position of the terminal with respect to the sagittal plane of the user.

[0015] By using additional data such as data relating to the acceleration and / or orientation of the terminal, provided for example by an accelerometer, a gyroscope, a Hall effect sensor or any other module known to those skilled in the art to generate data relating to the orientation and / or acceleration of a terminal. This embodiment advantageously makes it possible to make the method more reliable and to reinforce the confidence in its ability to determine the user's actual manuality.

[0016] According to a particular embodiment, the reception of said at least one image from the camera of the terminal takes place only when the data from the accelerometer indicate that the value of the movement / acceleration of the terminal is less than a given threshold.

[0017] This embodiment advantageously makes it possible to limit the algorithmic processing and the use of the computing power of the terminal by concentrating only on images representative of nominal / normal use of the terminal, which in particular makes it possible to limit the energy consumption of the method when it is implemented on a portable terminal operating on battery.

[0018] According to a particular embodiment, the determination of manuality is further implemented from a detection of at least one finger of the user on said at least one image.

[0019] This embodiment advantageously makes it possible to take into account other information relating to the actual use of the terminal by the user and in particular to determine whether the terminal is held in the dominant hand of the user, who uses his thumb to interact with the terminal, or in the weak hand of the user, who then uses the fingers of his strong hand (the hand corresponding to his manuality) to interact with the terminal.

[0020] The various embodiments or features mentioned above may be added independently or in combination with each other, to the method of determining- reduction of manuality as defined above.

[0021] The invention also relates to a terminal suitable for determining the manuality of a user of said terminal, equipped with a camera, characterized in that it is configured to implement the following autonomously: - initiate a request to determine the manuality of the user of said terminal, - receive at least one image of the user from the camera, - determine from said at least one image a relative position of the terminal with respect to the sagittal plane of the user, - determine the user's manuality from the determined relative position.

[0022] Such a device is particularly suitable for implementing the method for determining the manuality of a user according to any of the embodiments described above.

[0023] The invention also relates to a computer program comprising program code instructions for implementing the method for determining manuality according to any one of the particular embodiments described above, when this program is executed by a processor.

[0024] Such instructions can be stored permanently in a non-transitory memory medium of a device implementing the method for determining manuality according to the invention.

[0025] This program may use any programming language, and be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0026] The invention also relates to a recording medium or information medium readable by a computer, and comprising instructions of a computer program for implementing the method for determining manuality as mentioned above.

[0027] The recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM (Read Only Memory), for example a CD ROM (Compact Disc Read-Only Memory), a synthetic DNA (deoxyribonucleic acid) or a microelectronic circuit ROM, or a magnetic recording means, for example a mobile medium, a hard disk or an SSD (Solid State Drive).

[0028] On the other hand, the recording medium may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means, so that the computer program contained therein is remotely executable. The program according to the invention may be in par particular downloaded on a network, for example an Internet type network.

[0029] Alternatively, the recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the aforementioned manuality determination method.

[0030] According to an exemplary embodiment, the present technique is implemented by means of software and / or hardware components. With this in mind, in this document, the term “device” can correspond to a software component, a hardware component or a set of hardware and software components. Brief description of the drawings

[0031] Other characteristics and advantages will appear on reading particular embodiments of the invention, given as illustrative and non-limiting examples, and the appended drawings, among which:

[0032] [Fig.l] represents an example of architecture in which the method of determining manuality is implemented.

[0033] [Fig.2] represents the steps of a method for determining manuality, in one embodiment of the invention.

[0034] [Fig. 3] represents the steps of a method for determining manuality, in another embodiment of the invention.

[0035] [Fig.4] represents the steps of a method for determining manuality, in another embodiment of the invention.

[0036] [Fig.5] represents an example of a terminal for determining the manuality of a user of this terminal, in one embodiment of the invention. Detailed description

[0037] Description of an example of architecture in which the method for determining manuality is implemented

[0038] With reference to [Fig.l] an example of architecture is described in which the method for determining the manuality of a user is implemented. This architecture comprises:

[0039] - a portable terminal T for example a smartphone or a tablet, to which are associates: — a computing unit UC, — a screen E, for example rectangular in shape, — a camera C placed on the same side as the screen E so that the user is in the field of vision CVC of said camera, — a DDM device for determining the manuality of the terminal user.

[0040] The terminal T is held by a user U. Depending on the embodiments, the terminal can be held in the right hand or the left hand of the user or his two hands.

[0041] The user's face is "cut" vertically by its sagittal plane PS, the anatomical plane which divides the user's body U into right and left parts. The user's face is also "cut" according to a second plane, the frontal plane PF which anatomically separates the front and back of the user's face U and finally a third plane, the transverse plane PT which by convention in this text separates the user's face into an upper part and a lower part.

[0042] Depending on the usage configurations within this architecture, the terminal T is placed in a relative position (hereinafter called PV) with respect to the sagittal plane PS of the user. In the case of figure [Fig.l], the relative position PV of the terminal T is the left: that is to say that the terminal is to the left of the sagittal plane of the user U with respect to the point of view of the user U. In other usage configurations, the relative position of the terminal may be the right.

[0043] The terminal is placed at a distance d(UT) from the user corresponding to the distance between the point of intersection of the transverse, sagittal and frontal planes of the user's face U and the barycenter of the terminal T.

[0044] The terminal is inclined along axes x, y and z, where x corresponds to the axis parallel to the smallest side of the terminal screen, y corresponds to the axis parallel to the largest side of the terminal screen and z corresponds to the axis orthogonal to the plane formed by the x and y axes.

[0045] Depending on the possible usage configurations within this architecture, the terminal T has an inclination corresponding to a rotation according to the angles x, y or z.

[0046] For example, if the terminal is held vertically (when the longest side of the terminal is vertical from the point of view of the user U), it is likely to rotate along the z axis by 90° compared to a situation where the terminal is held horizontally (when the longest side of the terminal is horizontal from the point of view of the user U).

[0047] According to the embodiments, the camera C of the terminal T can be placed at different locations on the front of the terminal T, for example centered against the top side of the terminal when the latter is held vertically or placed in a corner of the terminal. The position of the terminal within the plane defined by the x and y axes will be defined by the xcam and ycam coordinates (not shown in the figure).

[0048] Description of an embodiment of the method for determining manuality according to the invention

[0049] Figure [Fig.2] represents the steps implemented by the method for determining manuality, in a particular embodiment of the invention, the method taking place in accordance with the architecture of figure [Fig.l].

[0050] The method begins when the DDM handedness detection device receives in E201 a request for determination of manuality from terminal T.

[0051] According to the embodiments, this request is triggered autonomously. The reasons for this request may vary according to the embodiments, for example when the terminal comes out of standby, when launching an application, regularly according to a frequency pre-established elsewhere in parameters of the terminal T, etc.

[0052] According to the embodiments, upon receiving this request, the manuality determination device receives information relating to the terminal T, such as the position of the camera C on the terminal T, via the coordinates xcam and ycam described in [Fig.l].

[0053] According to other embodiments, if the method for determining the user's manuality is implemented directly by the terminal T, step E201 consists of initializing the detection of the user's manuality.

[0054] The manuality detection device then receives at step E202 at least one image Im of the user U from the camera C of the terminal T.

[0055] According to the embodiments of the invention, the device for determining manuality receives several images Im.

[0056] In step E203, the manual device determines a relative position PR of the terminal T with respect to the sagittal plane PS of the user, from said image(s) received in E202.

[0057] In the context of this embodiment, the identification of the sagittal plane of the user can be obtained from the identification of the face of the user within the at least one image Im received in E202 using different face detection techniques (for example the Viola and Jones method, Eigenface, etc.) known to those skilled in the art.

[0058] The identification of the user's face and its general orientation with respect to the shooting format, that is to say whether the face appears vertically within an image taken in portrait or landscape mode, makes it possible to deduce the way in which the terminal is held by the user U, for example whether the terminal is held vertically or horizontally.

[0059] According to one embodiment, if the device for determining manuality identifies several faces in the image Im, it takes into account only the face appearing in the foreground, for example the face not hidden by a second face or the face occupying the most space within the image.

[0060] Face identification then makes it possible to establish the position of the user's sagittal plane, i.e. the plane which anatomically separates the user's face vertically into a left side and a right side.

[0061] According to the embodiments, the facial analysis makes it possible to determine the distance d(UT) between the user's face and the terminal. This distance can in particular be calculated by referring to the average value of the user's eye spacing U or to any other method known to those skilled in the art.

[0062] The analysis of the image makes it possible to deduce the orientation of the terminal T along the x, y and z axes as represented in figure [Fig. 1]. This analysis can in particular be carried out due to the deformation of the face induced by the misalignment of the camera with respect to the sagittal plane of the user and, more generally, by the misalignment of the axis formed by the intersection of the sagittal plane PS, the transverse plane PT and the frontal plane PF.

[0063] Determining the sagittal plane of the user, the distance between the terminal T and the user's face and the orientation of the terminal T makes it possible to determine a relative position PR of the terminal T with respect to the sagittal plane PS of the user.

[0064] According to other embodiments, the determination of the relative position PR of the terminal T with respect to the sagittal plane PS of the user includes the calculation of the position of the terminal T within the reference frame formed by the sagittal planes PS, frontal PF and transverse PF of the user.

[0065] In the case where the handedness detection device receives several images Im in E202 the device analyzes the different images and determines a relative position PR of the terminal T with respect to the sagittal plane PS of the user on the basis of all the images by cross-referencing the data via different data partitioning methods (e.g.: k-means, Voronoi diagram, etc.) known to those skilled in the art.

[0066] According to certain embodiments, if the analysis 203 does not make it possible to determine a relative position PR of the terminal T with respect to the sagittal plane PS of the user, for example due to a lack of data or due to insufficiently relevant data, the method of determining manuality ends, otherwise it continues.

[0067] In step E204, the device for determining the manuality determines the manuality Man of the user U from the relative position PR determined in E203.

[0068] This determination is based on the principle that the user's handedness corresponds to the position of the terminal relative to the user's sagittal plane. For example, if the terminal is positioned relatively to the left of the user's sagittal plane, it can be deduced that the user is left-handed, whereas if the terminal is positioned relatively to the right of the user's sagittal plane, the user is right-handed.

[0069] According to other embodiments, the determination of the user's manuality is based on the average of different relative positions PR of the terminal with respect to the sagittal plane of the user. To do this, the device for determining manuality establishes a cross-check of information, for example via an average calculation of these different relative positions PR or by using learning algorithms known to those skilled in the art which infer a relative position PR from the different relative positions PR, etc.

[0070] According to other embodiments, the determination of the user's manuality integrates additional data such as the detection of at least one finger (thumb or index finger) or the detection of the user's hand or arm crossing the field of vision of the camera C of the terminal T, which makes it possible to detect use by the left hand or the right hand or even use of the terminal by the user's leading hand with his thumb or by the user's non-leading hand but with the index finger of the user's leading hand.

[0071] In step E205, the device for determining the manuality sends to the terminal T the manuality Man determined in E204.

[0072] According to other embodiments, if no manuality could be determined, the device returns an error or proposes to restart the method of determining the manuality at step E202.

[0073] According to other embodiments, the manuality is sent to the terminal with a confidence index in the manuality where for example an index equal to 1 corresponds to a certainty in the manuality and 0 to a total uncertainty in the returned result.

[0074] Description of another embodiment of the method for determining the manuality of a user according to the invention

[0075] Figure [Fig.3] represents the steps implemented by the method for determining manuality according to another embodiment, the method taking place in an architecture similar to that described in [Fig.l], but where the terminal is additionally equipped with a module, for example an accelerometer or a Hall effect sensor or any other module known to those skilled in the art, capable of producing data relating to the orientation of the terminal.

[0076] The method begins when the manuality determination device DDM receives a request for manuality determination at E301. This is similar to step E201 described in [Fig.2].

[0077] In step E302, similarly to step E202 of the figure [Fig.2], the handedness detection device receives at least one image Im of the user U from the camera C of the terminal T. However, this at least one image is accompanied by data relating to the orientation of the terminal T from an accelerometer and / or a Hall effect sensor and / or any other device known to those skilled in the art capable of measuring the orientation of the terminal.

[0078] According to the embodiments of the invention, the handedness detection device receives several images Im and data corresponding to the orientation and movement of the terminal T at the time when each of said images was captured.

[0079] In step E303, the manual device determines a relative position PR of the terminal T with respect to the sagittal plane PS of the user from said image(s) Im and the orientation and / or movement data of the terminal T received in E302.

[0080] This step is similar to that described in E203 of figure [Fig.2] with the difference that the orientation data makes it possible to reinforce the precision of the determination of the relative position of the terminal with respect to the sagittal plane PS of the user.

[0081] According to one embodiment, the terminal orientation data will make it possible to determine, without analysis of the image or in addition to an analysis of the image, the general orientation of the image, for example whether an image is taken in portrait mode or in landscape mode.

[0082] According to one embodiment, the terminal orientation data will make it possible to determine, without analysis of the image or in addition to an analysis of the image, the orientation of the terminal along the x, y and z axes.

[0083] Steps E304 and E305 of figure [Fig.3] are respectively similar to steps E204 and E205 of figure [Fig.2]. For this purpose, they will not be described again.

[0084] Description of another embodiment of the method for determining the manuality of a user according to the invention

[0085] Figure [Fig.4] represents the steps implemented by the method for determining manuality according to another embodiment, the method taking place in an architecture similar to that described with reference to [Fig.l] but where the terminal is additionally equipped with a module, for example a gyroscope and / or an accelerometer, capable of producing data relating to the axial and / or angular acceleration of the terminal.

[0086] Step E402 is similar to the step described in E302 of the figure [Fig.3], with the difference that the device for determining the manuality DDM will receive said at least one image from the camera C of the terminal T, only when the data from the accelerometer indicate that the value of the movement of the terminal T is less than a given threshold.

[0087] Depending on the embodiments, the threshold may be fixed, configured at the terminal level or may evolve according to a learning logic known to those skilled in the art.

[0088] Steps E403, E404 and E405 are respectively similar to steps E203, E204 and E205 described with reference to figure [Fig.2]. For this purpose, they will not be described again.

[0089] Description of a terminal adapted to determine the manuality of a user of the terminal in one embodiment of the invention.

[0090] Figure [Fig.5] shows the simplified structure of a TDM terminal suitable for complete the manuality of a user of said terminal corresponding to a particular embodiment of the invention implemented in an architecture as illustrated in [Fig.l], where the device for determining manuality DDM is inseparable from the terminal T, called here TDM.

[0091] According to a particular embodiment of the invention, the actions executed by the TDM device, within the framework of the implementation of the method for determining the manuality of a user of a terminal of the present invention, are implemented by instructions of a computer program PG. For this, the TDM terminal has the conventional architecture of a computer and notably comprises a memory MEM, a processing unit UTR, equipped for example with a processor PROC, and controlled by the computer program PG stored in memory MEM. The computer program PG comprises instructions for implementing the steps of the method for determining the manuality of the user of the TDM terminal, in particular the aforementioned actions: - initiate a request to determine the manuality of the user of said terminal, - receive at least one image Im of the user from the camera C, - determine from said at least one image Im a relative position PR of the TDM terminal with respect to the sagittal plane PS of the user, - determine the user's manuality Man from the determined relative position.

[0092] According to other embodiments, based on a so-called cloud computing architecture, the processor PROC, the memory MEM can be associated with another terminal, different from TDM.

Claims

Claims

1. Method for determining the manuality (Man) of a user of a portable terminal (T) equipped with a camera (C), characterized in that it implements the following autonomously at the level of a manuality determination device (DDM): - receiving (E201) from said terminal (T), a request for determining the manuality of the user of said terminal, - receiving (E202) at least one image (Im) of the user from the camera (C), - determining (E203) from said at least one image (Im) a relative position (PR) of the terminal (T) relative to the sagittal plane (PS) of the user, - determining (E204) the manuality (Man) of the user from the determined relative position, - sending (E205) to the terminal (T) information relating to the manuality (Man) of the user.

2. Method for determining manuality according to claim 1, in which the determination of said relative position (PR) of the terminal (T) with respect to the sagittal plane of the user is preceded by a reception of data from a module integrated into the terminal (T) making it possible to measure the orientation and acceleration of the terminal, said data being used in the determination of the relative position (PR) of the terminal (T) with respect to the sagittal plane (PS) of the user.

3. Method for determining manuality according to claim 2, wherein the reception of said at least one image from the camera (C) of the terminal (T) takes place only when the data from the accelerometer indicate that the value of the movement of the terminal (T) is less than a given threshold.

4. Method for determining manuality according to any one of claims 1 to 3, in which the determination of manuality is further implemented from a detection of at least one finger of the user on said at least one image (Im).

5. Terminal (TDM) adapted to determine the manuality (Man) of a user of said terminal, equipped with a camera (C), characterized in that it is configured to implement the following autonomously: - initiate a request to determine the manuality of the user of said terminal, - receive at least one image (Im) of the user from the camera (C), - determine from said at least one image (Im) a relative position (PR) of the terminal (TDM) in relation to the sagittal plane (PS) of the user, - determine the manuality (Man) of the user from the determined relative position.

6. A computer program comprising program code instructions for implementing the method of determining handedness according to any one of claims 1 to 4, when executed on a computer.

7. Computer-readable information medium comprising instructions of a computer program for implementing the method for determining manuality according to any one of claims 1 to 4.

Images