Method and device for interacting with a user

Abstract

A user interaction process includes: triggering (610) a signal to require the user to perform a consent gesture validating the execution of an operation to be carried out within the application environment, the consent gesture being a hand clap; biometric authentication (620), or biometric identification, based on a consent gesture performed by the user; and execution (630) of the operation upon successful authentication or identification. Figure for the abbreviation: Fig. 6

Description

Title of the invention: Method and device for interaction with a user Technical field

[0001] The present description relates to a device and method for interacting with a user. Technical background

[0002] In certain applications, information (password, payment data, agreement to the transaction, etc.) must be entered by the user in order to authenticate and / or validate an electronic transaction or electronic operation to be executed.

[0003] However, having to enter such information while the user is in an application environment (for example, a virtual reality environment, a video game, an online commerce application, viewing multimedia content, etc.) can cause breaks or “frictions” in the user experience.

[0004] Furthermore, such entries must, depending on the case, be made using a device other than that used to access the application environment and the user must then change devices, to access their mobile telephone for example.

[0005] When, for example, the application environment is a virtual reality environment which the user accesses using a virtual reality headset, the authentication step can thus cause significant friction, or even completely disrupt the user's experience by making them remove the headset to access their mobile phone, for example.

[0006] There thus appears to be a need for a solution for interaction with a user, usable in an application environment when an electronic transaction or electronic operation must be validated by the user before being executed, which allows a high level of security and a seamless immersive experience ("frictionless"), in particular which does not require the user to leave the application environment or change interaction device. Summary

[0007] The scope of protection is defined by the claims.

[0008] According to a first aspect, the present description relates to a method of interaction with a user of an application environment, the method comprising: triggering a signal to require execution by the user of a consent gesture validating the execution of an operation to be executed in the context of the application environment, the consent gesture being a clap of the hands; biometric authentication, or respectively biometric identification, based on a gesture of consent made by the user; execution of the operation in the event of successful authentication, or respectively successful identification.

[0009] In one or more embodiments, the biometric authentication comprises: an acquisition of raw data representative of the consent gesture made by the user; a generation of an embedding vector representative of behavioral and / or morphological characteristics of the user on the basis of the raw data; an obtaining of an authentication decision on the basis of a comparison of a biometric signature of the user with the embedding vector, the biometric signature being representative of behavioral and / or morphological characteristics of the user.

[0010] Comparing the user's biometric signature with the embedding vector may include calculating the distance between the biometric signature and the embedding vector, with authentication being successful if the distance is less than an authentication threshold.

[0011] In one or more embodiments, the biometric identification comprises: acquiring raw data representative of the consent gesture made by the user; generating an embedding vector representative of behavioral and / or morphological characteristics of the user based on the raw data; obtaining an identification decision based on a comparison of biometric signatures representative of behavioral and / or morphological characteristics of respective users with the embedding vector.

[0012] The comparison of the biometric signatures with the embedding vector may comprise a calculation of the distance between each of the biometric signatures and the embedding vector, the identification being successful if the smallest of the distances obtained is less than an identification threshold, the user being identified as being the user for whom the smallest distance was obtained.

[0013] In one or more embodiments, the method comprises: an acquisition of raw data representative of one or more consent gestures made by the user for enrollment; a generation, based on the raw data, of an embedding vector representative of behavioral and / or morphological characteristics of the user for each gesture made by the user for enrollment; a generation of the biometric signature of the user from the embedding vector(s) obtained for the gesture(s) made by the user for enrollment.

[0014] The biometric signature can be calculated as the equibarycenter of the embedding vectors obtained for the gesture(s) performed by the user for enrollment.

[0015] The application environment may be a virtual reality environment.

[0016] The hand clapping gesture may include a first passing phase from an initial arm rest position to a clap position, a second clap phase in the clap position and a third phase of moving from the clap position to a final arm rest position.

[0017] According to a second aspect, the present description relates to a device comprising means (in particular software and / or hardware means) for implementing a method according to the first aspect.

[0018] The means may comprise electronic means. The electronic means may comprise, for example, one or more circuits configured to execute one or more or all of the steps of the method according to the first aspect. The electronic means may comprise, for example, at least one processor and at least one memory comprising program instructions configured to, when executed by the processor, cause the device to execute one or more or all of the steps of the method according to the first aspect.

[0019] According to another aspect, the present disclosure relates to a data processor-readable recording medium having recorded thereon a program comprising program instructions configured to cause the data processor to execute one or more or all of the steps of the method according to the first aspect.

[0020] According to another aspect, the present disclosure relates to a computer program comprising program instructions configured to cause a data processor to execute one or more or all of the steps of the method according to the first aspect. Brief description of the figures

[0021] Other characteristics and advantages will result from the detailed description which follows, carried out on the basis of embodiments and examples given for illustrative and non-limiting purposes, with reference to the appended figures.

[0022] [Fig.l] schematically represents a virtual reality system according to an exemplary embodiment.

[0023] [Fig.2] is a block diagram of an interaction device according to an exemplary embodiment.

[0024] [Fig.3] is a block diagram illustrating an interaction method according to an exemplary embodiment

[0025] [Fig.4] is a block diagram of a model usable for generating embedding vectors according to an exemplary embodiment

[0026] [Fig.5A] illustrates aspects of a method of training a model according to an exemplary embodiment.

[0027] [Fig.5B] illustrates aspects of a method of training a model according to a example of realization.

[0028] [Fig.6] is a flowchart illustrating an interaction method according to an exemplary embodiment.

[0029] [Fig.7] illustrates aspects of an interaction method according to an exemplary embodiment.

[0030] [Fig.8] illustrates aspects of an interaction method according to an exemplary embodiment. Detailed description

[0031] Various exemplary embodiments will now be described in more detail with reference to the drawings. However, the specific structural and / or functional details disclosed herein are used to enable an understanding of the various possible embodiments. However, those skilled in the art will understand that the exemplary embodiments may undergo various modifications and may be implemented without all of these details.

[0032] The present description relates to a device and method for interacting with a user.

[0033] The interaction device and method are applicable for example in a virtual reality environment, in particular when the user is wearing an immersion device in a virtual space. This immersion device is hereinafter called a “virtual reality headset”, “VR headset” (VR, for “Virtual Reality”) or more generically an “immersion device”.

[0034] The interaction solution is in fact particularly suitable for use during the restitution of audiovisual content in a virtual space by means of a device for immersion in a virtual space, such as a virtual reality headset. This interaction solution is applicable to any type of virtual reality (VR) system, including augmented reality (AR), mixed reality (MR) or extended reality (XR) systems.

[0035] In the context of this document, the expression "virtual reality" will cover all virtual reality technologies in the broad sense, including those of augmented reality, mixed reality or extended reality as well as variants or technologies derived from these technologies.

[0036] The interaction solution described here is however more generally usable for different applications or application environments involving collecting validation from a user at one time or another and requiring authentication (or identification).

[0037] By validation of a user, we mean here a response provided by the user for signify that he gives his consent for the execution of one or more operations. In the context of this document, we will speak indifferently of “consent” or “validation” or “acceptance” or “agreement” given by the user for an operation to be validated.

[0038] The operation to be validated can be any operation, simple or complex. The operation to be validated is for example a transaction (for example a banking transaction, a payment transaction, etc.), a transmission of a message and / or data or documents, a recording of entered data (validation of an entry), a creation of a user account, a configuration of software or peripheral, etc.

[0039] The operation to be validated may be an electronic operation to be executed in the context of the application environment or in relation to the application environment with which the user interacts.

[0040] User acceptance may also be requested when the user must indicate acceptance of general terms and conditions of sale on a website, acceptance of a license agreement required to benefit from access to a service, or in any other type of circumstances requiring prior agreement from the user.

[0041] The present invention thus relates to a device and method for interaction with a user, allowing transparent behavioral biometric authentication and / or identification in an application environment, this behavioral biometric authentication and / or identification being based on a consent gesture. Consent is expressed by a particular gesture which is here the clapping of hands and the different movement characteristics of which are analyzed to serve as a behavioral biometric factor during authentication (or identification). Consent is only considered to be validly given if a successful authentication (or respectively an identification) of the user has been carried out on the basis of this consent gesture.

[0042] This interaction solution makes it possible to streamline the user experience while offering the user a more fluid and often simpler interaction than a succession of clicks and / or entries in the application environment.

[0043] This interaction solution makes it possible to resolve the problems cited above by proposing strong and transparent authentication of the user since the authentication occurs automatically and transparently for the user when he makes the consent gesture.

[0044] Strong authentication means authentication based on the verification of at least two authentication factors of different types. There are three types of authentication factors: - biometric factors (based on biometric data); - knowledge factors (based on secret information known only to the user); - possession factors (based on the use of a user-specific hardware device).

[0045] In the context of the authentication method described in this document, strong authentication is possible based on biometric behavioral factors provided by the consent gesture on the one hand and on the basis of the user-specific interaction device on the other hand.

[0046] This interaction solution allows for a high level of security and a frictionless and immersive experience while remaining compliant with privacy.

[0047] In the case of its application to a virtual reality environment, the solution is also inexpensive since it only uses the data generated by sensors already available in the interaction devices (in particular in the VR headset and / or the controller) with the virtual reality environment and avoids the user interrupting the rendering of the audiovisual content by the VR headset or removing his VR headset or changing device to authenticate / identify himself.

[0048] Consent by hand clapping can be easily obtained by directly integrating the request to execute this consent gesture into the application environment, for example in a web page or more generally in content presented via the application environment.

[0049] In embodiments, the interaction solution uses a technique from the field of artificial intelligence (AI) for generating an "embedding" of the consent gesture, authentication being performed on the basis of this "embedding". The term "embedding" used in English terminology can be translated as "incorporation" or "embedding" in French. In this context, an embedding or an embedding is a digital vector representation (i.e. a vector of digital values) of a set of values ​​of representative parameters (of an object, movement, process, document, etc.) produced by an AI system based on neural networks.

[0050] Such a condensed digital vector representation is obtained by projection (or embedding) into a multi-dimensional space, for example into a 20-dimensional N=2 space. The values ​​of the representative parameters are projected into this space in such a way as to preserve certain properties or relationships inherent to these representative parameters.

[0051] This digital vector representation is generally a condensed representation (i.e. with a reduction in dimensionality) of the set of values ​​of initial representative parameters, the vector obtained having fewer components than the initial set has values.

[0052] Embeddings are thus used to encode in a continuous representation space complex and discrete information, making it easier for AI models to manipulate it.

[0053] For the purposes of authentication, a current embedding, calculated for a consent gesture made at a given time by the user, can be compared to a reference embedding, obtained during user enrollment, in order to determine whether the authentication is successful or not. In practice, a distance between the two embeddings is calculated and compared to a threshold (hereinafter called the authentication threshold) to determine whether the authentication is successful or not.

[0054] In this document, the expression “embedding vector” or “embedding” will be used to designate an “embedding” calculated for a consent gesture.

[0055] The consent gesture is used both as a means of expressing consent and as a behavioral biometric authentication factor.

[0056] Using a hand clap as a consent gesture makes it possible to take advantage of different characteristics specific to a user: the position of one hand relative to the other, the position of the first hand and / or that of the second hand, the position of the head, the speed of performing the gesture, the symmetry or non-symmetry of the hands and / or arms relative to the position of the head, and other characteristics which derive from these basic characteristics.

[0057] Furthermore, this is a gesture that is well known, so no learning of this particular gesture is necessary. Furthermore, it has been observed that this gesture allows reliable authentication given that this gesture is very much part of bodily automatisms and that a person performs it by almost always reproducing the same gesture.

[0058] The gesture is executed freely by the person according to their own bodily automatisms, without any constraint imposed by the interaction solution (for example, a hand alignment constraint), so that the discriminating behavioral factors, specific to the person, are preserved and usable for authentication.

[0059] The interaction solution relies exclusively on data acquired by gesture sensors and movement analysis, with no analysis of the sound produced by hand clapping being necessary or used. Such gesture sensors are available in most devices for interaction with an application environment.

[0060] Identification / authentication is carried out discreetly and not by voice and / or sound authentication. Identification / authentication does not require restrictive, specific / precise interaction (at reduced distance) with a specific acquisition device to carry out facial, palm, iris recognition, etc.

[0061] The interaction solution allows user enrollment and does not require retraining of the AI ​​model. User enrollment is rapid: in practice, 1 to 5 hand clapping gestures maximum are sufficient for such enrollment.

[0062] The interaction method allows authentication and / or identification and at the same time validation of an electronic operation to be executed in the context of an application environment.

[0063] This interaction solution with authentication and / or identification is inexpensive in that it is based solely on the equipment and sensors already available in equipment such as: for example, equipment for monitoring a private or public space, a commercial space or on equipment for rendering virtual reality environments, etc. More generally, the interaction solution with authentication and / or identification can be integrated into any application system including gesture sensors and requiring the collection of validation from a user for an operation to be executed.

[0064] Gesture sensors may be video sensors (cameras) and / or motion sensors (gyroscopes, accelerometers, etc.) and are used to capture gestures made by the user. Gesture analysis means (software and / or hardware) can be used to analyze, categorize and / or decompose the captured gestures. These gesture analysis means can be used to analyze and characterize the user's behavior for the purpose of performing behavioral biometric authentication.

[0065] [Fig.l] schematically represents a virtual reality system according to an exemplary embodiment.

[0066] The system includes an immersion device 120 (e.g., a VR headset) worn by a user 110. The immersion device 120 includes a display screen 125 (generally, on an inner face of the headset that is placed in front of the user's eyes) allowing the user to view the virtual environment in three dimensions (3D).

[0067] In the case of a virtual reality environment, virtual audiovisual content is digitally created and represented in a virtual space, which is generally three-dimensional. In the case of augmented reality, also called hybrid or mixed, virtual content (sounds, images, graphics, GPS positioning information, etc.) is mixed in the virtual space with real content resulting from a capture of the real world, for example by means of a camera. The feeling of immersion in the virtual space is generally reinforced by a stereoscopic or three-dimensional reproduction of the video component and by the spatialization of the audio component of the audiovisual content.

[0068] The immersion device 120 is in operational communication with a remote authentication server 140 associated with an authentication database 145. The immersion device 120 may be in communication with a remote content server 160 providing audiovisual content to be rendered by the immersion device. 120.

[0069] In one embodiment, communication with a remote server 140 and / or with the remote content server 160 is established via a telecommunications network 150, either directly or through a local device 130, of the personal computer type.

[0070] In one embodiment, the immersion device 120 executes an interaction program implementing an interaction method.

[0071] In one embodiment, the interaction program is downloaded into the immersion device 120 from the remote server 140 and then executed locally by the immersion device 120 to interact with the user and capture biometric behavioral data and other interaction data necessary for authentication.

[0072] The analysis of the captured data and / or the generation of the immersion vectors can also be carried out locally by the immersion device and / or by the remote server which then transmits the immersion vectors to the immersion device.

[0073] The authentication decision may also be made locally by the immersion device and / or by the remote server which transmits the authentication decision to the immersion device.

[0074] In certain embodiments, the execution of certain functions (in particular calculation or data storage functions) can be transferred to a local device 130 (of the personal computer type) connected via a local link with the immersion device 120, 200. In this case the immersion device and the local device 130 cooperate for the implementation of the interaction method.

[0075] If the immersion device is not recognized as a trusted terminal, the risk level is higher than in the case of authentication server-side verification. To reduce this risk, the authentication server can identify the user's immersion device (based on a possession factor, in addition to the other two user authentication factors) and verify its authenticity.

[0076] Different authentication procedures for the immersion device are possible. For example, the authentication server sends the immersion device a random challenge to be signed at each user authentication request. The immersion device signs the challenge and then transmits the signature of the challenge with the result of the user authentication carried out on the basis of the two authentication factors (biometric + secret sequence) to the authentication server. The authentication server verifies the authenticity of the immersion device on the basis of the received data and decides whether or not to validate the result of the user authentication. In this way, the final decision always rests with the server.

[0077] [Fig.2] is a block diagram of an interaction device 200 according to an example of realization.

[0078] In the example case of a virtual reality application, the interaction device may comprise an immersion device (VR headset) as described with reference to [Fig.l] and / or a controller and / or joystick and / or another interaction command input device.

[0079] The interaction device 200 may generally have the architecture of a computer, including constituents of such an architecture: data memory(s) 250, processor(s) 220, communication bus 270, communication interface(s) 240 for connecting this interaction device 200 to a telecommunications network or other local or remote equipment.

[0080] The interaction device 200 comprises a display screen 210 for the reproduction of images of audiovisual content and loudspeakers 211, 212 for the reproduction of the sound of the audiovisual content.

[0081] The interaction device 200 may comprise different acquisition devices: camera 231, infrared camera 234, depth camera, microphone 232, biometric sensor 235, gyroscope 233, accelerometer 236, etc.

[0082] The camera and / or the infrared camera and / or the gyroscope may for example be used to capture gestures made by the user. Gesture analysis means (software and / or hardware) may be used to analyze, categorize and / or decompose the captured gestures. For example, the memory 250 may comprise program instructions 260 configured to be executed by the processor and implement gesture analysis functions. These gesture analysis functions may be provided in the form of libraries adapted to the interaction device 200.

[0083] [Fig.3] is a block diagram illustrating an interaction method according to an example of realization.

[0084] The interaction method may be executed by means of an authentication program executed by an interaction device 200 described with reference to [Fig.l] or 2 and / or by an authentication server 140 described with reference to [Fig.l] or 2.

[0085] The functional blocks 310 to 313 correspond to functions or steps of a phase 31 of enrolling a user.

[0086] The functional blocks 320 to 325 correspond to functions or steps of a phase 32 of authentication of a user.

[0087] In block 310, a user is asked to perform one or more times a consent gesture, in this case a number N (for example N = 1 to 5) of hand claps. Raw data, representative of these consent gestures, are acquired by one or more gesture sensors.

[0088] The raw data may be stored in a database 300 in association with a user identification.

[0089] In block 311, preprocessing is applied to the raw data acquired in block 310 to generate preprocessed data of interest, comprising values ​​of characteristic parameters of interest for user recognition. The characteristic parameters of interest for user recognition comprise morphological and / or behavioral characteristics.

[0090] The preprocessing may further comprise filtering the raw data, for example to remove outliers or noise or other defect.

[0091] In block 312, for each consent gesture performed, the preprocessed data of interest obtained in block 311 are encoded as a feature vector and provided to a trained AI model to generate a corresponding embedding vector. Thus N embedding vectors are generated in a multidimensional space E.

[0092] Such an immersion vector is representative of behavioral and / or morphological characteristics of the user.

[0093] The N embedding vectors may be stored in the database 300 in association with the user identification and the raw data obtained in block 310.

[0094] In block 313, the N embedding vectors obtained in block 312 are combined to generate a biometric signature of the user. This biometric signature is itself an embedding vector in the same multidimensional space E used to generate the N embedding vectors. The biometric signature is used as a reference embedding vector or biometric template.

[0095] The biometric signature may be a combined embedding vector resulting from a combination of the N embedding vectors. For example, the biometric signature may be calculated as the barycenter (e.g., the equibarycenter) of the N embedding vectors.

[0096] The biometric signature may be stored in the database 300 in association with the user identification, the raw data obtained in block 310 and the N embedding vectors obtained in block 312.

[0097] The user's enrollment is then complete.

[0098] User enrollment can be done using a direct or indirect method.

[0099] According to the direct method, an enrollment session takes place. During this session, the user is prompted to perform the consent gesture one or more times in a row. From the raw data, preprocessed data of interest are extracted, which are coded in the form of vectors of morphological and / or behavioral characteristics. The morphological and / or behavioral characteristics are provided as input to the artificial intelligence model in order to obtain the corresponding embedding vector(s). From this embedding vector(s), an embedding vector serving as a biometric signature is calculated and saved as a biometric template.

[0100] According to the indirect method, the consent gesture can be used several times by the user to validate one or more operations, by collecting the raw data for these consent gestures but without performing authentication based on the gesture. After having detected enough of them (for example at least 3 consent gestures), the phase of obtaining the biometric template can be triggered automatically. From the raw data collected for the gestures, preprocessed data of interest are extracted which are coded in the form of vectors of morphological and / or behavioral characteristics. The preprocessed data of interest pass through the artificial intelligence model in order to obtain the corresponding embedding vector(s). From this(these) embedding vector(s), an embedding vector serving as a biometric signature is calculated and recorded as a biometric template.

[0101] Whatever the method used, the gesture of consent can be used as an authentication and / or identification factor from the next transaction.

[0102] Authentication of a user is based on the same principles as enrollment and is described with reference to blocks 320-323.

[0103] In block 320, for the purpose of authentication and consent collection, a user is asked to perform the expected consent gesture, i.e., a hand clap. Raw data, representative of this consent gesture, is acquired by one or more gesture sensors.

[0104] The raw data obtained at block 320 may be stored in the database 300 in association with a user identification.

[0105] As for block 311, at block 321, preprocessing is applied to the raw data acquired at block 320 to generate preprocessed data of interest, comprising values ​​of characteristic parameters of interest for user recognition. The characteristic parameters of interest for user recognition include morphological and / or behavioral characteristics.

[0106] The preprocessing may further comprise filtering the raw data, for example to remove outlier data or noise or other defects.

[0107] In block 322, the preprocessed data of interest obtained in block 321 is encoded as a feature vector and provided to the trained AI model (the same as in block 312) to generate a corresponding embedding vector. The current embedding vector is generated in the multidimensional space E.

[0108] Such an embedding vector is representative of behavioral characteristics and / or morphological characteristics of the user.

[0109] The current embedding vector may be stored in the database 300 in association with the user identification and the raw data obtained in block 320.

[0110] In block 323, the current embedding vector obtained in block 322 is compared with the user's biometric signature obtained in block 313. If the distance between the current embedding vector obtained in block 322 and the user's biometric signature obtained in block 313 is less than an authentication threshold, then the authentication is successful and the user's consent is deemed to have been given.

[0111] If the distance between the current embedding vector obtained in block 322 and the user's biometric signature obtained in block 313 is greater than this authentication threshold, then the authentication fails. If the distance is equal to the authentication threshold, the authentication decision may be failure or success.

[0112] When an operation must be validated, the user gives his consent by making the expected consent gesture (here the hand clap).

[0113] If the distance is below the predefined authentication threshold, then the user is recognized as a legitimate user and the transaction is validated. Thus, based on a single gesture, consent is given and user authentication is performed so that it can be certain that it is this user who has given his consent.

[0114] If the distance is above the authentication threshold, the transaction is not validated. Acquisition of a replacement authentication factor may be proposed.

[0115] Identification of a user can be carried out in a manner analogous to authentication:

[0116] - preprocessed data, including characteristic parameter values of interest for user recognition, are extracted from the raw data (as described for block 321);

[0117] - these pre-processed data are provided as input to the artificial intelligence model to obtain a current embedding vector (as described for block 322);

[0118] Then a distance calculation is performed between this current embedding vector and the biometric signatures of the users who have already been enrolled. The user whose biometric signature is at the smallest distance from the current embedding vector will be identified as the current user. His profile can be automatically selected. It can also be considered that above a certain identification threshold, no user is identified because it is considered that the gesture performed is that of a user outside the database 300.

[0119] The hand clapping gesture can be performed according to an execution protocol comprising the following 3 phases: - initial phase 1: transition from the initial resting position to the clap position: the user stretches his arms out in front of him to keep his hands visible in the headset until the clap is triggered; - phase 2 of clap: the user is in clap position and claps his hands once (i.e. a single clap); - final phase 3: transition from the clap position to a final rest position; the user returns to a rest position while waiting for confirmation that their clap has been taken into account.

[0120] The initial resting position may be different from the final resting position: in this case metrics can be used to detect these differences and exploit them as behavioral characteristics.

[0121] The initial resting position and / or the final resting position may be positions in which the arms are alongside the body. This makes it possible to limit the number of sensors required to capture data relating to these positions.

[0122] It is also conceivable that the user keeps the video game controller in his hands while performing the consent gesture: in such a case the clap in phase 2 will be done more by hitting the wrists against each other. Since no analysis of the sound produced by the consent gesture is necessary or used, it matters little that the sound produced in this way is lower.

[0123] The gesture execution protocol defined above is applicable to a large number of people and allows for a significant variety of morphologies and movements to be taken into account. The authentication model is enriched by this variety, this variety improving the discrimination of different users.

[0124] A start signal (visual and / or audible signal for example) can be emitted to require execution by the user of the consent gesture: from this start signal, the recording of the raw data is triggered.

[0125] This starting signal may correspond to information (text, sound, image, vibration, light signal, or other element) presented in content (sound, visual, etc.) rendered in the application environment. This starting signal may correspond to information (text, sound, image, vibration, light signal, or other element) presented via the interaction device, either in the application environment or at the same time as the application environment.

[0126] A counter can be triggered at the same time as the start signal so as to limit the duration of this recording to a maximum value (approximately 2 to 3 seconds for example).

[0127] The sensors that can be used to acquire the raw data can be sensors video in the VR headset, motion sensors in the VR headset and / or in a controller that the user holds in their hands.

[0128] As described for blocks 311 and 321, raw data acquired by the sensors are analyzed to extract preprocessed data of interest, comprising values ​​of characteristic parameters of interest for user recognition: these preprocessed data of interest may include behavioral and / or morphological characteristics.

[0129] Behavioral characteristics can be obtained for the three phases 1 to 3: the transition from the initial rest position to the clap position, the clap movement in the clap position, the transition from the clap position to the rest position. The behavioral data can include for example: a reaction time between the start signal and the start of the consent gesture, a speed of execution of the consent gesture, a duration of each phase, a total duration of execution of the consent gesture, a trajectory of the hands over all phases 1 to 3, trajectory of the clap in phase 2, etc.

[0130] Morphological characteristics can be obtained for the initial and / or terminal resting positions corresponding to phases 1 and 3. The morphological data can include, for example: the spatial position of each of the two hands, the spatial orientation of each of the two hands, the spatial position of the head, the spatial orientation of the head, size and length of the arms, spacing of the arms, spatial (a)symmetry of the arms.

[0131] In an exemplary embodiment, the raw data and / or behavioral characteristics may include:

[0132] - Position along three axes x,y,z of the head;

[0133] - Position along three axes x,y,z of the left hand;

[0134] - Position along three axes x,y,z of the right hand;

[0135] - Orientation along four axes w,x,y,z of the head;

[0136] - Orientation along four axes w,x,y,z of the left hand;

[0137] - Orientation along four axes w,x,y,z of the right hand.

[0138] In an exemplary embodiment, the extracted morphological features may include:

[0139] - User size (the head position reference is the ground);

[0140] - Distance between the head and the right hand;

[0141] - Distance between the head and the left hand;

[0142] - Distance between the 2 hands;

[0143] - Axial asymmetry of position x,y,z between the hands;

[0144] - Distance between the position of the right hand before and after the clap;

[0145] - Distance between the position of the left hand before and after the clap;

[0146] - Height of the clap position;

[0147] - Distance between the clap position and the head;

[0148] - Distance between the clap position and the resting position of the left hand before the clap ;

[0149] - Distance between the clap position and the position of the right hand before the clap;

[0150] - Position x,y,z of the clap under the head reference frame;

[0151] - Position x,y,z of the clap under the reference frame of the left hand;

[0152] - Position x,y,z of the clap under the right hand reference frame.

[0153] [Fig.4] shows a block diagram of an AI model and corresponding method allowing the extraction of characteristics from the consent gesture and the generation of embedding vectors from the extracted characteristics according to an exemplary embodiment.

[0154] This feature extraction method aims to extract information and / or measurements that are as discriminating as possible between users and to generate a feature vector that allows unambiguous distinction between several users.

[0155] For this purpose, the raw data acquired by the sensors are first converted into characteristic vectors which will be used by the artificial intelligence model.

[0156] Because feature vectors are generally not sufficient to simply and effectively generalize the distinction of users, an artificial intelligence model adapted to recognize discriminating characteristics (complex patterns or others) is used.

[0157] A single AI model is trained with an initial set of users and is then used for the identification and authentication of all users who have been enrolled, i.e. those for whom a biometric signature has been obtained as described with reference to [Fig.3].

[0158] The training of the AI ​​model can be based on a SNN (Siamese Neural Network). A Siamese neural network is an artificial neural network is a network comprising at least two identical sub-networks. The term "identical" here means that they have the same configuration and structure, with the same parameters and the same weights. The update of the parameters is reflected on both sub-networks. The Siamese network is used to train the AI ​​model. The Siamese network allows in particular efficient learning from little data. At the end of the training, one of the identical sub-networks is extracted and serves as an AI model for authentication as such.

[0159] These networks are used to find the similarity between inputs by comparing their feature vectors. This approach has the advantage of training a model AI from little data and take advantage of a generic AI model that can be used to authenticate any user without the need for retraining.

[0160] From all the raw data collected for consent gestures made by an initial set of users, the training of the AI ​​model can be carried out. The objective of training the AI ​​model is to minimize the distance between several embedding vectors of the same person, while maximizing the distance between the embedding vectors of different people so as to have an AI model that is sufficiently discriminating to be able to distinguish the different enrolled users. The confusion rate of one user compared to another must tend towards zero and the recognition rate of a user must tend towards 100% (the confusion matrix, which measures the quality of a classification system, is then the identity matrix).

[0161] Cost functions such as triplet loss or contrastive loss can be used to determine the adjustments to be made to the coefficients of the trained neural network. The triplet cost function is an objective function for machine learning algorithms where an input (called an anchor) is compared to a positive (truth) input and a negative (false) input. The objective of the contrastive cost function is to discriminate the characteristics of the input vectors.

[0162] Returning to [Fig. 4], at block 400, raw data, representative of a consent gesture made by a user, is acquired by one or more sensors. The raw data may be stored in a database in association with an identification of the user.

[0163] In block 410, a first preprocessing is applied to the raw data acquired in block 310 to extract morphological characteristics of interest for user recognition. The morphological characteristics are encoded in the form of a morphological characteristic vector.

[0164] In block 420, the morphological feature vector is provided as input to a first neural network configured to output an embedding vector of the morphological features. This first neural network may be a Multi-Layer Perceptron (MLP) network. The network may, for example, comprise a 10-node input layer, one or more 1024-node hidden layers, and a 10-node output layer. The cost function may be the “Triplet Loss” function and the optimization function may be the “Adam” function.

[0165] Examples of networks and training methods are described for example in this web page: https: / / towardsdatascience.com / how-do-we-train-neural-networks-edd985562b73.

[0166] At block 430, a second preprocessing is applied to the raw data acquired at block 310 to extract behavioral features of interest for user recognition. The behavioral features are encoded as a behavioral feature vector.

[0167] In block 440, the vector of behavioral characteristics is provided as input to a second neural network configured to generate as output an embedding vector of the behavioral characteristics. This second neural network may be an LSTM (Long-Short-Term-Memory) network or a recurrent neural network: such networks have a memory and are thus adapted to analyze time series, in this case the successive movements and / or positions of the hands and / or arms in order to extract behavioral characteristics of interest from the gesture performed by the user.

[0168] Examples of networks and training methods are described for example in this web page: https: / / towardsdatascience.com / lstm-how-to-train-neural-networks-to-write-like-lovecr aft-e56ell65f514.

[0169] In block 450, the behavioral feature embedding vector and the morphological feature embedding vector are concatenated to generate a concatenated embedding vector including all of the behavioral and morphological features.

[0170] In block 460, the concatenated embedding vector is provided as input to a third neural network configured to output a final embedding vector that accounts for both behavioral and morphological features in a single vector. This third neural network may be a Multi-Layer Perceptron (MLP) network. The network may include, for example, a 20-node input layer, one or more 1024-node hidden layers, and a 10-node output layer.

[0171] Figures 5A and 5B illustrate aspects of a method of training a model according to an exemplary embodiment.

[0172] Figures 5A and 5B show points in a two-dimensional space: each of these points (indicated by squares, triangles or circles) results from the projection (for example by principal component analysis) into this two-dimensional space of an embedding vector obtained for a user among users U08 to U016.

[0173] In [Fig.5A], we show the projections of the embedding vectors obtained before training the AI ​​model: we observe that the points are distributed in space so that the points corresponding to different users can be found in the same area of ​​space.

[0174] In [Fig.5B], we show the projections of the embedding vectors obtained after training the AI ​​model (the training having been carried out on a simplified set of approximately 20 users and for approximately 200 to 300 training epochs): we observe that the points are distributed in space so that the points corresponding to one user tend to move closer to each other and away from the points corresponding to another user. Discrimination between users is therefore possible.

[0175] [Fig.5B] also shows the points S08 to S016 corresponding to the projection of the biometric signatures obtained respectively for users U08 to U016. It is observed that the points corresponding to a user are positioned around the point corresponding to this biometric signature.

[0176] Like all biometric factors, the morphology and behavior of a user may change slightly over time. A regular update of the biometric template (i.e. the biometric signature calculated as described for block 313) can therefore be carried out.

[0177] There are different types of strategy for updating a biometric template. For example, one can select the embedding vectors from an authentication phase whose distance with the biometric template is either below or above a certain authentication threshold.

[0178] In the first case (embedding vector below an authentication threshold), the biometric template can be updated transparently for the user. Indeed, it is considered that if the distance is sufficiently small and therefore the confidence in the authentication sufficiently strong, then it is impossible for embedding vectors resulting from a false positive to be taken into account for the calculation of the biometric template and the biometric template will simply be reinforced.

[0179] In the second case (embedding vector above the authentication threshold), the update can take advantage of a false negative. Another replacement authentication factor can be used in order to be able to carry out the authentication: it is ensured that it is indeed the legitimate user by means of a replacement authentication factor. If this authentication succeeds, then we are in the case of a false negative and the update of the template on the basis of the embedding vector above the authentication threshold makes it possible to integrate into the biometric template additional characteristics that have not yet been taken into account in the biometric template.

[0180] [Fig.6] is a flowchart illustrating a method of interaction with a user according to an exemplary embodiment.

[0181] The method may be executed by a host device (e.g., a device interaction device), respectively a host system (e.g., an interaction device cooperating with an authentication server), serving as an interaction device (respectively system).

[0182] In the case of a virtual reality application, the interaction device may comprise the VR headset and / or the controller and / or a joystick and / or other interaction command input equipment.

[0183] Although the steps are described sequentially, those skilled in the art will understand that certain steps may be omitted, combined, performed in a different order and / or in parallel.

[0184] During step 600 (enrollment step) a biometric signature of the user is generated.

[0185] The generation of the user's biometric signature may comprise: - an acquisition of raw data representative of one or more consent gestures made by the user for enrollment; - a generation, based on the raw data, of an embedding vector representative of behavioral and / or morphological characteristics of the user for each gesture made by the user for enrollment; - a generation of the user's biometric signature from the embedding vector(s) obtained for the gesture(s) made by the user for enrollment.

[0186] The biometric signature can be calculated as the equibarycenter of the embedding vectors obtained for the gesture(s) performed by the user for enrollment.

[0187] The number of gestures performed by the user for enrollment can be between 1 and 3.

[0188] During step 610, a signal is triggered to require execution by the user of a consent gesture validating the execution of an operation to be executed, the consent gesture being a clap of the hands.

[0189] During step 620, a biometric authentication, or respectively a biometric identification, based on a consent gesture made by the user is carried out.

[0190] Biometric authentication may include: - an acquisition of raw data representative of the consent gesture made by the user; - a generation of an embedding vector representative of behavioral and / or morphological characteristics of the user based on the raw data; - obtaining an authentication decision based on a comparison of a biometric signature of the user with the embedding vector, the biometric signature being representative of behavioral and / or morpho- characteristics user logic.

[0191] Comparing the user's biometric signature with the embedding vector may include: calculating the distance between the biometric signature and the embedding vector, authentication being successful if the distance is less than an authentication threshold.

[0192] Biometric identification may include: - an acquisition of raw data representative of the consent gesture made by the user; - a generation of an embedding vector representative of behavioral and / or morphological characteristics of the user based on the raw data; - obtaining an identification decision based on a comparison of biometric signatures representative of behavioral and / or morphological characteristics of respective users with the embedding vector.

[0193] The comparison of the biometric signatures with the embedding vector comprises a calculation of the distance between each of the biometric signatures and the embedding vector, the identification being successful if the smallest of the distances obtained is less than an identification threshold, the user being identified as being the user for whom the smallest distance was obtained.

[0194] During step 630, an execution of the operation in the event of successful authentication, or respectively in the event of successful identification.

[0195] The hand clapping gesture, whether it is the one or ones performed during the enrollment step 600 or during the authentication (or respectively identification) step 620, comprises a first phase of passing from an initial arm rest position to a clap position, a second phase of clap in the clap position and a third phase of passing from the clap position to a final arm rest position.

[0196] [Fig.7] illustrates aspects of an interaction method with authentication and / or identification according to an exemplary embodiment.

[0197] This figure shows how the accuracy of authentication varies depending on the number N of gestures used to perform enrollment: this figure shows that maximum accuracy is achieved from N=3 gestures, with a very low rate of false positives and false negatives. By accuracy, we mean here the percentage of correct decisions, less the rate of false positives and false negatives).

[0198] [Fig.8] illustrates aspects of an interaction method with authentication and / or identification according to an exemplary embodiment.

[0199] This figure shows how the false positive rate FP and the false negative rate FN vary depending on the value of the authentication threshold used in the comparison of a current embedding vector with the biometric signature. The rate of FP false positive rate increases with the authentication threshold while FN negative rate decreases with the authentication threshold. Adjusting this value keeps both FP false positive rate and FN false negative rate below a threshold.

[0200] In the description of the different phases and methods, although the steps are described sequentially, those skilled in the art will understand that certain steps may be omitted, combined, carried out in a different order and / or in parallel.

[0201] One or more or all of the steps of one or more methods described in this document may be implemented by software or computer program and / or by hardware, for example by circuit, programmable or not, specific or not.

[0202] The functions, steps and methods described in this document may be implemented by software (e.g., via software on one or more processors, for execution on a general purpose or special purpose computer) and / or be implemented by hardware (e.g., one or more electronic circuits, and / or any other hardware component).

[0203] The present description thus relates to a software or computer program, capable of being executed by a host device (for example, an interaction device), respectively a host system (for example, an interaction device cooperating with an authentication server), serving as an interaction device (respectively system), by means of one or more data processors, this software / program comprising instructions for causing the execution by this host device (respectively system) of all or part of the steps of one or more methods described in this document. These instructions are intended to be stored in a memory of the host device (respectively system), loaded and then executed by one or more processors of this host device (respectively system) so as to cause the execution by this host device (respectively system) of the method.

[0204] This software / program may be coded using any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0205] The host device (respectively system) may be implemented by one or more physically distinct machines. The host device (respectively system) may generally have the architecture of a computer, including components of such architecture: data memory(s), processor(s), communication bus, hardware interface(s) for connecting this host device (respectively system) to a network or other equipment, user interface(s), etc.

[0206] In one embodiment, all or part of the steps of the interaction method or of another method described in this document are implemented by an authentication device provided with means for implementing these steps of this method.

[0207] These means may comprise software means (for example, instructions of one or more components of a program) and / or hardware means (for example, data memory(ies), processor(s), communication bus, hardware interface(s), etc.).

[0208] These means may comprise, for example, one or more circuits configured to execute one or more or all of the steps of one of the methods described herein. These means may comprise, for example, at least one processor and at least one memory comprising program instructions configured to, when executed by the processor, cause the device to execute one or more or all of the steps of one of the methods described herein.

[0209] Means implementing a function or a set of functions may correspond in this document to a software component, a hardware component or a combination of hardware and / or software components, capable of implementing the function or the set of functions, according to what is described below for the means concerned.

[0210] The present description also relates to an information medium readable by a data processor, and comprising instructions of a program as mentioned above.

[0211] The information carrier may be any hardware means, entity or device, capable of storing the instructions of a program as mentioned above. Usable program storage media include ROM or RAM memories, magnetic storage media such as magnetic disks and magnetic tapes, hard disks or optically readable digital data storage media, or any combination of these media.

[0212] In some cases, the computer-readable storage medium is not transient. In other cases, the information medium may be a transient medium (e.g., a carrier wave) for the transmission of a signal (electromagnetic, electrical, radio, or optical signal) carrying the program instructions. This signal may be conveyed via a suitable transmission means, wired or wireless: electrical or optical cable, radio or infrared link, or by other means.

[0213] An embodiment also relates to a computer program product comprising a computer-readable storage medium on which program instructions are stored, the program instructions being configured to cause the host device (respectively system) (e.g. a computer) to implement all or part of the steps of one or more methods described herein when the program instructions are executed by one or more processors and / or one or more several programmable hardware components of the host device (respectively system).

Claims

Claims

1. Method for interacting with a user of an application environment, the method comprising - triggering (610) a signal to require execution by the user of a consent gesture validating the execution of an operation to be executed in the context of the application environment, the consent gesture being a clap of the hands; - biometric authentication (620), or respectively biometric identification, based on a consent gesture made by the user; - execution (630) of the operation in the event of successful authentication, or respectively successful identification.

2. Method according to claim 1, in which the biometric authentication comprises: - an acquisition of raw data representative of the consent gesture made by the user; - a generation of an embedding vector representative of behavioral and / or morphological characteristics of the user on the basis of the raw data; - an obtaining of an authentication decision on the basis of a comparison of a biometric signature of the user with the embedding vector, the biometric signature being representative of behavioral and / or morphological characteristics of the user.

3. The method of claim 2, wherein comparing the user's biometric signature with the embedding vector comprises calculating the distance between the biometric signature and the embedding vector, authentication being successful if the distance is less than an authentication threshold.

4. Method according to claim 1, in which the biometric identification comprises: - an acquisition of raw data representative of the consent gesture made by the user; - a generation of an embedding vector representative of behavioral and / or morphological characteristics of the user on the basis of the raw data; - an obtaining of an identification decision on the basis of a comparison of biometric signatures representative of characteristics behavioral and / or morphological characteristics of respective users with the embedding vector.

5. The method of claim 4, wherein the comparison of the biometric signatures with the embedding vector comprises a distance calculation between each of the biometric signatures and the embedding vector, the identification being successful if the smallest of the distances obtained is less than an identification threshold, the user being identified as being the user for whom the smallest distance was obtained.

6. Method according to any one of claims 1 to 5, comprising - an acquisition of raw data representative of one or more consent gestures made by the user for enrollment; - a generation, on the basis of the raw data, of an embedding vector representative of behavioral and / or morphological characteristics of the user for each gesture made by the user for enrollment; - a generation of the biometric signature of the user from the embedding vector(s) obtained for the gesture(s) made by the user for enrollment.

7. Method according to claim 6, in which the biometric signature is calculated as the equibarycenter of the embedding vectors obtained for the gesture(s) performed by the user for enrollment.

8. Method according to any one of claims 1 to 7, in which the application environment is a virtual reality environment.

9. A method according to any one of claims 1 to 8, wherein the hand clapping gesture comprises a first phase of moving from an initial arm rest position to a clap position, a second phase of clap in the clap position and a third phase of moving from the clap position to a final arm rest position.

10. Device comprising means for implementing a method according to any one of the preceding claims.

Images