Notification method for guiding visual attention of user to specific interaction target by utilizing facial tactile feedback

Abstract

The invention provides a notification method for guiding visual attention of a user to a specific interaction target by utilizing facial tactile feedback, which belongs to the field of man-machine interaction and comprises the following steps: judging the distance between an object and the user and the azimuth angle relative to the current facing direction of the user by identifying a connecting line of the target object and the eyebrow position of the user; the user is guided to look at the corresponding direction by driving the feedback unit at the corresponding position in real time; the distance of the target object is indicated through the force, the amplitude, the duration, the driving quantity and the like of the tactile feedback unit; when the user begins to move towards the target object, continuously performing feedback change in real time according to the dynamic change of the user and the distance and orientation thereof from left to right, for example, the closer the feedback change is, the stronger the feedback change is; the multi-dimensional information notification of the source interaction target is displayed by controlling design parameters such as the force, the amplitude, the space-time mode and the duration of the tactile feedback unit; and judging whether the sight of the user is positioned to the target object or not through the eye movement focus, and stopping tactile feedback.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction, and in particular to a notification method for guiding a user's visual attention to a specific interaction target by using facial tactile feedback. Background Art

[0002] (For XR / VR / AR fields) In the use scenarios of head-mounted virtual (augmented, mixed) reality devices, some tasks require users to quickly shift their sight to interactive targets located in the peripheral field of vision or outside the field of vision in a complex visual environment. These interactive targets include but are not limited to people, objects, windows, icons, etc. that exist in virtual or real scenes. Task scenarios include 3D space navigation, finding objects in virtual / real environments, perceiving social objects, identifying virtual devices or windows, etc. Compared with vision and hearing, touch has the potential to provide more accurate spatial orientation information than hearing while avoiding excessive visual sensory burden.

[0003] Existing technologies have used tactile feedback provided to the head and face by wearable devices themselves (such as head-mounted displays HMD) for spatial guidance. Related studies have shown that specific tactile feedback for the head and face can be intuitively mapped with the user's relevant body movements and eye movements, helping people shift their attention to targets within and outside the line of sight. Facepush is a device that can generate a normal force on the left or right side of the user's face to guide the user's attention to objects outside the field of view, and the feedback intensity is mapped with the rotation angle. Masque adds 6 actuators to the facial area in contact with the HMD to provide lateral skin stretching. They designed several modes in which both the stretching direction and the stretching point change to guide the user to look or move to a specific direction in the room. Tseng et al. proposed a skin stroking tactile device installed inside the HMD, which produces stroking feedback on the eye circle and can be used as an off-screen indicator. Overall, considering the number and location of actuators, these devices use tactile stimulation to indicate the basic direction of body movement. Some studies have explored the use of tactile stimulation for navigation within and outside the line of sight. VirtualWhiskers has 2 robotic arms mounted on either side of the face and stimulates target points calculated based on the relative position of the face, covering 180° in the azimuth plane and 90° in the elevation plane. deJesusOliveira et al. deployed five vibration actuators on the forehead and two vibration actuators near the temples. The vibrotactile HMD points in the direction of the object in the azimuth plane (202.5° field of view). The vibration frequency peaks at the correct elevation angle (from −22.5° to 45°). In addition to the devices connected to the HMD, a desktop ultrasonic device can also provide tactile feedback to the face. Whiskers applies a set of tactile stimuli spaced 3.5 mm apart to the user's cheeks, center of forehead, and above the eyebrows. The direction of the tactile stimulus movement guides the user's gaze within a 60° field of view.

[0004] These studies include basic orientation cues for interactive targets, but lack an indication of the spatial distance of the interactive targets relative to the user. This results in the user being unable to accurately shift visual attention to a specific interactive target when multiple interactive targets are in the same orientation but at different distances relative to the user, resulting in a certain degree of occlusion of the interactive targets. In addition, the non-feedback unit array of the previous solution can provide a low tactile resolution. Finally, the aforementioned solution lacks the ability to provide changes in multiple tactile stimuli such as pressure, vibration, and temperature, and cannot provide rich tactile feedback effects. Summary of the invention

[0005] The present invention provides a notification method for guiding a user's visual attention to a specific interactive target using facial tactile feedback. The present invention provides a tactile stimulation with orientation to the surface of the user's skin around the eye by a tactile feedback unit that provides force, vibration and temperature feedback at the same time, such as a shape memory alloy (SMA) tactile feedback device, to help the user perform visual-tactile fusion human-computer interaction. Several tactile feedback units are integrated into an existing head-mounted display to provide tactile feedback to the user's forehead, cheeks, and temples around the eyes, helping the user to locate the interactive target from a specific spatial position and distance in a virtual / extended reality scene. When the system recognizes an object that needs to be indicated, the system calculates the relative orientation and relative distance of the current object relative to the center of the user's field of vision, helping the user's line of sight to quickly locate and reduce visual search time. In addition, the present invention can also display multi-dimensional information notifications of the source interactive target, such as urgency, event type, progress, emotional state, movement, etc., by controlling the tactile stimulation parameters such as the strength, amplitude, and spatiotemporal pattern of the tactile feedback unit.

[0006] Technical Solution

[0007] A notification method for guiding a user's visual attention to a specific interactive target using facial tactile feedback is a method that uses tactile feedback generated by a facial tactile array to indicate the relative orientation and relative distance of the interactive target relative to the center of the user's visual field, and presents multi-dimensional notification information through changes in tactile parameters.

[0008] The present invention provides a notification method for guiding a user's visual attention to a specific interactive target by using facial tactile feedback, comprising the following method steps: (1) By identifying the line connecting the target object and the user's brow center, the distance between the object and the user and the angle of the object relative to the current direction the user is facing are determined; (2) The feedback unit at the corresponding position is driven in real time to guide the user to look in the corresponding direction. When the user starts to move toward the target object, the feedback changes continuously in real time according to the dynamic changes of the user and his / her position, such as from left to right. (3) The distance of the target object is indicated by the strength, amplitude, duration, number of drives, etc. of the tactile feedback unit; when the user starts to move towards the target object, the feedback changes continuously in real time according to the dynamic changes in the distance between the user and the target object, such as the closer the closer the feedback is, the stronger the feedback is; (4) Display multi-dimensional information notifications of the source interaction target, such as urgency, information type, progress, emotional state, etc., by controlling the design parameters of the tactile feedback unit, such as strength, amplitude, spatiotemporal pattern, etc.; (5) Determine whether the user's gaze is located on the target object through eye movement focus and stop tactile feedback.

[0009] Preferably, step (2) is performed by tactile stimulation on the left side of the surface guiding the user to look to the left, and tactile stimulation on the right side of the surface guiding the user to look to the right.

[0010] Preferably, in step (3), the closer the target is to the user, the larger the tactile stimulation range is, and the principle of "near is larger and far is smaller" similar to vision is adopted to reduce the user's learning cost.

[0011] Preferably, in step (4), the multi-dimensional information notification of the source interactive target is displayed by controlling the design parameters such as the strength, amplitude, spatiotemporal pattern, etc. of the tactile feedback unit, including but not limited to urgency, information type, progress, emotional state, movement, etc., where the stronger the strength, the more urgent the event; two consecutive tactile stimulations represent social software information, etc.

[0012] Beneficial effects of the present invention: 1. The present invention provides real-time and dynamic prompts of relative orientation and distance, which can more accurately locate the interactive target of the information source in complex scenes.

[0013] 2. The present invention integrates multi-modal tactile feedback such as force, vibration, and heat to provide a rich tactile experience.

[0014] 3. The present invention displays multi-dimensional information notifications of the source interactive target, including but not limited to urgency, information type, progress, emotional state, movement, etc., by controlling design parameters such as strength, amplitude, and spatiotemporal pattern of the tactile feedback unit.

[0015] 4. The tactile feedback unit array of the present invention provides tactile stimulation with higher resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present disclosure, the following briefly introduces the drawings required to be used in some embodiments of the present disclosure. Obviously, the drawings described below are only drawings of some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings. In addition, the drawings described below can be regarded as schematic diagrams, and are not limitations on the actual size of the product involved in the embodiments of the present disclosure, the actual process of the method, the actual timing of the signal, etc.

[0017] Figure 1 A reference diagram for facial positioning; Figure 2 This is a schematic diagram of user usage scenario 1; Figure 3 This is a schematic diagram of user usage scenario 2; Figure 4 Schematic diagram of user usage scenario 3. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0020] The present invention is described in further detail below by way of examples.

[0021] Example 1

[0022] like Figure 1 As shown, the notification method for guiding the user's visual attention to a specific interaction target using facial tactile feedback includes the following method steps: (1) By identifying the line connecting the target object and the user's brow center, the distance between the object and the user and the angle of the object relative to the current direction the user is facing are determined; (2) Guiding the user to look in the corresponding direction by driving the feedback unit at the corresponding position in real time (for example, tactile stimulation on the left side of the face guides the user to look to the left); (3) The number, strength, amplitude, duration, number of drives, etc. of tactile feedback units are used to indicate the distance of the target object (for example, the closer the target is to the user, the larger the tactile stimulation range is, and the principle of "near is larger, far is smaller" similar to vision is adopted to reduce the user's learning cost); (4) By controlling the strength, amplitude, spatiotemporal pattern and other design parameters of the tactile feedback unit, the multi-dimensional information notification of the source interactive target is displayed, including but not limited to urgency, information type, progress, emotional state, etc. For example, the stronger the strength, the more urgent the event; two consecutive tactile stimulations represent social software information, etc.; when the user starts to move towards the target object, the feedback changes continuously in real time according to the dynamic changes of the distance and direction between the user and the target object, such as the closer the stronger, from left to right.

[0023] (5) Determine whether the user's gaze is located on the target object through eye movement focus and stop tactile feedback.

[0024] Example 2

[0025] like Figure 2 As shown, user usage scenario 1: social object recognition in XR scenario In XR scenarios, users need to perceive the positions of both virtual and real social objects. When users are immersed in a VR scene, they can perceive the relative position and distance of social objects in the real scene through tactile feedback without destroying the sense of immersion in VR. Users can perceive the social status and intention of social objects through tactile feedback. When users exit VR and see real social objects, the tactile feedback stops.

[0026] Specific steps: First, the camera identifies the person in the room, converts the spatial coordinates of the real object and the virtual object into a coordinate position centered on the user's forehead and the central line of sight as the X-axis, outputs the distance to the user based on the coordinate position, and then outputs the actuation parameters of the tactile feedback unit (for example: spatiotemporal mode, duration, pressure, temperature, etc.) based on the coordinate position, outputs the direction relative to the center of the user's field of vision, and the tactile feedback unit for the direction, starts to actuate the tactile unit based on the feedback parameters and unit, outputs the user's eye movement data of gazing at the social object, and finally stops actuation.

[0027] Example 3

[0028] like Figure 3 As shown, user scenario 2: message tracing for multiple virtual windows In the XR home scene, users can place multiple windows at different spatial locations at the same time. For example, there is a timer window in the kitchen and a TV window in the living room. When the user is watching TV, if a notification pops up on the kitchen timer, the user can accurately perceive through tactile feedback that the notification comes from a window farther away in the kitchen direction, thereby accurately locating the spatial source of the notification. And judge the progress of cooking by the strength of tactile stimulation.

[0029] Specific steps: first, when there is a message notification in the chat window, relevant information of the notification (urgency, information type, etc.) is obtained, and the actuation parameters of the tactile feedback unit (for example, spatiotemporal mode, duration, pressure, temperature, etc.) are set according to the relevant information; according to the message notification in the chat window, the spatial coordinates of the virtual window are output, and the coordinates are converted into coordinate positions centered on the center of the user's eyebrows and the central line of sight as the X-axis, and the distance to the user is output according to the coordinate position, and then the actuation parameters of the tactile feedback unit (for example, the number of drives, etc.) are output according to the coordinate position, and the tactile feedback unit for the orientation starts to actuate the tactile unit according to the feedback actuation parameters, outputs the user's eye movement data of staring at the window, and finally stops actuation.

[0030] Example 4

[0031] like Figure 4 As shown, user usage scenario 3: Finding objects in real scenes In some situations, such as when visiting a museum, the user wants to view a specific painting, but does not know the exact location and needs to search for it. The position information of the target painting relative to the center of the user's field of view can be determined through the combination of camera and indoor positioning, and the user can be helped to find the corresponding painting through tactile feedback prompts. When the user approaches the target object, the tactile feedback gradually increases, and the orientation changes dynamically, such as the tactile array from left to right, from bottom to top, etc., to provide navigation instructions until the target object appears in the user's field of view.

[0032] Specific steps: first, the camera recognizes paintings in multiple directions, recognizes the museum space information and the user's current position to obtain the spatial coordinates of the real object, and converts the coordinates into a coordinate position centered on the user's eyebrow center and the central line of sight as the X-axis. The distance to the user is output according to the coordinate position, and then the actuation parameters of the tactile feedback unit (for example: space-time mode, etc.) are output according to the coordinate position. The tactile feedback unit for the orientation starts to actuate the tactile unit according to the feedback actuation parameters, outputs the user's eye movement data gaze window, and finally stops actuation.

[0033] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0034] The above describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claim involved.

[0035] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A notification method for guiding a user's visual attention to a specific interactive target using facial tactile feedback, characterized in that: The method comprises the following steps: (1) By identifying the line connecting the target object and the user's brow center, the distance between the object and the user and the angle of the object relative to the current direction the user is facing are determined; (2) The feedback unit at the corresponding position is driven in real time to guide the user to look in the corresponding direction. When the user starts to move toward the target object, the feedback changes continuously in real time according to the dynamic changes of the user and his / her position, such as from left to right. (3) The distance of the target object is indicated by the strength, amplitude, duration, number of drives, etc. of the tactile feedback unit; when the user starts to move towards the target object, the feedback changes continuously in real time according to the dynamic changes in the distance between the user and the target object, such as the closer the closer the feedback is, the stronger the feedback is; (4) By controlling the design parameters of the tactile feedback unit, such as force, amplitude, spatiotemporal pattern, etc., the multi-dimensional information notification of the source interaction target can be displayed, such as urgency, information type, progress, emotional state, etc.

2. (5) Determine whether the user's gaze is located on the target object through eye movement focus and stop tactile feedback.

3. The method according to claim 1, characterized in that: Step (2) is that the tactile stimulation on the left side of the face guides the user to look to the left, and the tactile stimulation on the right side of the face guides the user to look to the right.

4. The method according to claim 1, characterized in that: Step (3) means that the closer the target is to the user, the larger the tactile stimulation range is, and the principle of "near is larger, far is smaller" similar to vision is adopted to reduce the user's learning cost.

5. The method according to claim 1, characterized in that: Step (4) controls the strength, amplitude, spatiotemporal pattern and other design parameters of the tactile feedback unit to display multi-dimensional information notifications of the source interactive target, including but not limited to urgency, information type, progress, emotional state, movement, etc., where the stronger the strength, the more urgent the event; two consecutive tactile stimulations represent social software information, etc.

Images