Subjective feeling of movement recognition assistance device, method, and program

Abstract

According to one embodiment of the present invention, when assisting recognition of a user's subjective feeling of movement when operating an avatar, operation information of the avatar input by the user is acquired, and on the basis of the acquired operation information, an operation form of the avatar is determined. Furthermore, avatar operation control information corresponding to the operation form is generated, and on the basis of the generated avatar operation control information, the operation of the avatar displayed on a display device is controlled; and stimulus control information corresponding to the operation form is generated, and a stimulus presentation device is controlled using the generated stimulus control information, such that a stimulus is presented, in synchronization with the operation of the avatar, to a muscle among the user's muscles that corresponds to the operation part of the avatar.

Description

Device, method and program for supporting the recognition of a sense of agency

[0001] One aspect of the present invention relates to an apparatus, method, and program for supporting a user's sense of agency, for example, by allowing the user to perceive an avatar's actions as the user's own actions.

[0002] In recent years, research has been conducted into systems that allow users who are unable to directly participate in meetings or classes due to illness or disability, for example, to communicate with other users in remote offices or schools using omnidirectional avatar robots that act as substitutes for the users. This research has proposed a method for using eye movements, which are less affected by disabilities, as an operation interface for physically disabled users, and controlling the movement of the avatar robot in real space in response to gaze input (see, for example, Non-Patent Document 1).

[0003] By using this operation interface, for example, in the metaverse space, a user can control the hand-waving and walking movements of their avatar, which is their alter ego.

[0004] Noriyasu, R. et al., "Movement Generation for Omnidirectional Avatar Robots Using Gaze Input," IEICE Technical Report; IEICE Technical Report 120.357 (2021): 9-12.

[0005] However, in the system described in Non-Patent Document 1, although it is possible to move an avatar using the user's gaze input, it is difficult for the user to perceive the sensation that their own body is moving through the avatar's movements, that is, a sense of agency.

[0006] For example, physically disabled people who have difficulty moving their joints voluntarily have difficulty matching their own physical movements with those of their avatar, making it difficult for them to recognize a sense of agency. Even able-bodied people, for example, when participating in a virtual event held in the Metaverse, may be able to control their own avatar, but they cannot perceive the avatar's movements as their own physical movements. This makes it difficult for them to fully experience the sense of realism and excitement of the event.

[0007] The present invention has been made in light of the above circumstances, and aims to provide a technique for improving the user's sense of agency when an avatar moves.

[0008] To solve the above problems, one aspect of the device or method for assisting in the recognition of a sense of agency in motion according to the present invention acquires avatar operation information input by a user, determines the operation form of the avatar based on the acquired operation information, generates avatar movement control information corresponding to the operation form, and controls the movement of the avatar displayed on a display device based on the generated avatar movement control information, and simultaneously generates stimulus control information corresponding to the operation form and controls a stimulus presentation device using the generated stimulus control information to present stimuli to muscle parts of the user that correspond to the moving parts of the avatar in synchronization with the movement of the avatar.

[0009] According to one aspect of the present invention, when a user moves an avatar, which is an avatar representing the user's alter ego, a stimulus is presented to a muscle part of the user that corresponds to the moving part of the avatar in synchronization with the movement of the avatar. As a result, the presented stimulus causes the muscle part of the user to move in synchronization with the movement of the avatar, thereby enabling the user to perceive a sense of agency in movement in accordance with the movement of the avatar.

[0010] That is, according to one aspect of the present invention, it is possible to provide a technology that improves the user's sense of agency when an avatar moves.

[0011] FIG. 1 is a diagram showing an example of the configuration of a system including a device for supporting a sense of agency in motion according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the hardware configuration of a device for supporting a sense of agency in motion according to an embodiment of the present invention. FIG. 3 is a block diagram showing an example of the software configuration of a device for supporting a sense of agency in motion according to an embodiment of the present invention. FIG. 4 is a flowchart showing an example of the processing procedure and processing content of assistance control processing executed by a control unit 1 of the device for supporting a sense of agency in motion shown in FIG. 3. FIG. 5 is a diagram showing an example of the configuration of a control table stored in a data storage unit of the device for supporting a sense of agency in motion shown in FIG. 3.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [One embodiment] (Summary) In one embodiment of the present invention, for example, a user US with a disability in the limbs is the user to be assisted, and when the user US operates an avatar, which is an avatar of the user US created in the metaverse space, stimulation is presented to muscle parts of the user US that correspond to the movement parts of the avatar in synchronization with the movement of the avatar.

[0014] In this way, even a user US with limb disabilities can feel as if the corresponding part of their body is moving in response to the avatar's movements, i.e., can recognize a sense of agency. In other words, it is possible to improve the sense of agency of physically disabled people.

[0015] (Configuration Example) (1) System FIG. 1 is a diagram showing an example of the configuration of a system for supporting a user's recognition of a sense of agency in movement, according to an embodiment of the present invention.

[0016] The system of one embodiment includes a control device, which is the core of the system, that is a motor subject recognition support device (hereinafter simply referred to as the support device) CS.

[0017] The support device CS is placed on a side table TB, and a monitor device MT is placed on the housing of the support device CS. A camera VM is attached to the upper end of the monitor device MT. The monitor device MT and camera CM are connected to the support device CS via, for example, a signal cable.

[0018] The monitor device MT displays, for example, a display screen of the metaverse space. The camera CM is, for example, a web camera, and captures a facial image including the eyes of the user US, which is used to estimate the gaze direction of the user US.

[0019] The system of one embodiment further includes an electrical stimulation device. The electrical stimulation device is composed of electrodes E1 and E2 attached to muscles related to the limbs of the user US to be supported, and an electrical stimulation generator ES that applies electrical stimulation signals to these electrodes E1 and E2. Note that each of the electrodes E1 and E2 is composed of a pair of electrodes consisting of an anode and a cathode.

[0020] The electrical stimulation generator ES is connected to the support device CS, for example, via a signal cable. It generates an electrical stimulation signal in accordance with an electrical stimulation control signal supplied from the support device CS. For example, a low-frequency stimulation signal suitable for muscular contraction is generated by superimposing a stimulation frequency of 15 to 200 Hz on a reference frequency of several kHz to 20 kHz. The electrical stimulation generator ES then selectively applies the low-frequency electrical stimulation signal to the electrodes E and E2, thereby selectively applying electrical stimulation (EMS: Electrical Muscle Stimulation) to muscles associated with the limbs of the user US, thereby causing muscle contraction.

[0021] The user US to be assisted sits or lies on his / her back on the bed BD in a state where he / she can see the monitor device MT and the camera CM.

[0022] (2) Support Device CS The support device CS is, for example, a general-purpose personal computer, and is capable of data communication with a server computer on the web or cloud, or with other terminals, via a network (not shown).

[0023] 2 and 3 are block diagrams showing an example of the hardware and software configurations of the support device CS, respectively.

[0024] The support device CS has a control unit 1 that uses a hardware processor such as a central processing unit (CPU), and this control unit 1 is connected via a bus 5 to a storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter, the interface will be abbreviated as I / F) unit 4.

[0025] The input / output I / F unit 4 has the function of receiving facial image data of the user US output from the camera CM, the function of outputting avatar display data to the monitor device MT, and the function of outputting an electrical stimulation control signal to the electrical stimulation generator ES.

[0026] The input / output I / F unit 4 may be provided with a communication interface function, which enables signals to be transmitted and received between the electrical stimulation generator ES and the camera CM using a low-power wireless communication interface such as Bluetooth (registered trademark), thereby increasing the flexibility in the installation of the support device CS, the electrical stimulation generator ES, and the camera CM.

[0027] The program storage unit 2 is configured by combining, for example, a nonvolatile memory such as a solid-state drive (SSD) as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a read-only memory (ROM), and stores middleware such as an operating system (OS), as well as application programs required to execute various controls according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0028] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as a RAM (Random Access Memory), and in the storage area thereof, a control table storage unit 31 is provided as the main storage unit according to one embodiment of the present invention.

[0029] The control table storage unit 31 stores control table data for controlling the movement of the avatar and the electrical stimulation of the user US. Specifically, the control table data associates avatar movement patterns with stimulation presentation control data in response to avatar operation commands. The avatar movement patterns include information indicating the target body parts of the avatar and the content of the movements. The stimulation presentation control data includes information indicating the electrical stimulation presentation body parts, the stimulation patterns, and the stimulation intensity.

[0030] The control unit 1 includes the following processing functions necessary to implement one embodiment: an image data acquisition processing unit 11, a gaze direction estimation processing unit 12, an operation command determination processing unit 13, an avatar movement control information generation processing unit 14, an avatar display control unit 15, a stimulus presentation control information generation processing unit 16, and an electrical stimulus generation control unit 17.

[0031] Each of the processing units 11 to 17 is realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 11 to 17 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0032] The image data acquisition processing unit 11 receives facial image data of the user US output from the camera CM via the input / output I / F unit 4, and temporarily stores the received facial image data in an image storage area (not shown) within the data storage unit 3.

[0033] The gaze direction estimation processing unit 12 estimates the gaze direction specified by the user US based on the movement of the eyeballs, based on the face image data.

[0034] The operation command determination processing unit 13 determines an operation command represented by the estimated gaze direction of the user US. For this determination, for example, a correspondence table (not shown) that associates gaze directions with operation commands is used.

[0035] The avatar movement control information generation processing unit 14 uses the determined operation command as a key to access the control table data in the control table storage unit 31, thereby obtaining an avatar movement pattern corresponding to the operation command. Specific examples of the avatar movement pattern will be described in the operation examples.

[0036] The avatar display control unit 15 generates display data for an avatar whose movements reflect the avatar movement patterns, and outputs the generated avatar display data from the input / output I / F unit 4 to the monitor device MT.

[0037] The stimulus presentation control information generation processing unit 16 accesses the control table storage unit 31 using the determined operation command as a key, thereby obtaining stimulus presentation control data corresponding to the operation command.

[0038] The electrical stimulation generation control unit 17 generates an electrical stimulation control signal based on the stimulation presentation control data, and outputs the generated electrical stimulation control signal to the electrical stimulation generator ES from the input / output I / F unit 4. An example of the process of generating the electrical stimulation control signal will also be described in an operation example.

[0039] (Example of Operation) Next, an example of operation of the support device CS configured as above will be described.

[0040] FIG. 4 is a flowchart showing an example of a processing procedure and processing content of the control for assisting in the recognition of a sense of agency of motion, which is executed by the control unit 1 of the assist device CS.

[0041] (1) Determination of operation command input by user US When assistance control is started, a facial image including the eye area of ​​the assistance target user US is captured in real time by the camera CM. The captured facial image data of the user US is output from the camera CM to the assistance device CS.

[0042] When the control unit 1 of the support device CS detects the start of the support process in step S10, under the control of the image data acquisition processing unit 11, in step S11, it receives the facial image data output from the camera CM via the input / output I / F unit 4 and temporarily stores this facial image data in the image storage area in the data storage unit 3 for image processing.

[0043] Next, in step S12, the control unit 1 of the support device CS, under the control of the gaze direction estimation processing unit 12, performs image processing on the stored face image data to estimate the gaze direction of the user US.

[0044] For example, the gaze direction estimation processing unit 12 first detects the eye region from the acquired face image data and cuts it into a rectangular region of a predetermined size. Next, the cut-out image of the eye region is grayscaled and binarized to detect the pupil, outer corner, and inner corner of the eye. Then, the current gaze direction of the user US is estimated from the detected center of gravity of the pupil and the positions of the outer corner and inner corner of the eye. The gaze direction is divided into, for example, nine regions, and it is determined which region the user US is gazing at based on the position of the center of gravity of the pupil.

[0045] In step S13, the control unit 1 of the support device CS determines whether the gaze direction is in a region other than the central region of the nine regions. If the gaze direction is in a region other than the central region, it determines that the user US has operated the avatar with their gaze, and then, under the control of the operation command determination processing unit 13, in step S14, refers to the correspondence table to determine the operation command corresponding to the gaze direction.

[0046] For example, depending on the direction of the line of sight, it is determined whether the operation command is "forward," "stop," "turn right," "turn left," or "raise hand."

[0047] (2) Avatar Motion Control The control unit 1 of the support device CS then generates avatar motion control information corresponding to the operation command under the control of the avatar motion control information generation processing unit 14. For example, in step S15, the avatar motion control information generation processing unit 14 accesses the control table data in the control table storage unit 31 using the operation command as a key, and reads out an avatar motion pattern corresponding to the operation command.

[0048] 5 shows an example of control table data stored in the control table storage unit 31. In this example, if the operation command is "move forward," "swing left and right arms alternately" and "move left and right legs alternately" are read out as avatar motion patterns. Also, if the operation command is "raise right arm," "raise right arm" is read out as the avatar motion pattern.

[0049] Subsequently, in step S16, the control unit 1 of the support device CS generates display image data of the avatar reflecting the avatar movement pattern under the control of the avatar display control unit 15, and outputs the generated display image data of the avatar to the monitor device MT from the input / output I / F unit 4. As a result, a moving image representing the avatar movement according to the operation of the user US is displayed on the monitor device MT.

[0050] (3) Control of Stimulus Presentation to User US In synchronization with the above-described avatar motion control, the control unit 1 of the support device CS executes a control process for presenting stimuli to the user US as follows.

[0051] That is, in step S17, the control unit 1 first accesses the control table data in the control table storage unit 31 using the operation command as a key under the control of the stimulus presentation control information generation processing unit 16, and reads out information representing the stimulus presentation target area and stimulus pattern corresponding to the operation command. At the same time, in step S18, the stimulus presentation control information generation processing unit 16 reads out information representing the stimulus intensity corresponding to the operation command from the control table data.

[0052] As a result, for example, in the example shown in Figure 5, when the operation command is "move forward," "alternately stimulate the left and right forearm extensors" and "alternately stimulate the left and right leg joint extensors" are read out as the stimulation target area and stimulation pattern, and pulse-like stimulation intensity information is read out as the stimulation intensity, in which the intensity of the electrical stimulation to be applied to the left forearm extensor and right leg joint extensor and the intensity of the electrical stimulation to be applied to the right forearm extensor and left leg joint extensor are specified alternately in time series.

[0053] The electrical stimulation intensity is set to a maximum value in advance so that the user US can recognize the joint movement through the electrical stimulation and does not feel pain or discomfort.

[0054] Next, in step S19, the control unit 1 generates an electrical stimulation control signal that reflects the target location of electrical stimulation, its presentation pattern, and stimulation intensity in accordance with the stimulation presentation control information under the control of the avatar display control unit 15. Then, the control unit 1 outputs the generated electrical stimulation control signal from the input / output I / F unit 4 to the electrical stimulation generator EM.

[0055] As a result, an electrical stimulation signal is generated from the electrical stimulation generator EM and applied to electrodes E1 and E2 attached to the muscle area to be stimulated. Therefore, for example, electrical stimulation is alternately applied to the user US's "left forearm extensor muscle" and "right leg extensor muscle," and to the "right forearm extensor muscle" and "left leg extensor muscle." This causes the user US's left and right forearms and left and right legs to move alternately in synchronization with the aforementioned avatar's movements. This allows the user US to perceive the avatar's movements as the movements of their own arms and legs. In other words, the user US can recognize a sense of agency over the movements.

[0056] (4) Assistance Termination Control In step S20, the control unit 1 of the assistance device CS monitors whether an assistance termination request is input, for example, by a gaze operation of the user US. If an assistance termination request is not input, the process returns to step S11, and thereafter, steps S11 to S19 are performed to repeatedly execute the above-described series of assistance controls in real time. On the other hand, if an assistance termination request is input, the control unit 1 terminates the assistance control and returns to a standby state.

[0057] As described above, in one embodiment, the movement of the avatar is controlled in response to input operations of the avatar using the user's gaze, and electrical stimulation is applied to muscle parts of the user corresponding to the moving parts of the avatar in synchronization with the movement of the avatar. This allows the user US to perceive the movement of the avatar as the movement of the corresponding part of their own body, thereby enabling them to recognize a sense of agency over the movement.

[0058] In addition, since the electrical stimulation intensity is set according to the target part of the avatar's movement and its movement pattern, the user US can feel the avatar's movement more realistically than if the electrical stimulation intensity were fixed to a constant value.

[0059] Furthermore, the control table data collectively manages the avatar movement patterns corresponding to the operation commands and the stimulation control data, so that the avatar movement and electrical stimulation control for the user US can be efficiently performed through relatively simple processing.

[0060] [Other Embodiments] (1) In one embodiment, an example was described in which five types of operation commands were defined: “forward,” “stop,” “turn right,” “turn left,” and “raise hand.” However, other commands such as “backward” and “bow” may also be added.

[0061] (2) In one embodiment, the example has been described in which the user to be assisted is a physically disabled person with limited limbs. However, the present invention is not limited to this, and can also be applied to users with limited or impaired limb function. In this case, the input interface for operation commands may be button operation on an input device, cursor operation, or oral voice input, or may be a device that detects limb movements using, for example, an acceleration sensor or an electromyographic sensor and inputs the detection results.

[0062] (3) This invention is applicable not only to physically disabled people but also to able-bodied people. For example, if an able-bodied person participates in a virtual event held in the Metaverse space as an avatar, the able-bodied person will be able to directly sense the movements of their avatar as their own body movements, which will enable them to experience a greater sense of realism and excitement in the virtual event.

[0063] (4) In one embodiment, the avatar is a virtual object generated on a computer, and a stimulus is applied to a corresponding part of the operator in accordance with the movement of the avatar. However, the present invention is not limited to this. The avatar may be a real object, such as a human or a remote robot, and a stimulus may be applied to the body of the operator in accordance with the movement of the real object.

[0064] (5) As an application example, for example, if an avatar collides with an obstacle such as a door in the Metaverse space, electrical stimulation may be applied to the part of the user that collided based on information describing the event. This would further enhance the sense of realism in the Metaverse.

[0065] (6) A head-up display or the like may be used as a monitor device, and various modifications may be made to the method of determining operation commands from facial image data, the method of controlling the movement of an avatar, the method of controlling the presentation of stimuli, and the like, without departing from the spirit of this invention.

[0066] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0067] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0068] US...user to be assisted CS...motor agency recognition assistance device (assistance device) MT...monitor device CM...camera ES...electrical stimulation generator E1, E2...electrodes 1...control unit 2...program storage unit 3...data storage unit 4...input / output I / F unit 5...bus 11...image data acquisition processing unit 12...gaze direction estimation processing unit 13...operation command determination processing unit 14...avatar movement control information generation processing unit 15...avatar display control unit 16...stimulus presentation control information generation processing unit 17...electrical stimulation generation control unit 31...control table storage unit

Claims

1. An apparatus for supporting the recognition of a sense of agency in movement, connected to a display device capable of displaying an avatar, and a stimulus presentation device that presents stimuli to a user's muscle parts, comprising: a first processing unit that acquires operation information of the avatar input by the user; a second processing unit that determines an operation form of the avatar based on the operation information; a third processing unit that generates avatar movement control information corresponding to the operation form and controls the movement of the avatar displayed on the display device based on the generated avatar movement control information; and a fourth processing unit that generates stimulus control information corresponding to the operation form and controls the stimulus presentation device using the generated stimulus control information, thereby presenting stimuli in synchronization with the movement of the avatar to muscle parts of the user that correspond to the movement parts of the avatar.

2. The motor agency recognition support device of claim 1, further comprising a memory unit that stores the avatar movement control information and the stimulus control information in correspondence with the operation form, and the third processing unit and the fourth processing unit respectively acquire the avatar movement control information and the stimulus control information from the memory unit.

3. The motor agency recognition support device of claim 1, wherein the avatar movement control information includes information indicating the movement target part of the avatar and the movement content, and the third processing unit moves the movement target part of the avatar in accordance with the movement content based on the avatar movement control information.

4. The device for supporting recognition of a sense of agency in exercise as described in claim 1, wherein the stimulus control information includes information representing a target part of the stimulus to be presented and the content of the stimulus to be presented, and the fourth processing unit applies electrical stimulation indicated by the content of the presentation to the target part of the muscle part of the user based on the stimulus control information.

5. The device for supporting the recognition of a sense of agency in exercise as described in claim 4, wherein the fourth processing unit applies the electrical stimulation having an intensity corresponding to the movement speed of the avatar to the target muscle part of the user in synchronization with the movement period of the avatar in accordance with the information representing the presentation content.

6. A method for supporting recognition of a sense of agency in movement executed by an information processing device connected to a display device capable of displaying an avatar, and a stimulus presentation device that presents stimuli to a user's muscle parts, comprising the steps of: acquiring operation information of the avatar input by the user; determining an operation form of the avatar based on the operation information; generating avatar movement control information corresponding to the operation form and controlling the movement of the avatar displayed on the display device based on the generated avatar movement control information; and generating stimulus control information corresponding to the operation form and controlling the stimulus presentation device using the generated stimulus control information to present stimuli in synchronization with the movement of the avatar to muscle parts of the user that correspond to the movement parts of the avatar.

7. A program for causing a processor provided in an apparatus for supporting the recognition of a sense of agency in motion to execute at least one of the processes executed by the first to fourth processing units provided in the apparatus for supporting the recognition of a sense of agency in motion described in any one of claims 1 to 5.

Images