Avatar control device, method, and program

Abstract

One aspect of the present invention is configured to: acquire muscle action data on a plurality of muscle parts of a user when controlling the motion of an avatar in response to a user operation; extract a first feature amount from the acquired muscle action data; determine, on the basis of the extracted first feature amount, whether each of the plurality of muscle parts has acted voluntarily under the consciousness of the user; and control the motion of the avatar on the basis of the muscle action data on the muscle part judged to have acted voluntarily.

Description

Avatar control device, method and program

[0001] One aspect of the present invention relates to an avatar control device, method, and program for controlling the behavior of an avatar in, for example, a metaverse space.

[0002] Generally, operating devices such as a mouse, keyboard, or joystick are used to control an avatar in the Metaverse space. Meanwhile, a more intuitive operating method has been proposed that utilizes muscle activity of specific body parts of the user. For example, a method for determining an operation command based on muscle activity signals involves calculating the root mean square (RMS) of muscle activity over a certain time window, and then determining that an operation has been performed and transmitting an operation command when the signal strength exceeds a preset threshold (see, for example, Patent Document 1 or Non-Patent Document 1).

[0003] Japanese Patent Application Publication No. 2015-2939

[0004] Isamu Kajitani, "Development of Myoelectric Interfaces," Prosthetic Hand Discussion Group, Proceedings 2007 (2007): 33-36.

[0005] However, the method of determining operation commands based on a preset muscle activity threshold has the following two problems.

[0006] (1) When determining separate avatar operation commands from multiple muscle activities, interference between muscle activities can lead to erroneous determination of an operation command based on the activity of a muscle other than the muscle that the user moved voluntarily. For example, when determining an operation command based on the activity of two paired muscles, such as flexion and extension of the hand, the flexion movement can increase the activity of both muscles, which can lead to erroneous determination of the muscle that was moved voluntarily.

[0007] (2) It is possible to control whether or not an avatar performs a certain action, but it is not possible to control the speed, amount, etc. of the action. For example, when making an avatar wave its hand, it is possible to control whether or not it waves, but it is not possible to control how fast and how large the wave should be.

[0008] This invention has been made with the above-mentioned circumstances in mind, and in one aspect, aims to provide a technology that enables accurate determination of muscle areas that are voluntarily moved, thereby enabling accurate control of the movement expression of an avatar.

[0009] In order to solve the above problems, one aspect of the avatar control device or avatar control method of the present invention is to, when controlling the movement of an avatar in response to user operation, acquire muscle activity data of multiple muscle parts of the user, extract a first feature from the acquired muscle activity data, determine whether or not each of the multiple muscle parts moved voluntarily without the user's awareness based on the extracted first feature, and control the movement of the avatar based on the muscle activity data of the muscle parts determined to have moved voluntarily.

[0010] According to one aspect of the present invention, even if multiple muscle parts are activated when a user operates an avatar, it is possible to determine which muscle part will be activated at will from among them, thereby enabling the corresponding part of the avatar to be accurately activated in accordance with the user's operation.

[0011] That is, according to one aspect of the present invention, it is possible to provide a technology that enables accurate determination of muscle parts that are voluntarily moved, thereby enabling accurate control of the movement expression of an avatar.

[0012] Fig. 1 is a diagram showing an example of the configuration of a system including an avatar control device according to an embodiment of the present invention. Fig. 2 is a block diagram showing an example of the hardware configuration of an avatar control device according to an embodiment of the present invention. Fig. 3 is a block diagram showing an example of the software configuration of an avatar control device according to an embodiment of the present invention. Fig. 4 is a flowchart showing an example of the processing procedure and processing content of avatar control processing executed by a control unit of the avatar control device shown in Fig. 3.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] [One Embodiment] (Configuration Example) (1) System FIG. 1 is a diagram showing an example of the configuration of a system including an avatar control device according to one embodiment of the present invention.

[0015] The system according to one embodiment includes, as its core component, an avatar control device CS that controls the behavior of an avatar created as an avatar of a user US in the metaverse space in response to operations by the user US.

[0016] To control the movements of the avatar, electromyographic sensors ES1 to ESi are attached to multiple muscle regions used by the user US to operate the avatar. The electromyographic sensors ES1 to ESi measure, on a time axis, action potentials generated when muscle fibers contract, and output the measured electromyographic signals to the avatar control device CS via, for example, a signal cable.

[0017] In addition, the collection ES of electromyogram sensors ES1 to ESi may be made into a unit, and the electromyogram signals measured by the electromyogram sensors ES1 to ESi may be output collectively to the avatar control device CS by a transmitter provided in the unit.

[0018] The avatar control system according to one embodiment also includes a monitor device MT. The monitor device MT is connected to the avatar control device CS via a signal cable and displays image data of the metaverse space output from the avatar control device CS.

[0019] (2) Avatar Control Device CS The avatar control device CS is, for example, a general-purpose personal computer, and is capable of data communication with a server computer on the web or cloud, or with other terminals, via a network (not shown).

[0020] 2 and 3 are block diagrams showing an example of the hardware and software configurations of the avatar control device CS, respectively.

[0021] The avatar control device CS includes a control unit 1 that uses a hardware processor such as a central processing unit (CPU). The avatar control device CS is connected to the control unit 1 via a bus 5, and is also connected to a storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 4.

[0022] The input / output I / F unit 4 has a function of receiving electromyogram signals output from the electromyogram sensors ES1 to ESi, and a function of outputting avatar display data to the monitor device MT.

[0023] The input / output I / F unit 4 may be provided with a communication interface function, which enables signals to be transmitted and received between the myoelectric sensors ES1 to ESi and the monitor device MT using a low-power wireless communication interface such as Bluetooth (registered trademark), thereby increasing the degree of freedom in the operation of the user US and the degree of freedom in the installation of the avatar control device CS and the monitor device MT.

[0024] The program storage unit 2 is configured by combining, for example, a nonvolatile memory such as a solid-state drive (SSD) as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a read-only memory (ROM), and stores middleware such as an operating system (OS), as well as application programs required to execute various controls according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0025] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), and the storage area thereof is provided with a muscle activity data storage unit 31, a judgment threshold storage unit 32, a voluntary movement muscle judgment model storage unit 33, and an avatar movement information storage unit 34, as storage units according to one embodiment of the present invention.

[0026] The muscle activity data storage unit 31 is used to store muscle activity data of multiple muscle sites used for operating the avatar of the user US.

[0027] The determination threshold memory unit 32 stores a threshold for determining the presence or absence of muscle activity for each of the plurality of muscle sites. The threshold is set in advance based on the measured values ​​of muscle activity measured in advance when the user US exerts force on each muscle site.

[0028] The voluntary action muscle determination model storage unit 33 stores a voluntary action muscle determination model that determines muscle sites that are voluntarily moved by the user US's avatar operation. A machine learning model is used for the voluntary action muscle determination model. The voluntary action muscle determination model is trained in advance so that, when a root mean square (RMS) value indicating the feature amount of muscle activity of multiple muscle sites is input, the model outputs determination information indicating whether each muscle site is voluntarily moved. The learning algorithm may be, for example, a supervised learning algorithm such as a support vector machine (SVM), but may also be a regression model or various neural networks.

[0029] The avatar movement information storage unit 34 stores avatar movement information in association with movement parameters proportional to the magnitude of muscle activity for each muscle part. The avatar movement information includes control parameters such as the magnitude and speed of the avatar's movement.

[0030] The control unit 1 includes, as processing functions necessary for implementing one embodiment, an electromyogram signal acquisition processing unit 11, a muscle activity measurement processing unit 12, a muscle activity analysis processing unit 13, a muscle activity determination processing unit 14, a voluntary movement muscle determination processing unit 15, an avatar movement parameter calculation processing unit 16, an avatar movement control processing unit 17, and an avatar display control processing unit 18.

[0031] Each of the processing units 11 to 18 is realized by causing a hardware processor in the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 11 to 18 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0032] The electromyogram signal acquisition processing unit 11 receives the electromyogram signals output from the electromyogram sensors ES1 to ESi via the input / output I / F unit 4, and converts the received electromyogram signals into digital signals at a predetermined sampling period.

[0033] The muscle activity measurement processing unit 12 generates muscle activity data representing the measurement results of muscle activity at i muscle locations for each time t based on the digitized electromyogram signal, and stores the generated muscle activity data in the muscle activity data storage unit 31 as a matrix e with t rows and i columns.

[0034] The muscle activity analysis processing unit 13 calculates the root mean square (RMS) of each muscle activity for each time window of a predetermined time T for the muscle activity data stored as a matrix e, and represents the calculation result as vector data r consisting of i elements. At the same time, the muscle activity analysis processing unit 13 also performs frequency analysis on each muscle activity for each time window T to calculate the gain in the low-frequency component, and represents the calculation result as vector data g consisting of i elements.

[0035] The muscle activity determination processing unit 14 compares each of the i RMS values ​​of the vector data r with a threshold value stored in the determination threshold memory unit 32 to determine whether the corresponding i muscle sites are undergoing muscle activity.

[0036] The voluntary movement muscle determination processing unit 15 targets the muscle parts determined to be in muscle movement by the muscle activity determination processing unit 14, inputs the RMS values ​​as explanatory variables into the voluntary movement muscle determination model, and obtains the determination result of whether or not voluntary movement is occurring, which is output as a target variable from the voluntary movement muscle determination model.

[0037] The avatar movement parameter calculation processing unit 16 targets the muscle parts that are determined to be moving voluntarily by the voluntary movement muscle determination processing unit 15, and calculates avatar movement parameters corresponding to the gain based on the vector data g obtained by the muscle activity analysis processing unit 13.

[0038] Avatar movement control processing unit 17 reads out avatar movement information corresponding to the voluntarily moving muscle part from the avatar movement information storage unit, and determines at least one of the magnitude (movement amount) and movement speed of the avatar movement for the avatar movement information, based on the avatar movement parameters calculated by avatar movement parameter calculation processing unit 16.

[0039] The avatar display control processing unit 18 generates avatar display data for muscle areas determined to be in muscle movement, which reflects at least one of the magnitude and speed of the avatar movement determined by the avatar movement control processing unit 17, and outputs the generated avatar display data from the input / output I / F unit 4 to the monitor device MT.

[0040] (Example of Operation) Next, an example of operation of the avatar control device CS configured as above will be described. Fig. 4 is a flowchart showing an example of the procedure and content of the avatar control process executed by the control unit 1 of the avatar control device CS.

[0041] (1) Learning of Voluntary Movement Muscle Determination Model The avatar control device CS learns the voluntary movement muscle determination model prior to system operation. In the learning process, for example, muscle activity of all muscle sites is measured when each muscle site is voluntarily moved, and the acquired muscle activity measurement values ​​are used as training data. Then, when vector data r of RMS values ​​consisting of i elements for each time window T is input, learning is performed so that a determination matrix consisting of i elements representing the results of determining whether muscle site i was moved voluntarily is output, thereby creating a voluntary movement muscle determination model. Data representing the learned voluntary movement muscle determination model is stored in the voluntary movement muscle determination model storage unit 33.

[0042] (2) Avatar Motion Control (2-1) Measurement of Muscle Activity When the user US starts operating the avatar and this is detected in step S10, the control unit 1 of the avatar control device CS first receives, in step S11, electromyogram signals continuously output from each of the electromyogram sensors ES1 to ESi via the input / output I / F unit 4 under the control of the electromyogram signal acquisition processing unit 11. At this time, the electromyogram signal acquisition processing unit 11 converts the received electromyogram signals into digital signals at a fixed sampling period. Note that if the electromyogram sensors ES1 to ESi have the function of transmitting digitized electromyogram signals, the above-mentioned digitization process is not necessary.

[0043] When the control unit 1 of the avatar control device CS acquires the electromyogram signal, in step S12, the muscle activity measurement processing unit 12 generates data representing muscle activity of an i-th muscle site for each time t based on the electromyogram signal. The muscle activity measurement processing unit 12 stores the generated muscle activity data in the muscle activity data storage unit 31 as a matrix e with t rows and i columns.

[0044] (2-2) Analysis of Muscle Activity While performing the muscle activity measurement process for each time t, the control unit 1 of the avatar control device CS determines in step S12 whether a preset time window T has elapsed.

[0045] Then, after a time corresponding to the time window T has elapsed, the control unit 1, under the control of the muscle activity analysis processing unit 13, first extracts a first feature quantity related to the muscle activity in step S14. Specifically, the muscle activity analysis processing unit 13 reads each muscle activity matrix e with t rows and i columns obtained during the time window T from the muscle activity data storage unit 31. Then, the muscle activity analysis processing unit 13 calculates the root mean square (RMS) of the muscle activity for each of the i muscle sites from each of the read muscle activity matrices e, and generates vector data r consisting of i elements based on the calculated RMS values. Note that the first feature quantity may be a value other than the RMS value as long as it represents the magnitude of muscle activity during the time window T.

[0046] At the same time, in step S15, the muscle activity analysis processor 13 extracts a second feature related to the muscle activity. Specifically, the muscle activity analysis processor 13 performs frequency analysis on the muscle activity of each of the i muscle sites based on the muscle activity matrix e with t rows and i columns. The processor then calculates the gain of a predetermined low-frequency component, for example, the average value of the gain in a predetermined low-frequency band, and generates vector data g consisting of i elements based on the calculated average gain. The bandwidth of the low-frequency band is set so that effective gain features can be extracted by acquiring muscle activity data in advance and performing frequency analysis on it.

[0047] (2-3) Determining the Presence or Absence of Muscle Activity Next, in step S16, under the control of the muscle activity determination processing unit 14, the control unit 1 of the avatar control device CS compares the i RMS values ​​constituting the vector data r with threshold values ​​th(i) pre-stored for each of the i muscle parts in the determination threshold memory unit 32, and determines the muscle parts whose RMS values ​​are equal to or greater than the threshold value th(i).

[0048] The muscle activity determination processor 14 then outputs the determination results as information indicating the presence or absence of muscle activity for each muscle site. For example, if the RSM value r(i) is greater than th(i), the processor outputs r(i) = 1, and if r(i) is less than th(i), the processor outputs r(i) = 0.

[0049] (2-4) Determination of Voluntarily Acting Muscles Subsequently, in step S17, under the control of the voluntary action muscle determination processing unit 15, the control unit 1 of the avatar control device CS determines whether or not the muscle site determined by the muscle activity determination processing unit 14 to be active, i.e., the muscle site determined to be r(i) = 1, is a muscle site that has been voluntarily acted under the consciousness of the user US.

[0050] For example, the voluntary action muscle determination processor 15 inputs, as an explanatory variable, vector data r whose elements are i RSM values ​​in the time window T generated by the muscle activity analysis processor 13, into the trained voluntary action muscle determination model stored in the voluntary action muscle determination model storage unit 33. Then, a determination matrix consisting of i elements indicating whether or not i muscle sites have been moved voluntarily is obtained, which is output as a dependent variable from the trained voluntary action muscle determination model. Specifically, the trained voluntary action muscle determination model outputs a determination matrix in which muscle sites determined to have moved voluntarily are represented as "1" and muscle sites determined not to have moved voluntarily are represented as "0."

[0051] The voluntary movement muscle determination processing unit 15, according to the determination matrix, holds r(i) = 1 for elements of the vector data r that correspond to muscle parts determined to have moved voluntarily, and corrects elements that correspond to muscle parts determined not to have moved voluntarily to r(i) = 0.

[0052] (2-5) Calculation of Avatar Movement Parameters Next, in step S18, under the control of the avatar movement parameter calculation processing unit 16, the control unit 1 of the avatar control device CS selects the muscle parts that have been determined to be moving voluntarily by the voluntary movement muscle determination processing unit 15, that is, the muscle parts determined to have r(i) = 1 in the vector data r.

[0053] Then, avatar movement parameter calculation processing unit 16 calculates movement parameter A(i) corresponding to the low-frequency gain g(i) of the muscle part that is moving voluntarily, based on the low-frequency gain vector data g generated by muscle activity analysis processing unit 13. As a calculation formula, for example, a formula that increases proportionally with the value of low-frequency gain g(i) is used.

[0054] The avatar movement parameter calculation processor 16 sets the movement parameter A(i) = 0 for muscle parts (r(i) = 0) that are determined not to be moving voluntarily by the voluntary movement muscle determination processor 15.

[0055] (2-6) Generation of Avatar Movement Information Next, in step S19, under the control of the avatar movement control processing unit 17, the control unit 1 of the avatar control device CS targets muscle parts other than those calculated as A(i) = 0 by the avatar movement parameter calculation processing unit 16, i.e., muscle parts that are moving voluntarily, and reads avatar movement information corresponding to these muscle parts from the avatar movement information storage unit 34.

[0056] Then, avatar movement control processor 17 determines avatar control parameters representing the magnitude (movement amount) and speed of the avatar movement based on the values ​​of movement parameters A(i) for the read avatar movement information. Note that when determining the avatar control parameters, both the magnitude and speed of the avatar movement may be determined, or either one of them may be determined.

[0057] (2-7) Avatar display control Then, in step S20, the control unit 1 of the avatar control device CS, under the control of the avatar display control processing unit 18, generates avatar display data based on the avatar movement information output from the avatar movement control processing unit 17, and outputs the generated avatar display data from the input / output I / F unit 4 to the monitor device MT.

[0058] As a result, a moving image of the avatar is displayed on the monitor device MT. At this time, the avatar moves at the parts corresponding to the muscle parts voluntarily moved by the user US, and the movement reflects the magnitude and speed of the movement of the voluntarily moved muscle parts.

[0059] (3) Control of Ending Avatar Operation The control unit 1 of the avatar control device CS determines in step S21 whether the avatar operation has ended each time a series of controls for the avatar movement in the time window T ends. If the result of this determination is that the avatar operation is continuing, the control unit 1 returns to step S11 and repeats the series of controls in steps S11 to S20. On the other hand, when the avatar operation of the user US ends, the control unit 1 ends the avatar control and returns to a standby state.

[0060] (Effects) As described above, in one embodiment, muscle activity data for i muscle sites used to operate the avatar of the user US is acquired at each time t based on the electromyogram signals output from the electromyogram sensors ES1 to ESi. Based on the acquired muscle activity data, vector data r is generated, with the RSM values ​​of the i muscle sites as elements, and vector data g is generated, with the low-frequency gains of the i muscle sites as elements. Then, muscle sites that are active are determined based on the vector data r, and a voluntary action muscle determination model is used to determine whether the active muscle sites are voluntarily moved without the user's consent. Furthermore, avatar movement parameters corresponding to the voluntarily moving muscle sites are calculated, avatar movement information is generated based on the parameters, and the magnitude and speed of the avatar movement in the avatar movement information are determined based on the low-frequency gain represented by the vector data g. Avatar display data is generated based on the avatar movement information and displayed on the monitor device MT.

[0061] Therefore, even if multiple muscle parts are activated when the user US operates the avatar, it is possible to determine which muscle part will be activated at will, thereby enabling the corresponding part of the avatar to be accurately activated in accordance with the user US's operation.

[0062] In addition, the magnitude and speed of the avatar movement are controlled based on the low-frequency gain value that represents the muscle activity of the muscle parts when the user US operates the avatar, making it possible to accurately reflect the user US's operation in the avatar's movement expression.

[0063] Other Embodiments (1) In one embodiment, the muscle activity data for each muscle region is frequency-analyzed to determine a low-frequency band gain value, and the magnitude and speed of the avatar's movement are controlled based on this low-frequency band gain value. However, the magnitude and speed of the avatar's movement may also be controlled based on the RSM value of the muscle activity determined for each muscle region. In this case, avatar movement parameter calculation processing unit 16 calculates movement parameters that increase proportionally with the magnitude of the RSM value, and determines the avatar movement parameters based on the calculated movement parameters.

[0064] (2) In one embodiment, the muscle activity determination threshold is set based on the measured value of muscle activity measured when the user US actually exerts force on a muscle site. However, this is not limiting, and for example, the average amplitude of muscle activity when the avatar is not being operated, i.e., when the user is at rest, may be measured, and a value three times the measured value may be set as the threshold.

[0065] (3) In one embodiment, a machine learning model is used as a voluntary movement muscle determination model. However, the present invention is not limited to the above example. For example, a determination threshold for the activity of a muscle site other than the muscle site voluntarily moved by the user US may be set, and the measured value of the muscle activity for each muscle site may be determined based on the determination threshold for whether the muscle site is a voluntary movement.

[0066] (4) In one embodiment, the avatar is a virtual object generated on a computer, and the movement of the avatar is controlled on a moving image in response to the operation of the user US. However, the present invention is not limited to this. For example, the avatar may be a real object such as a remote robot, and the movement of the real object may be controlled by the user US using gestures at a management center.

[0067] (5) Furthermore, a head-mounted display (HMD) or the like may be used as a monitor device, and various modifications may be made to the type and configuration of the electromyography sensor, the method of analyzing muscle activity, the method of determining voluntary action muscles, the method of measuring the magnitude and speed of movement of muscle parts, and the method of controlling the magnitude and speed of avatar movement, all of which are within the scope of the present invention.

[0068] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0069] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0070] US...User CS...Avatar control device MT...Monitor device ES, ES1 to ESi...Electromyographic sensors E1, E2...Electrodes 1...Control unit 2...Program memory unit 3...Data memory unit 4...Input / output I / F unit 5...Bus 11...Electromyogram signal acquisition processing unit 12...Muscle activity measurement processing unit 13...Muscle activity analysis processing unit 14...Muscle activity determination processing unit 15...Voluntary movement muscle determination processing unit 16...Avatar movement parameter calculation processing unit 17...Avatar movement control processing unit 18...Avatar display control processing unit 31...Muscle activity data storage unit 32...Determination threshold storage unit 33...Voluntary movement muscle determination model storage unit 34...Avatar movement information storage unit

Claims

1. An avatar control device that controls the movement of an avatar in response to user operation, comprising: a first processing unit that acquires muscle activity data of multiple muscle parts of the user; a second processing unit that extracts a first feature from the muscle activity data and determines whether or not the user consciously moved each of the multiple muscle parts based on the extracted first feature; and a third processing unit that controls the movement of the avatar based on the muscle activity data of the muscle parts determined to have moved voluntarily.

2. An avatar control device as described in claim 1, further comprising a fourth processing unit that identifies a muscle activity site among the plurality of muscle sites where muscle activity has occurred based on the first characteristic amount, and the second processing unit determines whether or not the voluntary movement has been performed for the identified muscle activity site.

3. The avatar control device according to claim 1, further comprising a fifth processing unit that extracts a second feature from the muscle activity data, and the third processing unit controls at least one of the movement amount and movement speed of the avatar based on the second feature.

4. The avatar control device of claim 1, wherein the second processing unit is equipped with a machine learning model trained to output, as a target variable, judgment information representing the result of judging whether or not the voluntary movement was performed for each of the plurality of muscle parts when the first feature is input as an explanatory variable.

5. An avatar control method for controlling the movement of an avatar in response to a user's operation using an information processing device, comprising the steps of: acquiring muscle activity data of a plurality of muscle parts of the user; extracting a first feature from the muscle activity data, and determining, based on the extracted first feature, whether or not each of the plurality of muscle parts moved voluntarily under the user's consciousness; and controlling the movement of the avatar based on the muscle activity data of the muscle parts determined to have moved voluntarily.

6. A program for causing a processor provided in an avatar control device to execute at least one of the processes executed by the first through fifth processing units provided in an avatar control device according to any one of claims 1 through 4.

Images