Avatar generation device, avatar generation method, and program
The avatar generation device synchronizes avatars with user movements to facilitate communication between sign language and voice users by prioritizing sign language gestures during speech and linked motions during silence, addressing the limitations of existing systems.
Patent Information
- Application Number
- JP2023079968
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-27
AI Technical Summary
Existing communication systems fail to enable seamless interaction between sign language users and voice users, as sign language users cannot see the movements of able-bodied individuals other than sign language gestures, necessitating the latter to learn sign language or act as interpreters.
An avatar generation device that uses audio and motion detection to synchronize avatars with user movements, allowing sign language users to communicate through synchronized sign language gestures and linked motions, while prioritizing sign language actions during speech and linked actions during silence.
Enables effective communication between sign language and voice users by allowing the latter to understand sign language through synchronized avatars, reducing the need for interpreters and enabling seamless interaction in virtual spaces.
Smart Images

Figure 2024164474000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an avatar generating device, an avatar generating method, and a program. [Background technology]
[0002] Patent Document 1 discloses an information processing device for communication between a voice user and a sign language user. This information processing device senses a message expressed by the voice user using voice language with a microphone. The information processing device converts the message into a video of an avatar making a gesture in sign language corresponding to the message. Furthermore, the information processing device converts the sign language actions of the sign language user into sound and outputs it to the voice user.
[0003] Patent Document 2 discloses a communication system that displays an avatar of an operator. In this communication system, lip-sync processing is performed to move the lips of the avatar in synchronization with response voice data. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2019-8570 A [Patent Document 2] Patent Publication No. 2021-56940 [Non-patent literature]
[0005] [Non-Patent Document 1] https: / / qiita.com / yukihiko_a / items / 43d09db5628334789fab [Non-Patent Document 2] https: / / www.moguravr.com / full-tracking-tool-matome / Summary of the Invention [Problem to be solved by the invention]
[0006] When a able-bodied person (also called a natural language user or a speech user) communicates with a sign language user, the able-bodied person needs to learn sign language. Alternatively, a able-bodied person who has learned sign language needs to act as an interpreter. In Patent Document 1, gestures in sign language corresponding to a message are converted into a video of an avatar. However, in Patent Document 1, the avatar only performs sign language gestures, which causes a problem in that the sign language user cannot see any other actions of the able-bodied person than the sign language.
[0007] The present disclosure has been made in consideration of the above-mentioned points, and has an object to provide an avatar generation device, an avatar generation method, and a program that enable sign language users and speech language users to communicate appropriately. [Means for solving the problem]
[0008] The avatar generation device according to the present embodiment includes an audio signal acquisition unit that collects speech of a first user with a microphone and acquires an audio signal, a first movement detection unit that detects a user movement of the first user using a first sensor, a linked movement generation unit that generates a linked movement of the first avatar so that the first avatar moves in linkage with the user movement detected by the first movement detection unit, a text conversion unit that converts the audio signal into text data, a sign language movement generation unit that converts the text data into sign language and generates a sign language movement of the first avatar, and an action management unit that causes the linked movement generation unit to perform the linked movement other than when the first user is speaking, and stops the linked movement generation unit while the first user is speaking, and causes the first avatar to perform the sign language movement.
[0009] The avatar generating method according to the present embodiment includes the steps of: collecting speech of a first user with a microphone to obtain an audio signal; detecting a user movement of the first user using a first sensor; generating a linked movement of the first avatar such that the first avatar moves in conjunction with the user movement; converting the audio signal into text data; converting the text data into sign language to generate a sign language movement of the first avatar; causing the linked movement generating unit to perform the linked movement except when the first user is speaking; and stopping the linked movement while the first user is speaking and causing the first avatar to perform the sign language movement.
[0010] The program according to the present embodiment is a program for causing a computer to execute an avatar generation method, the avatar generation method including the steps of: collecting speech of a first user with a microphone and acquiring an audio signal; detecting a user movement of the first user using a first sensor; generating a linked movement of the first avatar so that the first avatar moves in linkage with the user movement; converting the audio signal into text data; converting the text data into sign language to generate a sign language movement of the first avatar; causing the linked movement generation unit to perform the linked movement except when the first user is speaking; and stopping the linked movement while the first user is speaking and causing the first avatar to perform the sign language movement. Effect of the Invention
[0011] An object of the present disclosure is to provide an avatar generation device, an avatar generation method, and a program that enable a sign language user and a voice language user to communicate appropriately. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a diagram illustrating an overall configuration of an avatar generation device. [Diagram 2]FIG. 2 is a control block diagram showing a system configuration of the avatar generation device. [Diagram 3] FIG. 2 is a control block diagram showing a system configuration of the avatar generation device. [Figure 4] 1 is a flowchart showing a method for generating an avatar. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In addition, the following description and drawings are appropriately simplified for clarity of explanation.
[0014] FIG. 1 is a schematic diagram showing a system configuration of an avatar generation device according to the present embodiment. The avatar generation device 100 generates a first avatar A1 and a second avatar A2 for communication between a first user U1 and a second user U2. The avatar generation device 100 displays the first avatar A1 and the second avatar A2. As shown in FIG. 1, the first avatar A1 is an avatar of the first user U1, and the second avatar A2 is an avatar of the second user U2. The first user U1 is, for example, a voice user (natural language user), and the second user is a sign language user. For example, the first user U1 is assumed to be a normal person who cannot understand sign language. The second user U2 is assumed to be a hearing-impaired person who cannot hear voices.
[0015] By using the avatar generation device 100, a first user U1 and a second user U2 can communicate with each other through avatars A1 and A2. That is, in a virtual space, a first user U1 who uses natural language and a second user U2 who uses sign language converse with each other through avatars A1 and A2, respectively. The avatars A1 and A2 may be images of the users themselves, or may be animated images generated from the images of the users themselves. Also, the avatars A1 and A2 may be animated images selected regardless of the images of the users themselves.
[0016] The first avatar A1 moves in conjunction with the movement of the first user U1. For example, when the first user U1 makes a movement of raising his right hand, the avatar generation device 100 generates and displays data so that the first avatar A1 raises his right hand. This causes the movement of the first avatar A1 to move in conjunction with the movement of the first user U1. That is, the first avatar A1 moves in a manner that imitates the movement of the first user U1. Similarly, the second avatar A2 moves in conjunction with the movement of the second user U2. Note that, in the following description, an example in which the first user U1 and the second user U2 have a conversation will be described, but three or more people may have a conversation. That is, users other than the first user U1 and the second user U2 may participate in the conversation.
[0017] The avatar generation device 100 is a communication terminal such as a personal computer, a smartphone, or a tablet PC. The avatar generation device 100 may be a wearable device such as a head mounted display, a smart watch, or smart glasses. The avatar generation device 100 may be a device that displays the avatars A1 and A2 in AR (Augmented Reality), VR (Virtual Reality), or MR (Mixed Reality).
[0018] The first user U1 and the second user U2 may be in a remote location or in the same location. When the first user U1 and the second user U2 are in a remote location, each of them has a user terminal that serves as the avatar generation device 100. The two avatar generation devices 100 can communicate with each other wirelessly or via a wired connection. For example, the avatar generation device 100 is connected to a network by a wireless LAN (Local Area Network) device such as WiFi (registered trademark). The avatar generation device 100 can receive data from the network and can transmit data to the network. A publicly known communication protocol can be used for the network, so a description thereof will be omitted.
[0019] When the first user U1 and the second user U2 are in remote locations, the avatar generation device 100 for the first user U1 displays the second avatar A2 on a display. At this time, the avatar generation device 100 for the first user may display the second avatar A2 by superimposing it on the image of the first user U1 or the first avatar A1 of the first user U1. Of course, the avatar generation device 100 for the first user may display the second avatar A2 alone without displaying the first user U1 or the first avatar A1.
[0020] Similarly, the avatar generation device 100 of the second user U2 displays the first avatar A1 on the display. At this time, the avatar generation device 100 of the second user U2 may display the first avatar A1 by superimposing it on the image of the second user U2 or the second avatar A2. Of course, the avatar generation device 100 of the second user may display the first avatar A1 alone without displaying the second user U2 or the second avatar A2.
[0021] When the first user U1 and the second user U2 are in the same place, the avatar generation device 100 may be common to the first user U1 and the second user U2. For example, when a display device such as a large screen or monitor is present in the same room, the avatar generation device 100 may display the first avatar A1 and the second avatar A2 on the display device. Of course, even when the first user U1 and the second user U2 are in the same place, the first user U1 and the second user U2 may use their own avatar generation device 100. In other words, the avatar generation device 100 of the first user U1 and the avatar generation device 100 of the second user U2 may be a single physical device or different devices.
[0022] 2 is a control block diagram showing a system configuration including an avatar generation device 100. The system includes an avatar generation device 100, an avatar generation device 100A, and a server 300. The avatar generation device 100 is a user terminal used by a first user U1, and the avatar generation device 100A is a user terminal used by a second user U2. In the following explanation, it is assumed that the first user U1 and the second user U2 are in remote locations and communicate with each other via the server 300.
[0023] A first user U1 accesses a server 300 that provides a virtual space via the avatar generation device 100. The server 300 stores data corresponding to the virtual space in which the avatars A1 and A2 operate. The server 300 transmits data of the virtual space in which the avatars A1 and A2 operate to the avatar generation devices 100 and 100A.
[0024] The avatar generation device 100 and the avatar generation device 100A transmit and receive motion data, voice data, etc. The avatar generation device 100 may transmit and receive various data directly from the avatar generation device 100A, or may transmit and receive data via the server 300 or the like. Here, the description will be given assuming that the avatar generation device 100 transmits and receives data from the avatar generation device 100A via the server 300.
[0025] The avatar generation device 100 includes a movement detection unit 110, a voice input / output unit 120, an avatar processing unit 130, a storage unit 140, and a display unit 150. The movement detection unit 110 includes a camera 111 and a movement acquisition unit 112. The voice input / output unit 120 includes a microphone 121, a voice signal acquisition unit 122, and a speaker 123. The avatar processing unit 130 includes a movement information extraction unit 131, a sign language determination unit 132, a natural language conversion unit 133, a text conversion unit 134, a sign language movement generation unit 135, an avatar movement management unit 136, and a linked movement generation unit 137.
[0026] (Process of converting speech into sign language actions) The avatar generation device 100 converts the speech of the first user U1 into a sign language action. Then, the first avatar A1 performs the sign language action. The second user U2, who is a sign language user, can understand the speech of the first user U1 by visually recognizing the first avatar A1. The process in which the avatar generation device 100 converts the speech of the first user U1 into a sign language action will be described below.
[0027] The microphone 121 is a voice sensor that collects the speech of the first user U1. The microphone 121 may be built into a PC (Personal Computer) or a smartphone, or may be provided separately, such as a USB (registered trademark) microphone or a Bluetooth (registered trademark) microphone. The microphone 121 outputs the collected voice signal to the voice signal acquisition unit 122. When the voice signal acquisition unit 122 acquires the voice signal collected by the microphone 121, the voice signal acquisition unit 122 generates voice data by performing A / D conversion or the like on the voice signal. The voice signal acquisition unit 122 outputs the voice data to the text conversion unit 134.
[0028] The text conversion unit 134 converts the voice data into text data. The text conversion unit 134 can use a transcription program that performs voice analysis and transcribes the voice data into text.
[0029] The sign language action generation unit 135 generates a sign language action from the text data. The sign language action generation unit 135 converts the speech content of the first user U1 into a sign language and generates a sign language action. The sign language action generation unit 135 generates the sign language action using a sign language generation program.
[0030] For example, the storage unit 140 stores a conversion database for converting text data into sign language actions. The storage unit 140 stores sign language data such as finger alphabets and words that are elements of sign language actions. The storage unit 140 stores sign language actions in association with finger alphabets, words, etc. The sign language action generation unit 135 converts the text data into sign language actions by referring to the conversion database stored in the storage unit 140. The sign language action generation unit 135 outputs the sign language actions to the avatar action management unit 136.
[0031] The motion detection unit 110 detects the motion of the first user U1 using the camera 111. The camera 111 is an optical sensor such as a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor. The camera 111 captures an image of the first user U1. Here, the camera 111 captures moving images. The motion acquisition unit 112 acquires the motion of the first user U1 from the image captured by the camera 111. For example, the motion acquisition unit 112 acquires the motion of each part of the first user U1, such as the arm, finger, head, etc., by performing image analysis on the moving images.
[0032] Specifically, the motion acquisition unit 112 identifies and tracks the parts of the first user U1 in the image by image analysis. The motion acquisition unit 112 calculates the moving parts, the motion direction, the motion angle, the motion amount, and the like by sequentially comparing the frame images. When the first user U1 moves an arm, a finger, an eye, a mouth, a head, a foot, a leg, or the like, the motion detection unit 110 detects the motion. Note that the sensor used by the motion detection unit 110 is not limited to the camera 111, and may be another sensor such as a motion capture sensor. Furthermore, the motion detection unit 110 may use both the camera 111 and the motion capture sensor.
[0033] For example, Non-Patent Document 1 discloses tracking software that tracks faces and forward movement from video. Non-Patent Document 2 discloses technology for attaching multiple trackers to the body for tracking. The sensor unit of each track generates motion information such as the position, speed, and acceleration of each part of the body. The system then generates a VR of the entire body based on the motion information.
[0034] The linked action generating unit 137 generates a linked action for moving the first avatar A1 in conjunction with the action of the first user U1. The linked action may be data including a part to be moved, a direction of the action, an angle of the action, an amount of the action, and the like. For example, when the first user U1 raises his / her right arm, the action of raising the right arm of the first avatar A1 becomes the linked action. Of course, when the first user U1 moves not only his / her arm but also his / her fingers, eyes, mouth, head, feet, legs, and the like, the same part of the first avatar A1 is moved in the same manner. In this way, the linked action generating unit 137 generates action data for the first avatar A1 to perform a linked action that imitates the action of the first user U1. The linked action generating unit 137 outputs the linked action to the avatar action managing unit 136.
[0035] The avatar action management unit 136 manages the avatar action of the first avatar A1 based on the linked action and the sign language action. The avatar action management unit 136 controls the first avatar A1 so that the first avatar A1 selectively performs either the sign language action or the linked action. That is, the avatar action management unit 136 selects either the sign language action or the linked action and transmits it to the server 300 as the avatar action. When the first user U1 is speaking, the avatar action management unit 136 stops synchronizing the linked action by the linked action generation unit 137 in order to select the sign language action. When the first user U1 is not speaking, the avatar action management unit 136 selects the linked action by the linked action generation unit 137.
[0036] The avatar action management unit 136 causes the first avatar A1 to perform the linked action by the linked action generation unit 137 when the first user U1 is not speaking. When the first user U1 is not speaking, the avatar action management unit 136 synchronizes the linked action by the linked action generation unit 137. This causes the first avatar A1 to perform the linked action. On the other hand, when the first user U1 is speaking, the avatar action management unit 136 stops the linked action by the linked action generation unit 137 and causes the first avatar to perform a sign language action. In other words, during speaking, the avatar action management unit 136 manages the action of the first avatar A1 so that the sign language action is given priority. The avatar action management unit 136 may determine whether the first user U1 is speaking based on a sound signal picked up by the microphone 121.
[0037] The avatar action management unit 136 transmits the sign language action or the linked action as the avatar action to the server 300. The server 300 transmits the sign language action or the linked action to the avatar generation device 100A of the second user U2. The avatar generation device 100A includes a display unit 150A (see FIG. 3). The display unit 150A displays the first avatar A1 performing the sign language action or the linked action. The display unit 150A displays a moving image of the first avatar A1 performing the sign language action. This allows the second user U2 to visually recognize the first avatar A1. By visually recognizing the first avatar A1 performing the sign language action, the second user U2 can understand the content of the utterance of the first user U1.
[0038] The display unit 150 of the avatar generation device 100 may display the first avatar A1. The first user U1 can visually recognize the first avatar A1 performing a sign language action or a linked action. As described later, the display unit 150 may or may not display a second avatar A2 linked to the action of the second user U2.
[0039] At least a part of the processing of the avatar generation device 100 may be executed by the server 300 or the avatar generation device 100A. For example, the server 300 or the avatar generation device 100A may convert the sign language action of the second user U2 into text data and transmit it to the avatar generation device 100.
[0040] In this way, when the first user U1 is speaking, the avatar action management unit 136 stops synchronization of the first avatar A1 with the linked action corresponding to the user action of the first user U1. The first avatar A1 stops synchronization of the linked action and performs a sign language action. In other words, the avatar action management unit 136 manages the action of the first avatar A1 so that the first avatar A1 performs a sign language action while the first user U1 is speaking. The first avatar A1 performs a sign language action in synchronization with the speech of the first user U1. This allows the second user U2 to understand the content of the speech of the first user U1 without stress.
[0041] When the first user U1 is not speaking, the avatar action management unit 136 manages the action of the first avatar A1 so that the first avatar A1 performs a linked action. This allows the second user U2 to visually recognize the first avatar performing a linked action that imitates the user action of the first user U1 when the first user U1 is not speaking. Therefore, since the action of the first user U1 can be visually recognized, appropriate communication becomes possible. Since the second user U2 can visually recognize actions other than sign language actions, communication becomes easier.
[0042] (Processing to convert sign language into natural language) Next, the process of converting the sign language of the second user U2 into natural language will be described with reference to Figs. 2 and 3. Fig. 3 is a control block diagram showing the configuration of the avatar generation device 100A. Note that the configuration and processing of the avatar generation device 100A are basically the same as the configuration and processing of the avatar generation device 100, and therefore illustrations and descriptions thereof will be omitted as appropriate. In the following description, at least a part of the processing performed by the avatar generation device 100A may be performed by the avatar generation device 100. Conversely, at least a part of the processing performed by the avatar generation device 100 may be performed by the avatar generation device 100A.
[0043] Like the avatar generation device 100, the avatar generation device 100A includes a motion detection unit 110A, an avatar processing unit 130A, a storage unit 140A, a display unit 150A, etc. Note that although a part of the configuration of the avatar generation device 100A is different from that of the avatar generation device 100, the configuration of the avatar generation device 100A may be the same as that of the avatar generation device 100. In other words, each block of the avatar generation device 100A may be the same as or different from that of the avatar generation device 100.
[0044] The motion detection unit 110A detects the motion of the second user U2 using the camera 111A. The second user U2 is a sign language user. Thus, the motion of the second user U2 includes sign language motion and non-sign language motion. The avatar generation device 100A transmits motion data indicating the motion of the second user U2 to the avatar generation device 100 via the server 300. The avatar generation device 100A may also transmit image data of a moving image of the second user U2 as motion data to the avatar generation device 100. Alternatively, the avatar generation device 100A may transmit data extracted from the moving image by image analysis or the like to the avatar generation device 100 as motion data.
[0045] The avatar generation device 100 shown in Fig. 2 acquires motion data indicating the motion of the second user U2 via the server 300. The motion information extraction unit 131 extracts the motion of the second user U2 from the motion data. For example, the motion information extraction unit 131 performs image processing on the image data to extract the motions of the hands, arms, fingers, feet, legs, head, and the like of the second user U2. The motion information is information indicating the motion of each body part.
[0046] The movement information extraction unit 131 outputs the movement information to the sign language determination unit 132. The sign language determination unit 132 determines whether the movement of the second user U2 is a sign language movement based on the movement information. The sign language determination unit 132 extracts continuous movements and divisions from the movement information. The sign language determination unit 132 checks the movement information and the sign language data to determine whether the movement of the second user U2 matches the sign language movement. If the movement of the second user U2 is determined to be sign language, the sign language determination unit 132 converts the tracked movement information into text data in the order of the movement information and outputs it to the natural language conversion unit 133. The sign language determination unit 132 generates text data based on the sign language data in the storage unit 140.
[0047] The natural language conversion unit 133 converts the converted text data into a natural language and outputs it to the speaker 123. That is, the natural language conversion unit 133 converts the text data into an audio signal. The speaker 123 outputs the audio signal to the first user U1. This allows the first user U1 to hear the audio corresponding to the sign language. Note that the speaker 123 may be a headphone or an earphone as long as it is a device capable of outputting audio.
[0048] In this way, the avatar generation device 100 converts sign language actions into audio signals and outputs them to the first user U1. The first user U1 can hear the contents of the sign language spoken by the second user U2 as natural language audio. The first user U1, who cannot understand sign language, can understand the contents of the sign language. This enables appropriate communication.
[0049] The avatar action management unit 136 may output the movement of the second avatar A2 to the display unit 150 simultaneously with the voice output. The display unit 150 displays the avatar A2 of the second user U2. Here, the avatar action management unit 136 may stop the sign language movement of at least the head, arms, and fingers of the second avatar A2 and replace it with another movement. Alternatively, the avatar action management unit 136 may output the sign language movement as it is to the display unit 150. Also, the second user U2 may move the mouth of the second avatar A2 according to the contents of the sign language. This allows the display unit 150 to display the second avatar A2 as if the second avatar A2 is speaking the contents of the sign language. In other words, the mouth of the second avatar A2 changes according to the text data.
[0050] When the sign language determination unit 132 determines that the avatar A1 is a sign language, the avatar generation device 100 may automatically replace the movement of the first avatar A1 with a sign language action and transmit the sign language action to the server 300. This can help the first user U1 to have a conversation with the second user U2 in the virtual space even if only the first user U1 has the avatar generation device 100. In this case, the second user U2 may use a display device capable of displaying the first avatar A1. Of course, the avatar generation device 100 may transmit text data to the server 300. In this case, the avatar generation device 100A may convert the text data into a sign language action.
[0051] If the sign language determination unit 132 does not determine that the action is a sign language, the avatar generation device 100 displays the action received from the server 300 as is on the display unit 150. In other words, the display unit 150 displays the second avatar A2 so that the second avatar A2 performs a linked action linked to the user action.
[0052] Furthermore, the sign language determination unit 132 may identify the type of sign language. For example, different countries and regions have different sign languages. Even if the hand and finger movements are the same, the sign language may correspond to words with different meanings. Even if the word is the same, the hand and finger movements may differ. The sign language determination unit 132 may identify the type of sign language based on the movement. For example, the sign language determination unit 132 identifies the sign language depending on whether or not the sign language movement can be converted into text data. Alternatively, the sign language determination unit 132 identifies the sign language depending on whether or not the converted text data has an appropriate meaning.
[0053] (Sign Language Conversion) Furthermore, when the first user U1 and the second user U2 use different sign languages, the avatar generation device 100A may perform a process of converting the sign languages. The avatar generation device 100A may perform a process of converting a sign language action in one sign language into a sign language action in another sign language. For example, even if the first user U1 and the second user U2 are sign language users, if the first user U1 and the second user U2 are of different nationalities, the sign languages of the first user U1 and the second user U2 are different. In this case, the first user U1 and the second user U2 cannot have a conversation in sign language as it is. Therefore, in this embodiment, the avatar generation device 100A performs a process of converting different sign languages.
[0054] The process of converting sign language will be described below with reference to Fig. 3. Also, the process and configurations described with reference to Fig. 2 will not be described as necessary. For example, each component of the avatar processing unit 130A performs the same process as each component of the avatar processing unit 130.
[0055] The motion acquisition unit 112A acquires a user motion of the second user U2 and outputs it to the motion information extraction unit 131A and the linked motion generation unit 137A. The linked motion generation unit 137A generates a linked motion according to the user motion of the second user U2. The motion information extraction unit 131A extracts motion information of each part of the second user U2. The sign language determination unit 132A determines whether the motion of the second user U2 is a sign language motion based on the motion information. Furthermore, if the motion of the second user U2 is a sign language motion, the sign language determination unit 132A identifies the sign language of the second user U2 and determines whether it matches the sign language of the first user U1. For example, the sign language determination unit 132A acquires information for identifying the sign language of the first user U1 from the server 300 or the like. Then, the sign language determination unit 132A determines whether the sign language of the second user U2 is the same sign language as the sign language of the first user U1.
[0056] If the sign language of the first user U1 does not match, the text conversion unit 134A converts the sign language actions of the second user U2 into text data. The sign language action generation unit 135A generates sign language actions of the sign language used by the first user U1 based on the text data. This converts the content of the sign language actions of the second user U2 into sign language actions of the sign language used by the first user U1.
[0057] When the second user U2 is making a sign language movement, the avatar movement management unit 136A stops synchronization of the sign language movement before conversion, which is a linked movement by the linked movement generation unit 137A, and transmits the converted sign language movement as an avatar movement to the server 300. When the server 300 transmits the sign language movement to the avatar generation device 100, the display unit 150 displays the second avatar A2. As a result, the second avatar A2 makes the sign language movement in the sign language used by the first user U1. The avatar generation device 100 may also convert the sign language in the same way as the avatar generation device 100A. Sign language users who use different sign languages can also communicate with each other in a borderless manner using each other's avatars.
[0058] Furthermore, if the second user U2 is not performing a sign language action, the avatar action management unit 136A transmits the linked action as an avatar action to the server 300. If the server 300 transmits a linked action unrelated to a sign language action to the avatar generation device 100, the display unit 150 displays the second avatar A2. This causes the second avatar A2 to perform a linked action other than the sign language action. This allows the first user U1 and the second user U2 to communicate appropriately. In other words, the first user U1 can visually recognize actions other than the sign language action of the second user U2, making communication easier.
[0059] The first user U1 and the second user U2 may input their nationalities and the sign language they use in advance to the avatar generation devices 100, 100A. This allows the sign language determination unit 132 and the sign language action generation unit 135 to read out the sign language data they use from the storage unit 140. The sign language determination unit 132 can make accurate determinations. Furthermore, the sign language action generation unit 135 can accurately generate sign language actions. Sign language users of different nationalities may become 3D avatars and converse borderlessly by converting sign languages.
[0060] Next, a method for generating an avatar will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the method for generating an avatar by the avatar generation device 100.
[0061] The audio signal acquisition unit 122 acquires an audio signal using the microphone 121 (S101). For example, the microphone 121 picks up the voice of the first user U1, and the audio signal acquisition unit 122 acquires the audio signal. Next, the movement detection unit 110 acquires a user movement of the first user using the camera 111 (S102). The linked movement generation unit 137 generates a linked movement of the first avatar A1 based on the user movement (S103).
[0062] The text conversion unit 134 converts the voice data of the voice signal into text data (S104). The sign language action generation unit 135 generates a sign language action from the text data (S105). In this way, a sign language action corresponding to the content of the utterance of the first user U1 is generated. Note that if no voice data is detected or if the action is not a sign language action, the processes of steps S104 and S105 may be omitted.
[0063] The avatar action management unit 136 determines whether the first user U1 is speaking (S106). If the first user U1 is speaking (YES in S106), the avatar action management unit 136 selects a sign language action (S107). While the first user U1 is speaking, the avatar action management unit 136 causes the first avatar A1 to perform a sign language action. Then, the display unit 150 displays the first avatar A1 performing the sign language action as a moving image (S109). While the first user U1 is speaking, the avatar action management unit 136 causes the first avatar A1 to perform a sign language action. If the first user U1 is speaking, the first avatar A1 stops the linked action and performs the sign language action. Thus, the second user U2 can understand the content of the first user U1's speech through the sign language action.
[0064] If the first user U1 is not speaking (NO in S106), the avatar action management unit 136 selects the linked action (S108). That is, the avatar action management unit 136 causes the first avatar A1 to perform the linked action when the first user U1 is not speaking. The display unit 150 displays the first avatar A1 performing the linked action as a moving image (S109). When the first user U1 is not speaking, the first avatar A1 performs the linked action. Thus, the second user U2 can view the linked action that imitates the action of the first user U1. The users can communicate more appropriately.
[0065] A sign language user and a non-handicapped person can communicate smoothly. For example, the first user U1, who is a non-handicapped person, can communicate with the second user U2 without learning sign language. Furthermore, the first user U1 and the second user U2 can communicate with each other without the intervention of a non-handicapped person who has acquired sign language. When a visual supplementary tool such as text is used, it is necessary to focus on the text. According to this embodiment, the second user U2 can converse in sign language while looking at the first avatar A1. Since both are in a virtual space, appropriate communication is possible. Furthermore, the first user U1 can see the actions of the second user U2 other than the sign language actions.
[0066] In addition, accurate reading is possible by detecting three-dimensional sign language movements using volumetrics technology. In addition, the conversion process between sign language movements and text data may use an AI (Artificial Intelligence) model generated by machine learning.
[0067] Also, only the first avatar A1 of the first user U1 may be displayed to the second user U2. In other words, the second avatar A2 of the second user U2 does not need to be displayed. For example, the first user U1 may appear in front of the second user as a 3D avatar and talk to him / her using sign language. Since the user does not need to look at another sign language conversion tool, the user can experience a realistic face-to-face communication. Since the user can talk without looking at devices such as terminals other than the one he / she is talking to, the user can feel closer to the other person.
[0068] A part or all of the above processes may be executed by a computer program. The above-mentioned program may be stored and provided to a computer using various types of non-transitory computer readable media. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be provided to a computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium may provide the program to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.
[0069] Furthermore, the processing in the avatar generating devices 100, 100A can be realized as an application program (app) for a smartphone or the like. For example, a user can easily communicate by installing an app on a smartphone in advance. In this case, the app for sign language users and the app for able-bodied people may be separate programs. Also, a sign language user may input the sign language to be used in advance through the app. For example, a sign language user may select the sign language to be used from multiple sign languages using a pull-down menu or the like.
[0070] Although the present invention has been specifically described based on the embodiments, it goes without saying that the present invention is not limited to the above-mentioned embodiments and can be modified in various ways without departing from the spirit of the invention. Two or more of the above-mentioned embodiments can also be appropriately combined. [Explanation of symbols]
[0071] 100, 100A Avatar Generator 110, 110A Operation detection unit 111, 111A Camera 112, 112A operation acquisition section 120 Audio input / output section 121 Mike 122 Audio signal acquisition unit 123 Speaker 130, 130A avatar processing unit 131, 131A Motion information extraction unit 132, 132A Sign language determination unit 133 Natural Language Conversion Department 134, 134A Text conversion section 135, 135A Sign language action generation unit 136, 136A Avatar operation management section 137, 137A Interlocking operation generation section 140, 140A storage section 150, 150A display section U1 User 1 U2 Second User A1 1st avatar A2 2nd avatar
Claims
1. a voice signal acquisition unit that collects speech of a first user with a microphone and acquires a voice signal; a first action detection unit that detects a user action of the first user by using a first sensor; a linking action generating unit configured to generate a linking action of the first avatar such that the first avatar moves in linkage with the user action detected by the first action detecting unit; a text conversion unit for converting the voice signal into text data; a sign language action generation unit that converts the text data into a sign language and generates a sign language action of the first avatar; and an action management unit that causes the interlocking action generation unit to perform the interlocking action except when the first user is speaking, and stops the interlocking action generated by the interlocking action generation unit while the first user is speaking, and causes the first avatar to perform the sign language action.
2. a display unit that displays the first avatar to a second user who is a sign language user; A second action detection unit that detects an action of the second user; a sign language determination unit that determines whether the action of the second user is a sign language action; a natural language conversion unit that converts the sign language action into a natural language when the action of the second user is a sign language action; The avatar generating device according to claim 1 , further comprising: a voice output unit that outputs the voice in the natural language to the first user.
3. The sign language determination unit identifies the type of the sign language, The avatar generating device according to claim 2 , wherein the sign language action generating unit generates a sign language action for the identified sign language.
4. A step of collecting speech of a first user with a microphone and acquiring a speech signal; detecting a user action of the first user with a first sensor; generating a linked action of the first avatar such that the first avatar moves in conjunction with the user action; converting the speech signal into text data; converting the text data into a sign language to generate a sign language action of the first avatar; causing the linkage action generation unit to execute the linkage action during a time other than when the first user is speaking; stopping the linked action while the first user is speaking and causing the first avatar to perform the sign language action.
5. A program for causing a computer to execute an avatar generating method, The avatar generating method includes: A step of collecting speech of a first user with a microphone and acquiring a speech signal; detecting a user action of the first user with a first sensor; generating a linked action of the first avatar such that the first avatar moves in conjunction with the user action; converting the speech signal into text data; converting the text data into a sign language to generate a sign language action of the first avatar; causing the linkage action generation unit to execute the linkage action during a time other than when the first user is speaking; stopping the linked action while the first user is speaking and causing the first avatar to perform the sign language action.
Citation Information
Patent Citations
Barrier free distributed virtual reality system
JP1999003369A
Method, system and program for modifying image
JP2009077380A
Sensory Eyewear
JP2019535059A
Information processing device, information processing method, and program
JP2019008570A
Communication system, reception terminal device, and program thereof
JP2021056940A
