Information processing device, gesture transcription method and program

Abstract

An information processing device comprising: a feature extraction unit that is configured to extract a plurality of feature data indicating a specific motion from motion video data indicating a motion-including video; a feature synthesis unit that is configured to synthesize the plurality of feature data; a control unit that is configured to select, upon receiving a transfer request, feature data that corresponds to the transfer request from the synthesized feature data; and a feature transfer unit that is configured to transfer the selected feature data to input video data and generate output video data.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a gesture transcription method, and a program. [Background technology]

[0002] There is known a technique for converting video data of a person into video data showing a specific gesture such as nodding, smiling, etc. For example, Non-Patent Document 1 discloses a technique for extracting data showing a specific gesture from video data of a person and transcribing it onto video data of another person in real time. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Tomas Jakab, et.al. "Unsupervised Learning of Object Landmarks through Conditional Image Generation", NIPS, 2018 Summary of the Invention [Problem to be solved by the invention]

[0004] In video conferences and other similar events, adding facial expressions, blinks, nods, posture, interjections, eye contact, gazes, and other gestures to video footage of participants can be considered to enable smooth interpersonal relationships and the progress of the conference. However, conventional technology simply transcribes video data showing the gestures of a person from which the data is extracted into target video data in chronological order. Therefore, when combining multiple elements, such as a smile and a nod, a video equivalent to the number of element combinations is required, resulting in a problem of a large amount of video data being required.

[0005] The disclosed technology aims to reduce the amount of video data required to transcribe gestures. [Means for solving the problem]

[0006] The disclosed technology is an information processing device including a feature extraction unit configured to extract multiple feature data indicating specific gestures from gesture video data showing video including gestures; a feature synthesis unit configured to synthesize the multiple feature data; a control unit configured to receive a transcription request and select feature data corresponding to the transcription request from the synthesized feature data; and a feature transcription unit configured to transcribe the selected feature data to input video data to generate output video data. [Effects of the Invention]

[0007] The amount of video data required to transcribe gestures can be reduced. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of an information processing device according to a first embodiment of the present invention. [Figure 2] 10 is a flowchart showing an example of the flow of a feature transfer process according to Example 1 of an embodiment of the present invention. [Figure 3] FIG. 1 is a diagram for explaining an outline of a feature transfer process according to Example 1 of an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram for explaining a method of synthesizing feature data according to Example 1 of an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram illustrating an example of a functional configuration of an information processing device according to Example 2 of an embodiment of the present invention. [Figure 6] 10 is a flowchart showing an example of the flow of a feature transfer process according to Example 2 of an embodiment of the present invention. [Figure 7] FIG. 2 illustrates an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0010] Hereinafter, examples 1 and 2 will be described as specific examples of this embodiment.

[0011] Example 1 In this embodiment, an example will be described in which a plurality of feature data extracted from data showing a gesture video is synthesized, and the video is processed based on the synthesized feature data, thereby reflecting various gestures on people and the like appearing in the video.

[0012] 1 is a diagram showing an example of the functional configuration of an information processing device according to Example 1 of an embodiment of the present invention. The information processing device 10 according to this example includes a gesture video storage unit 11, a feature extraction unit 12, a feature synthesis unit 13, a control unit 14, an input video storage unit 15, a feature transfer unit 16, and an output video storage unit 17.

[0013] The gesture video storage unit 11 stores data representing gesture videos. The gesture videos are pre-recorded videos of a person's gestures. Gestures are actions that convey emotions, intentions, etc., such as facial expressions, blinking, nodding, posture, interjections, and gaze.

[0014] The feature extraction unit 12 extracts multiple pieces of feature data from the data representing the gesture video according to the content of the specific gesture. The extracted data is extracted for each gesture content, such as feature data for "smile" and feature data for "nod."

[0015] The feature synthesis unit 13 synthesizes the extracted multiple pieces of feature data. For example, the feature synthesis unit 13 synthesizes the feature data of "smile" and the feature data of "nod" to generate feature data of "smile and nod" which is a combination of the action of "smile" and "nod". The feature data may be, for example, vector data indicating the feature. Therefore, the feature synthesis unit 13 may synthesize the multiple pieces of feature data by vector synthesis.

[0016] Upon receiving a transcription request, the control unit 14 selects feature data corresponding to the transcription request from the synthesized feature data. The transcription request is a transcription request specifying a specific gesture, for example, through a user operation. The control unit 14 may select feature data from either synthesized feature data or non-synthesized feature data. For example, the control unit 14 may select feature data from either "smile" feature data, "nod" feature data, or "smile and nod" feature data.

[0017] The input video storage unit 15 stores data representing an input video, which is a video of a user captured by a capturing device such as a web camera.

[0018] The feature transfer unit 16 transfers the feature data output by the control unit 14 to the input video. For example, the feature transfer unit 16 transfers the feature data of "smiling and nodding" to an input video of a user with a neutral expression, thereby converting the video into video data representing the user nodding and smiling, and outputs the video.

[0019] The output video storage unit 17 stores the video data output by the feature transfer unit 16 .

[0020] Next, a description will be given of the operation of the information processing device 10 according to the present embodiment. The information processing device 10 executes a feature transfer process in response to a user operation or the like.

[0021] 2 is a flowchart showing an example of the flow of the feature transfer process according to Example 1 of the embodiment of the present invention. When the feature transfer process starts, the feature extractor 12 extracts a plurality of feature data from the gesture video (step S11).

[0022] Next, the feature synthesis unit 13 synthesizes the extracted plurality of feature data (step S12). When the control unit 14 receives a transcription request through a user operation or the like, it selects feature data corresponding to the transcription request from the synthesized feature data (step S13).

[0023] Next, the feature transfer unit 16 transfers the feature data to the input video to generate an output video (step S14). The generated output video is stored in the output video storage unit 17. Then, the information processing device 10 outputs the generated output video (step S15).

[0024] 3 is a diagram for explaining an outline of feature transfer processing according to Example 1 of the embodiment of the present invention. Feature data 101 is an example of feature data for "nodding." The feature data 101 is, for example, a feature vector characterized by conversion from a normal video 101a to a "nodding" video 101b.

[0025] The feature data 102 is an example of feature data for a "smile." The feature data 102 is, for example, a feature vector characterized by the transformation from a normal video 102a to a "smile" video 102b.

[0026] The feature data 103 is an example of the feature data of "smile and nod" that is a combination of the feature data of "smile" and the feature data of "nod." The feature data 103 is, for example, a feature vector characterized by the conversion from a normal video 103a to a "smile and nod" video 103b.

[0027] Here, the normal video 101a, the normal video 102a, and the normal video 103a may be the same video or may be different videos.

[0028] Video 104 is an example of an input video. Video 105 is an example of an output video. When feature data 103 having the feature of "smiling and nodding" is transferred to video 104, video 105 is generated, which includes an image of a person appearing in video 104 smiling and nodding.

[0029] Here, the person in the input video and the person in the gesture video may be the same person or different people. The object in the input video or the object in the gesture video may or may not be a person, and may be, for example, a non-human animal such as a dog or a cat.

[0030] 4 is a diagram for explaining a method for synthesizing feature data according to Example 1 of an embodiment of the present invention. The input and output video data and the transferred feature data are represented as vector data (video vectors and feature vectors) by edge processing and the like included in the video. For example, an input video 202a showing person A is characterized by a video vector 301a starting from the origin 201.

[0031] When the feature vector 302a having the characteristic of "smiling" is reflected in the input video 202a, a video 203a of smiling person A is generated. When the feature vector 303a having the characteristic of "nodding" is reflected in the video 203a, a video 204a of person A smiling and nodding is generated.

[0032] Similarly, input image 202b showing person B is characterized by image vector 301b starting from origin 201.

[0033] When the feature vector 302b having the characteristic of "smile" is reflected in the input video 202b, a video 203b of smiling person B is generated. When the feature vector 303b having the characteristic of "nodding" is reflected in the video 203b, a video 204b of person B nodding with a smile is generated.

[0034] Here, the feature vector 302a and the feature vector 302b may be the same vector. Similarly, the feature vector 303a and the feature vector 303b may be the same vector.

[0035] In step S12 of the feature transfer process described above, the feature synthesis unit 13 synthesizes, for example, the feature vector 302a and the feature vector 303a. Then, the feature transfer unit 16 transfers the synthesized feature vector to 202a to generate image 204a, and transfers the synthesized feature vector to 202b to generate image 204b.

[0036] According to this embodiment, by synthesizing a plurality of feature data extracted from data showing gesture video and processing the video based on the synthesized feature data, it is possible to reflect various gestures on people, etc. appearing in the video. Therefore, when combining a plurality of elements such as a smile and a nod, it is not necessary to have video corresponding to the number of element combinations, and therefore the amount of video data required to transcribe the gestures can be kept small.

[0037] Example 2 A second embodiment will be described below with reference to the drawings. The second embodiment differs from the first embodiment in that emotions are estimated based on an input video. Therefore, the following description of the second embodiment will focus on the differences from the first embodiment, and components having the same functional configuration as the first embodiment will be assigned the same reference numerals as those used in the description of the first embodiment, and descriptions thereof will be omitted.

[0038] This embodiment is an example for solving the following problem. That is, when feature data extracted based on a gesture video is transferred to an input video, the facial expressions of the source gesture video (for example, the normal video 101a, the normal video 102a, etc. shown in FIG. 3) and the input video need to match. For example, if the source gesture video has a neutral expression and the destination has a smiling expression, it is fine as long as the input video has a neutral expression. However, if the source gesture video has an angry expression and the destination has a smiling expression, the conversion may not be successful if the input video has a neutral expression.

[0039] Therefore, in this embodiment, emotions are estimated based on input video, and feature data corresponding to the estimated emotions is synthesized.

[0040] 5 is a diagram illustrating an example of a functional configuration of an information processing device according to Example 2 of an embodiment of the present invention. The information processing device 10 according to this example has a configuration in which a feeling estimation unit 18 is added to the information processing device 10 according to Example 1.

[0041] The emotion estimation unit 18 estimates an emotion based on the input video. For example, the emotion estimation unit 18 may estimate the emotion of a person appearing in the input video from the facial expression of the person. For example, the emotion of joy is estimated based on a video showing a person with a smiling expression.

[0042] Furthermore, the feature synthesis unit 13 according to this embodiment synthesizes feature data corresponding to the estimated emotion from among the extracted plurality of feature data.

[0043] 6 is a flowchart showing an example of the flow of feature transfer processing according to Example 2 of the embodiment of the present invention. When the feature transfer processing starts, the feature extractor 12 extracts a plurality of feature data from the gesture video (step S21).

[0044] Next, the emotion estimation unit 18 estimates an emotion based on the input video (step S22). Subsequently, the feature synthesis unit 13 synthesizes feature data corresponding to the estimated emotion from among the extracted plurality of feature data (step S23). When the control unit 14 receives a transcription request by a user operation or the like, it selects feature data corresponding to the transcription request from the synthesized feature data (step S24).

[0045] Then, the feature transfer unit 16 transfers the input video to the selected feature data to generate an output video (step S25). The generated output video is stored in the output video storage unit 17. Then, the information processing device 10 outputs the generated output video (step S26).

[0046] According to this embodiment, emotions are estimated based on input video, and feature data corresponding to the estimated emotions is synthesized. This allows feature data appropriate for the input video to be synthesized and used. For example, facial expression conversion can be performed appropriately using feature data based on a gesture video with the same facial expression as that of a person appearing in the input video.

[0047] <Hardware configuration> Finally, a description will be given of the hardware configuration of the information processing device 10 according to this embodiment. The information processing device 10 according to this embodiment is realized by, for example, the hardware configuration of a computer 500 shown in FIG.

[0048] Fig. 7 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 7 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B.

[0049] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0050] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when instructed to start the program. The CPU 1004 implements functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a program-based graphical user interface (GUI), etc. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operational instructions. The output device 1008 outputs calculation results. Note that the computer may be equipped with a graphics processing unit (GPU) or tensor processing unit (TPU) instead of the CPU 1004, or may be equipped with a GPU or TPU in addition to the CPU 1004. In this case, processing may be shared, with the GPU or TPU executing processing requiring special calculations and the CPU 1004 executing other processing.

[0051] The information processing device 10 according to this embodiment is realized by reading a program for causing the computer 500 to execute each of the above-described processes and executing the processes defined in the program. The program may be recorded on a recording medium 503a or the like, or may be provided via a network.

[0052] (Summary of the embodiment) This specification describes at least the information processing device, gesture transcription method, and program described in the following sections. (Section 1) a feature extraction unit configured to extract a plurality of feature data representing a specific gesture from gesture video data representing a video including the gesture; a feature synthesis unit configured to synthesize the plurality of feature data; a control unit configured to receive a transcription request and select feature data corresponding to the transcription request from the synthesized feature data; a feature transfer unit configured to transfer the selected feature data to input video data to generate output video data, Information processing device. (Section 2) the feature synthesis unit is configured to synthesize a plurality of vector data representing the plurality of feature data by vector synthesis; 2. The information processing device according to claim 1. (Section 3) a feeling estimation unit configured to estimate a feeling based on the input video data; the feature synthesis unit is configured to synthesize feature data corresponding to an estimated emotion from among the extracted plurality of feature data. 3. The information processing device according to claim 1 or 2. (Section 4) 1. A computer-implemented method for transcribing gestures, comprising: extracting a plurality of feature data representing a specific gesture from gesture video data representing a video including the gesture; a step of combining the plurality of feature data; receiving a transcription request and selecting feature data corresponding to the transcription request from the synthesized feature data; and transferring the selected feature data to input video data to generate output video data. Gesture transcription method. (Section 5) A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 3.

[0053] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0054] 10. Information processing equipment 11 Gesture image storage unit 12 Feature Extraction Unit 13 Feature synthesis unit 14 Control Unit 15 Input video storage section 16 Feature Transfer Section 17 Output video storage section 18 Emotion estimation part 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output Device

Claims

1. a feature extraction unit configured to extract a plurality of feature data representing a specific gesture from gesture video data representing a video including the gesture; a feature synthesis unit configured to synthesize, from among the plurality of feature data, feature data corresponding to an emotion estimated based on input video data; a control unit configured to receive a transcription request and select feature data corresponding to the transcription request from the synthesized feature data; a feature transfer unit configured to transfer the selected feature data to the input video data to generate output video data, Information processing device.

2. the feature synthesis unit is configured to synthesize a plurality of vector data representing the plurality of feature data by vector synthesis; The information processing device according to claim 1 .

3. a feeling estimation unit configured to estimate a feeling based on the input video data; the feature synthesis unit is configured to synthesize feature data corresponding to the emotion estimated by the emotion estimation unit from among the plurality of extracted feature data. The information processing device according to claim 1 .

4. 1. A computer-implemented method for transcribing gestures, comprising: extracting a plurality of feature data representing a specific gesture from gesture video data representing a video including the gesture; synthesizing, among the plurality of pieces of feature data, feature data corresponding to an emotion estimated based on input video data; receiving a transcription request and selecting feature data corresponding to the transcription request from the synthesized feature data; and transferring the selected feature data to input video data to generate output video data. Gesture transcription method.

5. A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 3.

Images