Information processing device, gesture transcription method and program

Abstract

This information processing device comprises: a feature extraction unit that extracts, from motion video data expressing a video containing motions, feature data of a specific motion; a control unit that receives a transfer request and outputs the feature data corresponding to the transfer request; and a feature transfer unit that generates output video data by transferring, to input video data, the feature data corresponding to the transfer request, and that outputs the generated output video data.

Description

[Technical field]

[0001] The present invention relates to an information processing device, a gesture transcription method, and a program. [Background technology]

[0002] There is known a technology for converting video data of a person into video data showing a specific gesture such as a nod, a smile, etc. For example, Non-Patent Document 1 discloses a technology for extracting data showing a specific gesture from video data of a person and transcribing the data to video data of another person in real time. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Tomas Jakab, et.al. "Unsupervised Learning of Object Landmarks through Conditional Image Generation", NIPS, 2018 Summary of the Invention [Problem to be solved by the invention]

[0004] In video conferences, etc., adding facial expressions, blinking, nodding, posture, interjections, gazes, and other gestures to the video of participants can be considered to enable smooth interpersonal relationships and progress of the conference. However, conventional technology only transfers video data showing the gestures of the person from which the gestures are extracted to the target video data in chronological order, which is problematic in that it is not possible to freely specify the content of the gestures or the timing of transfer to the video data.

[0005] The disclosed technology aims to improve the degree of freedom when reflecting gestures in video data. [Means for solving the problem]

[0006] The disclosed technology is an information processing device that includes a feature extraction unit that extracts feature data of a specific gesture from gesture video data that shows a video including the gesture, a control unit that receives a transcription request and outputs the feature data corresponding to the transcription request, and a feature transcription unit that transcribes the feature data corresponding to the transcription request onto input video data to generate output video data, and outputs the generated output video data. Effect of the Invention

[0007] This improves the degree of freedom when reflecting gestures in video data. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of the information processing device according to the first embodiment. [Diagram 2] 11 is a flowchart showing an example of the flow of a feature transfer process according to the first embodiment. [Diagram 3] FIG. 11 is a diagram illustrating an example of a functional configuration of an information processing device according to a second embodiment. [Figure 4] FIG. 13 is a diagram for explaining a method of adjusting a video image. [Diagram 5] 13 is a flowchart showing an example of the flow of a feature transfer process according to the second embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of a functional configuration of an information processing device according to a third embodiment. [Figure 7] 13 is a flowchart showing an example of the flow of a feature transfer process according to a third embodiment. [Figure 8] FIG. 11 is a diagram illustrating an example of a functional configuration of an information processing device according to a fourth embodiment. [Figure 9] 13 is a flowchart showing an example of the flow of a feature transfer process according to the fourth embodiment. [Figure 10] FIG. 13 is a diagram illustrating an example of a functional configuration of an information processing device according to a fifth embodiment. [Figure 11] 13 is a flowchart showing an example of the flow of a feature transfer process according to a fifth embodiment. [Figure 12]FIG. 2 illustrates an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0010] Hereinafter, examples 1 to 5 of this embodiment will be described.

[0011] Example 1 1 is a diagram illustrating an example of a functional configuration of an information processing device according to Example 1. The information processing device 10 according to the present embodiment includes a gesture video storage unit 11, a feature extraction unit 12, a control unit 13, an input video storage unit 14, a feature transfer unit 15, and an output video storage unit 16.

[0012] The gesture image storage unit 11 stores data showing a gesture image. The gesture image is a pre-recorded image of a person's gestures. A gesture is an action that conveys an emotion or intention, such as facial expression, blinking, nodding, posture, interjections, and gaze.

[0013] The feature extraction unit 12 extracts feature data from the data representing the gesture video according to the content of the specific gesture. The extracted data is extracted for each content of the gesture, such as smile feature data, blink feature data, etc.

[0014] Upon receiving the transcription request, the control unit 13 outputs feature data corresponding to the transcription request. The transcription request is a request for transcription that designates a specific gesture through a user operation or the like.

[0015] The input video storage unit 14 stores data representing an input video. The input video is a video of a user captured by a capturing device such as a Web camera.

[0016] The feature transfer unit 15 transfers the feature data output by the control unit 13 to the input video. For example, the feature transfer unit 15 transfers the smiling feature data to the input video of an expressionless user, thereby converting the input video into video data representing the smiling user and outputting the video data.

[0017] The output video storage unit 16 stores the video data output by the feature transfer unit 15 .

[0018] Next, a description will be given of the operation of the information processing device 10 according to the present embodiment. The information processing device 10 executes a feature transfer process in response to a user's operation or the like.

[0019] 2 is a flowchart showing an example of the flow of the feature transfer process according to the first embodiment. When the feature transfer process starts, the feature extractor 12 extracts feature data from the gesture video (step S11). Then, when the controller 13 receives a transfer request by a user's operation or the like, it outputs feature data corresponding to the transfer request (step S12).

[0020] Next, the feature transfer unit 15 transfers the feature data to the input video to generate an output video, and outputs the generated output video (step S13).

[0021] According to the information processing device 10 of the present embodiment, feature data extracted from a gesture video is transferred to an input video to generate an output video. By using, as the gesture video, video recorded for each gesture content such as facial expression, blinking, nodding, posture, interjection, gaze, etc., it is possible to reflect a specified gesture in the input video.

[0022] Example 2 A description will be given of Example 2 below with reference to the drawings. Example 2 differs from Example 1 in that the facial features of the gesture image and the input image are recognized and aligned. Therefore, the following description of Example 2 will focus on the differences from Example 1, and the same reference numerals as those used in the description of Example 1 will be used to denote components having the same functional configuration as Example 1, and the description thereof will be omitted.

[0023] 3 is a diagram illustrating an example of a functional configuration of an information processing device according to Example 2. The information processing device 10 according to this embodiment has a configuration in which a first part recognition unit 21, a second part recognition unit 22, and an image adjustment unit 23 are added to the information processing device 10 according to Example 1.

[0024] The first part recognition unit 21 receives a gesture video and recognizes and outputs the positions, sizes, and orientations of the facial parts. The second part recognition unit 22 receives an input video and recognizes and outputs the positions, sizes, and orientations of the facial parts. The video adjustment unit 23 adjusts the positions, sizes, or orientations of the feature data based on the gesture video, based on the positions, sizes, and orientations of the facial parts in the gesture video and the positions, sizes, and orientations of the facial parts in the input video.

[0025] 4 is a diagram for explaining a method of adjusting an image. Image adjustment unit 23 adjusts the position (x, y), the direction (θ), or the size (r) so as to match feature data 901 with input image data 902.

[0026] 5 is a flowchart showing an example of the flow of the feature transfer process according to the second embodiment. When the feature transfer process starts, the feature extractor 12 extracts feature data from the gesture video (step S21). Then, when the controller 13 receives a transfer request by a user's operation or the like, it outputs feature data corresponding to the transfer request (step S22).

[0027] Next, the first part recognition unit 21 recognizes parts of the gesture image (step S23). Next, the second part recognition unit 22 recognizes parts of the input image (step S24). Next, the image adjustment unit adjusts the feature data (step S25).

[0028] Next, the feature transfer unit 15 transfers the adjusted feature data to the input video to generate an output video, and outputs the generated output video (step S26).

[0029] According to the information processing device 10 of the present embodiment, the facial parts of the gesture video and the facial parts of the input video are recognized, and the position, orientation, or size of the feature data is adjusted. This makes it possible to avoid distortion of the output video when the facial position, size, orientation, etc. in the gesture video and the input video are different.

[0030] Example 3 Example 3 will be described below with reference to the drawings. Example 3 differs from Example 1 in that only images related to large movements such as nodding and posture are prepared as gesture images, and small movements such as facial expressions, blinking, and eye contact are expressed by processing the gesture images. Therefore, in the following description of Example 3, differences from Example 1 will be mainly described, and components having the same functional configuration as Example 1 will be assigned the same reference numerals as those used in the description of Example 1, and descriptions thereof will be omitted.

[0031] 6 is a diagram illustrating an example of a functional configuration of an information processing device according to Example 3. The information processing device 10 according to this embodiment is configured by adding a video processing unit 31 and a processed gesture video storage unit 32 to the information processing device 10 according to Example 1.

[0032] The video processing unit 31 processes the gesture video including large movements (first movements) using a technique such as MLS (Moving Least Square Approximation) and generates processed gesture video data by adding small movements (second movements) such as facial expressions, blinking, and eye contact (Reference [1]).

[0033] The processed gesture video data is stored in the processed gesture video storage unit 32. The feature extraction unit 12 extracts feature data from the processed gesture video data stored in the gesture video storage unit 11 and the processed gesture video data stored in the processed gesture video storage unit 32.

[0034] 7 is a flowchart showing an example of the flow of the feature transfer process according to the third embodiment. When the feature transfer process starts, the video processing unit 31 processes the gesture video (step S31). Then, the feature extraction unit 12 extracts feature data from the gesture video or the processed gesture video (step S32). Then, when the control unit 13 receives a transfer request by a user's operation or the like, it outputs the feature data corresponding to the transfer request (step S33).

[0035] Next, the feature transfer unit 15 transfers the adjusted feature data to the input video to generate an output video, and outputs the generated output video (step S34).

[0036] According to the information processing device 10 of the present embodiment, a gesture video including a large action (first action) is processed, and processed gesture video data is generated by adding small actions (second actions) such as facial expressions, blinking, and eye contact. This makes it possible to reduce the number of videos to be prepared as gesture videos.

[0037] For example, in the first embodiment, when multiple gesture elements such as nodding with a smile are to be combined, images for all patterns of the elements are required. In contrast, in the present embodiment, a processed gesture video combining multiple elements such as nodding with a smile can be generated by processing a gesture video of a large gesture (first motion) of nodding and adding a small gesture (second motion) element of smiling, so that the number of patterns of gesture videos required can be significantly reduced.

[0038] Example 4 A fourth embodiment will be described below with reference to the drawings. The fourth embodiment differs from the third embodiment in that only images related to large movements such as nodding and posture are prepared as gesture images, and small movements such as facial expressions, blinking, and eye contact are expressed by processing the input images. Therefore, the following description of the fourth embodiment will focus on the differences from the third embodiment, and the same reference numerals as those used in the description of the third embodiment will be given to components having the same functional configuration as the third embodiment, and the description thereof will be omitted.

[0039] 8 is a diagram illustrating an example of a functional configuration of an information processing device according to a fourth embodiment. The image processing unit 31 according to the present embodiment processes the input image data stored in the input image storage unit 14 by using a method such as MLS (Moving Least Square Approximation) to generate processed input image data to which small actions (second actions) such as facial expressions, blinking, and gaze are added.

[0040] Moreover, the feature transfer unit 15 according to this embodiment transfers the feature data to the processed input video data to generate output video data, and outputs the generated output video data.

[0041] 9 is a flowchart showing an example of the flow of the feature transfer process according to the fourth embodiment. When the feature transfer process starts, the feature extractor 12 extracts feature data from the gesture video (step S41). Then, when the controller 13 receives a transfer request by a user's operation or the like, it outputs feature data corresponding to the transfer request (step S42).

[0042] The video processing unit 31 processes the input video (step S43). The feature transfer unit 15 transfers the feature data to the processed input video to generate output video data, and outputs the generated output video data (step S44).

[0043] According to the information processing device 10 of the present embodiment, the input video is processed to generate processed input video data by adding small actions such as facial expressions, blinking, and eye movements. This makes it possible to reduce the number of videos to be prepared as gesture videos.

[0044] For example, in the first embodiment, when multiple gesture elements such as nodding with a smile are to be combined, images for all patterns of the elements are required. In contrast, in the present embodiment, feature data extracted from a gesture video of a large gesture (first gesture) of nodding is transferred to processed input video data to which a small gesture element (second gesture) of smiling is added, thereby generating output video data combining multiple elements such as nodding with a smile, and thus the number of patterns of gesture videos required can be significantly reduced.

[0045] Example 5 A fifth embodiment will be described below with reference to the drawings. The fifth embodiment differs from the third and fourth embodiments in that only images related to large movements such as nodding and posture are prepared as gesture images, and small movements such as facial expressions, blinking, and eye contact are expressed by processing the output images. Therefore, the following description of the fifth embodiment will focus on the differences from the third and fourth embodiments, and the same reference numerals as those used in the description of the third and fourth embodiments will be assigned to components having the same functional configuration as the third and fourth embodiments, and the description thereof will be omitted.

[0046] 10 is a diagram illustrating an example of a functional configuration of an information processing device according to Example 5. A video processing unit 31 according to this embodiment processes the output video data to which the feature data has been transferred by the feature transfer unit 15.

[0047] 11 is a flowchart showing an example of the flow of the feature transfer process according to the fifth embodiment. When the feature transfer process starts, the feature extractor 12 extracts feature data from the gesture video (step S51). Then, when the controller 13 receives a transfer request by a user's operation or the like, it outputs feature data corresponding to the transfer request (step S52).

[0048] The feature transfer unit 15 transfers the feature data to the processed input video data to generate output video data, and outputs the generated output video data (step S53). The video processing unit 43 processes the output video data to generate processed output video data (step S54). Then, the video processing unit 43 outputs the processed output video data (step S55).

[0049] According to the information processing device 10 of the present embodiment, the output video data is processed to generate processed output video data to which small actions such as facial expressions, blinking, and eye movements are added. This makes it possible to reduce the number of videos to be prepared as gesture videos.

[0050] For example, in the first embodiment, when multiple gesture elements such as nodding with a smile are to be combined, images for all patterns of the elements are required. In contrast, in the present embodiment, feature data extracted from a gesture image of a large gesture (first gesture) of nodding is transferred to input image data, and an element of a small gesture (second gesture) of smiling is added to the generated output image data, so that processed output image data combining multiple elements such as nodding with a smile can be generated, and the number of patterns of gesture images required can be significantly reduced.

[0051] (Example of Hardware Configuration According to the Present Embodiment) The information processing device 10 can be realized, for example, by making a computer execute a program in which the processing contents described in this embodiment are described. Note that this "computer" may be a physical machine or a virtual machine on the cloud. When a virtual machine is used, the "hardware" described here is virtual hardware.

[0052] The above program can be recorded on a computer-readable recording medium (such as a portable memory) and can be stored or distributed. The above program can also be provided via a network such as the Internet or e-mail.

[0053] Fig. 12 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 12 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B.

[0054] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0055] When a program start instruction is received, the memory device 1003 reads out and stores the program from the auxiliary storage device 1002. The CPU 1004 realizes the functions related to the device according to the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard, a mouse, a button, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the calculation result. Note that the above computer may be equipped with a GPU (Graphics Processing Unit) or a TPU (Tensor processing unit) instead of the CPU 1004, or may be equipped with a GPU or a TPU in addition to the CPU 1004. In that case, the processes may be shared and executed such that the GPU or the TPU executes processes that require special calculations and the CPU 1004 executes other processes.

[0056] References [1]:Naoto Nakazato, et.al. "Smart Face: Enhancing Creativity During Video Conferences using Real-time Facial Deformation", CSCW, 2014

[0057] (Summary of the embodiment) This specification describes at least the information processing device, gesture transcription method, and program described in the following items. (Section 1) a feature extraction unit that extracts feature data of a specific gesture from gesture video data showing a video including the gesture; a control unit that receives a transcription request and outputs the feature data corresponding to the transcription request; a feature transfer unit that transfers the feature data corresponding to the transfer request to input video data to generate output video data, and outputs the generated output video data. Information processing device. (Section 2) a first part recognition unit that receives the gesture video data, recognizes the positions, sizes and orientations of the facial parts, and outputs the recognized positions, sizes and orientations of the facial parts; a second part recognition unit that receives the input image data, recognizes the positions, sizes and orientations of facial parts, and outputs the recognized positions, sizes and orientations of facial parts; and an image adjustment unit that adjusts a position, size, or orientation of feature data based on the gesture image data based on a position, size, and orientation of a facial feature of the gesture image data and a position, size, and orientation of a facial feature of the input image data, the feature transfer unit transfers the adjusted feature data to the input video data to generate output video data, and outputs the generated output video data. 2. The information processing device according to claim 1. (Section 3) The gesture image data is data showing an image related to a first movement, A video processing unit is further provided for processing the gesture video data to add a second motion. 3. The information processing device according to claim 1 or 2. (Section 4) The gesture image data is data showing an image related to a first movement, Further comprising a video processing unit that processes the input video data and adds a second operation. 3. The information processing device according to claim 1 or 2. (Section 5) The gesture image data is data showing an image related to a first movement, A video processing unit is further provided for processing the output video data to add a second operation. 3. The information processing device according to claim 1 or 2. (Section 6) 1. A computer implemented method for transcribing gestures, comprising: Extracting feature data of a specific gesture from gesture video data showing a video including the gesture; receiving a transcription request and outputting the feature data corresponding to the transcription request; transferring the feature data corresponding to the transfer request to input video data to generate output video data, and outputting the generated output video data. Gesture transcription method. (Section 7) A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 5.

[0058] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0059] 10. Information processing device 11 Gesture image storage unit 12 Feature Extraction Unit 13 Control section 14 Input video storage section 15 Feature Transfer Section 16 Output video storage section 21 First part recognition unit 22 Second part recognition unit 23 Image adjustment section 31 Video Processing Department 32 Processed gesture image storage unit

Claims

1. a feature extraction unit that extracts feature data of a specific gesture from gesture video data showing a video including the gesture; a control unit that receives a transcription request and outputs the feature data corresponding to the transcription request; a feature transfer unit that transfers the feature data corresponding to the transfer request to input video data to generate output video data, and outputs the generated output video data; The gesture image data is data showing an image related to a first movement representing a large movement, A video processing unit processes the gesture video data and adds a second motion that is smaller than the first motion, The feature extraction unit is extracting the feature data from the processed gesture video data; Information processing device.

2. A feature extraction unit that extracts feature data of a specific gesture from gesture video data showing a video including the gesture; a control unit that receives a transcription request and outputs the feature data corresponding to the transcription request; a feature transfer unit that transfers the feature data corresponding to the transfer request to input video data to generate output video data, and outputs the generated output video data; The gesture image data is data showing an image related to a first movement representing a large movement, a video processing unit that processes the input video data and adds a second motion that is smaller than the first motion; The feature transfer unit includes: transferring the feature data corresponding to the transfer request to the processed input video data to generate output video data, and outputting the generated output video data; Information processing device.

3. A feature extraction unit that extracts feature data of a specific gesture from gesture video data showing a video including the gesture; a control unit that receives a transcription request and outputs the feature data corresponding to the transcription request; a feature transfer unit that transfers the feature data corresponding to the transfer request to input video data to generate output video data, and outputs the generated output video data; The gesture image data is data showing an image related to a first movement representing a large movement, A video processing unit processes the output video data and adds a second motion representing a motion smaller than the first motion. Information processing device.

4. a first part recognition unit that receives the gesture video data, recognizes the positions, sizes and orientations of the facial parts, and outputs the recognized positions, sizes and orientations of the facial parts; a second part recognition unit that receives the input image data, recognizes the positions, sizes and orientations of facial parts, and outputs the recognized positions, sizes and orientations of facial parts; and an image adjustment unit that adjusts a position, size, or orientation of feature data based on the gesture video data based on a position, size, and orientation of a facial feature of the gesture video data and a position, size, and orientation of a facial feature of the input video data, the feature transfer unit transfers the adjusted feature data to the input video data to generate output video data, and outputs the generated output video data. The information processing device according to claim 1 .

5. 1. A computer implemented method for transcribing gestures, comprising: A feature extraction step of extracting feature data of a specific gesture from gesture video data showing a video including the gesture; a control step of receiving a transcription request and outputting the feature data corresponding to the transcription request; a feature transfer step of transferring the feature data corresponding to the transfer request onto input video data to generate output video data, and outputting the generated output video data; The gesture image data is data showing an image related to a first movement representing a large movement, A video processing step of processing the gesture video data to add a second motion representing a motion smaller than the first motion, The feature extraction step includes: extracting the feature data from the processed gesture video data; Gesture transcription method.

6. A computer-implemented method for transcribing gestures, comprising: A feature extraction step of extracting feature data of a specific gesture from gesture video data showing a video including the gesture; a control step of receiving a transcription request and outputting the feature data corresponding to the transcription request; a feature transfer step of transferring the feature data corresponding to the transfer request onto input video data to generate output video data, and outputting the generated output video data; The gesture image data is data showing an image related to a first movement representing a large movement, a video processing step of processing the input video data to add a second motion representing a motion smaller than the first motion; The feature transfer step includes: transferring the feature data corresponding to the transfer request to the processed input video data to generate output video data, and outputting the generated output video data; Gesture transcription method.

7. A computer-implemented method for transcribing gestures, comprising: A feature extraction step of extracting feature data of a specific gesture from gesture video data showing a video including the gesture; a control step of receiving a transcription request and outputting the feature data corresponding to the transcription request; a feature transfer step of transferring the feature data corresponding to the transfer request onto input video data to generate output video data, and outputting the generated output video data; The gesture image data is data showing an image related to a first movement representing a large movement, a video processing step of processing the output video data to add a second motion representing a motion smaller than the first motion, Gesture transcription method.

8. A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 4.

Images