Motion data correction device and program

Abstract

To efficiently correct hand and finger movements included in motion data.SOLUTION: A motion data correction device comprises: an input unit for accepting input of motion data that records sign language expressions of a sign language speaker; a handprint selection unit for selecting handprint data indicating a handprint designated by a user; and a handprint correction unit for correcting a portion of the motion data that indicates hand and finger movements based on the handprint data.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to a motion data correction device and a program.

Background Art

[0002] A motion capture technique for recording the motion of a person or an object in a real space in motion data is known. For example, Patent Document 1 discloses a camera mounting device worn by a mover, which has a plurality of markers installed at each feature point of the mover, a plurality of fish-eye lens cameras, and each transmitter that transmits the captured images of each fish-eye lens camera in time series, and an image processing device that receives the captured images from the plurality of fish-eye lens cameras in time series and converts the positions of the markers in the captured images at each time into three-dimensional coordinates.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, it is difficult to accurately record sign language expressions. Since sign language expressions have different semantic contents due to minute differences in the shapes of fingers, it may be necessary to correct the motion data recorded by motion capture.

[0005] One aspect of the present disclosure aims to efficiently correct finger motions included in motion data in view of the above technical problems.

Means for Solving the Problems

[0006] A motion data correction device according to an aspect of the present disclosure includes an input unit that receives an input of motion data recording a sign language expression of a sign language speaker, a handshape selection unit that selects handshape data indicating a handshape specified by a user, and a handshape correction unit that corrects a part of the motion data indicating finger movements based on the handshape data.

Effect of the Invention

[0007] According to an aspect of the present disclosure, finger movements included in motion data can be efficiently corrected.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0009] Hereinafter, each embodiment of the present disclosure will be described with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0010] [Embodiment] One embodiment of the present disclosure is a sign language CG creation system that creates sign language CG animations using computer graphics (CG). A sign language CG animation is an animation in which a CG character modeled in CG performs sign language expressions.

[0011] In sign language expressions, a very large number of hand shapes are used properly depending on variations such as minute differences in the shape of fingers or the presence or absence of contact between fingers. That is, hand shapes have an important meaning in determining the semantic content conveyed by sign language.

[0012] When creating a sign language CG animation, the sign language expression is recorded as motion data by motion capture, and the sign language CG animation is presented by operating a CG character using the recorded motion data. In order to accurately convey the semantic content in the sign language CG animation, the hand shape of the actor when the sign language expression is recorded by motion capture must be accurately reproduced by the CG character. However, there are two problems that make this difficult.

[0013] The first problem is related to an optical motion capture system (see Patent Document 1) used for recording motion data. In an optical motion capture system, infrared light is irradiated onto reflection markers attached to an actor, and the reflected light from the reflection markers is imaged by an infrared camera to recognize the actor's motion.

[0014] FIG. 1 is a diagram for explaining an optical motion capture system. As shown in FIG. 1, the optical motion capture system includes a number of illuminations L and infrared cameras C arranged around an actor P. The illumination L illuminates the actor P. The infrared camera C is equipped with a strobe capable of irradiating infrared rays. A number of reflection markers are attached to the body of the actor P. The optical motion capture system irradiates infrared rays from the strobe of the infrared camera C onto the actor P and captures the reflected light from the reflection markers with the infrared camera C.

[0015] In an optical motion capture system, if a reflection marker attached to the actor P cannot be imaged by any of the infrared cameras C, the motion of the actor P cannot be accurately recorded. For example, if there is a finger tip grip or an overlap between hands, some of the reflection markers may be physically hidden from the field of view of the infrared camera C.

[0016] FIG. 2 is a diagram for explaining the problems of the prior art. FIG. 2(A) is a diagram showing an example of the state where the actor has an open hand. FIG. 2(B) is a diagram showing an example of the state where the actor has a clenched hand.

[0017] As shown in FIG. 2(A), a number of reflection markers M are attached to the hand H of the actor P. Note that in FIG. 2, only representative reflection markers are labeled. If the hand H is in an open state, many of the reflection markers M attached to the hand H can be imaged by the infrared camera C. On the other hand, as shown in FIG. 2(B), when the hand H is in a state where the fingertips are clenched, the reflection markers M attached to the fingertips cannot be imaged by the infrared camera C, and the accurate shape of the fingers cannot be recorded.

[0018] The second problem is caused by the difference in body structure between the actor and the CG character. It is common for the actor who serves as the basis of the motion and the CG character that reproduces the motion to have different body structures, such as height, arm or finger length. When the body structures of the actor and the CG character are different, the recorded contact of the actor's hand or finger is not accurately reproduced. In this case, the hand or finger of the CG character may be separated or buried.

[0019] FIG. 3 is a diagram for explaining the problems of the prior art. FIG. 3(A) is a diagram showing an example of a state where an actor holds a hand. FIG. 3(B) is a diagram showing an example of a state where the motion data recorded in FIG. 3(A) is reproduced by a CG character. FIG. 3(C) is a diagram showing an example of a state where an actor touches a finger. FIG. 3(D) is a diagram showing an example of a state where the motion data recorded in FIG. 3(C) is reproduced by a CG character.

[0020] As shown in FIG. 3(A), when imaging the state where the hand H1 of the actor P1 grips the fingertip, the reflective marker attached to the fingertip is not imaged. As a result, as shown in FIG. 3(B), the hand H2 of the CG character P2 does not form a sufficiently gripped shape. Further, as shown in FIG. 3(C), when imaging the state where the hand H1 of the actor P1 touches the index finger and the thumb, if the length of the finger of the actor P1 is different from the length of the finger of the CG character P2, as shown in FIG. 3(D), the index finger and the thumb of the CG character P2 do not come into contact.

[0021] Due to the above problems, even if a sign language CG animation is created based on the motion data recorded by the optical motion capture system, it may not result in a sign language expression that accurately conveys the meaning. Therefore, in order to obtain a sign language expression that accurately conveys the meaning, it is necessary to correct the finger movements included in the motion data. However, conventionally, the correction of the motion data has to be performed manually using a dedicated CG tool. Since the correction of the motion data is a highly difficult task, even a skilled CG animator or the like requires a long time to perform the work.

[0022] One embodiment of the present disclosure aims to provide an apparatus capable of efficiently correcting finger movements included in motion data. In one aspect, according to this embodiment, even an operator who is not proficient in dedicated CG tools can easily correct the hand shapes of motion data. In another aspect, sign language CG animations that accurately convey the meaning content can be created at low cost.

[0023] <Overall Configuration> The overall configuration of the sign language CG creation system in this embodiment will be described with reference to FIG. 4. FIG. 4 is a block diagram showing an example of the overall configuration of the sign language CG creation system.

[0024] As shown in FIG. 4, the sign language CG creation system 1000 includes a motion capture 10, a motion data correction device 20, an animation synthesis device 30, and a terminal device 40. The motion capture 10, the motion data correction device 20, the animation synthesis device 30, and the terminal device 40 are connected so as to be capable of data communication via a communication network N such as a LAN (Local Area Network) or the Internet.

[0025] The motion capture 10 is an example of a motion capture system that records the movements of the actor P. The motion capture 10 may be, for example, an optical motion capture system. The motion capture 10 generates motion data recording the sign language expressions of the actor P, who is an example of a sign language speaker.

[0026] The motion data correction device 20 is an example of an information processing device such as a personal computer, a workstation, or a server that corrects motion data. The motion data correction device 20 receives the input of the motion data generated by the motion capture 10 and corrects the motion data according to a correction instruction received from the terminal device 40. The motion data correction device 20 outputs the corrected motion data to the animation synthesis device 30.

[0027] The animation synthesis device 30 is an example of an information processing device such as a personal computer, a workstation, or a server that synthesizes sign language CG animation based on motion data. The animation synthesis device 30 synthesizes sign language CG animation in which sign language expressions are reproduced for a CG character based on the motion data corrected by the motion data correction device 20.

[0028] The terminal device 40 is an example of an information processing terminal such as a personal computer, a smartphone, or a tablet terminal that is operated by a user of the sign language CG creation system 1000. The terminal device 40 presents a correction screen provided by the motion data correction device 20 to the user. The terminal device 40 accepts an operation by the user via the correction screen and transmits a correction instruction for the motion data to the motion data correction device 20.

[0029] Note that the overall configuration of the sign language CG creation system 1000 shown in FIG. 4 is an example, and there can be various system configuration examples according to the application and purpose. For example, a plurality of motion captures 10, motion data correction devices 20, animation synthesis devices 30, or terminal devices 40 may be included in the sign language CG creation system 1000. For example, the motion data correction device 20 or the animation synthesis device 30 may be realized by a plurality of computers, or may be realized as a cloud computing service. The motion data correction device 20 and the animation synthesis device 30 may be realized by a stand-alone computer. The classification of devices such as the motion capture 10, the motion data correction device 20, the animation synthesis device 30, and the terminal device 40 shown in FIG. 4 is an example.

[0030] <Hardware Configuration> The hardware configuration of the sign language CG creation system 1000 in the present embodiment will be described with reference to FIG. 5.

[0031] ≪Computer≫ The motion data correction device 20, the animation composition device 30, and the terminal device 40 are realized by, for example, a computer. FIG. 5 is a block diagram showing an example of the hardware configuration of the computer.

[0032] As shown in FIG. 5, the computer 500 has a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an HDD (Hard Disk Drive) 504, an input device 505, a display device 506, a communication I / F (Interface) 507, and an external I / F 508. The CPU 501, the ROM 502, and the RAM 503 form a so-called computer. Each hardware of the computer 500 is interconnected via a bus line 509. Note that the input device 505 and the display device 506 may be connected to the external I / F 508 for use.

[0033] The CPU 501 is an arithmetic unit that realizes the control and functions of the entire computer 500 by reading programs and data from a storage device such as the ROM 502 or the HDD 504 onto the RAM 503 and executing the processing. The computer 500 may have a GPU (Graphics Processing Unit) in addition to or instead of the CPU 501.

[0034] The ROM 502 is an example of a non-volatile semiconductor memory (storage device) that can hold programs and data even when the power is turned off. The ROM 502 functions as a main storage device that stores various programs, data, etc. necessary for the CPU 501 to execute various programs installed in the HDD 504. Specifically, the ROM 502 stores boot programs such as BIOS (Basic Input / Output System) and EFI (Extensible Firmware Interface) that are executed when the computer 500 is started up, as well as data such as OS (Operating System) settings and network settings.

[0035] The RAM 503 is an example of a volatile semiconductor memory (storage device) in which programs and data are erased when the power is turned off. The RAM 503 is, for example, a DRAM (Dynamic Random Access Memory), an SRAM (Static Random Access Memory), or the like. The RAM 503 provides a work area in which various programs installed in the HDD 504 are expanded when executed by the CPU 501.

[0036] The HDD 504 is an example of a non-volatile storage device that stores programs and data. Programs and data stored in the HDD 504 include an OS, which is basic software that controls the entire computer 500, and applications that provide various functions on the OS. Note that the computer 500 may use a storage device (for example, an SSD: Solid State Drive or the like) that uses a flash memory as a storage medium instead of the HDD 504.

[0037] The input device 505 is a touch panel, operation keys or buttons, a keyboard or a mouse, a microphone that inputs sound data such as voice, etc., which are used by a user to input various signals.

[0038] The display device 506 is composed of a display such as a liquid crystal or an organic EL (Electro-Luminescence) that displays a screen, a speaker that outputs sound data such as voice, etc.

[0039] The communication I / F 507 is an interface for connecting to a communication network and enabling the computer 500 to perform data communication.

[0040] The external I / F 508 is an interface with an external device. Examples of the external device include a drive device 510 and the like.

[0041] The drive device 510 is a device for setting the recording medium 511. The recording medium 511 here includes media that record information optically, electrically, or magnetically, such as CD-ROMs, flexible disks, magneto-optical disks, etc. The recording medium 511 may also include semiconductor memories that record information electrically, such as ROMs and flash memories. Thus, the computer 500 can read and / or write to the recording medium 511 via the external I / F 508.

[0042] Note that various programs installed in the HDD 504 are installed, for example, when the distributed recording medium 511 is set in the drive device 510 connected to the external I / F 508 and the various programs recorded on the recording medium 511 are read by the drive device 510. Alternatively, various programs installed in the HDD 504 may be installed by being downloaded from another network different from the communication network via the communication I / F 507.

[0043] <Functional Configuration> The functional configuration of the motion data correction device 20 in this embodiment will be described with reference to FIG. 6. FIG. 6 is a block diagram showing an example of the functional configuration of the motion data correction device.

[0044] As shown in FIG. 6, the motion data correction device 20 in this embodiment includes an input unit 201, a data storage unit 202, a form storage unit 203, a form selection unit 204, an image acquisition unit 205, a form estimation unit 206, a parameter setting unit 207, a form correction unit 208, and an output unit 209.

[0045] The input unit 201, the form selection unit 204, the image acquisition unit 205, the form estimation unit 206, the parameter setting unit 207, the form correction unit 208, and the output unit 209 are realized, for example, by the processing executed by the CPU 501 on the RAM 503 by the program expanded from the HDD 504 shown in FIG. 5.

[0046] The data storage unit 202 and the signature storage unit 203 are realized by, for example, the RAM 503 or the HDD 504 shown in FIG. 5.

[0047] The input unit 201 receives the input of the motion data generated by the motion capture 10. The motion data records the sign language expression of the actor. The input unit 201 may request the motion capture 10 for the motion data and receive the motion data transmitted from the motion capture 10. The input unit 201 may also receive the motion data input to the input device 505 of the motion data correction device 20.

[0048] The data storage unit 202 stores the motion data received by the input unit 201. The data storage unit 202 also stores the corrected motion data corrected by the signature correction unit 208.

[0049] A plurality of signature templates are stored in advance in the signature storage unit 203. The signature templates may be classified into a plurality of groups in advance. For example, the signature templates may be classified into a group with similar signature shapes or a group with similar semantic contents of the signatures.

[0050] The signature template includes information indicating a predetermined signature (hereinafter also referred to as "signature information") and motion data indicating the signature indicated by the signature information (hereinafter also referred to as "signature data"). As an example, the signature information may be an image depicting the signature. The image depicting the signature may be a real image or an illustration. The signature information may include a plurality of signature images depicting one signature from a plurality of viewpoints.

[0051] The hand shape selection unit 204 selects hand shape data indicating the hand shape specified by the user from the hand shape templates stored in the hand shape storage unit 203. The hand shape selection unit 204 may select hand shape data corresponding to the hand shape information selected by the user on the correction screen. The hand shape selection unit 204 may select hand shape data indicating the hand shape estimated by the hand shape estimation unit 206.

[0052] The image acquisition unit 205 acquires an image in which a hand shape is captured (hereinafter also referred to as a "hand shape image"). The image acquisition unit 205 may acquire a reference image input to the terminal device 40 by the user as a hand shape image. The reference image may be, for example, an image extracted in advance from an image or video obtained by capturing the motion of an actor when motion data is recorded. The image acquisition unit 205 may receive, from the terminal device 40, an image of the user's hand shape captured by an imaging device (e.g., a web camera, etc.) of the terminal device 40 as a hand shape image.

[0053] The hand shape estimation unit 206 estimates the hand shape captured in the hand shape image based on the hand shape image acquired by the image acquisition unit 205. Specifically, the hand shape estimation unit 206 extracts skeleton information from the hand shape image, and estimates the hand shape captured in the hand shape image based on the extracted skeleton information.

[0054] FIG. 7 is a block diagram showing an example of the functional configuration of the hand shape estimation unit. As shown in FIG. 7, the hand shape estimation unit 206 includes a skeleton extraction unit 261 and a hand shape classification unit 262. The hand shape estimation unit 206 takes a hand shape image as input and outputs an estimation result of the hand shape captured in the hand shape image.

[0055] The skeleton extraction unit 261 performs finger joint estimation processing on the hand shape image and extracts skeleton information. As an example, the skeleton extraction unit 261 may perform finger joint estimation processing using MediaPipe Hands or the like. Note that MediaPipe Hands is a library capable of recognizing finger landmarks from images by machine learning.

[0056] Based on the skeletal information extracted by the skeletal extraction unit 261, the hand shape classification unit 262 classifies the hand shape imaged in the hand shape image into one of the hand shape templates. The hand shape classification unit 262 may classify the skeletal information into one of the hand shape templates using, for example, a trained neural network or the like. The hand shape classification unit 262 may calculate a score for each hand shape template and output the hand shape template with the maximum score as the estimation result. The hand shape classification unit 262 may output a predetermined number of hand shape templates with the highest scores as the estimation result.

[0057] Returning to FIG. 6 for explanation. The parameter setting unit 207 sets parameters (hereinafter also referred to as "modification parameters") used for modifying the hand shape. The parameter setting unit 207 may acquire the modification parameters input by the user on the modification screen.

[0058] In the present embodiment, the modification parameters include a frame range, a blend ratio, a number of transition frames, an interpolation method, and the like. The frame range is information indicating the range of frames for modifying the hand shape in the motion data. The blend ratio is the ratio for synthesizing the original motion data and the hand shape data. The number of transition frames is the number of frames for interpolating before and after the frame range in which the hand shape is modified. The interpolation method is the algorithm used for interpolation.

[0059] Based on the hand shape data selected by the hand shape selection unit 204, the hand shape modification unit 208 modifies the part indicating the finger movement in the motion data read from the data storage unit 202. The hand shape modification unit 208 modifies the motion data according to the modification parameters set by the parameter setting unit 207.

[0060] Specifically, the hand shape correction unit 208 corrects, using the hand shape data, a portion indicating finger movements included in the frame range set in the correction parameters. For example, the hand shape correction unit 208 may synthesize the original motion data and the hand shape data at the blend ratio set in the correction parameters. Note that when the blend ratio indicates that the original motion data and the hand shape data are not synthesized, the hand shape correction unit 208 may replace the original motion data with the hand shape data.

[0061] Also, for example, the hand shape correction unit 208 performs frame interpolation within the range of the number of transition frames set in the correction parameters. At this time, the hand shape correction unit 208 performs frame interpolation by the interpolation method set in the correction parameters.

[0062] The hand shape correction unit 208 stores the corrected motion data (hereinafter, also referred to as "corrected motion data") in the data storage unit 202. The hand shape correction unit 208 may update (i.e., overwrite) the original motion data with the corrected motion data, or may store the original motion data in a backup and store the corrected motion data as new motion data.

[0063] The output unit 209 reads out the motion data from the data storage unit 202 according to the user's operation. The output unit 209 may read out the corrected motion data corrected by the hand shape correction unit 208. The output unit 209 may read out the motion data not corrected by the hand shape correction unit 208. The output unit 209 uses the read motion data as the output of the motion data correction device 20. For example, the output unit 209 may transmit the motion data to the animation synthesis device 30.

[0064] <Processing Procedure> A motion data correction method executed by the motion data correction device 20 in this embodiment will be described with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the motion data correction method.

[0065] In step S1, the input unit 201 of the motion data correction device 20 receives the input of the motion data generated by the motion capture 10. Next, the input unit 201 stores the received motion data in the data storage unit 202.

[0066] In step S2, the motion data correction device 20 transmits screen data for displaying a correction screen to the terminal device 40. The motion data read from the data storage unit 202 is embedded in the screen data.

[0067] The terminal device 40 receives the screen data from the motion data correction device 20. Next, the terminal device 40 displays the correction screen on the display device 506 of the terminal device 40 based on the received screen data. The terminal device 40 may synthesize a sign language CG animation based on the motion data and display the sign language CG animation on the correction screen.

[0068] In step S3, the user determines whether the hand shape included in the motion data displayed on the correction screen needs to be corrected. If correction is necessary (YES), the motion data correction device 20 proceeds to step S4. On the other hand, if correction is not necessary (NO), the motion data correction device 20 proceeds to step S12.

[0069] In step S4, the user decides whether to automatically select a hand shape. If the hand shape is to be automatically selected (YES), the motion data correction device 20 proceeds to step S5. On the other hand, if the hand shape is not to be automatically selected (NO), the motion data correction device 20 proceeds to step S8.

[0070] In step S5, the image acquisition unit 205 of the motion data correction device 20 acquires the hand shape image input to the terminal device 40. The image acquisition unit 205 may acquire a reference image as the hand shape image, or may acquire the hand shape image captured by the imaging device of the terminal device 40.

[0071] The terminal device 40 determines whether to acquire the reference image as a hand image or to acquire the hand image captured by the imaging device according to the user's instruction. When the user instructs to acquire the reference image as a hand image, the terminal device 40 transmits the reference image input to the input device 505 to the motion data correction device 20. When the user instructs to acquire the hand image captured by the imaging device, the terminal device 40 captures the user's hand with the imaging device and transmits the obtained hand image to the motion data correction device 20.

[0072] In step S6, the hand shape estimation unit 206 of the motion data correction device 20 estimates the hand shape captured in the hand image based on the hand image acquired in step S5. Specifically, the hand shape estimation unit 206 extracts the skeleton information from the hand image and classifies the hand shape captured in the hand image into one of the hand shape templates based on the extracted skeleton information. The hand shape estimation unit 206 sends the estimation result indicating the estimated hand shape to the hand shape selection unit 204.

[0073] In step S7, the hand shape selection unit 204 of the motion data correction device 20 receives the hand shape estimation result from the hand shape estimation unit 206. Next, the hand shape selection unit 204 selects the hand shape template stored in the hand shape storage unit 203 based on the hand shape estimation result. The hand shape selection unit 204 may display a plurality of hand shape templates indicated in the hand shape estimation result on the correction screen and select the hand shape template specified by the user. Subsequently, the hand shape selection unit 204 reads the hand shape data from the selected hand shape template. Then, the hand shape selection unit 204 sends the read hand shape data to the hand shape correction unit 208.

[0074] In step S8, the hand shape selection unit 204 of the motion data correction device 20 reads the hand shape template from the hand shape storage unit 203. Next, the hand shape selection unit 204 presents the read hand shape template on the correction screen. The hand shape selection unit 204 may classify and present the hand shape templates for each group.

[0075] In step S9, the handshape selection unit 204 of the motion data correction device 20 selects the handshape template specified by the user on the correction screen. Next, the handshape selection unit 204 reads the handshape data from the selected handshape template. Then, the handshape selection unit 204 sends the read handshape data to the handshape correction unit 208.

[0076] In step S10, the parameter setting unit 207 of the motion data correction device 20 acquires the correction parameters input by the user on the correction screen. Next, the parameter setting unit 207 sends the acquired correction parameters to the handshape correction unit 208.

[0077] In step S11, the handshape correction unit 208 of the motion data correction device 20 receives the handshape data from the handshape selection unit 204. Next, the handshape correction unit 208 receives the correction parameters from the parameter setting unit 207. Subsequently, the handshape correction unit 208 corrects the part indicating the finger motion included in the motion data read in step S2 based on the handshape data and the correction parameters. Then, the handshape correction unit 208 stores the corrected motion data in the data storage unit 202.

[0078] In step S12, the output unit 209 of the motion data correction device 20 reads the motion data from the data storage unit 202 according to the user's operation. The output unit 209 uses the read motion data as the output of the motion data correction device 20. The motion data correction device 20 may transmit the output motion data to the animation synthesis device 30. The animation synthesis device 30 can synthesize a sign language CG animation based on the motion data received from the motion data correction device 20.

[0079] <User Interface> The user interface of the motion data correction device 20 in this embodiment will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of a correction screen.

[0080] As shown in FIG. 9, the correction screen 600 has a file menu 601, a file selection area 602, a file display column 603, a pre-correction data display column 604, a post-correction data display column 605, a frame number display column 606, a play button 607, a sequence bar 608, a current frame display section 609, a template display area 610, a right hand correction instruction area 620, and a left hand correction instruction area 630.

[0081] The file menu 601 displays a menu for instructing file operations related to the motion data. File operations include, for example, loading motion data, writing out motion data, specifying a folder to be the operation target, etc. The file menu 601 may display a menu related to automatic selection of hand shapes. The menu related to automatic selection includes, for example, loading a reference image, imaging a hand shape by an imaging device, etc.

[0082] The file selection area 602 is a display area for selecting a file or folder that is the target of file operations. The file selection area 602 displays the current folder in a tree structure in the left half and a list of files stored in the current folder in the right half.

[0083] When the user performs an operation of selecting a file in the file selection area 602, the motion data to be corrected is selected. The operation of selecting a file may be, for example, an operation of clicking or double-clicking a desired file in the right half of the file selection area 602, or an operation of dragging and dropping a desired file to the file display column 603 or the like.

[0084] The file display column 603 displays information indicating the motion data being corrected. The information indicating the motion data is, for example, the file name. The information indicating the motion data may include attribute information of the file. The attribute information may include the length of the file, the update date and time, comments, etc.

[0085] The pre-modification data display field 604 displays a preview of the motion data before modification. For example, the pre-modification data display field 604 may display a sign language CG animation synthesized based on the motion data before modification. The pre-modification data display field 604 displays the pre-modification motion data of the frame number currently displayed in the frame display portion 609 of the sequence bar 608.

[0086] The post-correction data display field 605 displays a preview of the post-correction motion data. For example, the post-correction data display field 605 may display a sign language CG animation synthesized based on the post-correction motion data. The post-correction data display field 605 displays post-correction motion data of the same frame number as the pre-correction data display field 604.

[0087] The frame number display field 606 displays the current frame number. The frame number display field 606 may display the frame number together with the total number of frames. The frame number displayed in the frame number display field 606 is the frame number currently displayed in the frame display portion 609 of the sequence bar 608.

[0088] The play button 607 is a button for starting playback of motion data. When the user presses the play button 607, the motion data displayed in the pre-correction data display field 604 and the post-correction data display field 605 start to be played in conjunction with each other. Note that the play button 607 switches to a stop button during playback. When the user presses the stop button, playback of the motion data displayed in the pre-correction data display field 604 and the post-correction data display field 605 stops.

[0089] The sequence bar 608 displays the time axis of the motion data to be corrected. The current frame display section 609 displays the current frame number in the sequence bar 608. When the playback of the motion data starts, the current frame display section 609 moves in the progress direction of the time axis. When the user selects a desired position on the sequence bar 608, the current frame display section 609 may move to that position and the current frame number may be updated.

[0090] The template display area 610 displays a list of hand templates. The template display area 610 may display a list of groups of hand templates and display the hand templates for each group. When a group is selected by the user, the template display area 610 may display the hand templates included in the selected group.

[0091] The hand image 611 included in the hand template may be displayed in the template display area 610. In FIG. 9, only representative hand images are labeled. When any of the hand images 611 is selected by the user in the template display area 610, the hand template corresponding to the hand image 611 is selected.

[0092] The right hand correction instruction area 620 includes a hand input field 621, frame range input fields 622-623, a blend ratio input field 624, transition frame number input fields 625-626, an interpolation method selection field 627, an apply button 628, and the like.

[0093] The hand specified by the user is input into the hand input field 621. For example, when a hand image 611 is selected in the template display area 610 in a state where the hand input field 621 is selected (for example, a state where a cursor is displayed), information indicating the hand template corresponding to the hand image 611 (for example, the name of the hand) is displayed in the hand input field 621.

[0094] In the frame range input fields 622-623, the frame range to be set in the correction parameters is input. For example, in the frame range input field 622, the frame number at the beginning of the frame range is input. For example, in the frame range input field 623, the frame number at the end of the frame range is input. The frame range may be input to the frame range input fields 622-623 by selecting the frame number with the sequence bar 608.

[0095] In the blend ratio input field 624, the ratio for synthesizing the motion data before correction and the gesture data is input. For example, the blend ratio may take a value from 0 to 1, and the closer it is to 0, the more components of the original motion data, and the closer it is to 1, the more components of the gesture data.

[0096] In the transition frame number input fields 625-626, the number of transition frames to be set in the correction parameters is input. In the transition frame number input field 625, the number of frames to be interpolated in front of the frame range is input. In the transition frame number input field 626, the number of frames to be interpolated behind the frame range is input.

[0097] The interpolation method selection field 627 displays options for the interpolation method and accepts the user's selection of the interpolation method. Examples of the options for the interpolation method include linear interpolation, spline interpolation, and autumn interval interpolation.

[0098] The apply button 628 is a button for correcting the motion data of the right hand part with the gesture input in the gesture input field 621 in the current frame. When the user presses the apply button 628, the motion data in the frame range set in the frame range input fields 622-623 is corrected according to the correction parameters set in the blend ratio input field 624, the transition frame number input fields 625-626, and the interpolation method selection field 627. After the correction is performed, the sign language CG animation synthesized with the corrected motion data is displayed in the corrected data display field 605.

[0099] The apply button 628 may store the corrected motion data in a temporary storage area and enable an undo operation. In this case, when a menu for writing out the motion data is selected in the file menu 601, corrected motion data may be generated and stored in the data storage unit 202.

[0100] The left - hand correction instruction area 630 includes a hand - shape input field 631, frame - range input fields 632 - 633, a blend - ratio input field 634, transition - frame - number input fields 635 - 636, an interpolation - method selection field 627, an apply button 638, and the like. Each input field included in the left - hand correction instruction area 630 is the same as each input field included in the right - hand correction instruction area 620, except that the hand (right hand or left hand) to be corrected is different.

[0101] <Effects of the Embodiment> The motion - data correction device 20 in the present embodiment corrects a part indicating finger movement in the motion data recording the sign language expression of a sign - language speaker based on hand - shape data indicating a hand - shape specified by a user. In one aspect, according to the present embodiment, since the user only needs to perform an operation of specifying a hand - shape, the hand - shape of the motion data can be efficiently corrected.

[0102] The motion - data correction device 20 may select hand - shape data indicating a hand - shape selected by a user from a plurality of predetermined hand - shape templates. The motion - data correction device 20 may also select hand - shape data indicating a hand - shape estimated from an image input by a user. According to the present embodiment, the target hand - shape can be specified from a large number of hand - shapes by a simple operation.

[0103] The motion data correction device 20 may replace the part indicating the finger motion with hand shape data. The motion data correction device 20 may replace the part indicating the finger motion with hand shape data within the frame range specified by the user. The motion data correction device 20 may synthesize hand shape data with the part indicating the finger motion. The motion data correction device 20 may synthesize hand shape data with the part indicating the finger motion at the frame range and ratio specified by the user. According to this embodiment, the user can appropriately correct the motion data with a desired correction method.

[0104] The motion data correction device 20 may interpolate before and after the part indicating the finger motion. The motion data correction device 20 may interpolate before and after the part indicating the finger motion with the number of frames and interpolation method specified by the user. According to this embodiment, it is possible to harmonize the part where the hand shape is corrected and the part that is not corrected.

[0105] [Supplementary Explanation] Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in this specification refers to a processor programmed to execute each function by software, such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) implemented by an electronic circuit, or an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array) designed to execute each function described above, and devices such as conventional circuit modules.

[0106] As described above in detail regarding the embodiments of the present disclosure, the embodiments disclosed this time are illustrative in all respects and not restrictive. The embodiments can be modified and improved in various forms without departing from the scope and gist of the appended claims. Matters described in the above plurality of embodiments can also adopt other configurations and can be combined within a non - conflicting range.

Explanation of Signs

[0107] 10 Motion capture 20 Motion data correction device 30 Animation synthesis device 40 Terminal device 201 Input unit 202 Data storage unit 203 Handprint storage unit 204 Handprint selection unit 205 Image acquisition unit 206 Handprint estimation unit 207 Parameter setting unit 208 Handprint correction unit 209 Output unit 1000 Sign language CG creation system

Claims

1. An input unit configured to receive an input of motion data recording a sign language expression of a sign language speaker, A handshape selection unit configured to select handshape data indicating a handshape specified by a user, A handshape correction unit configured to correct a part indicating a finger movement in the motion data based on the handshape data, A motion data correction device comprising the above.

2. The motion data correction device according to claim 1, wherein the handshape selection unit is configured to select the handshape data indicating the handshape selected by the user from a plurality of predetermined handshape templates. Motion data correction device.

3. The motion data correction device according to claim 1, wherein the handshape selection unit is configured to select the handshape data indicating the handshape estimated from an image input by the user. Motion data correction device.

4. The motion data correction device according to any one of claims 1 to 3, wherein the handshape correction unit is configured to replace the part indicating the finger movement with the handshape data. Motion data correction device.

5. The motion data correction device according to claim 4, wherein the handshape correction unit is configured to replace the part indicating the finger movement with the handshape data within a frame range specified by the user. Motion data correction device.

6. The motion data correction device according to any one of claims 1 to 3, wherein the handshape correction unit is configured to synthesize the handshape data with the part indicating the finger movement. Motion data correction device.

7. The motion data correction device according to claim 6, wherein the handshape correction unit is configured to synthesize the handshape data with the part indicating the finger movement within a frame range and ratio specified by the user. Motion data correction device.

8. The motion data correction device according to any one of claims 1 to 3, wherein the handshape correction unit is configured to interpolate before and after the part indicating the finger movement. Motion data correction device.

9. The motion data correction device according to claim 8, wherein the handshape correction unit is configured to interpolate before and after the part indicating the finger movement with a number of frames and an interpolation method specified by the user. Motion data correction device.

10. A program for causing a computer to function as the motion data correction device according to any one of Claims 1 to 3.

Images