Animation Production System

Abstract

To allow for animation filming in a virtual space.SOLUTION: An animation creation system includes: a virtual camera which images characters arranged in a virtual space; a user input detection unit which detects an input of a user from at least one of a controller and a head-mounted display worn by the user; a character control unit which controls motion of the characters in accordance with the input; and a preset storage unit which stores expressions of the characters. The character control unit sets expressions corresponding to an input which does not affect the motion of the characters to the characters.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an animation production system. [Background technology]

[0002] A virtual camera is arranged in a virtual space (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-146651 A Summary of the Invention [Problem to be solved by the invention]

[0004] There have been no attempts to film animation within a virtual space.

[0005] The present invention has been made in view of the above background, and has an object to provide a technique that enables animation to be shot within a virtual space. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problems is an animation production system comprising: a virtual camera that photographs a character placed in a virtual space; a user input detection unit that detects user input from at least one of a head-mounted display and a controller worn by the user; a character control unit that controls the movement of the character in response to the input; and a preset storage unit that stores facial expressions of the character, wherein the character control unit sets the facial expression to the character that corresponds to the input that does not affect the movement of the character.

[0007] Other problems and solutions disclosed in this application will be made clear in the description of the preferred embodiments of the invention and the drawings. Effect of the Invention

[0008] According to the present invention, animation can be shot in a virtual space. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram showing an example of a virtual space displayed on a head mounted display (HMD) worn by a user in the animation production system 300 of this embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of the overall configuration of an animation production system 300 according to an embodiment of the present invention. [Diagram 3] 2 is a diagram showing a schematic external view of an HMD 110 according to the present embodiment. FIG. [Figure 4] FIG. 2 is a diagram illustrating an example of the functional configuration of an HMD 110 according to the present embodiment. [Diagram 5] FIG. 2 is a diagram showing a schematic external view of a controller 210 according to the present embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of the functional configuration of a controller 210 according to the present embodiment. [Figure 7] FIG. 2 is a diagram showing the functional configuration of an image generating device 310 according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will be described below with reference to the preferred embodiments thereof.

[0011] [Item 1] a virtual camera for photographing a character placed in a virtual space; a user input detection unit that detects an input from the user through at least one of a head mounted display and a controller worn by the user; a character control unit that controls an action of the character in response to the input; A preset storage unit for storing the facial expressions of the character; Equipped with the character control unit sets, to the character, the facial expression corresponding to the input that does not affect the action of the character; An animation production system that features: [Item 2] The animation production system according to item 1, the character control unit registers the facial expression of the character created by the movement of the character in the preset storage unit; An animation production system that features: [Item 3] The animation production system according to item 1, the character control unit sets the facial expression to the character in response to an instruction from a user different from the user; An animation production system that features:

[0012] A specific example of an animation production system 300 according to an embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims. In the following description, the same elements in the description of the drawings will be given the same reference numerals, and duplicated descriptions will be omitted.

[0013] <Summary> FIG. 1 is a diagram showing an example of a virtual space displayed on a head mounted display (HMD) worn by a user in an animation production system 300 of this embodiment. In the animation production system 300 of this embodiment, a virtual character 4 and a virtual camera 3 are placed in the virtual space 1, and the character 4 is photographed using the camera 3. Also, a cameraman 2 (photographer character) is placed in the virtual space 1, and the cameraman 2 virtually operates the camera 3. In the animation production system 300 of this embodiment, a user creates animation while overlooking the virtual space 1 from a TPV (Third Person View), photographing the character 4 from an FPV (First Person View) as the cameraman 2, and acting as the character 4 from the FPV. Also, multiple characters 4 (characters 4-1 and 4-2 in the example of FIG. 1) can be placed in the virtual space 1, and the user can act while possessing the character 4. When multiple characters 4 are arranged, the user can also switch the object possessed by each character 4 (for example, characters 4-1 and 4-2). That is, in the animation production system 300 of this embodiment, one person can play multiple roles. Also, since the user can shoot while virtually operating the camera 2 as the cameraman 2, it is possible to realize natural camera work and enrich the expression of the video to be shot.

[0014] <Overall composition> FIG. 2 is a diagram showing an example of the overall configuration of an animation production system 300 according to an embodiment of the present invention. The animation production system 300 can be configured to include, for example, the HMD 110, the controller 210, and an image generation device 310 that functions as a host computer. The image generation device 310 can include a display device 311 such as a display, and an input device 312 such as a keyboard, a mouse, or a touch panel. The animation production system 300 can also include an infrared camera (not shown) for detecting the position, direction, and inclination of the HMD 110 and the controller 210. These devices can be connected to each other by wired or wireless means. For example, each device can be provided with a USB port and communication can be established by connecting with a cable, or communication can be established by wired or wireless means such as HDMI, wired LAN, infrared, Bluetooth (registered trademark), WiFi (registered trademark), etc. The image generation device 310 can be any device having a calculation processing function, such as a PC, a game machine, or a mobile communication terminal.

[0015] <hmd110> Fig. 3 is a diagram showing a schematic external view of the HMD 110 according to the present embodiment, and Fig. 4 is a diagram showing an example of the functional configuration of the HMD 110 according to the present embodiment.

[0016] The HMD 110 is attached to the user's head and includes a display panel 120 arranged in front of the user's left and right eyes. The display panel 120 may be an optically transmissive or non-transmissive display, but in this embodiment, a non-transmissive display panel capable of providing a greater sense of immersion is illustrated. The display panel 120 displays a left-eye image and a right-eye image, and a stereoscopic image can be provided to the user by utilizing the parallax between the two eyes. If it is possible to display a left-eye image and a right-eye image, it is possible to provide a left-eye display and a right-eye display separately, or to provide an integrated display for the left and right eyes.

[0017] The housing 130 of the HMD 110 includes a sensor 140. The sensor 140 may include, for example, a magnetic sensor, an acceleration sensor, or a gyro sensor, or a combination of these, though not shown, in order to detect movements such as the orientation and tilt of the user's head. When the vertical direction of the user's head is the Y axis, an axis that connects the center of the display panel 120 and the user and corresponds to the front-back direction of the user among the axes perpendicular to the Y axis is the Z axis, and an axis that is perpendicular to the Y axis and corresponds to the left-right direction of the user is the X axis, the sensor 140 can detect a rotation angle around the X axis (so-called pitch angle), a rotation angle around the Y axis (so-called yaw angle), and a rotation angle around the Z axis (so-called roll angle).

[0018] Instead of or in addition to the sensor 140, the housing 130 of the HMD 110 may be provided with a plurality of light sources 150 (e.g., infrared LEDs, visible LEDs). A camera (e.g., infrared camera, visible light camera) installed outside the HMD 110 (e.g., indoors, etc.) detects these light sources, thereby making it possible to detect the position, orientation, and inclination of the HMD 110 in a specific space. Alternatively, for the same purpose, the HMD 110 may be provided with a camera for detecting the light sources installed in the housing 130 of the HMD 110.

[0019] The housing 130 of the HMD 110 may also be equipped with an eye tracking sensor. The eye tracking sensor is used to detect the gaze direction and gaze point of the left and right eyes of the user. There are various possible methods for the eye tracking sensor, but for example, a method is possible in which the position of reflected light on the cornea created by irradiating the left and right eyes with weak infrared light is used as a reference point, the gaze direction is detected based on the position of the pupil relative to the position of the reflected light, and the intersection of the gaze directions of the left and right eyes is detected as the gaze point.

[0020] <Controller 210> Fig. 5 is a diagram showing a schematic external appearance of the controller 210 according to this embodiment. Fig. 6 is a diagram showing an example of the functional configuration of the controller 210 according to this embodiment.

[0021] The controller 210 can support the user in performing a predetermined input in a virtual space. The controller 210 can be configured as a set of a left-hand controller 220 and a right-hand controller 230. The left-hand controller 220 and the right-hand controller 230 can each have an operation trigger button 240, an infrared LED 250, a sensor 260, a joystick 270, and a menu button 280.

[0022] Operation trigger buttons 240 are arranged as 240a and 240b at positions that anticipate an operation of pulling the trigger with the middle finger and index finger when gripping grip 235 of controller 210. Frame 245 formed in a ring shape downward from both sides of controller 210 is provided with multiple infrared LEDs 250, and by detecting the positions of these infrared LEDs with a camera (not shown) provided outside the controller, the position, orientation, and tilt of controller 210 in a specific space can be detected.

[0023] The controller 210 may also include a sensor 260 for detecting movements such as the direction and tilt of the controller 210. Although not shown, the sensor 260 may include, for example, a magnetic sensor, an acceleration sensor, or a gyro sensor, or a combination of these. Furthermore, the controller 210 may include a joystick 270 and a menu button 280 on the upper surface thereof. The joystick 270 may be moved in 360 degrees around a reference point, and is assumed to be operated by the thumb when the grip 235 of the controller 210 is held. The menu button 280 is also assumed to be operated by the thumb in the same manner. Furthermore, the controller 210 may also include a vibrator (not shown) for applying vibrations to the hand of the user operating the controller 210. The controller 210 has an input / output unit and a communication unit in order to output information such as the contents of a user's input via a button or joystick, and the position, direction, and tilt of the controller 210 via a sensor, etc., and to receive information from a host computer.

[0024] Depending on whether the user is holding controller 210 and operating various buttons and joysticks, and on information detected by infrared LEDs and sensors, the system determines the movement and posture of the user's hands and can simulate the display and movement of the user's hands in the virtual space.

[0025] <Image Generating Device 310> 7 is a diagram showing a functional configuration of an image generating device 310 according to this embodiment. As the image generating device 310, a device such as a PC, a game machine, or a mobile communication terminal having a function of storing user input information transmitted from the HMD 110 or the controller 210 and information on the user's head movement or controller movement or operation acquired by a sensor or the like, performing a predetermined calculation process, and generating an image can be used. The image generating device 310 can include, for example, an input / output unit 320 for establishing a wired connection with peripheral devices such as the HMD 110 or the controller 210, and can include a communication unit 330 for establishing a wireless connection such as infrared, Bluetooth (registered trademark), or WiFi (registered trademark). Information on the movement of the user's head and the movement and operation of the controller 210 received from the HMD 110, the controller 210, and / or the input device 311 via the input / output unit 320 and / or the communication unit 330 is detected as input contents including the user's position, gaze, posture, and other movements, speech, operation, etc., in the control unit 340, and processing such as controlling the character 4 and generating an image is performed by executing a control program stored in the storage unit 350 according to the user's input contents. The user input detection unit 410 can also accept input from an input device 312 such as a keyboard or a mouse. The control unit 340 can be configured with a CPU, but by further providing a GPU specialized for image processing, information processing and image processing can be distributed to improve the efficiency of the overall processing. The image generation device 310 can also communicate with other calculation processing devices and have the other calculation processing devices share the information processing and image processing.

[0026] The control unit 340 includes a user input detection unit 410 that detects information on the user's head movement, the user's speech, and the movement and operation of the controller received from the HMD 110 and / or the controller 210, a character control unit 420 that executes a control program stored in a control program storage unit 460 for a character 4 previously stored in a character data storage unit 450 of the storage unit 350, a camera control unit 440 that controls a virtual camera 3 arranged in the virtual space 1 according to the character control, and an image generation unit 430 that generates an image of the virtual space 1 captured by the camera 3 based on the character control. Here, the control of the movement of the character 4 is realized by converting information such as the direction and inclination of the user's head and the movement of the hands detected via the HMD 110 and the controller 210 into the movement of each part of a bone structure created in accordance with the movement and constraints of the joints of the human body, and applying the movement of the bone structure by associating the bone structure with the previously stored character data. The camera 3 is controlled, for example, by changing various settings for the camera 3 (for example, the position of the camera 3 in the virtual space 1, the line of sight direction of the camera 3, the focal position, zoom, etc.) in response to the movement of the hand of the character 4. The image generating unit 430 registers action data representing the movement of the character 4 controlled by the character control unit 420 and the movement (operation) of the camera 3 controlled by the camera control unit 440 in the image data storage unit 470, and can generate an image in which the movement of the character 4 is virtually photographed by the camera 3. The image generating unit 430 can display the image on the display unit 61 of the control panel 6 arranged in the virtual space 1, and can also display the image on the display device 311. Furthermore, the image generating unit 430 can display the generated image on a display unit (not shown) equipped in the camera 3.

[0027] The storage unit 350 stores information related to the character 4, such as the attributes of the character 4, in addition to the image data of the character 4 in the character data storage unit 450. The control program storage unit 460 stores a program for controlling the movement and facial expression of the character 4 in the virtual space and for controlling objects such as the camera 3. The image data storage unit 470 stores images generated by the image generation unit 430. In this embodiment, the images stored in the image data storage unit 470 can be action data for generating a moving image. The action data can include, for example, 3D data for displaying the character 4 in the virtual space 1, pose data for specifying the state of the bone structure of the 3D data, and motion data for specifying the movement of the bone structure. The image generation unit 430 can also create (render) a moving image based on the action data, and register the moving image data resulting from the rendering in the image data storage unit 470.

[0028] <Video post-effects> The control unit 340 also includes an effect processing unit 480 that applies effects to the moving image. The effect processing unit 480 does not perform a simulation process of rendering in consideration of a light source in the virtual space 1, but applies effects to pixels of the moving image (two-dimensional moving image) generated by the image generating unit 430. The effect can employ any process that can be applied to any two-dimensional moving image. The effects can include, for example, a bloom effect, a depth of field effect, a vignetting effect, a color grading effect, a color curve effect, a diffusion filter, and the like. Parameters are set for the effect. The parameters can be input from an input device 312 such as a keyboard or a mouse.

[0029] The effect processor 480 can receive the designation of the type of effect, the parameters of the designated effect, and apply effect processing to the moving image based on the received parameters. For example, when a flare effect is selected, the effect processor 480 can receive the center position, size, color, and the like of the flare as parameters. The position to which the effect is applied can also be set by the user using the input device 312 such as a mouse. Note that the selection of the effect and the input of the parameters may be performed within the virtual space 1. For example, the control panel 6 can be provided with an effect selection section and a parameter setting section.

[0030] <Storing presets> The storage unit 350 can further include a preset storage unit 451. In the preset storage unit 451, various preset parameters can be recorded.

[0031] The preset storage unit 451 can store parameters related to post-effects applied by the effect processing unit 480 as presets. The effect processing unit 480 can accept a designation from the user and apply effect processing using the parameters of the designated preset. It is also possible to call up a preset stored in the preset storage unit 451 and apply it after making fine adjustments. In this case, the effect processing unit 480 can display the preset parameters on a control panel or the like, for example, and perform effect processing after accepting adjustments from the user for each parameter.

[0032] Furthermore, the preset storage unit 451 can store the facial expressions of the character 4 as presets. For example, the preset storage unit 451 can store parameters for transforming the face model of the character 4 as presets for each of a plurality of facial expressions. The character control unit 420 can change the facial expression of the character 4 by, for example, reading different facial expressions from the preset storage unit 451 according to the buttons and movements of the controller 210 and applying them to the character 4. Therefore, since the user can set the facial expression using the controller 210 or the like while possessing the character 4 and performing an act, for example, it is not necessary to recognize the face using the HMD 110, and the user can concentrate only on the body movement of the character 4. For example, the user may create the facial expressions of the character 4 in the virtual space 1 by operating the HMD 110 or the controller 210, and store the created facial expressions in the preset storage unit 451. Furthermore, when modeling the character 4 in advance, a plurality of facial expressions may be created and stored in the preset storage unit 451.

[0033] The application of the preset facial expressions to the character 4 may be performed by a user other than the user who is possessed by the character 4 and performing the role. In this case, the other user can select a facial expression while watching the movements of the character 4, and can also specify the timing to apply the facial expression to the character 4.

[0034] Furthermore, the preset storage unit 451 stores presets of body movements (actions) of the character 4, and the character control unit 420 can apply different movements according to, for example, the buttons and movements of the controller 210. For example, by pressing a certain button on the controller 210, the character can assume a predetermined pose or perform a preset motion. Therefore, for example, for a character 4 with a specific set pose or set action, the user who is the performer can make the character 4 assume a predetermined posture or perform a predetermined movement without having to take the pose or action every time.

[0035] Moreover, the preset actions can be not only short actions but also one-shot actions. Multiple takes of an action can be recorded.

[0036] As described above, according to the animation production system 300 of this embodiment, the user can operate the camera 3 as a cameraman 2 to shoot moving images in the virtual space 1. Therefore, since shooting can be performed by operating the camera 3 in the same way as in the real world, natural camera work can be realized, and richer expression can be imparted to the animated moving image.

[0037] Furthermore, according to the animation production system 300 of this embodiment, the video resulting from applying effect processing to the image generated by the image generation unit 430 can be displayed on the display device 311 and / or the display unit 61 in the virtual space 1. This allows the user to produce animation while visually checking the final result including the application of effects.

[0038] Furthermore, in the animation production system 300 of this embodiment, the display device 311 and input device 312 arranged outside the virtual space can be used to select the type of effect to be processed by the effect processor 480 and to accept input of effect parameters. Therefore, the producer who operates the character 4 or camera 3 and the producer who adjusts the effects can work independently. In other words, animation can be produced by dividing the producer responsible for the acting and camera work from the producer responsible for adjusting the final quality of the video.

[0039] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.

[0040] For example, in the present embodiment, the image generating device 310 is one computer, but this is not limiting, and the HMD 110 or the controller 210 may be provided with all or part of the functions of the image generating device 310. Also, some of the functions of the image generating device 310 may be provided in another computer communicably connected to the image generating device 310.

[0041] In addition, in this embodiment, a virtual space using virtual reality (VR) is assumed, but this is not limited to this, and the animation production system 300 of this embodiment can be directly applied to an augmented reality (AR) space or a mixed reality (MR) space.

[0042] In this embodiment, the grid 31 is divided into three parts vertically and horizontally, but it is possible to display dividing lines 32 that divide the grid into any number of parts.

[0043] In this embodiment, the user possessed by the cameraman 2 or the character 4 performs an operation to adjust the position of the grid 31, but this is not limiting, and the position of the grid 31 may be adjusted in a state where the virtual space 1 is overlooked (a state where the user is not possessed by the cameraman 2 or the character 4). In this case, the user may grip and move the grid 31. [Explanation of symbols]

[0044] 1. Virtual Space 2. Photographer 3. Camera 4. Characters 6. Control Panel 7 Screens 31 Grid 32 Partition Line 61 Display section 71 Display section 72 Play button 110 HMD 120 Display Panel 130 Housing 140 Sensors 150 light source 210 Controller 220 Left-handed controller 230 Right Hand Controller 235 Grip 240 Trigger Button 250 Infrared LEDs 260 Sensors 270 Joystick 280 Menu button 300 Animation Production System 310 Image generating device 311 Display device 312 Input Device 320 Input / output section 330 Communications Department 340 Control Unit 350 Storage section 410 User Input Detection Unit 420 Character Control Unit 430 Image Generation Unit 440 Camera Control Unit 450 Character data storage section 451 Preset storage section 460 Program Storage Unit 470 Image data storage unit 490 Video playback section

Claims

[Claim 1] a virtual camera for photographing a character placed in a virtual space; a user input detection unit that detects an input from the user through at least one of a head mounted display and a controller worn by the user; a character control unit that controls an action of the character in response to the input; A preset storage unit for storing the facial expressions of the character; Equipped with the character control unit sets the facial expression to the character in response to an instruction from a user different from the user; An animation production system that features:

Images