Animation Production System

Abstract

To capture an animation in a virtual space.SOLUTION: An animation production system includes a virtual camera to be disposed in a virtual space, a user input detecting section detecting an input of a user from at least one of a head-mounted display mounted on the user and a controller, and a control panel to be displayed in response to the input, for activating a function relating to animation production.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an animation production system. [Background technology]

[0002] A virtual camera is arranged in a virtual space (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-146651 A Summary of the Invention [Problem to be solved by the invention]

[0004] There have been no attempts to film animation within a virtual space.

[0005] The present invention has been made in view of the above background, and has an object to provide a technique that enables animation to be shot within a virtual space. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problem is an animation production method comprising the steps of: placing a virtual camera in a virtual space; placing one or more objects in the virtual space; a user input detection unit that detects user input from at least one of a head-mounted display and a controller worn by the user; accepting selection of at least one of the objects in response to the input; and deleting the object from the virtual space in response to the input.

[0007] Other problems and solutions disclosed in this application will be made clear in the description of the preferred embodiments of the invention and the drawings. Effect of the Invention

[0008] According to the present invention, animation can be shot in a virtual space. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram showing an example of a virtual space displayed on a head mounted display (HMD) worn by a user in the animation production system 300 of this embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of the overall configuration of an animation production system 300 according to an embodiment of the present invention. [Diagram 3] 2 is a diagram showing a schematic external view of an HMD 110 according to the present embodiment. FIG. [Figure 4] FIG. 2 is a diagram illustrating an example of the functional configuration of an HMD 110 according to the present embodiment. [Diagram 5] FIG. 2 is a diagram showing a schematic external view of a controller 210 according to the present embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of the functional configuration of a controller 210 according to the present embodiment. [Figure 7] FIG. 2 is a diagram showing the functional configuration of an image generating device 310 according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will be described below with reference to the preferred embodiments thereof.

[0011] [Item 1] Positioning a virtual camera in a virtual space; placing one or more objects within the virtual space; a user input detection unit that detects an input from the user through at least one of a head mounted display and a controller worn by the user; accepting a selection of at least one of the objects in response to the input; removing the object from the virtual space in response to the input; An animation production method comprising: [Item 2] The animation production method according to item 1, a step of displaying, in the virtual space, a list of the objects arranged in the virtual space; receiving, in response to the input, a selection of at least one of the objects from the list; An animation production method characterized by: [Item 3] The animation production method according to item 1, receiving a designation of the object to be locked; In the step of accepting the selection, the locked object is made unselectable; An animation production method characterized by:

[0012] A specific example of an animation production system 300 according to an embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims. In the following description, the same elements in the description of the drawings will be given the same reference numerals, and duplicated descriptions will be omitted.

[0013] <Summary> 1 is a diagram showing an example of a virtual space displayed on a head mounted display (HMD) worn by a user in an animation production system 300 of this embodiment. The animation production system 300 of this embodiment aims to create animation by placing a virtual character 4 and a virtual camera 3 in a virtual space 1 and photographing the character 4 with the camera 3.

[0014] A cameraman 2 (photographer character) is placed in the virtual space 1, and the cameraman 2 virtually operates a camera 3. In the animation production system 300 of this embodiment, the user creates animation by placing a character 4 and a camera 3 while overlooking the virtual space 1 from a TPV (Third Person View), taking pictures of the character 4 from an FPV (First Person View) as the cameraman 2, and acting as the character 4 from the FPV, as shown in FIG. 1. A plurality of characters 4 (characters 4-1 and 4-2 in the example of FIG. 1) can be placed in the virtual space 1, and the user can perform acting while possessing (setting an operation target) the selected character 4. When a plurality of characters 4 are placed, the user can switch the possessed target (operation target) to one of the characters 4 (for example, characters 4-1 and 4-2).

[0015] In this way, in the animation production system 300 of this embodiment, one person can play multiple roles. Also, since the person can shoot while virtually operating the camera 2 as the cameraman 2, it becomes possible to realize natural camera work and enrich the expression of the moving image to be shot.

[0016] <Overall composition> FIG. 2 is a diagram showing an example of the overall configuration of an animation production system 300 according to an embodiment of the present invention. The animation production system 300 can be configured to include, for example, the HMD 110, the controller 210, and an image generation device 310 functioning as a host computer. An infrared camera (not shown) for detecting the position, orientation, inclination, etc. of the HMD 110 and the controller 210 can also be added to the animation production system 300. These devices can be connected to each other by wired or wireless means. For example, each device can be provided with a USB port and communication can be established by connecting with a cable, or communication can be established by wired or wireless means such as HDMI, wired LAN, infrared, Bluetooth (registered trademark), WiFi (registered trademark), etc. The image generation device 310 may be any device having a calculation processing function, such as a PC, a game machine, or a mobile communication terminal.

[0017] <hmd110> Fig. 3 is a diagram showing a schematic external view of the HMD 110 according to the present embodiment, and Fig. 4 is a diagram showing an example of the functional configuration of the HMD 110 according to the present embodiment.

[0018] The HMD 110 is attached to the user's head and includes a display panel 120 arranged in front of the user's left and right eyes. The display panel 120 may be an optically transmissive or non-transmissive display, but in this embodiment, a non-transmissive display panel capable of providing a greater sense of immersion is illustrated. The display panel 120 displays a left-eye image and a right-eye image, and a stereoscopic image can be provided to the user by utilizing the parallax between the two eyes. If it is possible to display a left-eye image and a right-eye image, it is possible to provide a left-eye display and a right-eye display separately, or to provide an integrated display for the left and right eyes.

[0019] The housing 130 of the HMD 110 includes a sensor 140. The sensor 140 may include, for example, a magnetic sensor, an acceleration sensor, or a gyro sensor, or a combination of these, though not shown, in order to detect movements such as the orientation and tilt of the user's head. When the vertical direction of the user's head is the Y axis, an axis that connects the center of the display panel 120 and the user and corresponds to the front-back direction of the user among the axes perpendicular to the Y axis is the Z axis, and an axis that is perpendicular to the Y axis and corresponds to the left-right direction of the user is the X axis, the sensor 140 can detect a rotation angle around the X axis (so-called pitch angle), a rotation angle around the Y axis (so-called yaw angle), and a rotation angle around the Z axis (so-called roll angle).

[0020] Instead of or in addition to the sensor 140, the housing 130 of the HMD 110 may be provided with a plurality of light sources 150 (e.g., infrared LEDs, visible LEDs). A camera (e.g., infrared camera, visible light camera) installed outside the HMD 110 (e.g., indoors, etc.) detects these light sources, thereby making it possible to detect the position, orientation, and inclination of the HMD 110 in a specific space. Alternatively, for the same purpose, the HMD 110 may be provided with a camera for detecting the light sources installed in the housing 130 of the HMD 110.

[0021] The housing 130 of the HMD 110 may also be equipped with an eye tracking sensor. The eye tracking sensor is used to detect the gaze direction and gaze point of the left and right eyes of the user. There are various possible methods for the eye tracking sensor, but for example, a method is possible in which the position of reflected light on the cornea created by irradiating the left and right eyes with weak infrared light is used as a reference point, the gaze direction is detected based on the position of the pupil relative to the position of the reflected light, and the intersection of the gaze directions of the left and right eyes is detected as the gaze point.

[0022] <Controller 210> Fig. 5 is a diagram showing a schematic external appearance of the controller 210 according to this embodiment. Fig. 6 is a diagram showing an example of the functional configuration of the controller 210 according to this embodiment.

[0023] The controller 210 can support the user in performing a predetermined input in a virtual space. The controller 210 can be configured as a set of a left-hand controller 220 and a right-hand controller 230. The left-hand controller 220 and the right-hand controller 230 can each have an operation trigger button 240, an infrared LED 250, a sensor 260, a joystick 270, and a menu button 280.

[0024] Operation trigger buttons 240 are arranged as 240a and 240b at positions that anticipate an operation of pulling the trigger with the middle finger and index finger when gripping grip 235 of controller 210. Frame 245 formed in a ring shape downward from both sides of controller 210 is provided with multiple infrared LEDs 250, and by detecting the positions of these infrared LEDs with a camera (not shown) provided outside the controller, the position, orientation, and tilt of controller 210 in a specific space can be detected.

[0025] The controller 210 may also include a sensor 260 for detecting movements such as the direction and tilt of the controller 210. Although not shown, the sensor 260 may include, for example, a magnetic sensor, an acceleration sensor, or a gyro sensor, or a combination of these. Furthermore, the controller 210 may include a joystick 270 and a menu button 280 on the upper surface thereof. The joystick 270 may be moved in 360 degrees around a reference point, and is assumed to be operated by the thumb when the grip 235 of the controller 210 is held. The menu button 280 is also assumed to be operated by the thumb in the same manner. Furthermore, the controller 210 may also include a vibrator (not shown) for applying vibrations to the hand of the user operating the controller 210. The controller 210 has an input / output unit and a communication unit in order to output information such as the contents of a user's input via a button or joystick, and the position, direction, and tilt of the controller 210 via a sensor, etc., and to receive information from a host computer.

[0026] Depending on whether the user is holding controller 210 and operating various buttons and joysticks, and on information detected by infrared LEDs and sensors, the system determines the movement and posture of the user's hands and can simulate the display and movement of the user's hands in the virtual space.

[0027] <Image Generating Device 310> 7 is a diagram showing a functional configuration of an image generating device 310 according to this embodiment. As the image generating device 310, a device such as a PC, a game machine, or a mobile communication terminal having a function of storing user input information transmitted from the HMD 110 or the controller 210 and information on the user's head movement or controller movement or operation acquired by a sensor or the like, performing a predetermined calculation process, and generating an image can be used. The image generating device 310 can include, for example, an input / output unit 320 for establishing a wired connection with peripheral devices such as the HMD 110 or the controller 210, and can include a communication unit 330 for establishing a wireless connection such as infrared, Bluetooth (registered trademark), or WiFi (registered trademark). Information on the movement of the user's head and the movement and operation of the controller received from the HMD 110 and / or the controller 210 via the input / output unit 320 and / or the communication unit 330 is detected as input contents including the user's position, gaze, posture, and other movements, speech, operation, etc., in the control unit 340, and processing such as controlling the character 4 and generating an image is performed by executing a control program stored in the storage unit 350 according to the user's input contents. The control unit 340 can be configured with a CPU, but by further providing a GPU specialized for image processing, information processing and image processing can be distributed to improve the efficiency of the overall processing. The image generating device 310 can also communicate with other calculation processing devices and have the other calculation processing devices share the information processing and image processing.

[0028] The control unit 340 includes a user input detection unit 410 that detects information on the user's head movement, the user's speech, and the movement and operation of the controller received from the HMD 110 and / or the controller 210, a character control unit 420 that executes a control program stored in a control program storage unit 460 for a character 4 previously stored in a character data storage unit 450 of the storage unit 350, a camera control unit 440 that controls a virtual camera 3 arranged in the virtual space 1 according to the character control, an operation processing unit 480 that performs user operations in the virtual space 1, and an image generation unit 430 that generates an image of the virtual space 1 captured by the camera 3 based on the character control. Here, the control of the movement of the character 4 and the cameraman 2 is realized by converting information such as the direction and inclination of the user's head and the movement of the hands detected via the HMD 110 and the controller 210 into the movement of each part of a bone structure created in accordance with the movement and constraints of the joints of the human body, and applying the movement of the bone structure by associating the bone structure with the previously stored character data. The camera 3 is controlled, for example, by changing various settings for the camera 3 (for example, the position of the camera 3 in the virtual space 1, the shooting direction of the camera 3, the focus position, zoom, etc.) in accordance with the hand movements of the character 4.

[0029] The storage unit 350 stores information related to the character 4, such as the attributes of the character 4, in addition to the image data of the character 4 in the character data storage unit 450. The control program storage unit 460 stores a program for controlling the movement and facial expression of the character 4 in the virtual space and for controlling objects such as the camera 3. The image data storage unit 470 stores images generated by the image generation unit 430. In this embodiment, the images stored in the image data storage unit 470 can be action data for generating a moving image. The action data can include, for example, 3D data for displaying the character 4 in the virtual space 1, pose data for specifying the state of the bone structure of the 3D data, and motion data for specifying the movement of the bone structure. The image generation unit 430 can also create (render) a moving image based on the action data, and register the moving image data resulting from the rendering in the image data storage unit 470.

[0030] As shown in FIG. 1, the operation processing unit 480 displays an operation panel 7 in the virtual space 1, and can activate various functions in response to an operation of, for example, a finger 21 on the operation panel 7 in the virtual space 1. The operation panel 7 can be made to appear in the virtual space 1 when, for example, any button on the controller 210 is pressed or a predetermined operation is performed by the controller 210. The operation panel 7 can be freely arranged in the virtual space 1 by the user's operation, but can be moved to the vicinity of the user's hand or finger 21 in the virtual space 1 when, for example, any button on the controller 210 is pressed or a predetermined operation is performed by the controller 210. In this case, the operation panel 7 can be moved to a space where no object is arranged in the user's line of sight, for example, so that it is visible to the user in the user's field of view (a bird's-eye view in the case of FIG. 1, and when the user is operating the character 4, for example, the field of view of the character 4).

[0031] The operation panel 7 can include a display unit 71 and an instruction unit 72. The display unit 71 can display various types of information.

[0032] For example, a button 721 can be arranged on the instruction unit 72 of the operation panel 7. By pressing the button 721, various functions necessary for the performance of the character 4 or shooting by the camera 3 can be called up. For example, as shown in FIG. 1, pressing the button 721 can cause a list 711 of objects arranged in the virtual space 1 to be displayed on the display unit 71. Any function can be assigned to the button provided on the instruction unit 72, such as a function to call an asset panel (not shown) that displays a list of assets (objects) to be newly arranged, a function to display an image of the camera 3 on the display unit 71, or a function to display the field of view from the character 4 on the display unit 71.

[0033] For example, when there is a lot of information to be displayed on the display unit 71, such as when many objects are arranged in the virtual space 1, the display unit 71 can be provided with a scroll bar 712. The user can operate the scroll bar 712 using the finger 21 or the like, but in this embodiment, a lever 713 for operating the scroll bar 712 is provided on the display unit 71. The user can pinch and slide the lever 713 using the finger 21 or the like, and the display unit 71 can be scrolled according to the sliding of the lever 713. Instead of operating a planar controller such as a scroll bar displayed in a window system of a general operating system, an object (such as the lever 713) in the virtual space 1 that is held by hand can be operated, so that operability can be improved. That is, various three-dimensional controllers can be adopted in addition to the lever 713. For example, a knob used for volume control of a mixer device or the like may be provided on the display unit 71, and the display unit 71 may be scrolled by rotating the knob. The sliding direction of the lever 713 is the up-down direction in the example of FIG. 1, but it can also be set to be slidable in any direction, such as the horizontal direction or the depth direction.

[0034] Display unit 71 displays a list 711 of objects in virtual space 1, and each item is provided with a delete icon 714. When the user selects delete icon 714 with finger 21 or the like, the object related to that item can be deleted from virtual space 1. Note that it is also possible to select an item in list 711 and then press delete button 721 of instruction unit 72 to delete the object corresponding to the selected item.

[0035] Also, the user may select an object in the virtual space 1 with the finger 21 or the like (in the example of FIG. 1, the object indicated by the cone 8 may be shown to be selected), and the selected object may be deleted by pressing a delete icon 714 on the display unit 71 or a button 721 on the instruction unit 72. Also, a delete mark may be displayed near (for example, near the cone 8, etc.) the selected object (for example, the character 4-1 indicated by the cone 8 in FIG. 1), and the selected object may be deleted by selecting the delete mark with the finger 21 or the like.

[0036] Furthermore, when deleting an object from the virtual space 1, an object to be locked may be specified. For example, in Fig. 1, background image data 65 (such as a plate polygon to which an image is applied as a texture) can be locked so that it cannot be selected as a deletion target. In this case, the locked object may not be displayed in the list 711 of the display unit 71, or the delete icon 714 may not be displayed on the item corresponding to the locked object, or may be displayed as unavailable.

[0037] Also, for example, an object to be locked (e.g., image data 65) can be selected from a list 711 displayed on the display unit 71, and the selected object can be locked so that when specifying an object to be deleted in the virtual space 1, the locked object cannot be selected with the finger 21 or the like in the virtual space 1. Note that object locking can be used not only when deleting objects but also when performing general operations in the virtual space 1. For example, by locking background image data 65 in the virtual space 1, it is possible to prevent the background from being moved by mistake when selecting a character 4 or other object placed near the background.

[0038] As described above, according to the animation production system 300 of this embodiment, the user can operate the camera 3 as a cameraman 2 to shoot moving images in the virtual space 1. Therefore, since shooting can be performed by operating the camera 3 in the same way as in the real world, natural camera work can be realized, and richer expression can be imparted to the animated moving image.

[0039] Furthermore, according to the animation production system 300 of this embodiment, the characters 4 can be sequentially acted while switching between the characters 4 to be possessed in the virtual space 1. Therefore, animation can be efficiently shot.

[0040] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.

[0041] For example, in the present embodiment, the image generating device 310 is one computer, but this is not limiting, and the HMD 110 or the controller 210 may be provided with all or part of the functions of the image generating device 310. Also, some of the functions of the image generating device 310 may be provided in another computer communicably connected to the image generating device 310.

[0042] In addition, in this embodiment, a virtual space using virtual reality (VR) is assumed, but this is not limited to this, and the animation production system 300 of this embodiment can be directly applied to an augmented reality (AR) space or a mixed reality (MR) space.

[0043] In addition, in this embodiment, the operation mode is either the overhead view mode or not (S603), but a rendering mode for creating a final animation moving image based on the action data may be provided. [Explanation of symbols]

[0044] 1. Virtual Space 2. Photographer 3. Camera 4. Characters 110 HMD 120 Display Panel 130 Housing 140 Sensors 150 light source 210 Controller 220 Left-handed controller 230 Right Hand Controller 235 Grip 240 Trigger Button 250 Infrared LEDs 260 Sensors 270 Joystick 280 Menu button 300 Animation Production System 310 Image generating device 320 Input / output section 330 Communications Department 340 Control Unit 350 Storage section 410 User Input Detection Unit 420 Character Control Unit 430 Image Generation Unit 440 Camera Control Unit 450 Character data storage section 460 Program Storage Unit 470 Image data storage unit 480 Operation Processing Unit 510 Handle 520 Display section 530 Rail 540 Slider 550 Record button

Claims

1. placing a virtual camera in a virtual space; placing one or more objects within the virtual space; generating an animation video based on an image captured by the camera of the virtual space; a user input detection unit that detects an input from the user through at least one of a head mounted display and a controller worn by the user; accepting a selection of at least one of the objects in response to the input; removing the object from the virtual space in response to the input; accepting a designation of the object to be locked; Equipped with In the step of accepting the selection, the locked object is made unselectable; An animation production method characterized by:

2. 2. The animation production method according to claim 1, a step of displaying, in the virtual space, a list of the objects arranged in the virtual space; receiving, in response to the input, a selection of at least one of the objects from the list; An animation production method characterized by:

Images