Animation Production System
The system allows animation production in virtual space using a virtual camera and controller, enabling natural camera work and enhanced video expression through viewer-driven object replacement.
Patent Information
- Application Number
- JP2020128305
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-07-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-07-29
AI Technical Summary
There is a lack of techniques for filming animation in virtual space.
A system for producing animation in virtual space using a virtual camera, a head-mounted display, and a controller to control character motion, with an image generation unit that generates advertising videos by replacing objects based on viewer input.
Enables animation to be photographed in virtual space, allowing for natural camera work and enriched video expression.
Smart Images

Figure 0007588286000001 
Figure 0007588286000002 
Figure 0007588286000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an animation production system. [Background technology]
[0002] A virtual camera is arranged in a virtual space (see Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2017-146651 A Summary of the Invention [Problem to be solved by the invention]
[0004] There have been no attempts to film animation within a virtual space.
[0005] The present invention has been made in view of the above background, and has an object to provide a technique that enables animation to be shot within a virtual space. [Means for solving the problem]
[0006] The main invention of the present invention for solving the above problem is a system for producing animation for advertising videos, characterized in that it comprises a virtual camera that captures images of a character placed in a virtual space, a user input detection unit that detects user input from at least one of a head-mounted display and a controller worn by the user, a character control unit that controls the movement of the character in response to the input, an image data storage unit that records action data including the movement of the character captured by the camera, and an image generation unit that generates the advertising video while replacing objects in the virtual space in accordance with the viewer based on the action data.
[0007] Other problems and solutions disclosed in this application will be made clear in the description of the preferred embodiments of the invention and the drawings. Effect of the Invention
[0008] According to the present invention, animation can be shot in a virtual space. [Brief description of the drawings]
[0009] [Figure 1] FIG. 2 is a diagram showing an example of a virtual space displayed on a head mounted display (HMD) worn by a user in the animation production system 300 of this embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of the overall configuration of an animation production system 300 according to an embodiment of the present invention. [Diagram 3] 2 is a diagram showing a schematic external view of an HMD 110 according to the present embodiment. FIG. [Figure 4] FIG. 2 is a diagram illustrating an example of the functional configuration of an HMD 110 according to the present embodiment. [Diagram 5] FIG. 2 is a diagram showing a schematic external view of a controller 210 according to the present embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of the functional configuration of a controller 210 according to the present embodiment. [Figure 7] FIG. 2 is a diagram showing the functional configuration of an image generating device 310 according to the present embodiment. [Figure 8] FIG. 2 is a diagram illustrating the flow of video distribution processing by the animation production system of this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] The present invention will be described below with reference to the preferred embodiments thereof.
[0011] [Item 1] A system for producing animation for an advertising video, comprising: a virtual camera for photographing a character placed in a virtual space; a user input detection unit that detects an input from the user through at least one of a head mounted display and a controller worn by the user; a character control unit that controls an action of the character in response to the input; an image data storage unit for recording action data including the actions of the character captured by the camera; an image generating unit that generates the advertising video while replacing objects in the virtual space according to the viewer based on the action data; An animation production system comprising: [Item 2] The animation production system according to item 1, the image generation unit changes the character in accordance with the viewer; An animation production system that features: [Item 3] The animation production system according to item 1, the image generation unit changes model data of the character in accordance with the viewer; An animation production system that features: [Item 4] The animation production system according to item 1, the image generation unit changes an object other than the character arranged in the virtual space in accordance with the viewer; An animation production system that features:
[0012] A specific example of an animation production system 300 according to an embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims. In the following description, the same elements in the description of the drawings will be given the same reference numerals, and duplicated descriptions will be omitted.
[0013] <Summary> FIG. 1 is a diagram showing an example of a virtual space displayed on a head mounted display (HMD) worn by a user in an animation production system 300 of this embodiment. In the animation production system 300 of this embodiment, a virtual character 4 and a virtual camera 3 are placed in the virtual space 1, and the character 4 is photographed using the camera 3. Also, a cameraman 2 (photographer character) is placed in the virtual space 1, and the cameraman 2 virtually operates the camera 3. In the animation production system 300 of this embodiment, a user creates animation while overlooking the virtual space 1 from a TPV (Third Person View), photographing the character 4 from an FPV (First Person View) as the cameraman 2, and acting as the character 4 from the FPV. Also, multiple characters 4 (characters 4-1 and 4-2 in the example of FIG. 1) can be placed in the virtual space 1, and the user can act while possessing the character 4. When multiple characters 4 are arranged, the user can also switch the object possessed by each character 4 (for example, characters 4-1 and 4-2). That is, in the animation production system 300 of this embodiment, one person can play multiple roles. Also, since the user can shoot while virtually operating the camera 2 as the cameraman 2, it is possible to realize natural camera work and enrich the expression of the video to be shot.
[0014] <Overall composition> FIG. 2 is a diagram showing an example of the overall configuration of an animation production system 300 according to an embodiment of the present invention. The animation production system 300 can be configured to include, for example, an HMD 110, a controller 210, and an image generation device 310 functioning as a host computer. The animation production system 300 can also be configured to include an infrared camera (not shown) for detecting the position, orientation, inclination, etc. of the HMD 110 and the controller 210. These devices can be connected to each other by wired or wireless means. For example, each device can be provided with a USB port and communication can be established by connecting with a cable, or communication can be established by wired or wireless means such as HDMI, wired LAN, infrared, Bluetooth (registered trademark), WiFi (registered trademark), etc. The image generation device 310 may be a device having a calculation processing function, such as a PC, a game machine, or a mobile communication terminal. The image generation device 310 is connected to an advertisement distribution device 311 so as to be able to communicate with it. The advertisement distribution device 311 may be, for example, a server that provides advertisements together with content such as a web page including an advertisement space, or may be an advertisement distribution server that provides advertisements to web pages, apps, etc. In this embodiment, it is assumed that a moving image generated by an image generating device 310 is provided to an advertisement distribution device 311 as a moving image advertisement.
[0015] <hmd110> Fig. 3 is a diagram showing a schematic external view of the HMD 110 according to the present embodiment, and Fig. 4 is a diagram showing an example of the functional configuration of the HMD 110 according to the present embodiment.
[0016] The HMD 110 is attached to the user's head and includes a display panel 120 arranged in front of the user's left and right eyes. The display panel 120 may be an optically transmissive or non-transmissive display, but in this embodiment, a non-transmissive display panel capable of providing a greater sense of immersion is illustrated. The display panel 120 displays a left-eye image and a right-eye image, and a stereoscopic image can be provided to the user by utilizing the parallax between the two eyes. If it is possible to display a left-eye image and a right-eye image, it is possible to provide a left-eye display and a right-eye display separately, or to provide an integrated display for the left and right eyes.
[0017] The housing 130 of the HMD 110 includes a sensor 140. The sensor 140 may include, for example, a magnetic sensor, an acceleration sensor, or a gyro sensor, or a combination of these, though not shown, in order to detect movements such as the orientation and tilt of the user's head. When the vertical direction of the user's head is the Y axis, an axis that connects the center of the display panel 120 and the user and corresponds to the front-back direction of the user among the axes perpendicular to the Y axis is the Z axis, and an axis that is perpendicular to the Y axis and corresponds to the left-right direction of the user is the X axis, the sensor 140 can detect a rotation angle around the X axis (so-called pitch angle), a rotation angle around the Y axis (so-called yaw angle), and a rotation angle around the Z axis (so-called roll angle).
[0018] Instead of or in addition to the sensor 140, the housing 130 of the HMD 110 may be provided with a plurality of light sources 150 (e.g., infrared LEDs, visible LEDs). A camera (e.g., infrared camera, visible light camera) installed outside the HMD 110 (e.g., indoors, etc.) detects these light sources, thereby making it possible to detect the position, orientation, and inclination of the HMD 110 in a specific space. Alternatively, for the same purpose, the HMD 110 may be provided with a camera for detecting the light sources installed in the housing 130 of the HMD 110.
[0019] The housing 130 of the HMD 110 may also be equipped with an eye tracking sensor. The eye tracking sensor is used to detect the gaze direction and gaze point of the left and right eyes of the user. There are various possible methods for the eye tracking sensor, but for example, a method is possible in which the position of reflected light on the cornea created by irradiating the left and right eyes with weak infrared light is used as a reference point, the gaze direction is detected based on the position of the pupil relative to the position of the reflected light, and the intersection of the gaze directions of the left and right eyes is detected as the gaze point.
[0020] <Controller 210> Fig. 5 is a diagram showing a schematic external appearance of the controller 210 according to this embodiment. Fig. 6 is a diagram showing an example of the functional configuration of the controller 210 according to this embodiment.
[0021] The controller 210 can support the user in performing a predetermined input in a virtual space. The controller 210 can be configured as a set of a left-hand controller 220 and a right-hand controller 230. The left-hand controller 220 and the right-hand controller 230 can each have an operation trigger button 240, an infrared LED 250, a sensor 260, a joystick 270, and a menu button 280.
[0022] Operation trigger buttons 240 are arranged as 240a and 240b at positions that anticipate an operation of pulling the trigger with the middle finger and index finger when gripping grip 235 of controller 210. Frame 245 formed in a ring shape downward from both sides of controller 210 is provided with multiple infrared LEDs 250, and by detecting the positions of these infrared LEDs with a camera (not shown) provided outside the controller, the position, orientation, and tilt of controller 210 in a specific space can be detected.
[0023] The controller 210 may also include a sensor 260 for detecting movements such as the direction and tilt of the controller 210. Although not shown, the sensor 260 may include, for example, a magnetic sensor, an acceleration sensor, or a gyro sensor, or a combination of these. Furthermore, the controller 210 may include a joystick 270 and a menu button 280 on the upper surface thereof. The joystick 270 may be moved in 360 degrees around a reference point, and is assumed to be operated by the thumb when the grip 235 of the controller 210 is held. The menu button 280 is also assumed to be operated by the thumb in the same manner. Furthermore, the controller 210 may also include a vibrator (not shown) for applying vibrations to the hand of the user operating the controller 210. The controller 210 has an input / output unit and a communication unit in order to output information such as the contents of a user's input via a button or joystick, and the position, direction, and tilt of the controller 210 via a sensor, etc., and to receive information from a host computer.
[0024] Depending on whether the user is holding controller 210 and operating various buttons and joysticks, and on information detected by infrared LEDs and sensors, the system determines the movement and posture of the user's hands and can simulate the display and movement of the user's hands in the virtual space.
[0025] <Image Generating Device 310> 7 is a diagram showing a functional configuration of an image generating device 310 according to this embodiment. As the image generating device 310, a device such as a PC, a game machine, or a mobile communication terminal having a function of storing user input information transmitted from the HMD 110 or the controller 210 and information on the movement of the user's head or the movement or operation of the controller acquired by a sensor or the like, performing a predetermined calculation process, and generating an image can be used. The image generating device 310 can include, for example, an input / output unit 320 for establishing a wired connection with peripheral devices such as the HMD 110 or the controller 210, and can include a communication unit 330 for establishing a wireless connection such as infrared, Bluetooth (registered trademark), or WiFi (registered trademark). Information on the movement of the user's head and the movement and operation of the controller 210 received from the HMD 110 and / or the controller 210 via the input / output unit 320 and / or the communication unit 330 is detected as input contents including the user's position, gaze, posture, and other movements, speech, operation, etc., in the control unit 340, and processing such as controlling the character 4 and generating an image is performed by executing a control program stored in the storage unit 350 according to the user's input contents. The control unit 340 can be configured with a CPU, but by further providing a GPU specialized for image processing, information processing and image processing can be distributed to improve the efficiency of the overall processing. The image generating device 310 can also communicate with other calculation processing devices and have the other calculation processing devices share the information processing and image processing.
[0026] The control unit 340 includes a user input detection unit 410 that detects information on the user's head movement, the user's speech, and the movement and operation of the controller received from the HMD 110 and / or the controller 210, a character control unit 420 that executes a control program stored in a control program storage unit 460 for a character 4 previously stored in a character data storage unit 450 of the storage unit 350, a camera control unit 440 that controls a virtual camera 3 arranged in the virtual space 1 according to the character control, and an image generation unit 430 that generates an image of the virtual space 1 captured by the camera 3 based on the character control. Here, the control of the movement of the character 4 is realized by converting information such as the direction and inclination of the user's head and the movement of the hands detected via the HMD 110 and the controller 210 into the movement of each part of a bone structure created in accordance with the movement and constraints of the joints of the human body, and applying the movement of the bone structure by associating the bone structure with the previously stored character data. The camera 3 is controlled, for example, by changing various settings for the camera 3 (for example, the position of the camera 3 in the virtual space 1, the line of sight direction of the camera 3, the focal position, zoom, etc.) in accordance with the hand movement of the character 4. The image generating unit 430 registers, in the image data storage unit 470, action data representing the movement of the character 4 controlled by the character control unit 420 and the movement (operation) of the camera 3 controlled by the camera control unit 440, and can generate an image in which the movement of the character 4 is virtually captured by the camera 3.
[0027] The storage unit 350 stores information related to the character 4, such as the attributes of the character 4, in addition to the image data of the character 4 in the character data storage unit 450. The character data storage unit 450 also stores a texture image to be attached to the surface of the character 4. The character data storage unit 450 can also store transformation data for transforming the skeleton of the character 4. The character data storage unit 450 can also store several versions of model data in association with the same character. For example, even for the same character, different skin and bone model data can be stored for each race. The character data storage unit 450 can also store model data related to various objects (for example, desks, chairs, cups, beverages, etc.) placed in the virtual space 1 in addition to the character 4. The control program storage unit 460 also stores a program for controlling the movement and facial expression of the character 4 in the virtual space and for controlling objects such as the camera 3. The image data storage unit 470 stores images generated by the image generation unit 430. In this embodiment, the images stored in the image data storage unit 470 can be action data for generating a moving image. The action data may include, for example, 3D data for displaying the character 4 in the virtual space 1, pose data for specifying the state of the bone structure of the 3D data, motion data for specifying the movement of the bone structure, and the like. The image generating unit 430 may also create (render) a moving image based on the action data, and register the moving image data resulting from the rendering in the image data storing unit 470. The replacement information storing unit 480 stores information (replacement information) for replacing a part of the action data stored in the image data storing unit 470. The replacement information storing unit 480 may store the replacement information in association with an advertising target. The advertising target is information for specifying a viewer, and may be information indicating a segment of the viewer.The viewer segment may be any of various segments targeted for targeted advertising (e.g., conditions for the viewer's demographic, geographic, psychographic, and behavioral attributes, and interests such as being a fan of a particular character). The replacement information may include, for example, a replacement source object and a replacement destination object in association with viewer information indicating the viewer (the viewer's attributes, interests, etc.).
[0028] When generating an image to be transmitted to the advertisement distribution device 311, the image generation unit 430 can acquire information indicating a viewer, read out replacement information corresponding to the acquired viewer, perform rendering while replacing model data included in the action data based on the read out replacement information, and transmit the rendered moving image as an advertisement video to the advertisement distribution device 311. For example, the image generation unit 430 can acquire viewer information regarding a viewer viewing content provided by the advertisement distribution device 311, which is a Web server, by a known method. The image generation unit 430 may also receive designation of viewer information from a user of the image generation device 310.
[0029] The replacement information can include, for example, information indicating the character 4 in the virtual space 1 as the replacement source and information indicating the model data of the specific character as the replacement destination, in association with information indicating a viewer who is a fan of a specific character. This allows the image generating unit 430 to perform rendering based on the action data by replacing the main character 4 with the model data of a character that the viewer is a fan of, for a specific viewer.
[0030] Furthermore, the replacement information can include, for example, information indicating the character 4 in the virtual space 1 as the replacement source, and information indicating model data of a race corresponding to the country as the replacement destination, in association with information indicating a specific country. This allows the image generating unit 430, when performing rendering based on the action data, to perform rendering for a viewer residing in a specific country by replacing the main character 4 with model data of a race representative of the country in which the viewer resides.
[0031] The replacement information can include information specifying a beverage object placed in the virtual space 1 as a replacement source in association with information indicating the viewer, and information indicating model data representing an object of a specific product targeted at the viewer as a replacement destination. This makes it possible to perform replacement advertising in which an object placed according to the viewer is replaced with an object of an advertised product.
[0032] <Video transmission process> FIG. 8 is a diagram illustrating the flow of video distribution processing by the animation production system of this embodiment.
[0033] The user can operate the character 4 in the virtual space 1 to perform an action by operating the HMD 110 and the controller 210 (S101), and can also operate the cameraman 2 to control the camera 3 and capture the action of the character 4 (S102). The performance of the character 4 and the capture of the image by the camera 3 can be performed simultaneously, or the character 4 can be first performed, the action data can be recorded, and the action data can be played back and captured later by the camera 3. The actions of the character 4 (and all objects in the virtual space 1) captured by the camera 3 are stored as action data (S103).
[0034] The image generation unit 430 acquires viewer information that identifies the viewer to whom the video is to be delivered (S104), reads out replacement information corresponding to the acquired viewer information (S105), performs rendering while replacing objects included in the action data based on the read replacement information (S106), and transmits the rendered video to the advertisement delivery device 311 (S107).
[0035] As described above, according to the animation production system 300 of this embodiment, the user can operate the camera 3 as a cameraman 2 to shoot moving images in the virtual space 1. Therefore, since shooting can be performed by operating the camera 3 in the same way as in the real world, natural camera work can be realized, and richer expression can be imparted to the animated moving image.
[0036] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.
[0037] In addition, in this embodiment, a virtual space using virtual reality (VR) is assumed, but this is not limited to this, and the animation production system 300 of this embodiment can be directly applied to an augmented reality (AR) space or a mixed reality (MR) space.
[0038] In addition, in this embodiment, the image generating device 310 transmits moving images to the advertisement distribution device 311, but the image generating device 310 itself may be provided as an advertisement distribution system including an advertisement distribution unit that distributes advertisements.
[0039] Furthermore, when replacing an object in the virtual space based on the replacement information, the movement of the object (character 4) may be corrected. For example, when replacing with a character of a different height or build, the image generating unit 430 can correct the movement speed, angle, movement distance, etc. according to the length of bones such as arms and legs. Furthermore, the image generating unit 430 can set a limit to the range of movement for each character 4 and perform correction so that the character moves only within the range of movement. This makes it possible to express the difference in expression between a character with a stiff body and a character with a flexible body, for example. [Explanation of symbols]
[0040] 1. Virtual Space 2. Photographer 3. Camera 4. Characters 6. Control Panel 7 Screens 31 Grid 32 Partition Line 61 Display section 71 Display section 72 Play button 110 HMD 120 Display Panel 130 Housing 140 Sensors 150 light source 210 Controller 220 Left-handed controller 230 Right Hand Controller 235 Grip 240 Trigger Button 250 Infrared LEDs 260 Sensors 270 Joystick 280 Menu button 300 Animation Production System 310 Image generating device 311 Advertising distribution device 320 Input / output section 330 Communications Department 340 Control Unit 350 Storage section 410 User Input Detection Unit 420 Character Control Unit 430 Image Generation Unit 440 Camera Control Unit 450 Character data storage section 460 Control program storage section 470 Image data storage unit 480 Replacement Information Storage Unit
Claims
1. A system for producing animation for an advertising video, comprising: a virtual camera for photographing a character placed in a virtual space; a user input detection unit that detects an input from the user through at least one of a head mounted display and a controller worn by the user; a character control unit that controls an action of the character in response to the input; an image data storage unit for recording action data including the actions of the character captured by the camera; an image generating unit that generates the advertising video while replacing objects in the virtual space according to the viewer based on the action data; An animation production system comprising:
2. The animation production system according to claim 1, the image generation unit changes the character in accordance with the viewer; An animation production system that features:
3. The animation production system according to claim 1, the image generation unit changes model data of the character in accordance with the viewer; An animation production system that features:
4. The animation production system according to claim 1, the image generation unit changes an object other than the character arranged in the virtual space in accordance with the viewer; An animation production system that features:
Citation Information
Patent Citations
Image processing method and image processing program
JP2017146651A
Information processor and advertisement control program
JP2018185738A
JPP6709317B
