Program, electronic device and control method

A portable computer system with integrated virtual camera control and preview displays addresses the challenge of maintaining composition in virtual reality shooting by anticipating subject movements, ensuring continuous and satisfying video capture.

JP7588123B2Active Publication Date: 2024-11-21BANDAI NAMCO ENTERTAINMENT INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2022187045
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-11-24
Publication Date
2024-11-21
Estimated Expiration
2039-02-08

AI Technical Summary

Technical Problem

Existing methods for setting the motion of a virtual camera in virtual reality or augmented reality environments, such as for creating music videos with virtual idols, fail to maintain continuous shooting composition without interruptions due to dynamic actions of the subjects, requiring users to remember complex timing and actions.

Method used

A portable computer system with a camera and display, equipped with position and orientation acquisition, virtual space setting, subject object control, virtual camera control, and advance notice mechanisms, allowing users to shoot virtual videos with continuous composition by anticipating subject object movements through preview displays and synchronized music playback.

Benefits of technology

Enables users to capture virtual videos with desired compositions by providing real-time previews and synchronized controls, preventing image cutoffs and enhancing the user experience in virtual photography.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007588123000001
    Figure 0007588123000001
  • Figure 0007588123000002
    Figure 0007588123000002
  • Figure 0007588123000003
    Figure 0007588123000003
Patent Text Reader

Abstract

To provide a new support technology for videotaping a subject in a virtual three-dimensional space. [Solution] A user terminal 1500 is equipped with a camera, a display, and various sensors, and measures information about the camera's position and orientation in real space. The user terminal 1500 sets up a virtual three-dimensional space in which a subject object 4 and a virtual camera C1 that captures the subject object 4 are placed. The user terminal 1500 automatically controls the subject object 4 while controlling the position and orientation of the virtual camera C1 in conjunction with the measured position and orientation information. The user terminal 1500 generates an image of the virtual three-dimensional space captured by the virtual camera, displays it on the display as a viewfinder image for virtual capture, and records it. During virtual capture, the user terminal 1500 notifies the user of the details of the automatic control of the subject object 4 in a preview display 20.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a program for enabling a computer to realize a function for generating an image of a virtual three-dimensional space. [Background technology]

[0002] Most of the recent computer games, Augmented Reality (AR) images, and Virtual Reality (VR) images are created using 3DCG (3-Dimensional Computer Graphics). For example, game screens are generated by placing objects such as characters and backgrounds in a virtual three-dimensional space, and capturing images of these objects with a virtual camera.

[0003] The composition of a screen in a video is an important element for creating a dramatic effect, and in the case of a game, it also affects the ease of gameplay. Therefore, the setting of the position, orientation, and shooting angle of the virtual camera when creating 3DCG is a very important element. Patent Document 1 discloses a technology that maintains the expected composition by setting in advance the way the virtual camera moves as motion data for the virtual camera and correcting it according to the situation so that the game screen has the expected composition. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-212965 A Summary of the Invention [Problem to be solved by the invention]

[0005] When the motion of the virtual camera can be set in advance, the technique of Patent Document 1 can be an effective method. However, in a situation where a user uses a handheld virtual video camera to shoot a virtual subject (for example, a virtual idol singing and dancing in a virtual space) that exists in an augmented reality space or a virtual reality space, the technique of Patent Document 1 cannot be applied as is.

[0006] Specifically, if we consider a service that provides a user experience of virtually shooting a music video or promotional video of a virtual idol, in order to continuously shoot an entire song in the composition that the user desires, the user must remember all of the actions of the virtual idol and when. Otherwise, the camerawork will not be able to keep up with the dynamic actions of the virtual idol, resulting in so-called "cut-off" scenes.

[0007] The problem to be solved by the present invention is to provide a new support technique for videotaping a subject in a virtual three-dimensional space. [Means for solving the problem]

[0008] The first invention for solving the above-mentioned problems is a portable computer equipped with a camera (e.g., the camera 1520 in FIG. 2) and a display unit (e.g., the touch panel 1506 in FIG. 2), a position and orientation acquisition control means for controlling acquisition of information on the position and orientation of the camera in real space (for example, the control board 1550 in FIG. 2, the terminal processing unit 200 in FIG. 9, the virtual photography management unit 210, the position and orientation acquisition control unit 212, the actual photography condition data 705 in FIG. 21, and step S14 in FIG. 22); A virtual space setting means for setting a virtual three-dimensional space in which the subject object is placed (for example, the control board 1550 in FIG. 2, the terminal processing unit 200 in FIG. 9, the virtual photography management unit 210, the virtual space setting unit 214, the virtual three-dimensional space control data 710 in FIG. 21, step S12, step S20 in FIG. 22), Object control means for automatically controlling the object (for example, control board 1550 in FIG. 2, terminal processing unit 200 in FIG. 9, virtual photography management unit 210, object control unit 218, object control data 713 in FIG. 21, step S12, step S20 in FIG. 22), a virtual camera control means for controlling the position and attitude of a virtual camera that captures the virtual three-dimensional space in conjunction with the data acquired by the position and attitude acquisition control means (for example, the control board 1550 in FIG. 2, the terminal processing unit 200 in FIG. 9, the virtual photography management unit 210, the virtual camera control unit 220, the virtual camera control data 711 in FIG. 21, and step S14 in FIG. 22); A virtual space image generating means (e.g., the control board 1550 in FIG. 2, the terminal processing unit 200 in FIG. 9, the virtual photography management unit 210, the virtual space image generation control unit 222, the image generating unit 292, step S16 in FIG. 22) for generating an image of the virtual three-dimensional space captured by the virtual camera to be displayed on the display unit; This is a program for functioning as a notice control means (for example, control board 1550 in Figure 2, terminal processing unit 200 in Figure 9, virtual shooting management unit 210, notice control unit 234, steps S30 to S36 in Figure 22) that gives notice of automatic control by the subject object control means.

[0009] The term "computer" as used herein refers not only to a single computer, but also to a configuration in which multiple computers are linked together.

[0010] According to the first invention, when a user carries a portable computer equipped with a camera and a display unit and performs a shooting operation in the same manner as shooting with a handheld video camera (so-called camcorder), an image of the virtual space shot by a virtual camera whose position and attitude change in conjunction with the position and attitude of the computer is displayed on the display unit like a camera viewfinder image. In other words, virtual video shooting is possible with the computer as if it were a camcorder. During the virtual shooting process, the user is notified of the automatic control of the subject object. Therefore, the user can shoot while predicting how the next subject object will be controlled, making it easier to shoot with a desired composition. According to the first invention, a new support technology can be provided when shooting a video of a subject in a virtual three-dimensional space.

[0011] The second invention is a program of the first invention, in which the subject object control means automatically controls the subject object based on given control procedure data or a given control procedure program (hereinafter collectively referred to as "control procedure data") in which a control procedure for the subject object is defined, and the advance notice control means advances the automatic control by the subject object control means based on the control procedure data.

[0012] According to the second invention, the contents of the automatic control of the object are defined in advance in a time series by the control procedure data. From the user's point of view, if the user remembers the flow of the automatic control of the object, he or she can shoot a video with a more perfect composition. If the user repeatedly shoots a video of the same automatic control of the object, the video can be shot with a higher degree of satisfaction.

[0013] The third invention is a program of the second invention, which causes the computer to function as a music playback control means (e.g., control board 1550 in FIG. 2, terminal processing unit 200 in FIG. 9, virtual shooting management unit 210, music playback control unit 246, music data 540 in FIG. 10, step S20 in FIG. 22) that controls the playback of a given music piece, the control procedure data defining a control procedure synchronized with the playback of the music, and the subject object control means controlling the subject object in synchronization with the playback of the music by the music playback control means.

[0014] According to the third invention, the subject object can be automatically controlled in synchronization with the reproduction of music. For example, if the subject object is a virtual idol, the user can enjoy the feeling of virtually shooting the singing and dancing virtual idol with his / her camcorder. Even if the subject object is not a virtual idol but a virtual animal, the BGM synchronized with the movement of the animal can be reproduced as music to enhance the virtual shooting mood and provide the user with a wonderful virtual shooting experience.

[0015] A fourth invention is a program of any of the first to third inventions, wherein the object control means controls the movement of the object, and the advance notice control means advances at least one of the movement direction, movement distance, and movement path of the object.

[0016] According to the fourth invention, it is possible to provide the user with information for preventing cut-off caused by the movement of the subject object.

[0017] A fifth invention is a program of any of the first to fourth inventions, wherein the subject object is a character object, the subject object control means controls the movement of the subject object's body, and the advance notice control means advances the body movement of the subject object.

[0018] According to the fifth invention, it is possible to provide the user with information for preventing cut-off caused by the movement of the subject object.

[0019] A sixth invention is the program according to the fifth invention, wherein the advance notice control means performs control to distinguishably display a part related to the advance notice of the body movement of the subject object.

[0020] According to the sixth invention, it is possible to provide the user with information on the body part that will be moved, as a notice of the body movement of the subject object.

[0021] A seventh invention is a program of any of the first to sixth inventions, wherein the virtual space setting means places a plurality of the subject objects in the virtual three-dimensional space, the subject object control means automatically controls each of the plurality of subject objects, and the advance notice control means performs individual advance notice control for the subject objects.

[0022] According to the seventh aspect of the present invention, automatic control and advance notice can be performed for each of a plurality of subject objects. For example, if the subject objects are virtual idols, even in a service for enjoying virtual photography of a virtual idol group, it is possible to support the user in taking a virtual photo with a composition that he or she desires.

[0023] An eighth invention is the seventh invention program, wherein the advance notice control means selects at least one of the plurality of subject objects, and gives an advance notice about the selected subject object.

[0024] According to the eighth aspect of the invention, it becomes possible to give a notice of at least one of a plurality of subject objects.

[0025] A ninth invention is the program according to the eighth invention, wherein the advance notice control means selects a subject object that satisfies a predetermined condition from among the plurality of subject objects.

[0026] According to the ninth aspect of the present invention, it is possible to provide a notice of a subject object whose movement satisfies a predetermined condition at any given time, and therefore, for example, it is possible to provide a user with information in advance about a subject object and its movement that should be given priority in consideration from the viewpoint of action when considering a composition.

[0027] A tenth invention is an eighth invention program, wherein the virtual space setting means sets the virtual three-dimensional space as a space in which a concert is held by the multiple subject objects, the subject object control means automatically controls the singing action of each subject object, and the advance notice control means selects the subject object which will next perform the singing action from among the multiple subject objects.

[0028] According to the tenth invention, when virtually shooting a concert in which a plurality of subject objects take turns singing, information as to which subject object will sing next can be provided to the user.

[0029] An eleventh invention is the program of any of the seventh to tenth inventions, wherein the subject object control means controls the plurality of subject objects to perform a coordinated action, and the advance notice control means advances the coordinated action.

[0030] According to the eleventh aspect of the present invention, it is possible to provide the user with information for preventing cut-off caused by linked movements of a plurality of subject objects.

[0031] The 12th invention is a program of any of the 1st to 11th inventions, which causes the computer to function as a notice target control setting means (e.g., control board 1550 in Figure 2, terminal processing unit 200 in Figure 9, virtual shooting management unit 210, notice target control setting unit 230 in Figure 21, specific action type 725 in Figure 22, step S10 in Figure 22) that sets, based on a user's operational input, the control content to be the subject of the notice among the control contents for automatically controlling the subject object by the subject object control means, and the notice control means gives a notice of the control content set by the notice target control setting means.

[0032] According to the twelfth aspect of the present invention, the user can specify the contents of the automatic control to be notified. In other words, the user can be provided with a notice only about the automatic control that the user thinks is necessary, and since unnecessary notices are not given, an environment that makes it easy to concentrate on virtual photography can be created.

[0033] A thirteenth aspect of the invention provides a prediction means for predicting whether or not a specific part of the subject object is included in an image captured by the virtual camera when automatic control of the subject object control means relating to the content of the prediction is performed in the future, on the assumption that the current position and attitude of the virtual camera will continue, when the prediction control means makes a prediction (for example, the control board 1550 in FIG. 2, the terminal processing unit 200 in FIG. 9, the virtual photography management unit 210, the specific part setting unit 232, the specific part type 723 in FIG. 21, and step S10 in FIG. 22); A prediction notification control means for controlling to notify the user when a negative prediction is made by the prediction means (for example, the control board 1550 in FIG. 2, the device processing unit 200 in FIG. 9, the virtual photography management unit 210, the prediction notification control unit 242, step S38 in FIG. 22), The program according to any one of the first to twelfth aspects of the present invention for causing the computer to function as the program.

[0034] According to the thirteenth aspect of the present invention, it is possible to focus on a specific portion of the subject object, predict that the portion may be cut off if the image is left as is, and notify the user of this fact, thereby further enhancing user support.

[0035] A fourteenth invention is a portable electronic device equipped with a camera and a display unit, comprising: a position and attitude acquisition control means for controlling the acquisition of information on the position and attitude of the camera in real space; a virtual space setting means for setting a virtual three-dimensional space in which a subject object is placed; a subject object control means for automatically controlling the subject object; a virtual camera control means for controlling the position and attitude of a virtual camera that captures the virtual three-dimensional space in conjunction with data acquired by the position and attitude acquisition control means; a virtual space image generation means for generating an image of the virtual three-dimensional space captured by the virtual camera to be displayed on the display unit; and a warning control means for warning of automatic control by the subject object control means.

[0036] According to the fourteenth aspect of the present invention, it is possible to realize an electronic device that provides the same advantageous effects as the first aspect of the present invention. [Brief description of the drawings]

[0037] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a virtual photography experience providing system. [Diagram 2] FIG. 4 is a front view showing a configuration example of a user terminal. [Diagram 3] 1A and 1B are diagrams for explaining display control of a subject using augmented reality in a user terminal. [Figure 4] FIG. 13 is a diagram for explaining recording of a virtually captured augmented reality image. [Diagram 5] FIG. 13 is a diagram showing an example of a situation in which cut-off occurs. [Figure 6] FIG. 13 is a conceptual diagram showing an example of a preview. [Figure 7] FIG. 13 is a diagram showing an example of a preview icon used in preview display. [Figure 8] 13A and 13B are diagrams showing examples of prediction determination of cut-off and notification of cut-off prediction; [Figure 9] FIG. 4 is a functional block diagram showing an example of the functional configuration of a user terminal in the first embodiment. [Figure 10] 4 is a diagram showing an example of programs and data stored in a terminal storage unit in the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of the data structure of preview icon definition data (part 1). [Figure 12] FIG. 2 is a diagram showing an example of the data structure of preview icon definition data (part 2). [Figure 13] FIG. 3 is a diagram showing an example of the data structure of preview icon definition data. [Figure 14] FIG. 4 is a diagram showing an example of the data structure of preview icon definition data. [Figure 15] FIG. 4 is a diagram showing an example of the data configuration of subject object definition data. [Figure 16] FIG. 13 is a diagram showing an example of the data configuration of singing assignment data. [Figure 17] FIG. 4 is a diagram showing an example of a data structure of control procedure data. [Figure 18] FIG. 13 is a diagram showing an example of the data configuration of a focus action list. [Figure 19] FIG. 4 is a diagram showing an example of the data structure of performance procedure data. [Figure 20] FIG. 13 is a diagram showing an example of the data configuration of advance notice procedure data. [Figure 21] FIG. 4 is a diagram showing an example of the data structure of virtual photography management data. [Figure 22] 6 is a flowchart illustrating a process flow in the user terminal according to the first embodiment. [Figure 23] Flowchart continuing from Figure 22. [Figure 24] FIG. 11 is a functional block diagram showing an example of the functional configuration of a server system according to a second embodiment. [Diagram 25] FIG. 11 is a diagram showing an example of programs and data stored in a server storage unit in the second embodiment. [Figure 26] FIG. 11 is a functional block diagram showing an example of the functional configuration of a user terminal in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] An example of an embodiment of the present invention will be described below, but it goes without saying that the form to which the present invention can be applied is not limited to the following embodiment.

[0039] [First embodiment] FIG. 1 is a diagram illustrating an example of the configuration of a virtual photography experience providing system. The virtual photography experience providing system 1000 is a computer system that provides a user experience of virtually photographing objects that exist in a virtual three-dimensional space, and includes a server system 1100 and a user terminal 1500 that are connected to each other via a network 9 so that data communication is possible between them.

[0040] The network 9 refers to a communication path that allows data communication. In other words, the network 9 includes a dedicated line (dedicated cable) for direct connection, a LAN (Local Area Network) using Ethernet (registered trademark), etc., as well as a telephone communication network, a cable network, the Internet, and other communication networks, and the communication method may be either wired or wireless.

[0041] The server system 1100 includes, for example, a keyboard 1106 , a touch panel 1108 , and a storage 1140 , and the main unit 1101 is equipped with a control board 1150 .

[0042] The control board 1150 is equipped with various microprocessors such as a CPU (Central Processing Unit) 1151, a GPU (Graphics Processing Unit), and a DSP (Digital Signal Processor), various IC memories 1152 such as a VRAM, RAM, and ROM, and a communication device 1153. Note that a part or the whole of the control board 1150 may be realized by an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or an SoC (System on a Chip).

[0043] The server system 1100 realizes the function of providing various programs and data for providing the user with a virtual photography experience on the user terminal 1500 by the control board 1150 performing calculations based on predetermined programs and data.

[0044] Although only one user terminal 1500 is depicted in FIG. 1, in actual system operation, a configuration may be adopted in which a plurality of user terminals 1500 can access the server system 1100 at the same time.

[0045] Also, although the server system 1100 is depicted as if it were a single server device, it may be realized by a configuration of multiple devices. For example, the server system 1100 may be configured to include multiple blade servers each sharing a function, and connected to each other via an internal bus so that data communication is possible. Furthermore, the location of the hardware constituting the server system 1100 does not matter. It may be configured such that multiple independent servers installed in remote locations communicate data via the network 9 to function as the server system 1100 as a whole.

[0046] The user terminal 1500 is a computer system used by the user 2 to obtain a user experience of virtual photography, and is an electronic device (electronic device) that can access the server system 1100 via the network 9. The user terminal 1500 of this embodiment is a device called a smartphone, but may be a portable game device, a tablet computer, or the like, as long as it is a handheld computer system.

[0047] FIG. 2 is a front view showing an example of the configuration of a user terminal 1500 in this embodiment. The user terminal 1500 includes a direction input key 1502, a button switch 1504, a touch panel 1506 that functions as an image display device and a contact position input device, a speaker 1510, an internal battery 1509, a microphone 1512, a camera 1520, a control board 1550, and a memory card reader 1542 that can read and write data from a memory card 1540 that is a computer-readable storage medium. In addition, a power button, a volume control button, etc. (not shown) are provided. Also, an IC card reader that can read and write data contactlessly from an IC card type credit card or prepaid card that can be used to pay for system usage may be provided.

[0048] Camera 1520 is an image sensor unit with a variable zoom magnification. That is, camera 1520 includes components that enable digital moving image capture, such as an image sensor such as a CCD (Charge Coupled Device), a zoom mechanism, an autofocus mechanism, and an LSI that generates video data from a signal captured by the image sensor.

[0049] The control board 1550 is equipped with (1) a CPU 1551 and various microprocessors such as a GPU, DSP, etc., (2) various IC memories 1552 such as VRAM, RAM, ROM, etc., (3) a wireless communication module 1553 for wireless communication with mobile phone base stations and wireless LAN base stations connected to the network 9, (4) a positioning module 1555, (5) a three-axis acceleration sensor 1556, (6) a three-axis gyro 1557, (7) an interface circuit 1558, etc.

[0050] The positioning module 1555 acquires the position of the user terminal 1500. In this embodiment, the position coordinates in the real space can be acquired by using an existing positioning system. That is, the positioning module 1555 provides a positioning function in the user terminal 1500 by receiving a signal provided by the positioning system and outputting positioning information at a predetermined cycle (for example, every second). In this embodiment, a GPS (Global Positioning System) is used as the positioning system. Therefore, the positioning module 1555 can use an existing "GPS module" or "GPS receiver". The "positioning information" includes the positioning date and time (for example, UTC: Coordinated Universal Time), position coordinates (latitude, longitude, altitude), direction, etc. The positioning system to be used is not limited to GPS and can be appropriately selected. For example, the positioning module 1555 can be replaced with a configuration of a communication device that wirelessly connects to a wireless base station of a mobile phone or a wireless base station of a Wi-Fi network, and a direction sensor. That is, location information that has been previously determined for the currently connected wireless base station may be acquired and used as the location coordinates of the user terminal 1500 in real space.

[0051] The interface circuit 1558 includes a driver circuit for the touch panel 1506, a circuit for receiving signals from the directional input keys 1502 and the button switches 1504, an output amplifier circuit for outputting audio signals to the speaker 1510, an input signal generation circuit for generating an audio signal collected by the microphone 1512, a circuit for inputting image data of an image captured by the camera 1520, and a signal input / output circuit for the memory card reader 1542.

[0052] These elements mounted on the control board 1550 are electrically connected via a bus circuit or the like, and are connected so as to be able to read and write data and send and receive signals. Note that a part or all of the control board 1550 may be configured with an ASIC, FPGA, or SoC. The control board 1550 stores programs and various data for realizing the functions of a user terminal in an IC memory 1552.

[0053] In this embodiment, the user terminal 1500 is configured to download programs and various setting data from the server system 1100, but may be configured to read them from a storage medium such as a memory card 1540 obtained separately.

[0054] FIG. 3 is a diagram for explaining display control of a subject using augmented reality in the user terminal 1500. In FIG. The user terminal 1500 can use augmented reality to display an image of a subject object set in a virtual three-dimensional space on the touch panel 1506. That is, an AR marker 6 is set in the real space, and the user 2 adjusts the position and orientation of the user terminal 1500 and the zoom magnification of the camera 1520 so as to capture the AR marker 6 with the camera 1520. The AR marker 6 in the image of the real space captured by the camera 1520 (real space image) is subjected to image recognition processing, and is used in calculation processing of the relative position and relative orientation between the user terminal 1500 and the AR marker 6.

[0055] The user terminal 1500 can constantly measure and acquire the position and attitude changes of the camera 1520 from the calculated relative position and relative attitude, and the acceleration and attitude information of the user terminal 1500 obtained using the 3-axis acceleration sensor 1556 and the 3-axis gyro 1557. In addition, the information on the zoom magnification (corresponding to the shooting angle of view) can also be periodically acquired by the zoom operation of the camera 1520. Hereinafter, the parameter values ​​describing so-called shooting conditions such as the relative position, relative attitude, position change, attitude change, and zoom magnification, as well as the size and direction of the AR marker 6 in the captured image, attitude data, acceleration data of the user terminal 1500, zoom control values, sensor measurement values, and the like that are the basis for calculating these parameters will be collectively referred to as the "actual shooting conditions".

[0056] While acquiring the actual shooting conditions, the user terminal 1500 can place the subject object 4 in a virtual three-dimensional space and automatically control it. In this embodiment, the subject object 4 is a character object in the form of a human, and is an AR (Augmented Reality) character based on a captured image captured by the camera 1520. The user terminal 1500 automatically controls the subject object 4 to dance while singing music on the stage 5, using given music data prepared in advance and predetermined motion data for realizing motion synchronized with the music of the music data.

[0057] The "motion" referred to here is an action prepared for automatic control of the subject object 4. The type of action can be set as appropriate. For example, walking, running, jumping, crouching, singing action of opening and closing the mouth with a microphone close to the mouth, dance choreography, peace sign, turning, coordinated action with one or more other subject objects or background objects (for example, high-fiving, shaking hands, folding arms, hanging, throwing, lifting, pulling, etc.), etc. can be set. Note that movement of the subject object 4 without moving the body, for example, non-action movement by a mechanism on the stage (elevator, slider, wire, etc.), is also included in the motion.

[0058] The stage 5 is one of the background objects that constitute the space in which a concert is held by the subject objects 4. Note that background objects other than the stage 5 can also be set as appropriate.

[0059] A virtual camera C1 is placed in the virtual three-dimensional space, and its shooting position, shooting posture, and shooting angle of view are linked and controlled to parameter values ​​of the actual shooting conditions of the camera 1520. The user terminal 1500 then generates an image of the virtual three-dimensional space captured by the virtual camera C1, i.e., a virtual space image 90 (images of the subject object 4 and stage 5), and displays this image superimposed on a real space image 92 as an augmented reality image 94 on the touch panel 1506.

[0060] After inputting a predetermined operation to start recording to the user terminal 1500, the user 2 performs a shooting operation by moving the user terminal 1500 in real space, sometimes changing its posture, and sometimes changing the zoom ratio, in a manner similar to shooting with a portable video camera (handheld camcorder), regarding the subject object 4 dancing and singing a song (singing motion) displayed on the touch panel 1506 as a viewfinder image. In other words, the user 2 can enjoy a virtual shooting experience of shooting the concert of the subject object 4, regarding the augmented reality image 94 as a viewfinder image.

[0061] Then, when a recording start operation is performed, as shown in FIG. 4, the user terminal 1500 generates a series of recorded data 740 in which the generated augmented reality images 94 are continuously recorded in chronological order, and records this.

[0062] Now, through such a virtual shooting experience, the user 2 feels like he / she is a cameraman and wants to shoot a promotional video for the subject object 4. Naturally, the user 2 wants to shoot with a particular video composition. The user wants to shoot with a composition that captures the dynamic action of the subject object 4 in full view at the right time, captures a close-up of an attractive expression shown by the subject object 4 at the right time, and captures flashy stage effects (for example, spotlight action, smoke, back dance, etc.) at the right time.

[0063] However, subject object 4 is singing and dancing. For example, in the example of FIG. 5, subject object 4 is jumping at the same time that user 2 is taking a waist shot of subject object 4. If user 2 does not know that a jump action will occur here, he or she will not be able to align the camerawork with the jump of subject object 4. As a result, subject object 4 will jump out of the shooting frame and will be "cut off."

[0064] To continue shooting the image with the desired composition without "cutting off the image" until the subject object 4 finishes singing one song (excluding cuts intentionally made for the purpose of production), one must perfectly remember the song sung by the subject object 4, the choreography of the dance, and the contents and timing of the stage production. Although memorization can be considered one of the elements of the service for enjoying the virtual shooting experience exemplified in this embodiment, users who have just started using the service may find it difficult and may refrain from playing the service altogether.

[0065] Therefore, in this embodiment, the user 2 is given advance notice of what will happen in the virtual three-dimensional space, in particular, automatic control of the subject object 4, thereby supporting the user 2 so that he or she can take a virtual photograph with a composition that he or she desires.

[0066] Fig. 6 is a conceptual diagram showing an example of a preview. The situation shown in Fig. 6 is the same as that in Fig. 5. Control procedure data is prepared for various objects arranged in a virtual three-dimensional space including a subject object 4 so that the objects perform predetermined operations at predetermined timings in synchronization with a time code, which is information on the elapsed time when playing a song. The user terminal 1500 automatically controls the various objects based on this control procedure data.

[0067] Therefore, the user terminal 1500 displays a preview display 20, which notifies the user of the content of the next automatic control to be executed, in the shooting operation screen W6 displayed on the touch panel 1506 during virtual shooting.

[0068] The display format of the notice can be appropriately selected, but in this embodiment, the user terminal 1500 displays the notice display 20 in the form of an icon on the right edge of the shooting operation screen W6. 6 shows an example in which a moving direction / path forecast icon 21a, an action type forecast icon 21b, and a moving distance forecast icon 21c of the subject object 4 are displayed. Specifically, the moving direction / path forecast icon 21a is an icon indicating the direction in which the subject object 4 moves (upward in the illustrated example) by the direction of the arrow drawn on the icon. The action type forecast icon 21b is an icon indicating the type of automatic control of the subject object 4 (jump action in the illustrated example) by the initials drawn on the icon. The moving distance forecast icon 21c is an icon forecasting the moving distance by the number drawn on the icon, using the width of the shooting frame at that time as a reference length, and indicating how many frames the object will move (two frames in the illustrated example).

[0069] Before the subject object 4 makes a jump action, the user 2 knows in advance what will happen next in the virtual three-dimensional space, including the movement of the subject object 4, through such a notice display, and can prepare what kind of camera work should be performed next to continue shooting the video with the desired composition. In the example of Fig. 6, the user 2 tilts the user terminal 1500 upward in accordance with the jump action of the subject object 4, reduces the zoom factor if necessary, and can continue shooting with a composition that captures the entire subject object 4 making the jump action within the shooting frame without cutting off the image.

[0070] FIG. 7 is a diagram showing examples of preview icons 21, in which parenthetical numbers are assigned according to categories according to purpose, and examples of different types of preview icons 21 that fall into the same category are shown side by side.

[0071] The purpose and contents of the notice can be set appropriately according to basic settings of virtual photography, such as what the subject object 4 will be, what kind of virtual three-dimensional space will be, etc. In this embodiment, since the subject object 4 singing and dancing is virtually photographed, notice is given of automatic control of the subject object 4, which is the main subject of photography, and of stage direction, etc.

[0072] Specifically, a movement direction / movement route preview icon 21a, an action type preview icon 21b, a movement distance preview icon 21c, a front direction preview icon 21d, a pose type preview icon 21e, an eye-catching preview icon 21f, a facial expression preview icon 21g, a focus area preview icon 21h, a performance preview icon 21j, and a focus object preview icon 21k are used in different ways.

[0073] The movement direction / movement path advance notice icon 21a advances the movement direction and movement path of the object to be advanced. In this embodiment, the movement direction of the subject object 4 is indicated by the direction of an arrow, and the movement path is advanced by the direction and shape of the axis of the arrow.

[0074] The action type preview icon 21b previews the action type of the object to be previewed. In this embodiment, the type is indicated by the initial letter of the name of the action of the subject object 4 or a figure that evokes the action. In a case where multiple subject objects 4 appear, a coordinated action by the multiple subject objects can also be previewed using this type of preview icon.

[0075] The moving distance forecast icon 21c forecasts the moving distance of the object to be forecasted. In this embodiment, the moving distance of the subject object 4 is indicated by a rough multiple of the shooting frame. Note that the unit for expressing the moving distance is not limited to the shooting frame, and may be expressed using the body of the subject object 4. Alternatively, the moving distance forecast icon 21c may be displayed at the position of the moving destination.

[0076] The front direction preview icon 21d previews the front direction of the object to be previewed. In this embodiment, the front direction is set for each action of the subject object 4. Of course, the front direction may be the direction that the face of the subject object 4 will be facing next. The reference for indicating the direction may be set based on the stage 5, or may be a relative direction based on the shooting direction of the virtual camera C1. In this embodiment, the latter is used.

[0077] The pose type preview icon 21e previews the type of characteristic pose set for the object to be previewed. In this embodiment, the type is previewed with the silhouette of the pose of the subject object 4.

[0078] The eye-catching preview icon 21f notifies an action that is considered to have a particularly high eye-catching effect among the actions set for the object to be notified. In other words, it notifies an action that looks good in a video or on a screen. In this embodiment, the eye-catching preview icon 21f indicates the type of eye-catching action by using a figure that evokes the eye-catching action of the subject object 4, such as a blown kiss, a wink, an arm thrust, a V sign, a decisive step, a show-off, a costume change, or an equipment change. In a case where multiple subject objects 4 appear, eye-catching actions resulting from the cooperation of multiple subject objects 4 can also be used as targets for using this type of preview icon.

[0079] The facial expression preview icon 21g previews the appearance of a key facial expression of the subject object 4 as part of the choreography of the song. For example, in the case where a sad facial expression that appears only in accordance with the lyrics is set in the choreography of a fun and pop song, it is preferable to preview this sad facial expression.

[0080] The attention part preview icon 21h is a key action in the choreography of the song, and predicts the appearance of an action in which a specific part of the body moves in a distinctive way, such as clapping with both hands above the head, blowing a kiss, twisting the waist, swinging an instrument, etc.

[0081] The performance preview icon 21j previews the type of performance on the stage 5. For example, the performance preview icon 21j indicates the type using figures reminiscent of flashlights, smoke, stage transformation, the appearance of backing dance, and the like.

[0082] The attention object preview icon 21k indicates an object to be noted. In this embodiment, in a case where a plurality of subject objects 4 appear, the subject object 4 that is in charge of singing the main part of the song at that time, playing an instrument that is the highlight, and the subject object 4 that performs an action to be noted as choreography of the song are previewed. The action to be noted corresponds to, for example, an action (for example, the action with the largest action) that satisfies a predetermined condition at that time among the plurality of subject objects 4, or a linked action by the plurality of subject objects 4. Note that, for the attention object preview icon 21k for the linked action, either or both of a form in which the combination of subject objects 4 related to the linked action is previewed and the content of the linked action is not previewed, or a form in which the content of the linked action is previewed can be set.

[0083] In this embodiment, as one of the functions to support the user 2, the user terminal 1500 predicts the occurrence of cut-off and notifies the user when cut-off is predicted (when the prediction result is incompatible). "Cut-off" refers to a state in which a specific part of a subject object is not included in the camera's shooting frame by a predetermined ratio or more, or a state in which a specific part captured in a photographed image is not located within a specific range in the photographed image (for example, the center of the photographed image).

[0084] FIG. 8 is a diagram showing an example of prediction of cut-off and notification of cut-off prediction. The user terminal 1500 can determine objects included in the shooting range of the virtual camera C1 in the same manner as the hit determination check. In other words, it can determine whether or not each part object of the subject object 4 captured in the virtual space image 90 constituting the augmented reality image 94 displayed on the touch panel 1506 at that time is captured in the shooting frame.

[0085] When the user terminal 1500 determines that any of a predetermined main part object (e.g., the head) of the subject object 4, a part predicted by the attention part prediction icon 21h in the preview display 20, and a part separately specified by the user 2 as a specific part is not shown in the shooting frame, the user terminal 1500 determines that cut-off will occur if the current situation remains unchanged. Then, when the user terminal 1500 determines that cut-off will occur, it notifies the user that cut-off is predicted to occur. Specifically, the cut-off prediction notification 30 is displayed in the shooting operation screen W8. By displaying the cut-off prediction notification 30, attention to the preview display 20 is further drawn.

[0086] In addition, the prediction of the occurrence of cut-off may be determined not based on whether the part object is captured within the shooting frame, but based on whether the proportion of the part object in the captured image (the ratio of the area captured within the shooting frame to the area of ​​the captured image) is in a non-compliant state that satisfies specified non-compliant shooting conditions.

[0087] 9 is a functional block diagram showing an example of a functional configuration of a user terminal 1500 in this embodiment. The user terminal 1500 includes an operation input unit 100, a positioning unit 102, an acceleration measurement unit 106, an attitude measurement unit 108, a voice input unit 110, an image capture unit 120, a device processing unit 200, a sound output unit 390, an image display unit 392, a communication unit 394, and a device storage unit 500.

[0088] The operation input unit 100 outputs operation input signals corresponding to various operation inputs to the device processing unit 200. For example, this can be realized by a push switch, a joystick, a touch pad, a track ball, an acceleration sensor, a gyro, a CCD module, etc. The direction input key 1502, the button switch 1504, and the touch panel 1506 in Fig. 2 correspond to this.

[0089] The positioning unit 102 acquires location information of the user terminal 1500 and outputs it to the terminal processing unit 200. This corresponds to the positioning module 1555 in FIG.

[0090] The acceleration measurement unit 106 measures the acceleration occurring in the user terminal 1500, and outputs it to the terminal processing unit 200. This corresponds to the three-axis acceleration sensor 1556 in FIG.

[0091] The attitude measurement unit 108 measures the change in attitude of the user terminal 1500 and outputs the result to the terminal processing unit 200. The three-axis gyro 1557 in FIG.

[0092] The voice input unit 110 collects voice uttered by the user (user voice) and environmental sounds, and outputs a voice signal to the device processing unit 200. The microphone 1512 in FIG.

[0093] The photographing unit 120 photographs the outside world, generates image data thereof, and outputs the image data to the device processing unit 200. This corresponds to the camera 1520 in FIG.

[0094] The device processing unit 200 is realized by electronic components such as a microprocessor such as a CPU or GPU, or an IC memory, and controls input and output of data between each functional unit including the operation input unit 100 and the device storage unit 500. Then, it executes various arithmetic processing based on predetermined programs and data, operation input signals from the operation input unit 100, and various data received from the server system 1100, to control the operation of the user terminal 1500. This corresponds to the control board 1550 in FIG. 2. The device processing unit 200 in this embodiment has a virtual photography management unit 210, a timing unit 280, a sound generation unit 290, an image generation unit 292, and a communication control unit 294.

[0095] The virtual photography management unit 210 executes various processes for providing the user with a virtual photography experience on the user terminal 1500. Specifically, the virtual photography management unit 210 has a position and orientation acquisition control unit 212, a virtual space setting unit 214, a performance control unit 216, a subject object control unit 218, a virtual camera control unit 220, a virtual space image generation control unit 222, an augmented reality image generation control unit 224, a notice target control setting unit 230, a specific part setting unit 232, a notice control unit 234, a prediction unit 240, a prediction notification control unit 242, a recording control unit 244, and a music playback control unit 246. Of course, functional units other than these may also be included as appropriate.

[0096] The position and orientation acquisition control unit 212 controls the acquisition of information on the position and orientation of the camera in real space. Specifically, the position and orientation acquisition control unit 212 measures the position and orientation of the user terminal 1500 based on an image captured in real space by the imaging unit 120. More specifically, this corresponds to control for recognizing and analyzing the AR marker 6 from the captured image and calculating the relative position and relative orientation between the AR marker 6 and the user terminal 1500.

[0097] Furthermore, the position and orientation acquisition control unit 212 can perform control to measure information related to the position and orientation of the user terminal 1500 in real space based on measurement data measured by the positioning unit 102, the acceleration measuring unit 106, and the orientation measuring unit 108. When the positioning unit 102, the acceleration measuring unit 106, and the orientation measuring unit 108 are configured as external devices, the position and orientation acquisition control unit 212 performs control to acquire information related to the position and orientation measured and calculated by them using data communication with the external device.

[0098] The virtual space setting section 214 sets a virtual three-dimensional space in which one or more subject objects are placed.

[0099] The production control unit 216 performs given production control in the virtual three-dimensional space. For example, the production control unit 216 controls various production objects such as smoke, cherry blossom snow, and spotlight beams.

[0100] The subject object control unit 218 performs automatic control of objects set in the virtual three-dimensional space, such as the subject object 4 and the stage 5. Specifically, subject object control unit 218 automatically controls subject objects in synchronization with the playback of music, based on given control procedure data (or a given control procedure program) in which a control procedure for the subject objects in synchronization with the playback of music is defined. At that time, subject object control unit 218 executes the automatic control for each of a plurality of subject objects. More specifically, subject object control unit 218 controls the movement of the subject object, the movement of the subject object's body, the change in the orientation of the subject object, the control of the subject object taking a given pose, and the control of the subject object taking a given eye-catching action.

[0101] Virtual camera control unit 220 controls the position and orientation of the virtual camera that captures the virtual three-dimensional space in conjunction with the position and orientation of user terminal 1500 acquired by position and orientation acquisition control unit 212 (see FIG. 3).

[0102] The virtual space image generation control unit 222 performs control to generate an image of the virtual three-dimensional space captured by the virtual camera to be displayed on the image display unit 392. The virtual space image generation control unit 222, together with the image generation unit 292, forms a virtual space image generation means that generates an image of the virtual three-dimensional space captured by the virtual camera.

[0103] The augmented reality image generation control unit 224 performs control to generate an augmented reality image 94 (see FIG. 3) to be displayed on the image display unit 392.

[0104] The advance notice target control setting unit 230 sets, based on a user's operational input, the control contents to be the subject of advance notice among the control contents for automatically controlling the subject object by the subject object control unit 218. In this embodiment, this corresponds to the control for the user to select and specify the motion to be the subject of advance notice from among the motions of the subject object 4.

[0105] The specific part setting unit 232 sets a specific part of the subject object based on a user's operation input.

[0106] The preview control unit 234 performs control to preview the automatic control by the subject object control unit 218 and the production control by the production control unit 216. In the present embodiment, the control related to the display of the preview display 20 corresponds to this (see FIG. 6).

[0107] Specifically, the preview control unit 234 previews the automatic control by the subject object control unit 218 based on given control procedure data (or a given control procedure program) in which the control procedure of the subject object is defined. Then, the preview control unit 234 previews at least one of the moving direction, moving distance, moving path, body movement, control for changing the orientation of the subject object, control for making the subject object take a given pose, and control for making the subject object take a given eye-catching action of the subject object. Further, the preview control unit 234 performs control to identify and display the part related to the preview of the body movement of the subject object.

[0108] When there are a plurality of subject objects, the preview control unit 234 previews each subject object individually. When a plurality of subject objects are controlled to perform a linked operation, the linked operation is previewed collectively. Note that when N (N is an integer of N≧3) subject objects appear, cases where M subject objects that are performing a linked action at that time or M (M<N) subject objects that are gathered in a predetermined range are handled collectively are also included. For example, in a case where 5 subject objects appear and 2 are in the first group and 3 are in the second group, and each group performs a linked operation separately, each group referred to here corresponds to the target of individual preview.

[0109] The advance notice control unit 234 can also select at least one of the multiple subject objects and give an advance notice about the selected subject object. For example, the advance notice control unit 234 can select, as an advance notice target, an object whose movement satisfies a predetermined condition (e.g., the largest), an object selected according to a user's selection operation, or an object that next performs a noteworthy action such as a singing action or a musical performance action, among the multiple subject objects. Of course, as long as the predetermined condition is satisfied, multiple subject objects may be selected as advance notice targets.

[0110] Furthermore, the advance notice control unit 234 issues an advance notice regarding the control contents set by the advance notice target control setting unit 230. For example, when the user inputs a selection operation of a subject object to be the subject of the advance notice, an advance notice regarding the selected subject object is issued.

[0111] The prediction unit 240 predicts whether or not an incompatible state that satisfies a predetermined incompatible photographing condition will occur when the subject object control unit 218 performs automatic control related to the content of the notice, assuming that the current position and attitude of the virtual camera will continue, when the notice control unit 234 performs a notice. The incompatible photographing condition is, for example, that a specific part of the subject object is not included in the image captured by the virtual camera, and / or that the proportion of the specific part in the captured image is equal to or less than a predetermined value.

[0112] In other words, prediction unit 240 can predict whether or not a particular part of the subject object will be included in an image captured by the virtual camera when automatic control of the subject object related to the preview content is performed in the future. However, when the specific part setting section 232 sets a specific part of the subject object based on an operational input by the user, the prediction section 240 makes a prediction limited to the specific part.

[0113] The prediction notification control unit 242 performs control to notify the user when a non-conforming state is predicted by the prediction unit 240. In this embodiment, this corresponds to display control of the cut-off prediction notification 30 (see FIG. 8).

[0114] The recording control unit 244 controls the recording of images generated by the virtual space image generation control unit 222 and the augmented reality image generation control unit 224.

[0115] The music playback control unit 246 controls the playback of a given music piece.

[0116] The timekeeping unit 280 uses a system clock to keep track of the current date and time, time limit, and the like.

[0117] The sound generation unit 290 is realized, for example, by a digital signal processor (DSP), a processor such as a voice synthesis IC, or an audio codec capable of playing audio files, and generates sound signals for music, sound effects for performances, background music, and various operation sounds, and outputs them to the sound output unit 390.

[0118] The sound output unit 390 is realized by a device that outputs sound such as sound effects and background music based on the sound signal input from the sound generation unit 290. This corresponds to the speaker 1510 in FIG.

[0119] The image generation unit 292 generates various image data and performs control for displaying the generated images on the image display unit 392. In this embodiment, a virtual space image 90 and an augmented reality image 94 are generated (see FIG. 3).

[0120] The image display unit 392 displays various images based on the image signal input from the image generation unit 292. For example, the image display unit 392 can be realized by an image display device such as a flat panel display, a projector, or a head mounted display. In this embodiment, the touch panel 1506 in FIG. 2 corresponds to this.

[0121] The communication control unit 294 executes data processing related to data communication, and realizes data exchange with external devices via the communication unit 394.

[0122] The communication unit 394 realizes communication by connecting to the network 9. For example, it is realized by a wireless communication device, a modem, a TA (terminal adapter), a jack for a wired communication cable, a control circuit, etc., and corresponds to the wireless communication module 1553 in FIG.

[0123] The terminal storage unit 500 stores programs for implementing various functions for the terminal processing unit 200 to comprehensively control the user terminal 1500, various data, etc. It is also used as a working area for the terminal processing unit 200, and temporarily stores the results of calculations executed by the terminal processing unit 200 according to various programs, input data input from the operation input unit 100, etc. These functions are implemented by, for example, IC memories such as RAM and ROM, magnetic disks such as hard disks, optical disks such as CD-ROMs and DVDs, etc. The IC memory 1552 and memory card 1540 mounted on the control board 1550 in FIG. 2 correspond to these. A configuration using online storage is also possible.

[0124] FIG. 10 is a diagram showing an example of programs and data stored in the terminal storage unit 500. As shown in FIG. The terminal storage unit 500 stores a terminal program 502, preview icon definition data 510, virtual photography package data 520, virtual photography management data 700, and a current date and time 800. Of course, other programs and data than these can also be stored as appropriate.

[0125] The terminal program 502 is a program that is read and executed by the terminal processing unit 200 to cause the user terminal 1500 to function as the virtual photography management unit 210 (see FIG. 9).

[0126] The preview icon definition data 510 is prepared for each type of preview icon 21, and stores various data for initially setting the preview icon of that type. 11 to 14 are diagrams showing examples of the data configuration of preview icon definition data 510. The preview icon definition data 510 includes a unique preview ID 511, metadata 512, icon data 513, and application requirements 515. Of course, data other than these may also be included as appropriate.

[0127] The metadata 512 stores the ID (name) of the motion or effect to which the preview icon is applied, numerical information (e.g., angle, direction, length ratio) related to the preview icon, and information indicating the category of other preview icons to be displayed together with the preview icon. For example, the metadata 512 of the movement direction / movement path preview icon 21a and the movement distance preview icon 21c sets information on the type of action associated with them (see Figs. 11 and 13). For example, the metadata 512 of the preview icon to be accompanied by the display of the movement direction / movement path preview icon 21a and the movement distance preview icon 21c specifies them as related categories (see Fig. 12).

[0128] The application requirements 515 define the conditions that must be met for the preview icon to be displayed. For example, in the application requirements 515 of a preview icon intended to notify a direction (e.g., a movement direction / movement route preview icon 21a, a front direction preview icon 21d, etc.; see Figure 7), the relative positional relationship and relative orientation relationship that must be satisfied in relation to the actual shooting conditions of the virtual camera C1 are specified.

[0129] For example, the application requirements 515 of a warning icon intended to warn of distance (e.g., travel distance warning icon 21c; see Figure 7) specify a relative length ratio (e.g., the relative length ratio of the travel distance when the length of the shooting frame at that time is taken as the reference length) that must be satisfied in relation to the actual shooting conditions of the virtual camera C1.

[0130] Returning to Fig. 10, the virtual photography package data 520 is a collection of various initial setting data required for providing a virtual photography experience to the user on the user terminal 1500. The virtual photography package data 520 is prepared, for example, by a manufacturer or an operating company of the virtual photography experience providing system 1000, and distributed by the server system 1100. It may also be sold through online shopping. Although only one piece of virtual photography package data 520 is shown in Fig. 10, multiple pieces of virtual photography package data 520 may be prepared, and the user 2 may be able to select the data to use.

[0131] The virtual shooting package data 520 includes a unique package ID 521, virtual three-dimensional space initial setting data 522, specific target initial setting 528, subject object definition data 530, music data 540, vocal assignment data 550, control procedure data 560, and a noteworthy action list 570. Of course, data other than these may be included as appropriate.

[0132] The virtual three-dimensional space initial setting data 522 stores various data that define the initial setting of the virtual three-dimensional space in which the subject object 4 and the virtual camera C1 are placed. For example, the data includes stage object data 524 that defines the stage 5, and performance object data 526 that defines objects for performance display. In addition, the data may include data related to various objects that form the background of the virtual three-dimensional space, as appropriate.

[0133] The specific target initial setting 528 stores an initial value for setting some kind of limitation on the preview display 20. The initial value may be "undefined", which essentially means that there is no limitation.

[0134] The subject object definition data 530 is prepared for each type of subject object 4, and stores various data necessary for displaying and operating the subject object of that type. 15, one subject object definition data 530 includes subject object ID 531, which is unique identification information, model data 532, motion definition data 534, pose definition data 536, and eye-catching action definition data 538. Of course, data other than these may also be included as appropriate.

[0135] Motion definition data 534 is prepared for each type of motion that the subject object performs in virtual photography. In this embodiment, subject object 4 dances while singing, so motion definition data 534 is prepared for each type of motion that an idol is likely to perform on stage at a concert, such as singing motion, choreography motion, walking on stage, jumping, going up stairs, and going down stairs. Each motion definition data 534 stores a unique motion ID (or a motion name) and motion data in association with each other.

[0136] The pose definition data 536 is data for realizing a choreography of a piece of music or a characteristic pose specific to the subject object 4. A plurality of pose definition data 536 can be set for one subject object 4.

[0137] Eye-catching action definition data 538 is data for realizing the choreography of a piece of music or an eye-catching action specific to the subject object. A plurality of eye-catching action definition data 538 can be set for one subject object 4.

[0138] The music data 540 is a diagram showing an example of the data structure of a music piece to be played back in virtual shooting using the virtual shooting package. In this embodiment, the music piece data 540 is data for playing back the audio of a popular song sung by the subject object 4 in accordance with a predetermined time code.

[0139] The vocal assignment data 550 is prepared when the music data 540 is a popular song to be sung by a plurality of subject objects 4, and indicates which subject object 4 is in charge of singing each of a plurality of parts set in the popular song.

[0140] 16, for example, singing assignment data 550 stores part delimitation timing 552 described based on a time code in association with assigned subject object ID 554. Part delimitation timing 552 indicates the timing based on the time code at which each part starts.

[0141] Returning to FIG. 10, control procedure data 560 is prepared for each subject object 4 that appears in virtual photography using the virtual photography package, and is data that defines a control procedure for automatic control of the subject object in synchronization with the music played in virtual photography using the virtual photography package.

[0142] As shown in FIG. 17, for example, the control procedure data 560 stores an applicable character ID 561 indicating the subject object 4 to which the data is applied, an automatic control start timing 562 based on the time code of the music data 540, an execution start motion ID 564, a motion control setting value 566, and a focus area 568 in association with each other.

[0143] The motion control setting values ​​566 store detailed setting values ​​for controlling the corresponding motion. For example, in the case of a motion involving movement, data on the direction of movement and the distance of movement is included. Also included is a setting as to which direction is the front when performing each motion. The attention part 568 indicates a part that makes a characteristic movement in the corresponding motion. The control procedure data 560 can also be said to be a given control procedure program.

[0144] Returning to Fig. 10, noteworthy action list 570 is data that lists actions that should be noted at any given time during the automatic control of all subject objects 4. For example, as shown in Fig. 18, action start timing 572 based on the time code of music data 540 and assigned character ID 574 indicating subject object 4 responsible for the noteworthy action are stored in association with each other. Of course, data other than these can also be included as appropriate.

[0145] The rendering procedure data 580 is data for realizing various rendering displays in synchronization with the playback of music. For example, as shown in Fig. 19, the rendering start timing 582 based on the time code of the music data 540 is stored in association with the rendering content 584. The rendering content 584 stores an object ID specifying the rendering object to be used and control data for the object.

[0146] The notice procedure data 590 is prepared for each subject object 4, and is data that defines the timing and contents of notice display 20 related to automatic control of the subject object 4 in a time series. For example, as shown in FIG. 20, an application character ID 591 indicating a subject object to which the data is applied, notice timing 592 based on the time code of music data 540, and display notice icon setting 594 are stored in association with each other. Of course, data other than these may also be included as appropriate. For example, the duration of notice display 20 may be included in association with each other.

[0147] The advance notice timing 592 is prepared in association with the automatic control start timing 562 (see FIG. 17) of the control procedure data 560 of the subject object 4 to be advanced notice. The advance notice timing 592 is set to a timing that is a given time before the corresponding automatic control start timing 562. The given time can be set appropriately. For example, it is preferable to set it to about 2 to 3 seconds before. The given time is not limited to a fixed value, and a different time may be used depending on the content of the automatic control.

[0148] A category of a preview icon or a type of an individual preview icon is set in the display preview icon setting 594. When the user terminal 1500 performs a new preview display 20 upon arrival of the preview timing 592, if a category is set in the display preview icon setting 594, the user terminal 1500 selects and displays a preview icon that satisfies the application requirements 515 by comparing with the actual shooting conditions of the virtual camera C1 at that time from among the preview icons (determined by the front part of the preview ID 511; see FIG. 11) belonging to the category.

[0149] Therefore, by setting the type of preview icon and its application requirements 515, it becomes possible to notify user 2 of the relative direction in which subject object 4 will move next and the relative direction it will face when viewed from virtual camera C1, based on the relative relationship between the shooting direction of virtual camera C1 and the movement direction and forward direction of subject object 4.

[0150] Returning to FIG. 10, the virtual photography management data 700 stores various data for managing the execution of the virtual photography.

[0151] 21, the virtual photography management data 700 includes a user account 701, a used package ID 703, actual photography condition data 705, virtual three-dimensional space control data 710, specific target setting data 720, advance notice control data 730, cut-off prediction notification control data 732, real space image data 734, virtual space image data 736, augmented reality image data 738, recording data 740, and a time code counter 742. Of course, data other than these may also be included as appropriate.

[0152] Data describing the latest actual shooting conditions is stored in the actual shooting condition data 705. For example, information on the relative position and relative orientation with respect to the AR marker 6 measured by the own device, the control value of the zoom mechanism of the camera 1520 (corresponding to the zoom magnification), etc. are included. When the position and orientation are measured by an external device, the position information and orientation information acquired from the external device are stored.

[0153] Virtual three-dimensional space control data 710 stores various data for managing and controlling objects in the virtual three-dimensional space. In this embodiment, this data includes virtual camera control data 711, subject object control data 713 for each subject object 4, stage object control data 715 for controlling the stage 5, and performance object control data 717. Of course, data other than these may also be included as appropriate.

[0154] The specific target setting data 720 is setting data related to limitations of the preview display 20. It is set before or during the start of virtual photography. The specific target setting data 720 includes, for example, a specific subject object ID 721, a specific body part type 723, and a specific action type 725. Of course, data other than these may also be included as appropriate.

[0155] One or more body parts or actions selected from candidates presented to the user before the start of play are set in the specific body part type 723 and the specific action type 725. The candidates for the specific body part are not limited to parts such as hands, feet, and face, but can also include a group including a wide range of parts such as the upper body, lower body, and whole body.

[0156] Specifically, when the user terminal 1500 detects an input of a predetermined specific target setting operation, it displays a menu screen for specifying a specific subject object, a specific body part, and a specific action type.

[0157] Then, in the menu screen for selecting the specific subject object, a list of subject objects 4 appearing in the virtual photography to be performed is presented, and the user is prompted to select one of them as the specific subject object. Next, in the menu screen for selecting a specific body part, a list of body parts is presented, and the user is prompted to select one or several body parts as the specific body parts. In the menu screen for selecting a specific action type, a list of motions, poses, and eye-catching actions of the subject object 4 set to the specific subject object is presented, and the user is prompted to select one or several body parts as the specific action. When presenting display examples of various lists, the subject object definition data 530 (see FIG. 15) is appropriately referenced.

[0158] The preview display control data 730 is data relating to display control of the preview display 20. The cut-off prediction notice control data 732 is data related to display control of the cut-off prediction notice 30 .

[0159] Next, the operation of the virtual photography experience providing system 1000 of this embodiment will be described. 22 and 23 are flowcharts for explaining the flow of processing in the user terminal 1500. The flow of processing explained here is realized by the user terminal 1500 executing the terminal program 502 in the terminal processing unit 200.

[0160] 22, the user terminal 1500 first accepts a user's selection / designation of a specific subject object, a specific body part, and a specific action to be subject to selective preview display (step S10). The selection results are stored as a specific subject object ID 721, a specific body part type 723, and a specific action type 725 (see FIG. 21).

[0161] Next, the user terminal 1500 initializes the virtual three-dimensional space (step S12). In this embodiment, the user terminal 1500 references the virtual three-dimensional space initial setting data 522 to place the stage 5, which is modeled after a concert hall, the subject object 4, and the virtual camera C1 in their initial states.

[0162] Next, the user terminal 1500 starts control of acquiring position and orientation information and capturing images using the camera 1520, and starts interlocking control of the virtual camera C1 (step S14). Next, the user terminal 1500 starts generating the virtual space image 90 and the augmented reality image 94, and displaying the shooting operation screen including the augmented reality image 94 (step S16).

[0163] Then, when input of a predetermined recording start operation is detected, the user terminal 1500 starts recording the augmented reality image 94 (step S18), and starts playing music, automatic control of the subject object 4, and performance control (step S20).

[0164] When playback of the music starts, the user terminal 1500 uses the time code counter 742 (see FIG. 21) to measure the time that has elapsed since the start of playback of the music. After this, automatic control of the subject object 4 and control of the stage production are performed based on the value indicated by the time code counter 742.

[0165] When the time code counter 742 matches any of the notice timings 592 in the notice procedure data 590 (see FIG. 20) applied to the subject object 4 indicated by the specific subject object ID 721 (YES in step S30), the user terminal 1500 primarily selects the notice icon indicated by the display notice icon setting 594 associated with the notice timing (step S32).

[0166] Next, the user terminal 1500 secondarily selects, from among the primarily selected preview icons, the attention part preview icon 21h corresponding to the part indicated by the specific part type 723, and secondarily selects the action type preview icon 21b corresponding to the action indicated by the specific action type 725 (step S34). Note that, if the specific action type 725 is an action related to movement, the movement direction movement path preview icon 21a and the movement distance preview icon 21c are secondarily selected together with the corresponding action type preview icon 21b.

[0167] Next, the user terminal 1500 starts control to display the secondary selected preview icon on the shooting operation screen for a certain period of time (step S36; see FIG. 6). Then, the user terminal 1500 executes a cut-off prediction determination and notifies the user of the prediction result (step S38; see FIG. 8).

[0168] Next, the user terminal 1500 refers to the singing assignment data 550 (see FIG. 16), and when it reaches a predetermined time (e.g., about 2 to 3 seconds) before the next part separation timing 552 (YES in step S40), it selects the attention object preview icon 21k (see FIG. 7) of the subject object 4 indicated by the assigned subject object ID 554 (step S42), and starts control to display this on the shooting operation screen for a certain period of time (step S44).

[0169] 23, the user terminal 1500 refers to the attention action list 570 (see FIG. 18), and when it reaches a predetermined time (e.g., about 2 to 3 seconds) before the next action start timing 572 (YES in step S50), selects the attention object preview icon 21k of the subject object 4 indicated by the assigned character ID 574 (step S52). Then, the user terminal 1500 starts control to display the selected preview icon for a certain period of time on the shooting operation screen (step S54).

[0170] User terminal 1500 repeatedly executes steps S30 to S54 until it detects a predetermined input of an operation to end recording (NO in step S70). If an input of an operation to end recording is detected (YES in step S70), user terminal 1500 ends recording (step S72) and ends the series of processes.

[0171] As described above, according to this embodiment, a new support technique for shooting a video of a subject in a virtual three-dimensional space can be provided.

[0172] Specifically, the user 2 can control the virtual camera C1 in conjunction with the user terminal 1500 (portable computer) held by hand in the same manner as when shooting with a handheld video camera. During the virtual shooting process, the user 2 is notified of the automatic control of the subject object 4. Therefore, the user 2 can perform shooting operations while mentally preparing, which makes it easier to shoot with a desired composition.

[0173] Second Embodiment Next, a second embodiment to which the present invention is applied will be described. In the following, differences from the first embodiment will be mainly described, and the same components as those in the first embodiment will be given the same reference numerals as those in the first embodiment, and duplicated explanations will be omitted.

[0174] In the first embodiment, the virtual photography management unit 210 (see FIG. 9) is realized by the user terminal 1500, but in the second embodiment, the virtual photography management unit 210 is realized by the server system 1100. Therefore, the user terminal 1500 in this embodiment is positioned as a terminal that accepts operation input and displays virtually captured images.

[0175] 24 is a functional block diagram showing an example of a functional configuration of a server system 1100 in this embodiment. The server system 1100 includes an operation input unit 100s, a server processing unit 200s, a sound output unit 390s, an image display unit 392s, a communication unit 394s, and a server storage unit 500s.

[0176] The operation input unit 100s is a means for inputting various operations for managing the server, and corresponds to the keyboard 1106 in FIG.

[0177] The server processing unit 200s is realized by electronic components such as processors serving as arithmetic circuits, such as a CPU, GPU, ASIC, and FPGA, as well as IC memories, and controls input and output of data between each functional unit including the operation input unit 100s and the server storage unit 500s.The server processing unit 200s performs various types of arithmetic processing based on predetermined programs and data, operation input signals from the operation input unit 100s, data received from the user terminal 1500, etc., and performs integrated control of the operation of the server system 1100.

[0178] The server processing unit 200s includes a user management unit 202, a virtual photography management unit 210, a timer unit 280s, a sound generation unit 290s, an image generation unit 292s, and a communication control unit 294s. Of course, other functional units may also be included as appropriate.

[0179] The user management unit 202 performs processing related to user registration procedures and stores and manages information on each user associated with a user account.

[0180] The timekeeping unit 280s uses a system clock to keep track of the current date and time, time limit, and the like.

[0181] The sound generation unit 290s is realized by executing an IC or software that generates or decodes audio data, and generates or decodes audio data such as for system management of the server system 1100, operation sounds, and music. Then, the sound signal related to the system management is output to the sound output unit 390s.

[0182] 1, the sound output unit 390s corresponds to a speaker (not shown) provided in the main device or the touch panel 1108.

[0183] The image generating unit 292s generates images and outputs image signals for displaying the images on the image display unit 392s. In this embodiment, the image generating unit 292s is responsible for part or all of the functions of generating images such as images related to system management of the server system 1100 and images used for virtual photography (virtual space images 90 and augmented reality images 94).

[0184] The communication control unit 294s executes data processing related to data communication, and realizes data exchange with an external device via the communication unit 394s.

[0185] The communication unit 394s realizes communication by connecting to the network 9. For example, it is realized by a wireless communication device, a modem, a TA (terminal adapter), a jack for a wired communication cable, a control circuit, etc. In the example of FIG. 1, it corresponds to the communication device 1153.

[0186] The server storage unit 500s stores programs and various data for implementing various functions for the server processing unit 200s to comprehensively control the server system 1100. It is also used as a working area for the server processing unit 200s, and temporarily stores the results of calculations executed by the server processing unit 200s according to various programs. This function is implemented by, for example, IC memory such as RAM or ROM, magnetic disks such as hard disks, optical disks such as CD-ROMs or DVDs, online storage, etc. In the example of FIG. 1, this corresponds to storage media such as IC memory 1152 and hard disks mounted on the main unit, and storage 1140.

[0187] 25 is a diagram showing examples of programs and data stored in the server storage unit 500s. The server storage unit 500s stores a server program 501, a distribution client program 503, preview icon definition data 510, and virtual shooting package data 520. The server storage unit 500s also stores user management data 600, virtual shooting management data 700, and current date and time 800 as data that is generated and managed sequentially. The server storage unit 500s can also store other programs and data (for example, timers, counters, various flags, etc.) as appropriate.

[0188] The server program 501 is a program for realizing the functions of the user management unit 202 and the virtual photography management unit 210 by being read and executed by the server processing unit 200s.

[0189] The delivery client program 503 is an original client program provided to the user terminal 1500 in this embodiment.

[0190] The user management data 600 is prepared for each registered user, and stores various registration information and service usage history related to the user.

[0191] FIG. 26 is a functional block diagram showing an example of the functional configuration of the user terminal 1500 of this embodiment. The device processing unit 200 of this embodiment includes an operation signal transmission control unit 261, an image display control unit 262, and a position and orientation information provision control unit 263.

[0192] The operation signal transmission control unit 261 executes processing for transmitting various data and requests to the server system 1100 in response to operations performed on the operation input unit 100 .

[0193] The image display control unit 262 performs control for displaying a shooting operation screen and the like based on various data received from the server system 1100 .

[0194] The position and orientation information provision control unit 263 performs control to provide information on the position and orientation of the own device, and information on zoom control in the image capture unit 120, to the server system 1100. Specifically, it performs various processes to periodically transmit information on the relative position, relative orientation, and zoom magnification with respect to the AR marker 6 to the server system 1100.

[0195] The terminal storage unit 500 in this embodiment stores a client program 504. Other data required for virtual photography can be downloaded from the server system 1100 as appropriate and stored.

[0196] The client program 504 is application software that the device processing unit 200 reads out and executes to realize the functions of the user terminal computing unit 260. In this embodiment, the client program 504 is a copy of the distribution client program 503 (see FIG. 25) provided by the server system 1100. The client program 504 may be a dedicated client program, or may be configured by a web browser program, a plug-in that realizes interactive image display, or the like.

[0197] The process flow relating to virtual photography by the server system 1100 in this embodiment is similar to the process flow relating to virtual photography by the user terminal 1500 in the first embodiment. That is, the execution subject of the flowcharts explained with reference to FIGS. 22 and 23 should be changed from user terminal 1500 to server system 1100.

[0198] According to this embodiment, the same effects as those of the first embodiment can be obtained.

[0199] [Modifications] Although the embodiment to which the present invention is applied has been described above, the forms to which the present invention can be applied are not limited to the above-described forms, and constituent elements can be added, omitted, or modified as appropriate.

[0200] [Variation 1] For example, in the above embodiment, the number of subject objects 4 is illustrated as one, but a plurality of objects may be used at the same time.

[0201] [Variation 2] In the above embodiment, an example was shown in which the image in the final form of the virtual photography is the augmented reality image 94, but it is also possible to omit the synthesis with the real space image 92 and configure the final form of the virtual photography to be the virtual space image 90. In that case, instead of an AR image, the 3DCG image itself of the virtually photographed virtual three-dimensional space is displayed or recorded.

[0202] [Variation 3] Furthermore, in the above embodiment, the configuration in which the AR marker 6 is used to measure the position and orientation of the user terminal 1500 has been exemplified, but the position and orientation may be measured using a markerless augmented reality technique.

[0203] [Variation 4] In the above embodiment, an example was shown in which a virtual character having a human form was photographed singing and dancing to music, but the subject object 4 may be an animal, a plant, a landscape, a vehicle, etc. Depending on the subject object, it may be possible to omit playing music or omit motion control synchronized with music as appropriate.

[0204] [Variation 5] In addition, in the above embodiment, an example was shown in which the preview display 20 is executed by referring to the preview procedure data 590, but it is also possible to configure it to be executed by referring to the control procedure data 560 (see Figure 17) or the performance procedure data 580 (see Figure 19) of the subject object 4.

[0205] To give a specific example of referring to the control procedure data 560, the advance notice timing is regarded as a predetermined time before the next automatic control start timing 562. Then, the operation type advance notice icon 21b corresponding to the motion indicated by the execution start motion ID 564 is selected.

[0206] Next, metadata 512 of the selected action type preview icon is referenced (see FIG. 12), and a preview icon of a related category set in metadata 512 is selected. At that time, if information relating to the movement direction, movement distance, and front direction is set in motion control setting values ​​566 (see FIG. 17) of control procedure data 560, a preview icon of a related category is selected based on a relative relationship with the latest state of virtual camera C1 (virtual camera control data 711; see FIG. 21).

[0207] For example, if movement direction information is set in the motion control setting value 566, a preview icon in a related category is selected in which the relative directional relationship between the movement direction and the shooting direction of the virtual camera C1 satisfies the application requirement 515.

[0208] When a movement distance is set, a preview icon in a related category is selected whose length ratio based on the movement distance and the length of the shooting frame geometrically calculated from the shooting angle of view of virtual camera C1 (e.g., screen vertical length, screen horizontal length, screen diagonal length, etc.) satisfies application requirement 515.

[0209] If information on the forward direction is set in the motion control setting value 566, a preview icon in a related category is selected in which the relative directional relationship between the forward direction and the shooting direction of the virtual camera C1 satisfies the application requirement 515.

[0210] Furthermore, if information on the type of facial expression to be applied is stored in association with the control procedure data 560, the facial expression preview icon 21g can be selected based on that information.

[0211] When referring to the performance procedure data 580 without referring to the preview procedure data 590, a predetermined time before the performance start timing 582 is regarded as the preview timing, and the performance name (performance ID) included in the performance content 584 is selected from the performance preview icon 21j (see Figure 7) included in the metadata 512.

[0212] [Variation 6] In the above embodiment, an example has been shown in which information on position and orientation is acquired by measuring the position and orientation of the user terminal 1500. In contrast, in cases where information on the position and orientation of the virtual camera C1 linked to the virtual camera C1 can be acquired by an external device and where a virtual space image (VR image) is sufficient as the image to be virtually captured, a stationary game device in which a game screen is viewed on a video monitor or a VR head-mounted display, a so-called home game device, can also be used as the user terminal 1500.

[0213] Specifically, the user holds a game controller in his / her hand as a substitute for a handheld camera used in virtual photography. The position of the game controller in real space can be obtained from an image captured by a camera connected to a stationary game device or information measured by a distance measuring sensor, and the orientation of the game controller in real space can be obtained based on measurement data from a three-axis gyro and a three-axis acceleration sensor mounted on the game controller.

[0214] [Variation 7] Furthermore, although the advance notice display 20 in the above embodiment is realized by displaying an advance notice icon as a symbol mark, the advance notice may be realized in other ways. For example, the advance notice display 20 may be realized using text information (for example, text of the name of the action, a number and unit of the distance traveled, etc.). Alternatively, this may be realized by displaying a semi-transparent image or a contour image of the subject object 4 at a predetermined time in the future.

[0215] [Variation 8] In the above embodiment, the method of determining whether or not the currently captured image can be cut off is used to determine whether or not the image can be cut off. Alternatively, the following determination method may be used. That is, while leaving the current position and orientation of the virtual camera unchanged, an image is provisionally drawn in which only the movement and position of the subject object 4 are a predetermined time into the future. If this provisionally drawn image includes either the part predicted by the attention part prediction icon 21h in the preview display 20 or the part separately specified as a specific part by the user 2, the prediction of the occurrence of the cut-out may be determined to be negative, and if not, the prediction of the cut-out may be determined to be positive. In other words, even if the subject object is included in the current shooting frame, if the cut-out occurs depending on the future movement of the subject object, it may be considered that the cut-out has occurred.

[0216] In this way, when determining cut-off prediction based on future images under predetermined assumptions, if there is a high possibility that cut-off will occur as a result of a future action (such as a large jump) of the subject object 4, the user can be notified of this. Based on this notification, the user can prevent cut-off from occurring by operating to widen the shooting range or adjusting the shooting direction in advance.

[0217] In addition, in predicting whether or not an object is cut off, a more detailed condition can be added by determining whether or not the object is captured within the photographic frame. For example, the reference area facing the virtual camera and the in-frame area included in the captured frame may be calculated for the part of the image predicted by the attention part prediction icon 21h in the prediction display 20 and for the part separately designated by the user 2 as a specific part, and if the in-frame area does not reach a predetermined ratio (e.g., 80%) of the reference area, the prediction of the occurrence of the cut-off may be judged to be positive.

[0218] For example, in the above embodiment, "within the shooting frame" when "determining whether or not an object appears within the shooting frame" refers to the entire shooting frame, but it is also possible to set a judgment area in a specified range of the shooting frame (for example, the center (3 / 4 or 2 / 3 range vertically and horizontally)) and determine whether or not an object appears within that judgment area. [Explanation of symbols]

[0219] 4…Subject object 5. Stage 6...AR marker 20...Preview 21a…Movement direction / route notification icon 21b…Action type preview icon 21c…Travel distance forecast icon 21d…Front direction warning icon 21e… Pose type preview icon 21F…Eye-catching preview icon 21g…Expression preview icon 21h…Attention area preview icon 21j…Preview icon 21k…Featured object preview icon 30…Notification of cut-off prediction 90…Virtual space image 94…Augmented reality images 200...Terminal processing section 210…Virtual Photography Management Department 212... Position and orientation acquisition control unit 214…Virtual space setting section 216…Production control unit 218...Subject object control unit 220...Virtual camera control unit 222...Virtual space image generation control unit 224...Augmented reality image generation control unit 230…Preview target control setting section 232...Specific part setting section 234...Preview control unit 240…Prediction Department 242...Prediction notification control unit 244…Recording control unit 246...Music playback control unit 500…Device memory section 502...Terminal program 510...Preview icon definition data 513...Icon data 515…Applicable requirements 520...Virtual shooting package data 522…Virtual 3D space initial setting data 530...Subject object definition data 534...Motion definition data 540...Song data 560...Control procedure data 562…Automatic control start timing 564…Start motion ID 566...Motion control setting value 580... Production procedure data 590...Preliminary procedure data 592…Preview timing 594…Display preview icon setting 700…Virtual photography management data 710…Virtual 3D space control data 713…Subject object control data 720…Specific target setting data 721…Specific subject object ID 723…Specific part type 725: Specific action type 730…Preview display control data 732…Broken image prediction notification control data 740…Recording data 1000...Virtual photography experience system 1500: User terminal

Claims

1. A program for causing a portable computer having a camera and a display unit to execute control for allowing a user to operate the computer and enjoy virtual photography by photographing a virtual subject object, the program comprising: a virtual space setting means for setting a virtual three-dimensional space in which the subject object is placed; a subject object control means for automatically controlling the subject object; an image display control means for controlling a virtual camera that captures the virtual three-dimensional space in conjunction with the capture direction of the camera, generating an image of the virtual three-dimensional space based on the virtual camera, and displaying the image on the display unit, thereby performing image display control for the virtual capture; a notice control means for giving a notice of automatic control by the subject object control means; A program for causing the computer to function as a

2. A program for causing a portable computer having a camera and a display unit to execute control for allowing a user to operate the computer and enjoy virtual photography by photographing a virtual subject object, the program comprising: a virtual space setting means for setting a virtual three-dimensional space in which the subject object is placed; a subject object control means for automatically controlling the subject object; an image display control means for controlling a virtual camera that captures the virtual three-dimensional space in conjunction with the capture direction of the camera, and performing AR control for generating an image of the virtual three-dimensional space based on the virtual camera and displaying the image on the display unit, thereby performing image display control for the virtual capture; a notice control means for giving a notice of automatic control by the subject object control means; A program for causing the computer to function as a

3. the subject object control means automatically controls the subject object based on given control procedure data or a given control procedure program (hereinafter collectively referred to as "control procedure data") in which a control procedure for the subject object is defined; the advance notice control means gives an advance notice of the automatic control by the subject object control means based on the control procedure data.

3. The program according to claim 1 or 2.

4. a music playback control means for controlling the playback of a given music piece; and causing the computer to function as the control procedure data defines a control procedure synchronized with the reproduction of the music piece, the subject object control means controls the subject object in synchronization with the reproduction of the music by the music reproduction control means; The program according to claim 3.

5. the subject object control means performs control to move the subject object; the advance notice control means advances at least one of a moving direction, a moving distance, and a moving path of the subject object; The program according to any one of claims 1 to 4.

6. the subject object is a character object, the subject object control means controls a body movement of the subject object; the advance notice control means advances a body movement of the subject object; The program according to any one of claims 1 to 5.

7. the notice control means performs control for identifying and displaying a part related to the notice of the body movement of the subject object; The program according to claim 6.

8. the virtual space setting means arranges a plurality of the subject objects in the virtual three-dimensional space; the subject object control means automatically controls each of the plurality of subject objects; the advance notice control means performs an individual advance notice control for the subject object; The program according to any one of claims 1 to 7.

9. the advance notice control means selects at least one of the plurality of subject objects and gives an advance notice about the selected subject object; The program according to claim 8.

10. the advance notice control means selects a subject object whose movement satisfies a predetermined condition from among the plurality of subject objects; The program according to claim 9.

11. the virtual space setting means sets the virtual three-dimensional space as a space in which a concert by the plurality of subject objects is to be held, the subject object control means automatically controls a singing action of each of the subject objects; the notice control means selects, from among the plurality of subject objects, the subject object which will next perform a singing action; The program according to claim 9.

12. the subject object control means controls the plurality of subject objects to perform a coordinated action; The advance notice control means advances the linkage operation. The program according to any one of claims 8 to 11.

13. a notice target control setting means for setting, based on a user's operation input, a control content to be a target of the notice among the control contents for automatically controlling the subject object by the subject object control means; and causing the computer to function as The advance notice control means issues an advance notice regarding the control contents set by the advance notice target control setting means. The program according to any one of claims 1 to 12.

14. a prediction means for predicting whether or not a specific part of the subject object will be included in an image captured by the virtual camera when automatic control of the subject object control means relating to the content of the prediction is performed in the future, on the assumption that the current position and attitude of the virtual camera will continue when the prediction control means makes a prediction; a prediction notification control means for controlling a notification to a user when a negative prediction is made by the prediction means; The program according to any one of claims 1 to 13, for causing the computer to function as a

15. The operation of the user in the virtual shooting is a handheld operation. The program according to any one of claims 1 to 14.

16. A portable electronic device that includes a camera and a display unit and controls a user to enjoy virtual photography by photographing a virtual subject object, a virtual space setting means for setting a virtual three-dimensional space in which the subject object is arranged; a subject object control means for automatically controlling the subject object; an image display control means for controlling a virtual camera that captures the virtual three-dimensional space in conjunction with a capture direction of the camera, generating an image of the virtual three-dimensional space based on the virtual camera, and displaying the image on the display unit, thereby performing image display control for the virtual capture; a notice control means for notifying the subject object control means of automatic control; Electronic equipment equipped with

17. A portable electronic device that includes a camera and a display unit and controls a user to enjoy virtual photography by photographing a virtual subject object, a virtual space setting means for setting a virtual three-dimensional space in which the subject object is arranged; a subject object control means for automatically controlling the subject object; an image display control means for controlling a virtual camera that captures the virtual three-dimensional space in conjunction with a capture direction of the camera, and performing AR control for generating an image of the virtual three-dimensional space based on the virtual camera and displaying the image on the display unit, thereby performing image display control for the virtual capture; a notice control means for notifying the subject object control means of automatic control; Electronic equipment equipped with

18. A control method for controlling a portable computer having a camera and a display unit to allow a user to enjoy virtual photography by photographing a virtual subject object, comprising the steps of: setting a virtual three-dimensional space in which the subject object is arranged; automatically controlling the subject object; a virtual camera that captures the virtual three-dimensional space is controlled in conjunction with a capture direction of the camera, and an image of the virtual three-dimensional space based on the virtual camera is generated and displayed on the display unit, thereby performing image display control for the virtual capture; performing a control to provide a notice of the automatic control of the subject object; A control method comprising:

19. A control method for controlling a portable computer having a camera and a display unit to allow a user to enjoy virtual photography by photographing a virtual subject object, comprising the steps of: setting a virtual three-dimensional space in which the subject object is arranged; automatically controlling the subject object; a virtual camera that captures the virtual three-dimensional space is controlled in conjunction with a shooting direction of the camera, and an AR control is performed to generate an image of the virtual three-dimensional space based on the virtual camera and display the image on the display unit, thereby performing an image display control for the virtual photography; performing a control to provide a notice of the automatic control of the subject object; A control method comprising:

Citation Information

Patent Citations

  • Program, and image processor with computer for executing program

    JP2015212965A

  • Information processing method, apparatus, and program for implementing that information processing method in computer

    JP2018175505A

  • Representation method and operation object

    WO2007077851A1