Information processing apparatus, control method for information processing apparatus, and program

Abstract

To provide an information processing apparatus, a control method for the information processing apparatus, and a program capable of preventing unnecessary objects from entering the shooting range of a virtual camera in a XR space.SOLUTION: A display control apparatus 100 comprises a CPU 106 that is configured to: determine whether or not an operation for selecting an icon 618 has been performed; and display, if it is determined that the operation for selecting the icon 618 has been performed, a shooting angle 619 indicating the shooting range of a virtual camera 610 by superimposing it on a virtual space 600 or an augmented reality space 609.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] In recent years, the development of cross-reality (XR) has been remarkable. Cross-reality is a technology that realizes the fusion of the real world and the virtual world, and is a general term for virtual reality (VR), augmented reality (AR), and mixed reality (MR). Virtual reality is a technology in which a computer generates (simulates) an environment in which a user can interact with virtual objects. The simulation content includes images, sounds, tactile feedback, and other pseudo-sensory information, and provides the user with a recognition similar to the real space. In the following description, the three-dimensional (3D) space generated by such a simulation is referred to as a "virtual space". The user can visually recognize the virtual space through a head-mounted display device or other display devices.

[0003] The user can interact with virtual objects in the virtual space through a motion sensor or a controller. In the virtual space, the user can communicate with other users through a virtual character called an avatar. Furthermore, some virtual reality platforms provide a camera function that can capture the virtual space with a virtual camera. In the camera function, it is possible to leave memories by taking pictures of friends or scenery of the avatar display in the virtual space with a virtual camera in the same way as a camera in the real space. Augmented reality is a simulation technology that superimposes virtual objects on the real space. In the following description, the three-dimensional space constituted by the augmented reality simulation is referred to as an "augmented reality space". Some augmented reality platforms also provide a camera function that can capture the augmented reality space with a virtual camera.

[0004] Furthermore, mixed reality is a simulation technology that further extends augmented reality, superimposes things that are not actually in that location onto the real world and the virtual world for display, and can be experienced from free positions and angles. In the following description, the three-dimensional space constituted by the simulation of mixed reality is referred to as the "mixed reality space". Some mixed reality platforms also provide a camera function that can capture the mixed reality space with a virtual camera. In the following description, when the virtual space, augmented reality space, and mixed reality space are not distinguished and collectively referred to, they are denoted as "XR space".

[0005] By the way, when taking a photo with a camera in the real space, a person who is the subject of the photo cannot clearly grasp whether they are within the shooting range. Therefore, there are many cases where the person who is the subject of the photo moves into the shooting range according to the instructions of the photographer. In such cases, for example, a person who is not the subject of the photo may enter the shooting range casually, causing the photographer to accidentally include the person who is not the subject of the photo in the photo and unable to take the expected image. Also, a person who is not the subject of the photo may not notice the shooting and accidentally enter the shooting range, resulting in being photographed without permission.

[0006] These events can also occur similarly when taking a photo with a virtual camera in the XR space. Regarding this point, some platforms provide functions to hide the avatars of other users close to the user or to hide the avatars of other users who are not registered as friends. Also, Patent Document 1 discloses a technique of using the parameters of a camera for shooting in the real space to display assist information regarding positions and movements for good shooting in the augmented reality space of the AR glasses worn by the wearer.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] However, in the technology disclosed in the above-mentioned Patent Document 1, assist information is not displayed on the AR glasses of a person who is not the shooting target. Also, in the function of making the avatars of other users close to the user invisible, depending on the distance from the user, the avatars of other users may be displayed. Furthermore, in the function of making the avatars of other users who are not registered as friends invisible, the avatars of other users will be displayed regardless of whether they are the shooting target or not as long as they are registered as friends. Therefore, even if the above-described technologies and functions are applied, there is a possibility that avatars that are not the shooting target may enter the shooting range of the virtual camera in the XR space.

[0009] The present invention has been made in view of the above problems. An object of the present invention is to provide an information processing apparatus, a control method for the information processing apparatus, and a program that can prevent unnecessary objects from entering the shooting range of a virtual camera in the XR space.

Means for Solving the Problems

[0010] In order to achieve the above object, the information processing apparatus of the present invention includes: a first determination means for determining whether an instruction operation has been performed; and a first display control means for, when the first determination means determines that the instruction operation has been performed, superimposing and displaying a shooting angle of view indicating the shooting range of a virtual camera on the XR space.

Effects of the Invention

[0011] According to the present invention, it is possible to prevent unnecessary objects from entering the shooting range of a virtual camera in the XR space.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the present embodiments. For example, each part constituting the present invention can be replaced with any configuration capable of exhibiting the same function. Also, any component may be added. Also, any two or more configurations (features) in the present embodiment can be combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] Hereinafter, this embodiment will be described with reference to FIGS. 1 to 8. FIG. 1(a) is a diagram showing the appearance of one side of a display control device 100 which is an example of the information processing apparatus of the present invention. The display 101 is a display unit that displays images and various information. As will be described later, the display 101 is integrally configured with a touch panel 102a so that touch operations on the display surface of the display 101 can be detected. The display control device 100 can display VR content or AR content on the display 101. The touch panel 102a, together with buttons 102b, 102c, 102d, 102e, etc., constitutes an operation unit 102 described later. The button 102b is a power button used when the user switches the power of the display control device 100 on and off.

[0015] The buttons 102c and 102d are volume buttons used when the user increases or decreases the volume of the sound output from the sound output unit 103 described later. The button 102e is a home button used when the user causes the home screen to be displayed on the display 101. The audio output terminal 103a is a headphone jack, which is a terminal for outputting sound to headphones, an external speaker, etc. The speaker 103b is a built-in speaker in the main body that emits sound. FIG. 1(b) is a diagram showing the appearance of the other side of the display control device 100. The lens 104 is a lens of a camera capable of taking pictures. Note that the format of the image taken through the lens 104 may be a still image or a moving image.

[0016] FIG. 1(c) is a block diagram showing a configuration example of the display control device 100. The display control device 100 can be configured using a smartphone or the like. In the display control device 100, a CPU 106, a memory 107, a non-volatile memory 108, an image processing unit 109, a display 101, an operation unit 102, a storage medium I / F 110, an external I / F 111, and a communication I / F 112 are connected to an internal bus 105. In the display control device 100, further, an audio output unit 103, a posture detection unit 113, a self-position and surrounding environment estimation unit 114, and an imaging unit 115 are connected to the internal bus 105. Each unit connected to the internal bus 105 is enabled to exchange data with each other via the internal bus 105.

[0017] The CPU 106 is a control unit that controls the entire display control device 100 and consists of at least one processor or circuit. The memory 107 is composed of, for example, a RAM (such as a volatile memory using semiconductor elements). The CPU 106 controls each unit of the display control device 100 using the memory 107 as a work memory according to a program stored in the non-volatile memory 108, for example. In the non-volatile memory 108, image data, music data, other data, various programs for the CPU 106 to operate, and the like are stored. The non-volatile memory 108 is composed of, for example, a flash memory or a ROM. The image processing unit 109 performs various image processes on the image data stored and saved in the non-volatile memory 108 and a storage medium 116 described later, the image data acquired via the external I / F 111, the image data acquired via the communication I / F 112, and the like under the control of the CPU 106.

[0018] The image processing performed by the image processing unit 109 includes A / D conversion processing, D / A conversion processing, encoding processing of image data, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, color conversion processing, and the like. Further, regardless of whether it is an omnidirectional image, the image processing unit 109 also performs various image processes such as panorama expansion of a wide-range image having a wide range of data, mapping processing, and conversion. Note that the image processing unit 109 may be configured by dedicated circuit blocks for performing specific image processing. Also, depending on the type of image processing, it is also possible for the CPU 106 to perform image processing according to a program without using the image processing unit 109.

[0019] On the display 101, a GUI screen composed of an image and a GUI (Graphical User Interface) or the like is displayed under the control of the CPU 106. The CPU 106 generates a display control signal according to a program so that the image signal generated for display on the display 101 is output to the display 101, and controls each part of the display control device 100. On the display 101, an image is displayed based on the image signal output to the display 101 in this way. Note that the configuration of the display control device 100 itself may include up to an interface for outputting an image signal for display on the display 101, and the display 101 may be configured by an external monitor (such as a television).

[0020] The operation unit 102 is an input device for receiving user operations. The operation unit 102 includes a touch panel 102a, buttons 102b, 102c, 102d, 102e, and a gesture detection unit 102f described later. Note that the operation unit 102 may include a character information input device such as a keyboard, a pointing device such as a mouse, a dial, a joystick, a touch sensor, a touch pad, and the like. The touch panel 102a is an input device configured to be planar by being superimposed on the display 101 and outputting coordinate information corresponding to the touched position.

[0021] The memory medium I / F 110 is capable of mounting a memory medium 116 such as a memory card, CD, or DVD, and based on the control of the CPU 106, reads data from the mounted memory medium 116 and writes data to the memory medium 116. The external I / F 111 is an interface for inputting and outputting image signals and audio signals by connecting to an external device via a wired cable or wirelessly. The communication I / F 112 is an interface for transmitting and receiving various data such as files and commands by communicating with an external device via the Internet 117 or the like. Further, the communication I / F 112 can also communicate with wireless connection controllers 118, 119, etc. described later.

[0022] In the audio output unit 103, audio of image (video) data, audio of music data, operation sounds, incoming call sounds, various notification sounds, etc. are output. The audio output unit 103 includes an audio output terminal 103a and a speaker 103b. In the audio output unit 103, audio output by wireless communication or the like may be performed. The attitude detection unit 113 detects the attitude of the display control device 100 with respect to the direction of gravity and the inclination of the attitude with respect to each axis of yaw, roll, and pitch. Based on the attitude detected by the attitude detection unit 113, the CPU 106 can determine whether the display control device 100 is held horizontally, vertically, upward, downward, or in an oblique attitude.

[0023] At least one of an acceleration sensor, a gyro sensor, a geomagnetic sensor, an azimuth sensor, an altitude sensor, etc. is used in the attitude detection unit 113, but a plurality of them may be used in combination. The self-position and surrounding environment estimation unit 114 estimates the self-position and the surrounding environment. The self-position indicates the position of the display control device 100 or the head-mounted adapter 120 described later in space. The self-position is represented by, for example, three parameters representing the position in a coordinate system defined with three mutually orthogonal axes as the X-axis, Y-axis, and Z-axis with a predetermined position within a predetermined range of space as the origin. The self-position may further be represented by adding three parameters representing the attitude (orientation).

[0024] The term "surrounding environment" refers to an area where there are objects that become obstacles within the range where the user who possesses the display control device 100 or wears the head-mounted adapter 120 described later exists. The area is represented by a plurality of sets of three parameters indicating positions in a coordinate system defined with three mutually orthogonal axes as the X-axis, Y-axis, and Z-axis, respectively, with a predetermined position within a predetermined range of space as the origin. The imaging unit 115 is a camera having a lens 104. The images acquired by the imaging unit 115 are utilized for various detection processes by the self-position and surrounding environment estimation unit 114, the gesture detection unit 102f described later, and the like. Note that the display 101 can also output an image of the outside world.

[0025] Also, as described above, the operation unit 102 includes the touch panel 102a. The CPU 106 can detect the following operations or states with respect to the touch panel 102a. · A finger or pen that was not touching the touch panel 102a newly touches the touch panel 102a, that is, the start of a touch (hereinafter referred to as "Touch-Down"). · A state in which a finger or pen is touching the touch panel 102a (hereinafter referred to as "Touch-On"). · A state in which a finger or pen that is touching the touch panel 102a is moving (hereinafter referred to as "Touch-Move"). · A finger or pen that was touching the touch panel 102a has left the touch panel 102a, that is, the end of a touch (hereinafter referred to as "Touch-Up"). · A state in which nothing is touching the touch panel 102a (hereinafter referred to as "Touch-Off").

[0026] When a touch-down is detected, a touch-on is detected simultaneously. After a touch-down, usually, the touch-on is continuously detected unless a touch-up is detected. Even when a touch-move is detected, a touch-on is detected simultaneously. If the touch position does not move even though a touch-on is detected, the touch-move is not detected. When it is detected that all fingers and pens that were touching the touch panel 102a have touched up, a touch-off is detected. These operations and states, and the position coordinates on the touch panel 102a where the fingers and pens are touching are notified to the CPU 106 through the internal bus 105. The CPU 106 determines what kind of touch operation has been performed on the touch panel 102a based on the notified information.

[0027] Regarding a touch-move, the CPU 106 can also determine for each vertical and horizontal component on the touch panel 102a based on the above-described change in the position coordinates about the moving direction of the finger or pen moving on the touch panel 102a. If the CPU 106 detects a touch-move of a predetermined distance or more, it shall be determined that a slide operation has been performed. An operation where the user quickly moves a finger or pen a certain distance while touching the touch panel 102a and then quickly releases it from the touch panel 102a is called a flick. In other words, a flick is an operation where the user quickly traces the touch panel 102a as if flicking it with a finger or pen.

[0028] When it is detected that a touch-and-move has been performed at a distance equal to or greater than a predetermined distance and at a speed equal to or greater than a predetermined speed, and a touch-up is detected as it is, it is determined that a flick operation has been performed (it is determined that there has been a flick operation following a slide operation). Further, a touch operation in which the user touches a plurality of locations (for example, two points) simultaneously and brings the touch positions closer to each other is called a "pinch-in". Also, a touch operation in which the user touches a plurality of locations (for example, two points) simultaneously and moves the touch positions away from each other is called a "pinch-out". The pinch-out and pinch-in are collectively referred to as a "pinch operation (or simply a pinch)". Note that the CPU 106 can detect touch operations often used in smartphones via the touch panel 102a. Such touch operations include, in addition to the flick operation and pinch operation described above, a single-tap operation, a double-tap operation, a long-press operation, and a swipe operation.

[0029] The touch panel 102a may use any of various methods such as a resistive film method, a capacitance method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, an image recognition method, and an optical sensor method. Further, the touch panel 102a has a method of detecting a touch when there is contact of a finger or a pen with the touch panel 102a, and a method of detecting a touch when there is approach of a finger or a pen to the touch panel 102a, and any of these methods may be used. Also, the operation unit 102 includes a gesture detection unit 102f. The CPU 106 can acquire a gesture image of the user's hand or the like via the imaging unit 115, and the gesture detection unit 102f can detect a predetermined gesture.

[0030] FIG. 1(d) is a diagram showing the appearance of a head-mounted adapter 120 into which the display control device 100 can be inserted. The display control device 100 can be used as a head-mounted display by being inserted into the head-mounted adapter 120. Note that the head-mounted display may be a video see-through type or an optical see-through type. The insertion port 121 is an insertion port for the user to insert the entire display control device 100 into the head-mounted adapter 120. The display control device 100 with the display surface of the display 101 facing the headband 122 side (i.e., the user side) is inserted into the insertion port 121. The user can fix the head-mounted adapter 120 to his / her head by the headband 122. Thus, the user can visually recognize the display 101 of the display control device 100 in a state where the head-mounted adapter 120 into which the display control device 100 is inserted is worn on the head, that is, in a state where the display control device 100 is not held.

[0031] In this case, when the user moves his / her head or the whole body, the posture of the display control device 100 also changes. The posture detection unit 113 detects the change in the posture of the display control device 100 at this time. The CPU 106 performs VR display processing or AR display processing based on the result of this detection. In this case, the posture detection unit 113 detecting the posture of the display control device 100 is equivalent to detecting the posture of the user's head (i.e., the direction in which the user's line of sight is directed). Note that in the display control device 100, the user's line of sight, expression, etc. may be detected, and the detection information may be used as an operation event.

[0032] FIG. 2 is a diagram showing an example of the appearance of controllers 118 and 119 capable of communicating with the display control device 100. The grip-type controller 118 has a hold portion 201 and an operation surface 202. In the grip-type controller 118, the user holds the hold portion 201 with the hand and operates the members on the operation surface 202. Thereby, an operation event is notified from the grip-type controller 118 to the display control device 100. Note that the grip-type controller 118 may be used by the user holding one in each hand.

[0033] Further, the ring-type controller 119 has a ring portion 211 and a ring operation portion 212. The ring portion 211 is worn on the finger 213 of the user. The ring operation portion 212 may be a member such as a push-button or a rotary dial, or a member capable of detecting finger contact such as an optical track pad. The ring operation portion 212 notifies the display control device 100 of an operation event when operated by the user. Hereinafter, the case where the user wears the head-mounted adapter 120 into which the display control device 100 is inserted and uses the controller 118 will be described, but the case where the user uses the controller 119 is the same.

[0034] FIG. 3 is a flowchart showing an example of OS mode processing executed by the display control device 100. When the display control device 100 detects that it has been inserted into the head-mounted adapter 120 or when the user performs a start operation with the controller 119, the OS mode processing in FIG. 3 is started. The OS mode processing (control method of the information processing apparatus) in FIG. 3 is realized by the CPU 106 (computer) expanding and executing a program stored in the non-volatile memory 108 in the memory 107. Note that this also applies to each process shown in the flowcharts of FIGS. 5, 7, and 8 described later. When the OS mode processing in FIG. 3 is started, first, in step S301, the CPU 106 displays on the display 101 a screen for the user to select an application included in the display control device 100.

[0035] FIG. 4 is a diagram showing an example of a screen displayed by OS mode processing, which is a screen for selecting an application included in the display control device 100. In the screen shown in FIG. 4, an application selection screen 401 is superimposed and displayed on the virtual space 400. Icons 402 to 407 indicating various applications are displayed on the application selection screen 401. When the user performs an operation of selecting the icons 402 to 407, the application corresponding to the selected icon is launched. Hereinafter, a case where the virtual space application is launched by selecting the icon 402 and the photo application is launched by selecting the icon 403 will be described. Note that the photo application is an application that reproduces images stored in the display control device 100. Also, there may be a plurality of virtual space applications.

[0036] Even when the user wears the head-mounted adapter 120 into which the display control device 100 is inserted, operations such as selection and movement of GUI objects displayed on the display 101 can be performed by using the controller 118. Further, the user can move the pointer 408 to an arbitrary position by operating the controller 118. The user can select an icon by operating a member of the controller 118 with the pointer 408 brought close to (or superimposed on) an arbitrary icon. Note that these points are the same when the user uses the controller 119. Also, the user can move the pointer 408 to an arbitrary position by the above-described posture detection unit 113, that is, according to the posture of the user and the direction of the line of sight.

[0037] Return to the description of FIG. 3. In step S302, the CPU 106 determines whether a selection operation of the virtual space application has been performed. If the selection operation of the icon 402 has been performed, the CPU 106 determines that the selection operation of the virtual space application has been performed. In this case, the process proceeds to step S303. On the other hand, if the selection operation of the icon 402 has not been performed, the CPU 106 determines that the selection operation of the virtual space application has not been performed. In this case, the process proceeds to step S304. In step S303, the CPU 106 executes the virtual space application process described below.

[0038] In step S304, the CPU 106 determines whether a selection operation of the photo application has been performed. If the selection operation of the icon 403 has been performed, the CPU 106 determines that the selection operation of the photo application has been performed. In this case, the process proceeds to step S305. On the other hand, if the selection operation of the icon 403 has not been performed, the CPU 106 determines that the selection operation of the photo application has not been performed. In this case, the process proceeds to step S306. In step S305, the CPU 106 executes the playback mode process. In the playback mode process, the CPU 106 performs image playback on all of the images stored in the storage medium 116 of the display control device 100.

[0039] In step S306, the CPU 106 determines whether any other operations have been performed. This determination is made based on user operations in the controller 118 or the like. Note that other operations include, for example, pairing operations with other display control devices (e.g., smartphones), operations related to automatic transfer settings of captured images to paired smartphones, and the like. When the CPU 106 determines that other operations have been performed, the process proceeds to step S307. On the other hand, when the CPU 106 determines that no other operations have been performed, the process proceeds to step S308. In step S307, the CPU 106 performs other processes based on the other operations determined in step S306 or the like. In step S308, the CPU 106 determines whether an end operation for the OS mode has been performed. This determination is made based on user operations in the controller 118 or the like. When the CPU 106 determines that the end operation for the OS mode has not been performed, the process returns to step S302. On the other hand, when the CPU 106 determines that the end operation for the OS mode has been performed, the OS mode process in FIG. 3 ends.

[0040] FIG. 5 is a flowchart showing an example of the virtual space application process in step S303. When the virtual space application process in FIG. 5 starts, first, in step S501, the CPU 106 performs a startup process for the virtual space application. The startup process for the virtual space application is realized by the CPU 106 expanding and executing a program stored in the non-volatile memory 108 in the memory 107. Also, the CPU 106 initializes flags, control variables, etc. for the execution of the virtual space application. In step S502, the CPU 106 executes a login process based on the user's account information. Through the login process, the CPU 106 loads the user's avatar, attribute information, etc. from a server that provides the service of the virtual space application.

[0041] In step S503, the CPU 106 displays a virtual space or the like on the display 101. FIGS. 6(a) to 6(h) are diagrams showing an example of a screen displayed by the virtual space application. FIG. 6(a) is a diagram showing an example of a screen in virtual space display. In the screen shown in FIG. 6(a), a virtual space 600, a virtual character 601, and a virtual animal 602 are displayed. The virtual character 601 and the virtual animal 602 may be avatars of other users who share the virtual space 600 with the user by logging in to the virtual space 600, or may be those whose behaviors such as movement are controlled by a server that provides the services of the virtual space application. Note that each screen shown in FIG. 6 is a first-person view screen on which the avatar display of the user wearing the head-mounted adapter 120 is not performed, but may be a third-person view screen on which the avatar display of the user himself / herself is performed.

[0042] Returning to the description of FIG. 5. In step S504, the CPU 106 performs a GUI display for performing various operations in the virtual space application. In FIG. 6(a), the virtual space application GUI 603 is an example of such a GUI display. In the screen shown in FIG. 6(a), the virtual space application GUI 603 is displayed superimposed on the virtual space 600. Icons 604 to 608 indicating various functions are arranged and displayed on the virtual space application GUI 603. The function indicated by the icon 604 is a chat function that enables communication with other users.

[0043] The function indicated by icon 605 is an emotion function that can express emotions. The function indicated by icon 606 is a camera function that can perform shooting with a virtual camera in virtual space 600. The function indicated by icon 607 is a space display switching function that switches between virtual space display and augmented reality space display. The function indicated by icon 608 is an end function that terminates the virtual space application. Note that information on which space display is currently in effect is displayed on icon 607 of the space display switching function. In the screen shown in Fig. 6(a), since virtual space 600 is displayed, information 607a indicating virtual space display is displayed on icon 607 of the space display switching function.

[0044] Also, Fig. 6(e) is a diagram showing an example of a screen in augmented reality space display. In the screen shown in Fig. 6(e), augmented reality space 609 is displayed as a background image. Further, in the screen shown in Fig. 6(e), the above-described virtual character 601, virtual animal 602, and virtual space application GUI 603 are displayed superimposed on augmented reality space 609. In the screen shown in Fig. 6(e), since augmented reality space 609 is displayed, information 607b indicating augmented reality space display is displayed on icon 607 of the space display switching function.

[0045] Returning to the description of Fig. 5. In step S505, CPU 106 determines whether an operation has been performed on virtual space application GUI 603 (excluding icons 606 and 608). If CPU 106 determines that an operation has been performed on virtual space application GUI 603, the process proceeds to step S506. On the other hand, if CPU 106 determines that no operation has been performed on virtual space application GUI 603, the process proceeds to step S507. In step S506, CPU 106 performs processing such as various setting switches based on the operation determined in step S505. For example, when a selection operation of icon 604 is performed, CPU 106 performs processing for communication with other users, and when a selection operation of icon 607 is performed, CPU 106 performs processing for switching the display between virtual space and augmented reality space.

[0046] In step S507, the CPU 106 determines whether a startup operation for the camera mode has been performed. When a selection operation on the icon 606 is performed, the CPU 106 determines that a startup operation for the camera mode has been performed. In this case, the process proceeds to step S508. On the other hand, when no selection operation on the icon 606 is performed, the CPU 106 determines that a startup operation for the camera mode has not been performed. In this case, the process proceeds to step S509. In step S508, the CPU 106 executes camera mode processing, which will be described later. In step S509, the CPU 106 determines whether other operations or the like have been performed. This determination is made based on user operations or the like in the controller 118. When the CPU 106 determines that other operations or the like have been performed, the process proceeds to step S510. On the other hand, when the CPU 106 determines that no other operations or the like have been performed, the process proceeds to step S511.

[0047] In step S510, the CPU 106 performs other processing based on the other operations or the like determined in step S509. Examples of other processing include processing for switching the display range of the virtual space 600 displayed on the display 101 when the user's viewpoint moves due to, for example, a posture movement of the head-mounted adapter 120. In step S511, the CPU 106 determines whether an end operation for the virtual space application has been performed. When no selection operation on the icon 608 is performed, the CPU 106 determines that an end operation for the virtual space application has not been performed. In this case, the process returns to step S505. On the other hand, when a selection operation on the icon 608 is performed, the CPU 106 determines that an end operation for the virtual space application has been performed. In this case, the virtual space application processing in FIG. 5 ends, and the process returns to the OS mode processing in FIG. 3.

[0048] FIG. 7 is a flowchart showing an example of the camera mode process in step S508. When the camera mode process in FIG. 7 starts, first, in step S701, the CPU 106 executes an initialization process for the camera mode. In this process, various parameters set by the user when the camera mode was last used are called. In step S702, the CPU 106 displays a virtual camera 610, which will be described later. In step S703, the CPU 106 displays a live view 611, which will be described later, superimposed on the virtual camera 610. In step S704, the CPU 106 displays a virtual camera GUI, which will be described later.

[0049] FIG. 6(b) is a diagram showing an example of a screen in the camera mode in the virtual space display. In the screen shown in FIG. 6(b), a virtual space 600 is displayed as a background image. Further, in the screen shown in FIG. 6(b), in addition to the virtual person 601 and the virtual animal 602 described above, a virtual camera 610, a live view 611, the user's virtual hand 612, a virtual lens 613, and icons 614 to 617 are displayed superimposed on the virtual space 600. A live view 611 of the virtual camera 610 is displayed superimposed on the virtual camera 610. Also, the state in which the virtual camera 610 is held by the user's virtual hand 612 is displayed. Further, around the virtual camera 610, a virtual lens 613 and icons 614 to 617 indicating various functions are displayed. Here, the icons 614 to 617 correspond to the virtual camera GUI described above.

[0050] The virtual camera GUI is composed of the icon 614 of the shooting start button, the icon 615 of the camera mode end button, the icon 616 for displaying the setting menu screen, and the icon 617 for starting the playback mode, but is not limited to this configuration. Also, Fig. 6(f) is a diagram showing an example of the screen in the camera mode in the augmented reality space display. In the screen shown in Fig. 6(f), the augmented reality space 609 is displayed as the background image. Further, in the screen shown in Fig. 6(f), the above-described virtual character 601, virtual animal 602, virtual camera 610, live view 611, the user's virtual hand 612, virtual lens 613, and icons 614 to 617 are displayed superimposed on the augmented reality space 609. Note that in the virtual space 600 and the augmented reality space 609, the user can perform touch operations often used on a smartphone in the real space with the virtual hand 612 or the like. That is, in the virtual space 600 and the augmented reality space 609, the user can perform single-tap operations, double-tap operations, long-press operations, flick operations, swipe operations, pinch operations, etc. with the virtual hand 612 or the like.

[0051] Returning to the description of Fig. 7. In step S705, the CPU 106 determines whether an operation of the setting menu has been performed. When the selection operation of the icon 616 is performed, the CPU 106 determines that an operation of the setting menu has been performed. In this case, the process proceeds to step S706. On the other hand, when the selection operation of the icon 616 has not been performed, the CPU 106 determines that an operation of the setting menu has not been performed. In this case, the process proceeds to step S707. In step S706, the CPU 106 displays a setting menu (not shown).

[0052] As a result, the user can switch settings such as the format of the image to be captured, the shooting parameters, and the format of the metadata to be attached to the captured image, with respect to the virtual camera 610. The format of the image captured by the virtual camera 610 may be a still image or a moving image. Furthermore, the format of the captured image may be a normal image (2D), or an image of 360° or VR180. The shooting parameters may be the shutter speed, aperture, ISO sensitivity, exposure compensation, etc. Also, settings for image processing such as background blur and beauty effect may be switched. Note that the explanation of the format of the metadata to be attached to the captured image will be given together with the explanations of steps S710 and S711 described later.

[0053] In step S707, the CPU 106 determines whether a shooting operation has been performed. When the selection operation of the icon 614 is performed, the CPU 106 determines that a shooting operation has been performed. In this case, the process proceeds to step S708. On the other hand, when the selection operation of the icon 614 has not been performed, the CPU 106 determines that a shooting operation has not been performed. In this case, the process proceeds to step S713. In step S708, the CPU 106 performs a shooting process. In the shooting process, when the CPU 106 performs shooting of a 2D image, it performs an image shooting process based on the image currently displayed in the live view 611 of the virtual camera 610. Also, when the CPU 106 performs shooting of a 360° image, it shoots the surrounding space visible from the virtual camera 610 as an image. Note that the detailed explanation of the shooting process will be described later.

[0054] In step S709, the CPU 106 determines whether shooting is being performed in virtual space display (VR display). When the CPU 106 determines that shooting is being performed in virtual space display, the process proceeds to step S710. On the other hand, when the CPU 106 determines that shooting is not being performed in virtual space display, that is, when it determines that shooting is being performed in augmented reality space display (AR display), the process proceeds to step S711. In step S710, the CPU 106 attaches metadata based on the virtual space to the captured image. In step S711, the CPU 106 attaches metadata based on the augmented reality space to the captured image.

[0055] The metadata to be provided here may be in any format as long as it contains information that can distinguish between virtual images and non-virtual images. The CPU 106 may only add binary flag information that can distinguish between virtual images and non-virtual images to the captured image, or may add information related to the space of the virtual space and the non-virtual space (for example, position information within the space) and time information to the captured image. Also, the metadata may be in the Exchangeable image file format (Exif). In the case of the Exif format, in addition to the position information, the CPU 106 may add parameters such as the shutter speed at the time of shooting to the captured image based on the Exif specifications.

[0056] In step S712, the CPU 106 executes a saving process. In the saving process, the CPU 106 saves the captured image of the virtual camera 610 to the storage medium 116. At this time, the CPU 106 saves the captured image in a region dedicated to the virtual space application in the storage medium 116. The CPU 106 may further save the captured image in a region that can be played back in the playback mode process in the OS mode process in the storage medium 116. Also, the CPU 106 may directly save the captured image of the virtual camera 610 to an external device or server. In step S713, the CPU 106 determines whether a playback operation has been performed. The CPU 106 determines that a playback operation has been performed when a selection operation of the icon 617 is performed. In this case, the process proceeds to step S714. On the other hand, when the CPU 106 determines that the selection operation of the icon 617 has not been performed, it determines that the playback operation has not been performed. In this case, the process proceeds to step S715.

[0057] In step S714, the CPU 106 executes playback mode processing. In the playback mode processing, the CPU 106 plays back the images stored in the dedicated area for the virtual space application among the images stored in the storage medium 116 of the display control device 100. In step S715, the CPU 106 determines whether any other operations have been performed. This determination is made based on user operations and the like in the controller 118. If the CPU 106 determines that any other operations have been performed, the process proceeds to step S716. On the other hand, if the CPU 106 determines that no other operations have been performed, the process proceeds to step S717.

[0058] In step S716, the CPU 106 performs other processing based on the other operations and the like determined in step S715. In this processing, the CPU 106 changes the display range of the live view 611 of the virtual camera 610, for example, according to user operations in the controller 118 or changes in the posture of the head mount adapter 120. In step S717, the CPU 106 determines whether an operation to end the camera mode has been performed. If the CPU 106 determines that the selection operation of the icon 615 has not been performed, it determines that the operation to end the camera mode has not been performed. In this case, the process returns to step S705. On the other hand, if the CPU 106 determines that the selection operation of the icon 615 has been performed, it determines that the operation to end the camera mode has been performed. In this case, the camera mode processing in FIG. 7 ends, and the process returns to the virtual space application processing in FIG. 5.

[0059] FIG. 8 is a flowchart showing an example of the photographing process in step S708. When the photographing process in FIG. 8 is started, first, in step S801, the CPU 106 (first determination means) determines whether an operation (instruction operation) for fixing the photographing target has been performed (determination step). The CPU 106 determines that an operation for fixing the photographing target has been performed when a selection operation of the icon 618 for fixing the photographing target, which will be described later, is performed. In this case, the process proceeds to step S802. On the other hand, when the selection operation of the icon 618 for fixing the photographing target has not been performed, the CPU 106 determines that the operation for fixing the photographing target has not been performed. In this case, the process proceeds to step S809. Note that the icon 618 for fixing the photographing target is a dedicated icon for the operation of fixing the photographing target, but it may also be an existing icon to which the function of the operation of fixing the photographing target is added.

[0060] In step S802, the CPU 106 (first display control means) superimposes and displays a radially-shaped photographing angle of view 619, which will be described later, in the virtual space 600 or the augmented reality space 609 (display control step). As a result, the radially-shaped photographing angle of view 619 is also superimposed and displayed in the virtual space 600 or the augmented reality space 609 on the display control device of another user logged in to the virtual space 600 or the augmented reality space 609. Further, the CPU 106 displays through the photographing angle of view 619. By the transparent display of the photographing angle of view 619, the background within the photographing angle of view 619 is visible in the virtual space 600 or the augmented reality space 609. Note that step S802 may be performed by a server that provides a service of a virtual space application.

[0061] In step S803, the CPU 106 (fixing means) fixes the object to be photographed. Specifically, in the virtual space 600 or the augmented reality space 609, the CPU 106 fixes an object that fits within the shooting angle 619 of the virtual camera 610, that is, a virtual person 601 or a virtual animal 602, as the object to be photographed by the virtual camera 610. As a result, the user can intentionally fix the object to be photographed by the virtual camera 610. Further, the tracking function of the virtual camera 610 becomes possible. Also, even if an object enters the shooting angle 619 of the virtual camera 610 intentionally (for example, for mischief) or accidentally (for example, passing by), it will not be fixed as the object to be photographed by the virtual camera 610. If an object different from the user's intention is fixed as the object to be photographed, the user performs a predetermined operation of flicking or swiping the object with the virtual hand 612 in the live view 611 of the virtual camera 610. The CPU 601 (excluding means) excludes the object targeted by the flick operation or swipe operation by the user from the object to be photographed. Note that the predetermined operation may be a touch operation other than the flick operation and the swipe operation.

[0062] In step S804, the CPU 106 determines whether there is an entry of a non-photographing target object within the shooting angle 619. If the CPU 106 determines that there is an entry of a non-photographing target object within the shooting angle 619, the process proceeds to step S805. On the other hand, if the CPU 106 determines that there is no entry of a non-photographing target object within the shooting angle 619, the process proceeds to step S806. In step S805, the CPU 106 (non-display means) makes the non-photographing target object non-displayed in the live view 611 of the virtual camera 610. As a result, photographing of the non-photographing target object is prevented by the virtual camera 610. Also, since the non-photographing target object is made non-displayed before shooting, distortion that occurs when deleting the non-photographing target object from the photographed image can be avoided. Note that the subsequent process returns to step S804.

[0063] In step S806, the CPU 106 determines whether to specify a specific shooting target object. This determination is made based on a user operation or the like in the controller 118. If the CPU 106 determines to specify a specific shooting target object, the process proceeds to step S807. On the other hand, if the CPU 106 determines not to specify a specific shooting target object, the process proceeds to step S809. In step S807, the CPU 106 (second determination means) determines whether the user has performed a specific operation (specifying operation) on the shooting target object.

[0064] Note that the specific operation may be an operation in which the user single-taps, double-taps, or long-presses the shooting target object with the virtual hand 612 in the live view 611 of the virtual camera 610. Also, the specific operation may be performed on at least one of the shooting target objects. Note that the specific operation may be a touch operation other than the single-tap operation, double-tap operation, and long-press operation. If the CPU 106 determines that the user has performed a specific operation on the shooting target object, the process proceeds to step S808. On the other hand, if the CPU 106 determines that the user has not performed a specific operation on the shooting target object, the process returns to step S806.

[0065] In step S808, the CPU 106 (second display control means) executes a shooting target specifying process. In the shooting target specifying process, the CPU 106 displays a specific icon around the shooting target object of the virtual camera 610 that has become the target of a specific operation by the user in the virtual space 600 or the augmented reality space 609. Note that instead of displaying a specific icon, the CPU 106 may blink-display the shooting target object that has become the target of a specific operation by the user, or display it with a highlighted border. In this way, the CPU 106 can explicitly display the shooting target object desired by the user among the shooting target objects of the virtual camera 610. Also, other users can recognize the shooting target object of the virtual camera 610 by the display of a specific icon or the like. Note that the CPU 106 may explicitly display all of the shooting target objects of the virtual camera 610. In step S809, the CPU 106 (shooting means) performs shooting with the virtual camera 610. Thereafter, the shooting process in FIG. 8 ends, and the process returns to the virtual space application process in FIG. 5.

[0066] Figures 6(c) and (d) are diagrams showing an example of screen transition in the camera mode in the virtual space display. Figure 6(c) is a diagram showing an example of the screen when the shooting target is fixed. In the screen shown in Figure 6(c), a virtual camera 610 is displayed superimposed on the virtual space 600. A live view 611 is superimposed and displayed on the virtual camera 610. In the live view 611 of the virtual camera 610, the virtual space 600, the virtual person 601, and the virtual animal 602 are displayed as image information entering from the virtual lens 613. In the screen shown in Figure 6(c), an icon 618 for fixing the shooting target is superimposed and displayed on the virtual space 600. When the user selects and operates the icon 618, a shooting angle of view 619 indicating the shooting range of the virtual camera 610 is displayed (step S802). At this time, the shape of the shooting angle of view 619 of the virtual camera 610 is radial spreading from the virtual lens 613. Also, since the shooting angle of view 619 of the virtual camera 610 is displayed transparently, the virtual person 601 and the virtual animal 602 within the shooting angle of view 619 of the virtual camera 610 and the background within the shooting angle of view 619 of the virtual camera 610 become visible.

[0067] At the same time, the virtual person 601 and the virtual animal 602 within the shooting angle of view 619 of the virtual camera 610 are fixed to the shooting target object (step S803). Figure 6(d) is a diagram showing an example of an image when the non-shooting target object 620 enters the shooting angle of view 619 of the virtual camera 610 when the shooting target is fixed. When the shooting target object is fixed, even if the non-shooting target object 620 enters the shooting angle of view 619 of the virtual camera 610, the non-shooting target object 620 is not displayed in the live view 611 of the virtual camera 610 (step S805).

[0068] Also, by the shooting target specifying process in step S808, a specific icon 621 is displayed above the head of the virtual character 601 which is the shooting target object. Note that the display position of the specific icon 621 is not limited to above the head of the virtual character 601 which is the shooting target object, and it may be around the virtual character 601 which is the shooting target object (for example, beside the face, etc.). FIGS. 6(g)(h) are diagrams showing an example of screen transition in the camera mode in the augmented reality space display. The description of FIGS. 6(g)(h) is omitted because it is the same as the description of FIGS. 6(c)(d) above.

[0069] As described above, in the display control device 100, when the user makes a selection operation on the icon 618, the shooting angle of view 619 indicating the shooting range of the virtual camera 610 is superimposed and displayed on the virtual space 600 and the augmented reality space 609. Thereby, in the virtual space 600 and the augmented reality space 609, other users can visually recognize the shooting angle of view 619 of the virtual camera 610 in the display control device 100, so that it is possible to prevent unnecessary objects from entering the shooting range of the virtual camera 610. Furthermore, consideration for other users who do not want to be photographed becomes possible.

[0070] As described above, the preferred embodiments of the present invention have been explained. However, the present invention is not limited to the above-described embodiments, and various modifications and changes are possible within the scope of the gist thereof. For example, the CPU 106 (notification means) may notify other users of the non-photographing target object 620 that has entered the photographing angle of view 619 to that effect. Further, the display control device 100 may perform MR display processing for displaying the composite reality space (XR space). The display control device 100 is not limited to a smartphone, and may be, for example, smart glasses, a tablet terminal, a digital camera, or the like. In the case of a digital camera, for example, existing buttons such as a release button or an AF lock button may be used as the icon 618 for fixing the photographing target by adding functions such as an operation for fixing the photographing target. The present invention can also be realized by supplying a program for realizing one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors of a computer of the system or device read and execute the program. Further, the present invention can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0071] The disclosure of the present embodiment includes the following configurations, methods, and programs. (Configuration 1) A first determination means for determining whether an instruction operation has been performed, and a first display control means for, when the first determination means determines that the instruction operation has been performed, superimposing and displaying a photographing angle of view indicating the photographing range of the virtual camera on the XR space. An information processing apparatus characterized by comprising. (Configuration 2) The information processing apparatus according to Configuration 1, wherein the instruction operation is an operation performed on a dedicated icon for the instruction operation or an operation performed on an existing icon to which the function of the instruction operation has been added. (Configuration 3) The information processing apparatus according to Configuration 1 or 2, wherein the photographing angle of view of the virtual camera is radial extending from the virtual lens of the virtual camera. (Configuration 4) The information processing apparatus according to any one of Configurations 1 to 3, wherein the first display control means displays through the shooting angle of view of the virtual camera. (Configuration 5) The information processing apparatus according to any one of Configurations 1 to 4, further comprising fixing means for fixing an object within the shooting angle of view of the virtual camera as the shooting target object of the virtual camera when the shooting angle of view of the virtual camera is displayed. (Configuration 6) The information processing apparatus according to Configuration 5, further comprising excluding means for excluding an object that has become the target of a predetermined operation from among the objects within the shooting angle of view of the virtual camera when the shooting angle of view of the virtual camera is displayed, as the shooting target object of the virtual camera. (Configuration 7) The information processing apparatus according to Configuration 6, wherein the predetermined operation is a flick or swipe operation performed by the user on an object. (Configuration 8) A second determination means for determining whether an explicit operation has been performed, and a second display control means for explicitly displaying the shooting target object of the virtual camera in the XR space when the second determination means determines that the explicit operation has been performed. The information processing apparatus according to any one of Configurations 5 to 7 is characterized by this. (Configuration 9) The information processing apparatus according to Configuration 8, wherein the second display control means performs display of a specific icon around the shooting target object of the virtual camera, blinking display of the shooting target object of the virtual camera, or display that lights up the outline of the shooting target object of the virtual camera. (Configuration 10) The information processing apparatus according to Configuration 8 or 9, wherein the explicit operation is a single tap, double tap, or long press operation performed by the user on at least one of the objects within the shooting angle of view of the virtual camera when the shooting angle of view of the virtual camera is displayed. (Configuration 11) The information processing apparatus according to any one of Configurations 8 to 10, wherein the second display control means explicitly displays the object targeted by the explicit operation as an object to be photographed by the virtual camera in the XR space. (Configuration 12) The information processing apparatus according to any one of Configurations 8 to 10, wherein the second display control means explicitly displays all objects within the shooting angle of view of the virtual camera as objects to be photographed by the virtual camera in the XR space when the shooting angle of view of the virtual camera is displayed. (Configuration 13) A non-display means for making non-display an object that has entered the shooting angle of view of the virtual camera in a live view showing the shooting range of the virtual camera, after the shooting angle of view of the virtual camera is displayed, and a shooting means for shooting a live view of the virtual camera, the information processing apparatus according to any one of Configurations 1 to 12. (Configuration 14) The information processing apparatus according to any one of Configurations 1 to 13, further comprising a notification means for notifying the user of an object that has entered the shooting angle of view of the virtual camera that the object has entered the shooting angle of view of the virtual camera. (Configuration 15) An information processing system having a plurality of information processing apparatuses that share an XR space and a server that controls the display of the XR space in the plurality of information processing apparatuses, each of the plurality of information processing apparatuses determines whether an instruction operation has been performed, and the server controls to display, in the XR space of each of the plurality of information processing apparatuses, a shooting angle of view indicating the shooting range of the virtual camera of the information processing apparatus on which the instruction operation has been performed when any one of the plurality of information processing apparatuses determines that the instruction operation has been performed. The information processing system is characterized by this. (Method 1) A determination step of determining whether an instruction operation has been performed, and a display control step of superimposing and displaying a shooting angle of view indicating the shooting range of a virtual camera in an XR space when it is determined by the determination step that the instruction operation has been performed. A control method for an information processing apparatus is characterized by this. A program for causing a computer to execute each means of the information processing apparatus according to any one of Configurations 1 to 14 (Program 1).

Explanation of Signs

[0072] 100 Display control device (information processing apparatus) 106 CPU (first determination means) (first display control means) 600 Virtual space (XR space) 609 Extended reality space (XR space) 610 Virtual camera 619 Shooting angle of view

Claims

1. a first determination means for determining whether or not a pointing operation has been performed; and a first display control means for, when the first determination means determines that the instruction operation has been performed, superimposing and displaying a shooting angle of view indicating a shooting range of a virtual camera on an XR space.

2. 2 . The information processing apparatus according to claim 1 , wherein the instruction operation is an operation performed on a dedicated icon for the instruction operation, or an operation performed on an existing icon to which a function of the instruction operation has been added.

3. The information processing apparatus according to claim 1 , wherein the imaging angle of the virtual camera is radially expanded from a virtual lens of the virtual camera.

4. 2. The information processing apparatus according to claim 1, wherein the first display control means displays the imaging angle of the virtual camera in a transparent manner.

5. 2 . The information processing apparatus according to claim 1 , further comprising a fixing unit that fixes an object that is within the shooting angle of view of the virtual camera when the shooting angle of view of the virtual camera is displayed, to an object that is a subject of shooting by the virtual camera.

6. 6. The information processing device according to claim 5, further comprising an exclusion means for excluding an object that has been the subject of a predetermined operation from among objects that are within the shooting angle of view of the virtual camera when the shooting angle of view of the virtual camera is displayed, from objects to be photographed by the virtual camera.

7. The information processing apparatus according to claim 6 , wherein the predetermined operation is a flick or swipe operation performed by a user on an object.

8. A second determination means for determining whether an explicit operation has been performed; and a second display control means for explicitly displaying an object to be photographed by the virtual camera in the XR space when the second determination means determines that the explicit operation has been performed.

9. The information processing device according to claim 8, characterized in that the second display control means displays a specific icon around the object being photographed by the virtual camera, blinks the object being photographed by the virtual camera, or displays the border of the object being photographed by the virtual camera with light.

10. The information processing device according to claim 8, characterized in that the explicit operation is a single tap, double tap, or long press operation performed by a user on at least one of the objects that is within the shooting angle of view of the virtual camera when the shooting angle of the virtual camera is displayed.

11. The information processing apparatus according to claim 8 , wherein the second display control means explicitly displays the object that is the target of the explicit operation in the XR space as a target object to be photographed by the virtual camera.

12. The information processing device according to claim 8, characterized in that the second display control means, when the shooting angle of view of the virtual camera is displayed, explicitly displays all objects that are within the shooting angle of view of the virtual camera in the XR space as objects to be photographed by the virtual camera.

13. a non-display means for non-displaying an object that has entered the shooting angle of view of the virtual camera after the shooting angle of view of the virtual camera has been displayed, in a live view showing a shooting range of the virtual camera; The information processing apparatus according to claim 1 , further comprising: a photographing unit for photographing a live view of the virtual camera.

14. 2 . The information processing apparatus according to claim 1 , further comprising a notification unit that notifies a user of an object that has entered the field of view of the virtual camera that the object has entered the field of view of the virtual camera.

15. a determination step of determining whether or not a command operation has been performed; and a display control step of superimposing and displaying a shooting angle of view indicating a shooting range of a virtual camera on an XR space when the determination step determines that the instruction operation has been performed.

16. 2. A program for causing a computer to execute each of the means of the information processing apparatus according to claim 1.

Images