Information processing device, information processing system, method for controlling an information processing device, and control program for an information processing device

Abstract

This invention provides a technology that can reduce the operational burden on users who wear a display device on their heads when taking images. [Solution] The information processing device includes an imaging unit, an acquisition unit that acquires positional relationship information between a user-worn display device and a user-operable imaging device, a determination unit that determines the display content to be displayed on the display device based on the positional relationship information acquired by the acquisition unit, and an output unit that outputs the display content determined by the determination unit. The display content includes an image showing the user's field of view when the display device is not worn or an image taken of an object operated by the user, and includes an image taken by the imaging unit or is generated based on an image taken by the imaging device.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, a control method for an information processing apparatus, and a control program for an information processing apparatus.

Background Art

[0002] In recent years, activities in a virtual world such as a metaverse displayed by a display device worn on the head, such as an HMD (Head Mounted Display), have become common. Also, it is known that a virtual world platform provides functions related to shooting, such as shooting an object, setting a shooting function, or operating an object in the virtual world by operating a controller attached to the HMD by hand.

[0003] However, wearing a display device on the head blocks the view of the user in the real world. As a result, the user cannot see their hands and has to operate the controller in a blind state. Also, since the shooting operation using the controller has a different imaging device and operation method from that in the real world, it is necessary to get used to the operation method. Therefore, the burden on the user increases. Also, in the virtual world, there are multiple operation targets such as a virtual camera and an avatar corresponding to the user in the real world. Therefore, it is conceivable to switch the operation target, but such a switching operation also increases the burden on the user.

[0004] Therefore, a technique has been proposed for detecting the positional relationship between an object or a controller in the real world and displaying the detected object or controller in the virtual world (see Patent Document 1). Also, a technique has been proposed for detecting an input from an HMD or a controller and switching between an avatar operation and a virtual camera operation in the virtual world (see Patent Document 2).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] However, while the technology described in Patent Document 1 allows users to grasp the approximate location of objects and controllers in the virtual world, it does not display detailed information such as the shape and position of the hands and fingers used to operate the controller. Therefore, controller button operations must be performed by touch. Furthermore, Patent Document 2 involves switching operations that do not occur in the real world. Thus, the technologies described in these patent documents do not reduce the burden on the user operating the controller.

[0007] Therefore, in one aspect, the present invention aims to provide a technology that can reduce the operational burden on users who have a display device attached, specifically regarding image capture. [Means for solving the problem]

[0008] One aspect of the present invention is, An acquisition unit that acquires positional relationship information between a display device equipped with an imaging unit that can be worn by a user and an imaging device that can be operated by the user, A determination unit determines the display content to be displayed on the display device based on the positional relationship information acquired by the acquisition unit, The system comprises an output unit that outputs the display content determined by the determination unit, The above display content is, This includes an image showing the user's field of view when the display device is not attached, or an image captured by the user's operating object. The image includes an image captured by the imaging unit, or is generated based on an image captured by the imaging device. It is an information processing device. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technology that can reduce the operational burden on users who are using a display device for taking photos. [Brief explanation of the drawing]

[0010] [Figure 1] Schematic diagram of an imaging system according to one embodiment. [Figure 2] 1. Explanatory diagram of the display content as described above. [Figure 3] Flowchart of the process according to one embodiment [Figure 4] 1. Explanatory diagram of the display content as described above. [Figure 5] Flowchart of the process according to one embodiment [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] This embodiment relates to the display of an HMD when using a shooting mode function in a virtual world, such as the metaverse. Here, a virtual world refers to a two-dimensional or three-dimensional space represented by images displayed on a computer, for example. The user wears an HMD and overlays the virtual world displayed on the HMD onto the real world. In this way, the user can experience Augmented Reality, Virtual Reality, or Mixed Reality. An avatar is, for example, a computer-generated representation of the user. An avatar is generally not the user themselves, but is represented by two-dimensional or three-dimensional computer graphics. The shooting mode function in a virtual world is a function that captures objects visible to the user in the virtual world as still images. Generally, it is assumed that virtual objects existing within the field of view of a virtual camera placed at the avatar's position are captured. The shooting mode function may also have the ability to freely place the object to be photographed and the viewpoint information of the virtual camera, as implemented in video games, within the virtual world, and to freely set shooting conditions such as light source information and color information.

[0013] [Overview of the shooting system 100] Using Figure 1, an example of the configuration of the imaging system 100 (an example of an "information processing system") will be explained. The imaging system 100 consists of a display device 101, an imaging device 105, a user terminal 110 (an example of an "information processing device"), and a virtual world management server 112.

[0014] The display device 101 (an example of a "user-wearable display device") is, for example, a head-mounted display (HMD), and its form may be goggles, glasses, or a helmet. The display device 101 includes a display unit 102 that displays a virtual world, an imaging unit 103 that captures images of the real world, a communication unit 104 that communicates with the user terminal 110, and a control unit 117 that performs overall control of the display device 101, including these components.

[0015] The control unit 117 is composed of, for example, a CPU and a storage medium such as ROM, RAM, or HDD (Hard Disk Drive). The communication unit 104 is composed of a communication module that enables short-range communication using infrared communication or Bluetooth® standards (an example of "direct communication"), and a communication module that can connect to telephone communication networks such as mobile phones or networks such as Wi-Fi®. The display unit 102 is composed of a display module that displays images to the user, and the imaging unit 103 is composed of an imaging module. The storage medium stores the operating system (OS), various programs, various tables, etc., and the CPU loads the programs stored in the storage medium into the working area of ​​the main memory and executes them. Through the execution of such programs, each component is controlled, thereby realizing various functions that match predetermined purposes, as described later.

[0016] The imaging device 105 (an example of a "user-operable imaging device") is, for example, a digital camera or a terminal with a shooting function such as a smartphone. The imaging device 105 includes an imaging unit 119 that captures a landscape within the shooting angle of view, a display unit 106 that displays the image captured by the imaging unit 119, and a detection unit 108 that detects whether the user is looking through the viewfinder. The imaging device 105 also includes an operation unit 109 that receives operation input to the display unit 106 or operation buttons, a communication unit 107 that communicates with the user terminal 110, and a position and orientation detection unit 120 that detects the position and orientation of the imaging device 105. The imaging device 105 also includes a control unit 118 that performs overall control of the imaging device 105, including these components. In this embodiment, the imaging device 105 is assumed to be, for example, a digital camera. In other words, the imaging device 105 is composed of, for example, a viewfinder, a display device within the viewfinder, a display unit 106 on the back, an eye sensor capable of detecting eye contact with the viewfinder (an example of a "sensor capable of detecting approach to the viewfinder"), and operation buttons. The imaging device 105 may also be a smartphone. In such a case, the imaging device 105 is provided with a touch panel covering the entire front surface.

[0017] The control unit 118 is configured to include, for example, a CPU and a storage medium such as a ROM, a RAM, or a HDD (Hard Disk Drive). The communication unit 107 is configured to include, for example, a communication module enabling short-range communication such as infrared communication or Bluetooth standard, and a communication module connectable to a telephone communication network such as a mobile phone or a network such as Wi-Fi. The imaging unit 119 is configured to include, for example, a photographing module and a sensor capable of detecting the position and orientation such as an IMU (Inertial Measurement Unit). The position and orientation detection unit 120 is configured to include, for example, a sensor capable of detecting the position and orientation, and detect the position and orientation using the output of the sensor. Note that the position and orientation detection unit 120 may acquire the position and orientation of the imaging device 105 using the image data captured by the imaging unit 119.

[0018] In the storage medium of the imaging device 105, an operating system (OS), various programs, various tables, etc. are stored, and the CPU loads the program stored in the storage medium into the working area of the main storage device and executes it. Then, by controlling each component through the execution of the program, various functions meeting predetermined purposes as described later can be realized. Note that hereinafter, the imaging device 105 reflected in the video displayed on the display unit 102 of the display device 101 is also referred to as a virtual camera.

[0019] The user terminal 110 includes a communication unit 113 that communicates with the display device 101, the imaging device 105, and the virtual world management server 112. The user terminal 110 also includes a position detection unit 111 that detects the positional relationship between the display device 101 and the imaging device 105, and an image processing unit 114 that renders object data according to the rendering information and generates an image representing the virtual world. The user terminal 110 also includes a control unit 121 that performs overall control of the user terminal 110 including these components. Note that the user terminal 110 may be provided with a display unit including a display module.

[0020] The control unit 121 comprises a CPU and a storage medium such as ROM, RAM, or HDD (Hard Disk Drive). The communication unit 113 comprises a communication module that enables short-range communication using infrared communication or Bluetooth standards, and a communication module that can connect to telephone communication networks such as mobile phones or networks such as Wi-Fi. The storage medium stores the operating system (OS), various programs, various tables, etc., and the CPU loads the programs stored therein into the working area of ​​the main memory and executes them. Through the execution of the programs, each component is controlled, thereby realizing various functions that match the predetermined purpose, as will be described later.

[0021] The virtual world management server 112 is, for example, a computer and includes a communication unit 115 that communicates with the user terminal 110 and a video management unit 116 that manages images representing the virtual world. More specifically, the video management unit 116 manages the display information displayed by the display unit 102 of the display device 101. For example, the video management unit 116 sets the object data to be displayed by the display unit 102 to a subject 200 detected from real-world image data captured by the imaging device 105. The video management unit 116 also generates rendering information for rendering the object data.

[0022] The video management unit 116 includes a processor such as a CPU, main memory such as RAM and ROM, and auxiliary storage devices such as an EPROM, a hard disk drive, and removable media. The communication unit 115 includes a communication module (e.g., a Network Interface Card) that can connect to a network such as Wi-Fi. The removable media may be, for example, a USB memory stick or a disk recording medium such as a CD or DVD. The auxiliary storage device stores the operating system (OS), various programs, various tables, etc. The CPU loads the programs stored in the storage medium into the working area of ​​the main memory and executes them, and through the execution of the programs, each component is controlled, thereby realizing various functions that match the predetermined purpose, as described later.

[0023] [Posture patterns and displayed content] Using Figure 2, an example of the display content shown by the display unit 102 of the display device 101 when the user wears the display device 101 on their head will be explained. Figure 2(A) shows a case where the user's posture is such that their eyes are focused on the viewfinder of the imaging device 105 and they are taking a picture using the imaging device 105. In this case, the display unit 102 of the display device 101 will display, for example, a subject 200 that is within the field of view of the imaging device 105 (display content 201). This allows the user to recognize the subject 200 in the virtual world. The user's posture shown in Figure 2(A) is also referred to as Pattern 1.

[0024] Figure 2(B) shows the case where the user's posture is such that their eyes are facing the back of the imaging device 105 and they are using the display unit 106 or operation unit 109 located on the back of the imaging device 105 (hereinafter also referred to as Pattern 2). In this case, the imaging device 105 is within the field of view of the imaging unit 103 of the display device 101. This posture is equivalent to the posture in the real world when taking a picture using the display unit 106 on the back of the imaging device 105. In addition to taking pictures, it is also thought that in this posture the user is operating the display unit 106 or operation unit 109 to set menus, etc. In this case, accurate operation would be difficult if the position of the user's hands was unknown, so it is necessary to display the movement of the user's hands during menu setting operations in the virtual world. Therefore, when the user's posture is Pattern 2, the display unit 102 switches to a video see-through method. That is, the imaging device 105, which is an object located within the field of view of the imaging unit 103 of the display device 101, is displayed on the display unit 102 in real time. Then, the display unit 106 on the back of the imaging device 105 displayed in the virtual space (hereinafter, the imaging device 105 displayed in the virtual space may be referred to as a virtual camera) can superimpose the image of the subject 200 that is within the field of view of the actual imaging device 105 (display content 202).

[0025] Figure 2(C) shows a case where the user's gaze is directed away from the imaging device 105 being held, and the imaging device 105 is not present in the line of sight; in other words, the user is not using the imaging device 105 (hereinafter also referred to as Pattern 3). In this case, the imaging device 105 is not within the field of view of the imaging unit 103 of the display device 101. In this case, the display unit 102 of the display device 101 can display all objects that are within the field of view of the imaging unit 103 (display content 203).

[0026] [Processing Example 1] Figure 3 illustrates an example of a flowchart for sequence processing in the imaging system 100. This type of processing is achieved when the CPU of each device constituting the imaging system 100 loads a program into the working area of ​​the main memory and executes it, and each component is controlled through the execution of the program.

[0027] In S101, the control unit 118 of the imaging device 105 determines whether or not it is providing the shooting mode function. If the control unit 118 determines that it is providing the shooting mode function, the process proceeds to S102; otherwise, the process ends. In S102, the control unit 118 of the imaging device 105 establishes a connection between the imaging device 105 and the user terminal 110 as an interface for performing shooting operations.

[0028] In S103, the position detection unit 111 (an example of an "acquisition unit") in the user terminal 110 acquires position information (an example of "positional relationship information") of the display device 101 and the imaging device 105. More specifically, the position detection unit 111 receives images captured by the imaging unit 103 of the display device 101 via the communication unit 113. Alternatively, the position detection unit 111 receives the reaction signal of the eye sensor of the imaging device 105 from the imaging device 105 via the communication unit 113. Alternatively, the position detection unit 111 receives records of short-range wireless communication, such as infrared, between the display device 101 and the imaging device 105 from the display device 101 or the imaging device 105 via the communication unit 113.

[0029] In S104, the position detection unit 111 (an example of a "determination unit") uses the position information acquired in S103 to determine whether the relative positions of the display device 101 and the imaging device 105 are within a predetermined distance. The predetermined distance is a value indicating that the display device 101 and the imaging device 105 are in close proximity, for example, a few centimeters. If the position detection unit 111 determines that the relative positions are within the predetermined distance, the process proceeds to S105; otherwise, the process proceeds to S107. More specifically, when the position detection unit 111 receives an image captured by the imaging unit 103, it determines whether the image is in a light-shielded state. If the position detection unit 111 determines that the image is in a light-shielded state, it assumes that the imaging unit 103 is blocked by the imaging device 105, and determines that the distance between the display device 101 and the imaging device 105 is within a predetermined distance.

[0030] Alternatively, if the position detection unit 111 receives a response signal from the eye sensor of the imaging device 105, it determines that the distance between the display device 101 and the imaging device 105 is within the predetermined distance. Alternatively, if the position detection unit 111 receives a record of short-range wireless communication, such as infrared communication, between the display device 101 and the imaging device 105, it determines that the distance between the display device 101 and the imaging device 105 is within the predetermined distance if the wireless communication was successful.

[0031] In S105, the position detection unit 111 determines that the user's posture is pattern 1 (see Figure 2), and the process proceeds to S106. In S106, the display unit 102 of the display device 101 displays the subject 200 within the shooting angle of the imaging device 105. More specifically, the communication unit 113 of the user terminal 110 transmits information to the imaging device 105 indicating that the user's posture is pattern 1. Then, the communication unit 107 of the imaging device 105 transmits the video being captured by the imaging device 105, the position and posture information of the imaging device 105, and camera setting information such as sensor size and zoom magnification (these are also called camera information) to the user terminal 110. The communication unit 113 of the user terminal 110 further transmits this camera information to the virtual world management server 112. The communication unit 115 of the virtual world management server 112 receives the position and posture information and camera setting information. Then, the video management unit 116 detects the subject 200 in the video based on the received camera information and sets this subject 200 as object data. Such detection may be performed, for example, by machine learning techniques.

[0032] Furthermore, the video management unit 116 uses position and orientation information to generate rendering information for, for example, a three-dimensional image of the subject 200. Alternatively, the video management unit 116 generates rendering information including brightness differences or color gradations of the image, for example, according to the sensor size of the imaging device 105. Alternatively, the video management unit 116 generates rendering information for an enlarged or reduced image of the subject 200, for example, according to the zoom information of the imaging device 105. In this way, the video management unit 116 generates display information such as object data and rendering information to be displayed by the display device 101. Then, the communication unit 115 transmits this display information to the user terminal 110.

[0033] In the user terminal 110, the communication unit 113 receives this display information. The image processing unit 114 then renders the object data included in the received display information according to the rendering information. The communication unit 113 (an example of an "output unit") then transmits the rendered image data to the display device 101. The display device 101 then receives the rendered image data in the communication unit 104, and the display unit 102 displays this image data.

[0034] If, in S104, the position detection unit 111 determines that the positional relationship between the display device 101 and the imaging device 105 exceeds a predetermined distance, then in S107, the position detection unit 111 acquires the image captured by the imaging unit 103 of the display device 101 via the communication unit 113. The position detection unit 111 then determines whether or not the imaging device 105, configured to allow the user to operate the display unit 106 or the operation unit 109, exists within the field of view of the imaging unit 103. "Configured to allow operation" means, for example, that the user's hand overlaps with the imaging device 105 and is captured in the image. If the position detection unit 111 determines that the imaging device 105 exists, the process proceeds to S108; otherwise, the process proceeds to S111. This determination is achieved, for example, by identifying the imaging device 105 from the image using mechanical methods.

[0035] In S108, the position detection unit 111 determines that the user's posture is pattern 2, and processing proceeds to S109. In S109, the communication unit 113 of the user terminal 110 sends a request to the display device 101 to switch the display method of the display unit 102 to the video see-through method. The communication unit 104 of the display device 101 receives this request, and the control unit 117 performs control to switch the display method of the display unit 102 to the video see-through method. That is, the display unit 102 of the display device 101 displays the real-time image being captured by the imaging unit 103 of the display device 101. The display device 101 also acquires display information of the subject 200 that is within the field of view of the imaging device 105 in S110 in the same way as in S106. The display unit 102 then superimposes and displays this subject 200 on the display unit 106 on the back of the virtual camera that is visible in the image being displayed in real time. The image data of this subject 200 is generated in the virtual world management server 112. Specifically, the communication unit 115 of the virtual world management server 112 receives the image captured by the imaging unit 103 from the display device 101. The virtual world management server 112 then estimates the position of the display unit 106 of the imaging device 105 using the image captured by the imaging unit 103. It then generates image data in which the subject 200 is placed at this estimated position. To improve the accuracy of estimating the position of the display unit 106, a marker member may be provided on the display unit 106 on the back of the imaging device 105 in the real world. The process then ends.

[0036] On the other hand, if the position detection unit 111 determines in S107 that the imaging device 105 is not present in the image captured by the imaging unit 103, then in S111 the position detection unit 111 determines that the user's posture is pattern 3. Then, in S112, the communication unit 113 of the user terminal 110 requests the display device 101 to display the image being captured by the imaging unit 103. The communication unit 104 of the display device 101 receives this request, and the display unit 102 displays the real-time image being captured by the imaging unit 103 of the display device 101.

[0037] [One aspect of action / effect] According to the shooting system 100 described above, the display unit 102 of the display device 101 automatically switches the image displayed according to the user's posture. This image accurately reproduces the user's field of view of the real world as it would appear without the display device 101. Therefore, the operational burden on the user regarding shooting while wearing the display device 101 is reduced.

[0038] <Variation> [Posture patterns and displayed content] In the modified version, we will focus on explaining the differences from the above embodiment. The position detection unit 111 in the modified version also determines the positional relationship between the display device 101 and the imaging device 105, similar to the above embodiment. Based on this positional relationship, the position detection unit 111 determines that the user's posture is one of the three patterns described above. Furthermore, in the modified version, operations on the imaging device 105 in the real world are reflected in the virtual world. In addition, the target of these operations is automatically switched according to the posture pattern.

[0039] The switching of the target of operation will be explained using Figure 4. Figure 4(A) shows the case where the user's posture is such that they are taking a picture using the viewfinder of the imaging device 105. In this case, the display unit 102 of the display device 101 displays, for example, a subject 200 that is within the field of view of the imaging device 105 (display content 401). The control unit 118 also sets the imaging device 105 as the target of operation for the operation unit 109. In other words, the control unit 118 sets the virtual camera as the target of operation in the virtual world. The user posture shown in Figure 4(A) is also referred to as Pattern 1.

[0040] Figure 4(B) shows the user's posture as if taking a picture using the display unit 106 on the back of the imaging device 105 (hereinafter also referred to as Pattern 2). In this case, the display unit 102 of the display device 101 displays, for example, a subject 200 that is within the field of view of the imaging device 105 (display content 402). The control unit 118 sets the target of operation of the operation unit 109 in the virtual world to a virtual camera, as in the case of Pattern 1.

[0041] Figure 4(C) shows the user's posture in which at least one image is not being captured using the imaging device 105 (hereinafter also referred to as pattern 3). At this time, the display unit 102 of the display device 101 displays, for example, the scenery that exists within the field of view of the imaging unit 103 of the display device 101 (display content 403). The control unit 118 also sets the imaging unit 103 of the display device 101 as the target of operation for the operation unit 109. That is, the control unit 118 sets the target of operation input in the virtual world to the avatar 400 that corresponds to the user in the virtual world or the user's viewpoint.

[0042] [Processing Example 2] Figure 5 illustrates another example of a flowchart for the sequence processing of the imaging system 100. This type of processing is achieved when the CPU of each device constituting the imaging system 100 loads a program into the working area of ​​the main memory and executes it, and each component is controlled through the execution of the program. Note that S201 to S205, S207, S208, and S210 correspond to S101 to S105, S107, S108, and S111, respectively, and therefore their explanation is omitted.

[0043] In S206, the control unit 118 of the imaging device 105 receives operation input for the display unit 106 or the operation unit 109. The operation information received by the control unit 118 is, for example, setting information for shooting, such as the zoom magnification of the imaging device 105. The control unit 118 uses this operation information to change the setting of the shooting mode function. The control unit 118 also receives the position and orientation information of the imaging device 105 detected by the position and orientation detection unit 120 as operation information. The control unit 118 also obtains from the user terminal 110 via the communication unit 107 that the user's orientation is pattern 1. The control unit 118 then sets the target of the above operation on the imaging device 105 to the virtual camera. The communication unit 107 then transmits the operation information, including the operation target information, to the user terminal 110. The user terminal 110 then transmits this operation information to the virtual world management server 112 via the communication unit 113.

[0044] Then, the communication unit 115 receives this operation information in the virtual world management server 112. The video management unit 116 stores the received operation information in the storage medium as operation information for the virtual camera. If this operation information is already stored in the storage medium, the video management unit 116 updates this operation information and stores it. The video management unit 116 then applies the latest operation information to the virtual camera and generates display information to be displayed on the display device 101. More specifically, the video management unit 116 detects the subject 200 included in the imaging data captured by the imaging device 105 and sets this subject 200 as object data. If setting information such as zoom magnification is updated, the video management unit 116 generates display information such as object and rendering information to which this updated setting information is applied. Also, if the position and orientation information of the imaging device 105 is updated, this updated position and orientation information is applied to the imaging data captured by the imaging device 105 to generate display information such as object and rendering information. Subsequent processing is performed in the same manner as in S106. Then, the display unit 102 of the display device 101 displays the image in which the user's operation on the operation unit 109 has been applied to the subject 200. The same process is performed in S209.

[0045] Meanwhile, in S211, the control unit 118 of the imaging device 105 receives operation input to the display unit 106 or operation buttons as operation information. The control unit 118 also receives the position and orientation information of the imaging device 105 detected by the position and orientation detection unit 120 as operation information. The control unit 118 also obtains from the user terminal 110 via the communication unit 107 that the user's orientation is pattern 3. The control unit 118 then sets the avatar 400 as the target of the operation on the imaging device 105 in the virtual world. The communication unit 107 then transmits the operation information, including the target of the operation, to the user terminal 110. The user terminal 110 then transmits this operation information to the virtual world management server 112 via the communication unit 113.

[0046] In the virtual world management server 112, the communication unit 115 receives this operation information. The communication unit 115 also receives the image captured by the imaging unit 103 of the display device 101. The video management unit 116 then composites an image of an avatar 400, which represents the user, onto the image data captured by the imaging unit 103 of the display device 101. The position where the avatar 400 is placed is, for example, an area where no objects are visible. Alternatively, the virtual world management server 112 may provide a virtual user viewpoint in the image instead of the avatar 400. The virtual world management server 112 then generates rendering information that applies the received operation information to the avatar 400. That is, for example, if the control unit 118 of the imaging device 105 receives a press operation for a button indicating the right direction, it generates an image in which the object shown on the right side of the image captured by the imaging unit 103 of the display device 101 is moved to the front. The video management unit 116 stores the operation information as operation information for the avatar 400 in a storage medium. Furthermore, if this operation information is already stored in the storage medium, the video management unit 116 updates and stores this operation information. The subsequent processing is the same as in S206. Then, the display unit 102 of the display device 101 displays the video in which the user's operation on the operation unit 109 has been applied to the avatar 400.

[0047] [One aspect of action / effect] In the modified version of the shooting system 100, when the user is looking through the viewfinder of the imaging device 105 (Pattern 1), the target of operation in the virtual world is automatically set to the virtual camera. Similarly, when the user is using the display unit 106 on the back of the imaging device 105 (Pattern 2), the target of operation is also set to the virtual camera. Furthermore, the subject 200 captured by the imaging device 105 is displayed on the display unit 102 of the display device 101. Therefore, the user can easily perform shooting using the virtual camera and configure shooting settings by operating the operation unit 109.

[0048] On the other hand, if the user's posture is one in which the imaging device 105 is not being used (Pattern 3), the target of operation in the virtual world is automatically set to the avatar 400. This avatar 400 is then displayed on the display unit 102 of the display device 101. Therefore, while wearing the display device 101, the user can search for a target to be photographed in the virtual world by operating the control unit 109 to change the viewpoint of the avatar 400. In this way, even with the modified image capture system 100, the operational burden on the user regarding image capture while wearing the display device 101 is reduced.

[0049] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or recording medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0050] Therefore, in order to realize the functional processing of the present invention on a computer, the program code itself supplied to and installed on the computer also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention. In this case, the form of the program is not limited as long as it has the functionality of a program, such as object code, a program executed by an interpreter, or script data supplied to an OS. As a recording medium for supplying the program, for example, a hard disk, a magnetic recording medium such as magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory may be used. Furthermore, as a method of supplying the program, a computer program that forms the present invention may be stored on a server on a computer network, and a connected client computer may download the computer program and run it.

[0051] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0052] The disclosures herein include the following information processing devices, information processing systems, methods for controlling information processing devices, and control programs for information processing devices. [Item 1] An acquisition unit that acquires positional relationship information between a display device equipped with an imaging unit that can be worn by a user and an imaging device that can be operated by the user, A determination unit determines the display content to be displayed on the display device based on the positional relationship information acquired by the acquisition unit, The system comprises an output unit that outputs the display content determined by the determination unit, The above display content is, This includes an image showing the user's field of view when the display device is not attached, or an image captured by the user's operating object. The image includes an image captured by the imaging unit, or is generated based on an image captured by the imaging device. Information processing device. [Item 2] The aforementioned positional relationship information is: First information indicating that the imaging unit is shielded from light by the imaging device, Second information indicating the approach of the display device to the viewfinder of the imaging device, and Third information obtained by direct communication between the display device and the imaging device, including at least one of the following: The information processing device described in item 1. [Item 3] When the acquisition unit acquires the first information or the second information, or when the acquisition unit acquires the third information and the distance between the display device and the imaging device, as determined based on the acquired third information, is within a predetermined distance, The determination unit determines that the image captured by the imaging device matches the display content. The information processing device described in item 2. [Item 4] When the acquisition unit acquires fourth information as positional relationship information indicating that the imaging device is included in the first image captured by the imaging unit, The determination unit determines that the first image and the second image captured by the imaging device, which is superimposed on the display portion on the back of the imaging device included in the first image, constitute the display content. The information processing device described in item 3. [Item 5] The display unit or operation unit of the imaging device included in the first image is configured to be operable by the user. The information processing device described in item 4. [Item 6] If the acquisition unit acquires positional relationship information that does not include any of the first information, second information, third information, and fourth information, The determination unit determines that the image captured by the imaging unit matches the display content. Information processing device as described in item 4 or 5. [Item 7] The information processing device described in item 2, An information processing system comprising the aforementioned imaging device, The imaging device includes a control unit that receives user input and sets the target of operation in the virtual world. When the acquisition unit acquires the first information or the second information, or when the acquisition unit acquires the third information and the distance between the display device and the imaging device, acquired based on the acquired third information, is within a predetermined distance, or when the fourth information indicating that the imaging device is included in the image captured by the imaging unit is acquired as the positional relationship information, The control unit sets the target of operation for the imaging device to the imaging device, The determination unit determines that the image data captured by the imaging device, in which the operation received by the imaging device is reflected, is the display content. If the acquisition unit acquires positional relationship information that does not include any of the first information, second information, third information, and fourth information, The control unit sets the target of operation for the imaging device to the display device, The determination unit determines that the image captured by the imaging unit of the display device, in which the operation received by the imaging device is reflected, is the display content. Information processing system. [Item 8] A method for controlling an information processing device, The acquisition unit includes an imaging unit and an acquisition step of acquiring positional relationship information between a user-worn display device and a user-operable imaging device, The determination unit performs a determination step of determining the display content to be displayed on the display device based on the positional relationship information acquired by the acquisition unit, The output unit includes an output step of outputting the display content determined by the determination unit, The above display content is, This includes an image showing the user's field of view when the display device is not attached, or an image captured by the user's operating object. The image includes an image captured by the imaging unit, or is generated based on an image captured by the imaging device. A method for controlling an information processing device. [Item 9] A program for causing a computer to execute each step in a control method for an information processing device, wherein the control method is: The acquisition unit includes an imaging unit and an acquisition step of acquiring positional relationship information between a user-worn display device and a user-operable imaging device, The determination unit performs a determination step of determining the display content to be displayed on the display device based on the positional relationship information acquired by the acquisition unit, The output unit includes an output step of outputting the display content determined by the determination unit, The above display content is, This includes an image showing the user's field of view when the display device is not attached, or an image captured by the user's operating object. The image includes an image captured by the imaging unit, or is generated based on an image captured by the imaging device. program. [Explanation of Symbols]

[0053] 1, 2, 3: Pose patterns, 100: Shooting system, 101: Display device, 102: Display unit, 103: Image capture unit, 104: Communication unit, 105: Image capture device, 106: Display unit, 107: Communication unit, 108: Detection unit, 109: Operation unit, 110: User terminal, 111: Position detection unit, 112: Virtual world management server, 113: Communication unit, 114: Image processing unit, 115: Communication unit, 116: Video management unit, 117: Control unit, 118: Control unit, 119: Image capture unit, 120: Position and pose detection unit, 121: Control unit, 200: Subject, 201-203, 401-403: Display content, 400: Avatar

Claims

1. An acquisition unit that acquires positional relationship information between a display device equipped with an imaging unit that can be worn by a user and an imaging device that can be operated by the user, A determination unit determines the display content to be displayed on the display device based on the positional relationship information acquired by the acquisition unit, The system comprises an output unit that outputs the display content determined by the determination unit, The above display content is, This includes an image showing the user's field of view when the display device is not attached, or an image captured by the user's operating object. The image includes an image captured by the imaging unit, or is generated based on an image captured by the imaging device. Information processing device.

2. The aforementioned positional relationship information is: First information indicating that the imaging unit is shielded from light by the imaging device, Second information indicating the approach of the display device to the viewfinder of the imaging device, and Third information obtained by direct communication between the display device and the imaging device, including at least one of the following: The information processing apparatus according to claim 1.

3. When the acquisition unit acquires the first information or the second information, or when the acquisition unit acquires the third information and the distance between the display device and the imaging device, as determined based on the acquired third information, is within a predetermined distance, The determination unit determines that the image captured by the imaging device matches the display content. The information processing apparatus according to claim 2.

4. When the acquisition unit acquires fourth information as positional relationship information indicating that the imaging device is included in the first image captured by the imaging unit, The determination unit determines that the first image and the second image captured by the imaging device, which is superimposed on the display portion on the back of the imaging device included in the first image, constitute the display content. The information processing apparatus according to claim 3.

5. The display unit or operation unit of the imaging device included in the first image is configured to be operable by the user. The information processing apparatus according to claim 4.

6. If the acquisition unit acquires positional relationship information that does not include any of the first information, second information, third information, and fourth information, The determination unit determines that the image captured by the imaging unit matches the display content. The information processing apparatus according to claim 5.

7. The information processing apparatus according to claim 2, An information processing system comprising the aforementioned imaging device, The imaging device includes a control unit that receives user input and sets the target of operation in the virtual world. When the acquisition unit acquires the first information or the second information, or when the acquisition unit acquires the third information and the distance between the display device and the imaging device, acquired based on the acquired third information, is within a predetermined distance, or when the fourth information indicating that the imaging device is included in the image captured by the imaging unit is acquired as the positional relationship information, The control unit sets the target of operation for the imaging device to the imaging device, The determination unit determines that the image data captured by the imaging device, in which the operation received by the imaging device is reflected, is the display content. If the acquisition unit acquires positional relationship information that does not include any of the first information, second information, third information, and fourth information, The control unit sets the target of operation for the imaging device to the display device, The determination unit determines that the image captured by the imaging unit of the display device, in which the operation received by the imaging device is reflected, is the display content. Information processing system.

8. A method for controlling an information processing device, The acquisition unit includes an imaging unit and an acquisition step of acquiring positional relationship information between a user-worn display device and a user-operable imaging device, The determination unit performs a determination step of determining the display content to be displayed on the display device based on the positional relationship information acquired by the acquisition unit, The output unit includes an output step of outputting the display content determined by the determination unit, The above display content is, This includes an image showing the user's field of view when the display device is not attached, or an image captured by the user's operating object. The image includes an image captured by the imaging unit, or is generated based on an image captured by the imaging device. A method for controlling an information processing device.

9. A program for causing a computer to execute each step in a control method for an information processing device, wherein the control method is: The acquisition unit includes an imaging unit and an acquisition step of acquiring positional relationship information between a user-worn display device and a user-operable imaging device, The determination unit performs a determination step of determining the display content to be displayed on the display device based on the positional relationship information acquired by the acquisition unit, The output unit includes an output step of outputting the display content determined by the determination unit, The above display content is, This includes an image showing the user's field of view when the display device is not attached, or an image captured by the user's operating object. The image includes an image captured by the imaging unit, or is generated based on an image captured by the imaging device. program.

Images