electronic machines

Abstract

To allow users to easily obtain information about parts of an image that are not depicted (for example, parts not shown in a photograph). [Solution] The electronic device of the present invention is characterized by comprising: an acquisition means for acquiring location information associated with an image selected by the user from among one or more images; and a control means for controlling the display device to display an image of a virtual space with the location corresponding to the location information as the viewpoint when the location information is acquired by the acquisition means.

Description

Technical Field

[0006] , , ,

[0001] The present invention relates to electronic devices, and particularly to a technology for displaying images etc. posted on SNS (Social Networking Service).

Background Art

[0002] Users of SNS can post photos taken at tourist destinations they visited etc. so that other users can view them. By viewing the posted photos, users of SNS may obtain information about tourist destinations etc. and decide their destinations.

[0003] In addition, a digital twin space that reproduces the real space has been proposed as a virtual space to be experienced (provided) to users by a display device such as an HMD (Head Mounted Display). By providing the digital twin space to users, users can virtually visit tourist destinations etc.

[0004] Patent Document 1 discloses displaying a pre-prepared 3D image when making a reservation for an accommodation facility etc. Patent Document 2 discloses a technology for making recommendations regarding places in the real world based on the behavior history of users in the metaverse.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, because 2D (two-dimensional) photographs have a limited field of view, users may not be able to obtain sufficient information and may not be able to make a suitable decision about their destination simply by viewing 2D photographs. While the technology disclosed in Patent Document 1 allows users to view 3D (three-dimensional) images of accommodations, the field of view is still limited in these 3D images, so users may not be able to obtain sufficient information. Users can obtain sufficient information if they move to the location where the photograph was taken within the digital twin space, but such movement is not easy.

[0007] The present invention aims to enable users to easily obtain information about parts of an image that are not depicted (for example, parts that are not visible in a photograph). [Means for solving the problem]

[0008] The electronic device of the present invention is characterized by comprising: acquisition means for acquiring location information associated with an image selected by a user from among one or more images; and control means for controlling the display device to display an image of a virtual space with the location corresponding to the location information as the viewpoint when the location information is acquired by the acquisition means. [Effects of the Invention]

[0009] According to the present invention, users will be able to easily obtain information about parts of an image that are not depicted (for example, parts that are not shown in a photograph). [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram showing the configuration of an information processing system. [Figure 2] This is a block diagram showing the configuration of the HMD, PC, and controller. [Figure 3] This is a flowchart showing the process on the PC. [Figure 4] This is a schematic diagram of the list screen. [Figure 5] This is a schematic diagram of the browsing screen. [Figure 6] This is a schematic diagram of a virtual space image. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While multiple features are described in the embodiments, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] <System Configuration> Figure 1 is a schematic diagram showing the configuration of the information processing system 1 according to this embodiment. The information processing system 1 includes an HMD (Head Mounted Display) 100, a PC (Personal Computer) 110, and a controller 120. The information processing system 1 also includes an SNS (Social Networking Service) server 200 and a virtual space server 300. Although the HMD 100, PC 110, and controller 120 are shown as separate components, two or three of these components may be integrated into a single unit.

[0013] The HMD100 is a display device that can be worn on the user's head (a head-mounted display device). The HMD100 displays images received from the PC110.

[0014] The PC110 is connected to the HMD100 and controls the HMD100. The connection between the PC110 and the HMD100 may be a wired connection using a USB cable or the like, or a wireless connection using Bluetooth® or Wi-Fi (Wireless Fidelity)®. The components of the PC110 may be installed inside the HMD100. The electronic device capable of controlling the HMD100 is not limited to a PC, but may also be a smartphone or tablet device, for example.

[0015] The controller 120 is connected to the PC 110 and transmits operation signals to the PC 110 in response to user input to the controller 120. The PC 110 performs various controls, such as controlling the HMD 100, according to the operation signals from the controller 120. The connection between the controller 120 and the PC 110 may be a wired connection using a USB cable or the like, or a wireless connection using Bluetooth® or Wi-Fi (Wireless Fidelity)®. The controller 120 may also communicate with the HMD 100. Multiple controllers may be used for one PC 110 (one HMD 100).

[0016] As shown in Figure 1, the controller 120 is a ring-shaped device that can be worn and supported on the user's finger. The user can use the controller 120 to indicate a position on the image (screen) displayed on the HMD 100, and can change the indicated position by moving the controller 120. The HMD 100 displays a point indicating the indicated position, or a virtual ray, which is a straight line (line segment), extending from the controller 120 to the indicated position. The controller 120 is equipped with physical buttons, and the user can perform menu confirmation and selection operations by pressing these buttons.

[0017] Note that the controller 120 is not limited to a ring-shaped device. For example, it may be a hand-held device held by hand, a bracelet-shaped device worn on the arm, or a glove-shaped device such as a glove. Also, the operating member provided on the controller 120 is not limited to a button and may include any one or more of a track pad, a touch panel, a wheel, and a track ball. The operations that the controller 120 can accept are not limited to pressing a button and may include a slide operation, a flick operation, a touch operation, etc. The controller 120 may be attached to an object held by hand or the like.

[0018] The SNS server 200 accepts submissions of content such as images, music, comments, etc. from various users and provides the submitted content to various users. For example, the SNS server 200 accepts submissions of images from the PC 110 and transmits the submitted images to the PC 110. Note that the SNS server 200 also accepts submissions of content such as images, music, comments, etc. from various users from other electronic devices such as smartphones and tablet terminals. Also, the SNS server 200 provides the submitted content to other electronic devices such as various smartphones and tablet terminals.

[0019] The virtual space server 300 provides a virtual space to various users. For example, the virtual space server 300 transmits data of the virtual space (for example, data representing a three-dimensional (3D) model of the virtual space) to the PC 110 via the SNS server 200. The virtual space server 300 may communicate with the PC 110 without going through the SNS server 200. The virtual space is, for example, a digital twin space which is a three-dimensional space that reproduces the real space. The virtual space is not limited to a digital twin space and may be, for example, a metaverse space that is not a digital twin space.

[0020] <Configuration of HMD> Figure 2 is a block diagram showing the configuration of the HMD100, PC110, and controller120. The HMD100 includes an HMD control unit 201, a display unit 202, an HMD attitude sensor unit 203, a non-volatile memory 204, a working memory 205, and a gaze sensor unit 206.

[0021] The HMD control unit 201 is, for example, a CPU and controls various parts of the HMD 100. When the HMD control unit 201 receives an image (image data) from the PC 110, it displays the image on the display unit 202. Alternatively, instead of the HMD control unit 201 controlling the entire HMD 100, multiple hardware components may share the processing to control the entire HMD 100.

[0022] The display unit 202 displays an image. For example, the display unit 202 has a display panel such as an LCD panel or an OLED panel, and displays an image on the display panel. When the user is wearing the HMD 100, the display unit 202 is positioned in front of the user's eyes. The display unit 202 has a right display unit positioned in front of the right eye and a left display unit positioned in front of the left eye, and by displaying two images with parallax in the right display unit and the left display unit respectively, the user can view the image in 3D.

[0023] The HMD attitude sensor unit 203 acquires information on the attitude (and position) of the HMD 100. The HMD attitude sensor unit 203 may also acquire information on the attitude (and position) of the user (the user wearing the HMD 100) corresponding to the attitude (and position) of the HMD 100. The HMD control unit 201 transmits the information (attitude (and position) information) acquired by the HMD attitude sensor unit 203 to the PC 110. The HMD attitude sensor unit 203 includes, for example, an inertial measurement unit (IMU). The IMU includes at least one of an acceleration sensor, an angular acceleration sensor, a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, and a geomagnetic sensor.

[0024] The non-volatile memory 204 is an electrically erasable and recordable non-volatile memory. The non-volatile memory 204 stores programs executed by the HMD control unit 201, among other things.

[0025] The working memory 205 is used as a buffer memory for temporarily holding various data, as well as as image display memory for the display unit 202 and as a work area for the HMD control unit 201.

[0026] The gaze sensor unit 206 acquires information to detect the user's gaze. For example, the gaze sensor unit 206 has a camera (image sensor) that captures images of the user's eyes while they are wearing the HMD 100. The HMD control unit 201 transmits the information acquired by the gaze sensor unit 206 (for example, an image of the user's eyes) to the PC 110.

[0027] <Controller Configuration> As shown in Figure 2, the controller 120 includes a controller control unit 221, an operation unit 222, a communication unit 223, and a controller attitude sensor unit 224.

[0028] The controller control unit 221 is, for example, a CPU, and controls each part of the controller 120. Alternatively, instead of the controller control unit 221 controlling the entire controller 120, multiple hardware components may share the processing to control the entire controller 120.

[0029] The operation unit 222 has operating elements such as buttons and detects user operations such as button presses. When the operation unit 222 detects a user operation, it transmits an operation signal corresponding to that user operation to the PC 110 via the communication unit 223.

[0030] The communications unit 223 communicates with PC110.

[0031] The controller attitude sensor unit 224 acquires information on the attitude (and position) of the controller 120. The controller attitude sensor unit 224 transmits the acquired information (attitude (and position) information) to the PC 110 via the communication unit 223. The controller attitude sensor unit 224 has, for example, an IMU including at least any one of an acceleration sensor, an angular acceleration sensor, a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, and a geomagnetic sensor.

[0032] <Configuration of PC> As shown in FIG. 2, the PC 110 includes a PC control unit 211, a non-volatile memory 212, a working memory 213, a communication unit 214, and a recording medium 215.

[0033] The PC control unit 211 is, for example, a CPU and controls each part of the PC 110. Instead of the PC control unit 211 controlling the entire PC 110, a plurality of hardware may share the processing to control the entire PC 110.

[0034] The PC control unit 211 transmits the image (image data) to be displayed on the HMD 100 to the HMD 100. The PC control unit 211 can also generate video (video data) of the virtual space and transmit it to the HMD 100. In that case, the PC control unit 211 determines the range of the virtual space to be generated as video based on the information (attitude (and position) information of the HMD 100) acquired by the HMD attitude sensor unit 203. By doing so, the range of the virtual space to be displayed can be changed according to the changes in the position and attitude of the HMD 100 (user), as if the user were in the virtual space.

[0035] The PC control unit 211 synthesizes (superimposes) an item (such as a point or a ray) indicating the user's instructed position on the image to be displayed on the HMD 100 based on the information (attitude (and position) information of the controller 120) acquired by the controller attitude sensor unit 224. ントローラー120の姿勢(および位置)の情報)に基づいて、ユーザーの指示位置を示すアイテム(点やレイなど)を、HMD100で表示する画像に合成(重畳)する。

[0036] The PC control unit 211 performs various controls in response to operation signals transmitted from the operation unit 222. Furthermore, the PC control unit 211 detects the gaze of the user wearing the HMD 100 based on information acquired by the gaze sensor unit 206 (for example, an image of the user's eyes) and performs various controls based on the user's gaze. For example, the PC control unit 211 considers the user's gaze on a button displayed on the HMD 100 as a button press and performs various controls accordingly.

[0037] The non-volatile memory 212 is an electrically erasable and recordable non-volatile memory. The non-volatile memory 212 stores programs executed by the PC control unit 211, etc.

[0038] The working memory 213 is used as a buffer memory for temporarily holding various data, as well as as image display memory for the display unit 202 and as a work area for the PC control unit 211.

[0039] The communications unit 214 communicates with the controller 120, the SNS server 200, and other devices.

[0040] The recording medium 215 stores various types of data.

[0041] Figure 3 is a flowchart illustrating the processing performed by PC110. The processing shown in Figure 3 is achieved when the PC control unit 211 loads a program stored in the non-volatile memory 212 into the working memory 213 and executes it. For example, when PC110 starts up and a user performs an operation to launch an SNS application installed on PC110, the processing shown in Figure 3 begins. While the SNS application is running, the processing shown in Figure 3 is repeated at predetermined intervals. When a user performs an operation to terminate the SNS application, the processing shown in Figure 3 ends.

[0042] In step S301, the PC control unit 211 displays a list of content posted to the SNS server 200 on a display device connected to the PC 110 (in this embodiment, the display unit 202 of the HMD 100). For example, the PC control unit 211 obtains thumbnails of posted images from the SNS server 200 via the communication unit 214, generates video data of a list screen with the obtained thumbnails arranged, and transmits the video data of the list screen to the HMD 100.

[0043] In step S302, the PC control unit 211 determines whether a browsing operation has been performed to select one or more images (thumbnails) displayed on the list screen. The PC control unit 211 waits for a browsing operation to be performed, and if it determines that a browsing operation has been performed, it proceeds to step S303. A browsing operation is, for example, a user operation in which the controller 120 indicates the position of the thumbnail to be selected and then presses a predetermined button (OK button) on the controller 120. A browsing operation is not particularly limited and may also be a user operation in which the user looks at the thumbnail to be selected and then presses a predetermined button on the controller 120. A browsing operation may also be an operation in which the controller 120 indicates the position of the thumbnail to be selected or the user looks at the thumbnail for a predetermined period of time.

[0044] In step S303, the PC control unit 211 requests the SNS server 200, via the communication unit 214, to send the image selected in the browsing operation in step S302. In response to this request, the SNS server 200 sends the image (image data) selected in the browsing operation in step S302 to the PC 110.

[0045] In step S304, the PC control unit 211 includes the image received in step S303. The browsing screen is displayed on a display device connected to the PC 110 (in this embodiment, the display unit 202 of the HMD 100). For example, the PC control unit 211 generates video data of the browsing screen and transmits it to the HMD 100.

[0046] In step S305, the PC control unit 211 determines whether a return operation has been performed to return the displayed screen to the previous screen. If the PC control unit 211 determines that a return operation has been performed, it proceeds to step S301; otherwise, it proceeds to step S306. A return operation is, for example, a user operation such as pressing a predetermined button (return button) on the controller 120. A return operation is not particularly limited. For example, the browsing screen may include an item for a return operation, and the return operation may be a user operation that specifies (selects) such an item, similar to a browsing operation.

[0047] In step S306, the PC control unit 211 determines whether a spatial display operation (VR (Virtual Reality) display operation) has been performed to display images of a virtual space such as a digital twin space or another metaverse space. If the PC control unit 211 determines that a spatial display operation has been performed, it proceeds to step S307; otherwise, it proceeds to step S305. A spatial display operation is, for example, a user operation such as pressing a predetermined button (VR button) on the controller 120. Spatial display operations are not particularly limited. For example, the browsing screen may include items for spatial display operations, and a spatial display operation may be a user operation that specifies (selects) such items, similar to a browsing operation.

[0048] In step S307, the PC control unit 211 determines whether the display device connected to the PC 110 is an HMD. If the PC control unit 211 determines that the display device connected to the PC 110 is an HMD, it proceeds to step S308; otherwise (for example, if the display device connected to the PC 110 is a regular PC monitor), it proceeds to step S305. The display of images in the virtual space is suitable for display on an HMD. The process in step S307 restricts the display devices that display images in the virtual space to HMDs. As part of the determination in step S307, it is also possible to determine whether the display device connected to the PC 110 is a mobile device. By doing so, smartphones and tablet devices can be included as display devices that display images in the virtual space.

[0049] In step S308, the PC control unit 211 acquires location information (for example, information indicating the location where the image was taken) associated with the image (the image of the viewing screen) received in step S303. For example, in step S303, image data is received with location information attached as metadata, and in step S308, metadata is acquired (extracted) from the image data. Alternatively, in step S303, image data may be received without location information attached, and in step S308, location information may be acquired from the SNS server 200 via the communication unit 214.

[0050] In step S309, the PC control unit 211 determines whether the image received in step S303 (the image on the viewing screen) is a specific image that meets at least one of the following conditions. If the PC control unit 211 determines that the image received in step S303 is a specific image, it proceeds to step S305; otherwise, it proceeds to step S309. Condition 1: The image contains a person. Condition 2: The image does not have the required information (permissions) granted to it. Condition 3: The image shows a location outside the designated area. Condition 4: It is not an image of real space. Condition 5: It must be a video (not a still image).

[0051] The processing in step S309 also restricts the display of video in the virtual space. Conditions 1 to 3 protect the privacy of the poster. For example, by using condition 1, it is possible to prevent others from knowing the poster's actions in detail through the display of video in the virtual space. By using condition 2, the poster can choose whether or not to display video in the virtual space based on whether or not they have certain information (permissions). By using condition 3, the areas where video in the virtual space can be displayed can be restricted to areas that have been pre-registered on the SNS server 200 by a tourism association or similar organization.

[0052] When using Condition 1, it is necessary to detect people from the image. The method for detecting people from an image is not particularly limited, and various existing methods can be used. A person who satisfies conditions such as the image size being above a threshold or being located in the center of the image may be detected as the main subject. For this reason, Condition 1 may also be a condition that the image contains a person as the main subject. If the poster's face image is pre-registered on the SNS server 200, only the poster can be detected from the image. For this reason, Condition 1 may also be a condition that the image contains the poster.

[0053] When using condition 2, the specified information (permissions) may be information that can only be granted by store account users or specific posters (specified individuals) such as influencers. This protects the privacy of general users. The specified information may also be location information obtained in step S308. In that case, condition 2 is satisfied if obtaining location information fails in step S308. Condition 2 may also be a condition that the image is accompanied by information indicating that the display of video in the virtual space is not permitted.

[0054] In step S310, the PC control unit 211 requests data of a predetermined size, including the location corresponding to the location information acquired in step S308, from the virtual space server 300 via the communication unit 214 and the SNS server 200, as virtual space data. The predetermined size range is, for example, a hemispherical (dome-shaped) range centered on the location corresponding to the location information acquired in step S308. In response to this request, the virtual space server 300 transmits the virtual space data (data of the predetermined size range) to the PC 110 via the SNS server 200. The data for the entire virtual space could also be transmitted, but by limiting the virtual space data to data of a predetermined size range, the amount of data transmitted can be reduced.

[0055] In step S311, the PC control unit 211 determines whether it has successfully received data from the virtual space. If the PC control unit 211 determines that it has successfully received data from the virtual space, it proceeds to step S312; otherwise, it proceeds to step S305. Depending on the connection status between PC 110, SNS server 200, and virtual space server 300, it may fail to receive data from the virtual space. Also, if data from the virtual space that is not on the virtual space server 300 is requested, it will fail to receive the virtual space data. For this reason, the process in step S311 is performed.

[0056] If the data for the virtual space is already prepared on the recording medium 215 of the PC110, steps S310 and S311 may be omitted.

[0057] In step S312, the PC control unit 211 displays a virtual space image on a display device connected to the PC 110 (in this embodiment, the display unit 202 of the HMD 100) with the viewpoint corresponding to the position information acquired in step S308, based on the virtual space data received in step S311. For example, the PC control unit 211 generates virtual space image data and transmits it to the HMD 100.

[0058] With the virtual space image displayed, a user wearing the HMD100 can turn to the right to see the area to the right that is not depicted in the image received in step S303 (the image on the viewing screen). Similarly, by turning to the left, up, or down, the user can see the areas to the left, up, or down that are not depicted in the image received in step S303. In addition, the user can move within the virtual space by performing a movement operation to move themselves (the viewpoint). In this way, the user can easily obtain information about parts that are not depicted in the image (for example, parts that are not shown in a photograph). If the virtual space is a digital twin space, the image received in step S303, and the location information acquired in step S308 indicates the shooting location of the image received in step S303, the display of the virtual space image allows the user to experience as if they were actually at the shooting location.

[0059] In step S313, the PC control unit 211 determines whether a return operation has been performed to return the displayed screen to the previous screen. If the PC control unit 211 determines that a return operation has been performed, it proceeds to step S304; otherwise, it proceeds to step S314. A return operation is, for example, a user operation such as pressing a predetermined button (return button) on the controller 120. The return operation is not particularly limited. For example, an item for the return operation may be displayed along with the virtual space image, and the return operation may be a user operation that specifies (selects) the item, similar to a browsing operation.

[0060] In step S314, the PC control unit 211 determines whether a movement operation to change the viewpoint has been performed. The PC control unit 211 waits for the movement operation to be performed, and if it determines that a movement operation has been performed, it proceeds to step S315. If the PC control unit 211 does not perform a movement operation, it proceeds to step S313. A movement operation is, for example, a user operation in which the controller 120's predetermined stick (movement stick) is tilted in the direction the user wants to move, such as up (back), down (front), left, or right. The movement operation is not particularly limited. For example, an item for the movement operation may be displayed along with the virtual space image, and the movement operation may be a user operation in which the user specifies (selects) the item, similar to a browsing operation.

[0061] In step S315, the PC control unit 211 requests data of a predetermined size, including the position after movement corresponding to the movement operation in step S314, from the virtual space server 300 via the communication unit 214 and the SNS server 200, as virtual space data. In response to this request, the virtual space server 300 transmits the virtual space data (data of a predetermined size) to the PC 110 via the SNS server 200.

[0062] In step S316, the PC control unit 211 displays a virtual space image, with the position after the movement operation in step S314 as the viewpoint, on a display device connected to the PC 110 (in this embodiment, the display unit 202 of the HMD 100), based on the virtual space data received in step S315.

[0063] Note that the processing of PC110 may be modified as appropriate; for example, at least one of the processes in steps S306 to S308 may be omitted. Without displaying the browsing screen, the display on HMD100 may be switched from the list screen display to the virtual space image display in response to a user operation to select an image on the list screen. The function of displaying the virtual space image may or may not be part of the functions of the SNS application. If the VR application that displays the virtual space image is a separate application from the SNS application, the SNS application and the VR application will cooperate to perform the processing shown in Figure 3. For example, when the process proceeds to step S310, the SNS application launches the VR application.

[0064] Figure 4(A) is a schematic diagram showing an example of the list screen displayed in step S301 of Figure 3. In the list screen of Figure 4(A), thumbnails of multiple posted images are displayed. There is no particular limit to the number of images posted, but for the sake of simplicity, let's assume that six images have been posted by different users. In the list screen of Figure 4(A), six thumbnails 401 to 406 are displayed, each corresponding to one of the six images. Thumbnails 401, 402, and 405 are thumbnails corresponding to images (still images) in the virtual space, while thumbnails 403, 404, and 406 are thumbnails corresponding to still images in the real world. Images in the virtual space include, for example, images obtained from screenshots of virtual space video or images taken with a virtual camera. The list screen may also include thumbnails of videos in the real world, videos in the virtual space, illustrations, animations, etc.

[0065] The PC control unit 211 may select and set one of several display modes for the list screen in response to user operations or other factors. The multiple display modes include a mode that displays a list screen showing images that trigger the display of virtual space video and images that do not trigger the display of virtual space video, and a mode that displays a list screen showing only images that trigger the display of virtual space video.

[0066] Here, let's assume that only condition 1 is used in step S309 of Figure 3. In that case, the list screen in Figure 4(A) shows trigger images and non-trigger images. Thumbnail 404 is a thumbnail of an image in which the poster is pictured, and is a non-trigger image. Thumbnails 401-403, 405, and 406 are thumbnails of trigger images. Therefore, if a mode is set to display a list screen that shows only trigger images, the list screen in Figure 4(B) will be displayed in step S301 of Figure 3. The list screen in Figure 4(B) does not include thumbnail 404, but includes thumbnails 401-403, 405, and 406. According to the list screen in Figure 4(B), the user can easily select an image that will trigger the display of video in the virtual space.

[0067] As shown in Figure 4(C), in the mode that displays a list screen showing only the trigger image, the PC control unit 211 may determine the order of multiple images (multiple thumbnails) on the list screen so that images from the real world are placed before images from the virtual world. As shown in Figure 4(D), in the mode that displays a list screen showing only the trigger image, the PC control unit 211 may adopt a list screen that shows only images from the real world. According to the list screens in Figures 4(C) and 4(D), the user can easily select an image from the real world that will trigger the display of video in the virtual world.

[0068] As shown in Figure 4(E), the list screen may include an item 410 associated with a trigger image (thumbnail). Item 410 is a button for starting the display of virtual space video. The form and arrangement of item 410 are not particularly limited. Item 410 may also be displayed on the list screen that shows only the trigger image. The PC control unit 211 may, in response to a user operation to specify (select) an item 410, perform control to display the virtual space video based on the image associated with that item 410 on the display unit 202. For example, after displaying the list screen on the display unit 202 in step S301 of Figure 3, the PC control unit 211 may proceed to step S306 in response to a user operation to specify an item 410. In situations where virtual space video is not displayed, such as when the display device connected to the PC 110 is not an HMD, the PC control unit 211 may not display item 410 on the list screen, or may display it in an invalid state (e.g., grayed out). This prevents users from selecting item 410 even though the virtual space image is not displayed.

[0069] Figure 5(A) is a schematic diagram showing an example of the browsing screen displayed in step S304 of Figure 3. The browsing screen in Figure 5(A) includes the selected image 500 and items 501-503. While a detailed explanation is omitted, the viewing screen in Figure 5(A) also includes items that correspond to various functions commonly used on social networking services (SNS). For example, in addition to items 501-503, it includes items such as a "Like" button to indicate that you like, enjoy, or support the posted content, a comment button to comment on a post, and a share button to share on other SNS platforms.

[0070] As shown in Figure 5(B), when a post contains multiple images, an item 504 is included to indicate which image is currently being displayed. Figure 5(B) shows an example where a post contains seven images. Item 504 consists of seven circles (dots), each corresponding to one of the seven images, and the circle corresponding to the currently displayed image is displayed in a different manner (different color, brightness, etc.) from the six circles corresponding to the remaining six images. Users can switch between the displayed images by user actions such as touching or gazing at the circles, or pressing the directional keys on the controller 120. Note that when a post contains only one image, the item indicating which image is currently being displayed is not included.

[0071] Item 501 is a button for the return operation in step S305, and items 502 and 503 are buttons for the spatial display operation in step S306. The form and arrangement of items 501 to 503 are not particularly limited. After the PC control unit 211 displays the browsing screen on the display unit 202 in step S304, it proceeds to step S301 in response to a user operation specifying item 501. The PC control unit 211 also proceeds to step S307 in response to a user operation specifying item 502 or item 503. In situations where the display device connected to the PC 110 is not an HMD, and the virtual space image is not displayed, the PC control unit 211 may not display items 502 and 503 on the browsing screen, or may display them in an invalid state (e.g., grayed out). This prevents the user from specifying items 502 and 503 even though the virtual space image is not displayed.

[0072] Figure 6(A) is a schematic diagram showing an example of a virtual space image displayed in step S312 of Figure 3. In Figure 6(A), items 601a-601d and 602 are displayed along with the virtual space image 600. As described above, the range of the virtual space image 600 changes with changes in the posture of the HMD 100 (changes in the orientation of the user's face while wearing the HMD 100), but the positions of items 601a-601d and 602 relative to the virtual space image 600 are fixed. Items 601a-601d are buttons for the movement operation in step S314, and item 602 is a button for the return operation in step S313. Note that the form and arrangement of items 601a-601d and 602 are not particularly limited.

[0073] After displaying the virtual space image 600 on the display unit 202 in step S312, the PC control unit 211 proceeds to step S315 in response to a user operation specifying item 601a, item 601b, item 601c, or item 601d. If item 601a is specified, the viewpoint moves forward; if item 601b is specified, the viewpoint moves backward; if item 601c is specified, the viewpoint moves to the left; and if item 601d is specified, the viewpoint moves to the right. The PC control unit 211 also proceeds to step S304 in response to a user operation specifying item 602.

[0074] As shown in Figure 6(B), the PC control unit 211 may display an item 603 indicating the existence of a post on the display unit 202 along with the virtual space image 600. Item 603 is a virtual window of the list screen. In this way, the user can understand whether there are any posts corresponding to their current location in the virtual space. Similar to steps S301 to S312 in Figure 3, an image is selected from item 603 (virtual window of the list screen). If this occurs, the PC control unit 211 may transition the virtual space image 600 to the image of the viewpoint based on the selected image. This further improves convenience. In this case, the user will have the feeling of having teleported within the virtual space.

[0075] The form and position of the item indicating the presence of a post are not particularly limited. For example, as shown in Figure 6(C), an item 604 (icon) indicating the presence of a post may be displayed at a position corresponding to the post's location information. This allows the user to navigate to the location corresponding to the post's location information by relying on item 604. The PC control unit 211 may also shift its viewpoint to the position of item 604 in response to a user operation specifying item 604. This further improves convenience.

[0076] Furthermore, the real or virtual space may be divided into multiple areas in advance, and advertisements may be pre-associated with each area. Then, as shown in Figure 6(C), the PC control unit 211 may display the advertisement 605 on the display unit 202 along with the virtual space image 600 based on the location information acquired in step S308. The advertisement 605 is an advertisement associated with an area in the real space that includes the location indicated by the location information, or an advertisement associated with an area in the virtual space that includes the location (viewpoint) corresponding to the location information. In this way, the user can easily obtain information related to their current viewpoint (for example, information about surrounding facilities) from the advertisement 605. The advertising fee may be paid to the poster of the selected image, or to the SNS or virtual space service provider.

[0077] As described above, according to this embodiment, location information associated with the image selected by the user is acquired, and a video of a virtual space with the location corresponding to that location information as the viewpoint is displayed on the display device. In this way, the user can easily obtain information about parts that are not depicted in the image (for example, parts that are not shown in the photograph).

[0078] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0079] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0080] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0081] For example, I described an instance where images posted on social media are selected, but this is not the only example. For instance, a user may store images they have taken on the recording medium 215 and review them later, but may not remember what parts of the image were like that were not visible. Therefore, The present invention can also be applied to cases such as selecting an image stored in the recording medium 215.

[0082] Furthermore, although an example was described in which the virtual space image is not displayed when a video is selected, the virtual space image may be displayed even when a video is selected. When a video is selected, the PC control unit 211 may acquire location information that represents the video as location information associated with the video. Representative location information for a video may include, for example, location information corresponding to the thumbnail of the video, location information corresponding to the first frame of the video, or location information of distinctive objects (landmarks) shown in the video.

[0083] When a video consisting of multiple scenes (a composite video created by stitching together multiple videos) is selected, the PC control unit 211 may acquire multiple location pieces corresponding to each of the scenes as location information associated with the video. In this case, the PC control unit 211 displays a virtual space image on the display unit 202 with the viewpoint set to a location corresponding to one of the multiple location pieces. The PC control unit 211 may also switch the viewpoint between multiple locations corresponding to each of the multiple location pieces in response to a user operation to switch the viewpoint of the virtual space image. This further improves convenience.

[0084] Furthermore, if a post (image content) containing images of multiple locations located more than a predetermined distance apart is selected on social media, the PC control unit 211 may display a virtual space image on the display unit 202 corresponding to the selected location among the multiple locations. Examples of posts containing images of multiple locations located more than a predetermined distance apart on social media include posts listing recommended restaurants serving the same type of cuisine, or posts listing representative tourist attractions in each country (or prefecture). In the case of such posts, the system may display a virtual space image based on the location information of the image displayed at the time VR display (item 502) is selected. When a user is viewing images of a location of interest among multiple locations, selecting VR display (item 502) allows the displayed image to transition to a virtual space image related to the image the user was viewing. Alternatively, in response to the selection of VR display (item 502), the system may display not only the image displayed at the time VR display (item 502) is selected, but also multiple images from a single post, allowing the user to select which image's virtual space image to display. This allows users to view all the images and then, if they find an image of a place that interests them, select that image to choose which virtual space image to display.

[0085] The user operation for switching viewpoints is not particularly limited; it may be a user operation by pressing a predetermined button (viewpoint switching button) on the controller 120, or it may not be. As shown in Figure 6(D), the PC control unit 211 may display viewpoint switching items 606a and 606b on the display unit 202 along with the virtual space image 600. In this case, in response to a user operation specifying item 606a, the PC control unit 211 switches the viewpoint to the next viewpoint from among multiple positions corresponding to multiple position information in a predetermined order (e.g., scene order). Also, in response to a user operation specifying item 606b, the PC control unit 211 switches the viewpoint to the position immediately preceding the current viewpoint from among multiple positions corresponding to multiple position information in a predetermined order.

[0086] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0087] This embodiment includes the following configurations, methods, and programs. (Composition 1) A means for obtaining location information associated with one or more images selected by the user, When the location information is acquired by the acquisition means, a control means controls the display device to display an image of a virtual space with the location corresponding to the location information as the viewpoint. An electronic device characterized by having the following features. (Configuration 2) The one or more images mentioned above are posted images. The electronic device according to configuration 1, characterized by the features described above. (Composition 3) When the location information is acquired by the acquisition means, the control means acquires data of a predetermined size including the location corresponding to the location information as data for the virtual space, and controls the display device to display the image of the virtual space based on the acquired data. The electronic device according to configuration 1 or 2, characterized by the above. (Composition 4) The control means controls the display device to display the image of the virtual space on the display device if the display device is a head-mounted display device, but does not control the display device to display the image of the virtual space on the display device if the display device is not a head-mounted display device. An electronic device according to any one of configurations 1 to 3, characterized by the features described herein. (Composition 5) The aforementioned virtual space is a space that reproduces the real world. An electronic device according to any one of configurations 1 to 4, characterized by the above. (Composition 6) The control means is capable of controlling the display of a list screen of one or more images, and selects the list screen to display from a plurality of list screens, including a first list screen that shows images that trigger the display of video in the virtual space and images that do not trigger the display of video in the virtual space, and a second list screen that shows only images that trigger the display of video in the virtual space. An electronic device according to any one of configurations 1 to 5, characterized by the above. (Composition 7) The electronic device according to configuration 6, characterized in that, when the second list screen displays images of real space and images of virtual space, the control means determines the order in which the plurality of images are arranged on the second list screen such that the images of real space are placed before the images of virtual space. (Composition 8) The second list screen mentioned above displays only images captured from real space. The electronic device according to configuration 6, characterized by the features described therein. (Composition 9) If an image containing a person is selected by the user, the control means does not control the display device to display the image of the virtual space. An electronic device according to any one of configurations 1 to 8, characterized by the above. (Composition 10) If the user selects an image that does not have the required information attached, the control means does not control the display device to display the image of the virtual space. An electronic device according to any one of configurations 1 to 9, characterized by the features described herein. (Composition 11) If the location information acquired by the acquisition means indicates a location outside the predetermined area, the control means does not control the display device to display the image of the virtual space. An electronic device according to any one of configurations 1 to 10, characterized by the above. (Composition 12) If the user selects an image that is not an image of the real world, the control means will not control the display device to display the image of the virtual world. An electronic device according to any one of configurations 1 to 11, characterized by the features described herein. (Composition 13) If a video is selected by the user, the control means does not control the display device to show the video of the virtual space. An electronic device according to any one of configurations 1 to 12, characterized by the above. (Composition 14) When a video is selected by the user, the acquisition means acquires location information representing the video, such as location information corresponding to the video's thumbnail, location information corresponding to the first frame of the video, or location information of a feature object shown in the video. An electronic device according to any one of configurations 1 to 12, characterized by the above. (Composition 15) When a video consisting of multiple scenes is selected by the user, the acquisition means acquires multiple location information corresponding to each of the multiple scenes. When the acquisition means acquires the plurality of location pieces, the control means controls the display device to display an image of the virtual space with the viewpoint corresponding to one of the plurality of location pieces. An electronic device according to any one of configurations 1 to 12, characterized by the above. (Composition 16) When the acquisition means acquires the plurality of location information, the control means switches the viewpoint between the plurality of locations corresponding to each of the plurality of location information in response to the first user operation. The electronic device according to configuration 15, characterized by the features described herein. (Composition 17) When the acquisition means acquires the plurality of location information, the control means controls the display device to display the first item together with the image of the virtual space, with the viewpoint being one of the locations corresponding to the plurality of location information. The first user operation includes a user operation that specifies the first item. The electronic device according to configuration 16, characterized by the features described above. (Composition 18) The control means is Control the display device to show a list screen of one or more images, or a viewing screen of an image selected by the user. In response to a second user operation while the list screen or browsing screen is displayed, the display on the display device is controlled to switch from the display of the list screen or browsing screen to the display of the video in the virtual space. An electronic device according to any one of configurations 1 to 17, characterized by the features described herein. (Composition 19) The aforementioned browsing screen includes the second item, The second user operation includes a user operation that specifies the second item. The electronic device according to configuration 18, characterized by the features described above. (Composition 20) The control means, under predetermined circumstances, does not control the display device to show the image of the virtual space even if the second user operation is performed, and controls the browsing screen not to display the second item, or to display it in an invalid state. The electronic device according to configuration 19, characterized by the features described above. (Composition 21) The aforementioned list screen includes a third item associated with the image in the aforementioned list screen, The second user operation includes a user operation that specifies the third item. An electronic device according to any one of configurations 18 to 20, characterized by the above. (Composition 22) The control means, under predetermined circumstances, does not control the display device to show the image of the virtual space even if the second user operation is performed, and does not display the third item on the list screen, or displays it in an invalid state. The electronic device according to configuration 21, characterized by the features described above. (Composition 23) After the display on the display device switches from the list screen or the browsing screen to the display of the virtual space video, the control means controls the display on the display device to return from the display of the virtual space video to the screen that was on before the switch to the display of the virtual space video, in response to a third user operation. An electronic device according to any one of configurations 18 to 22, characterized by the above. (Composition 24) The control means controls the display of an item representing at least one of the one or more images along with the video of the virtual space. An electronic device according to any one of configurations 1 to 23, characterized by the features described herein. (Composition 25) The control means controls the display device to display, along with the image of the virtual space, an advertisement associated with an area including the location indicated by the location information or an area including the viewpoint. An electronic device according to any one of configurations 1 to 24, characterized by the features described herein. (method) A step to obtain location information associated with one or more images selected by the user, If the location information is acquired in the acquisition step, a control step is performed to control the display device to display an image of a virtual space with the location corresponding to the location information as the viewpoint. A method for controlling electronic equipment, characterized by having the following features. (program) A program for causing a computer to function as one of the electronic devices described in any of configurations 1 to 25. [Explanation of Symbols]

[0088] 110: PC (Personal Computer) 211: PC Control Unit

Claims

1. A means for acquiring location information associated with one or more images selected by the user, When the location information is acquired by the acquisition means, a control means controls the display device to display an image of a virtual space with the location corresponding to the location information as the viewpoint. An electronic device characterized by having the following features.

2. The one or more images mentioned above are posted images. The electronic device according to feature 1.

3. When the location information is acquired by the acquisition means, the control means acquires data of a predetermined size, including the location corresponding to the location information, as data for the virtual space, and controls the display device to display the image of the virtual space based on the acquired data. The electronic device according to feature 1.

4. The control means controls the display device to display the image of the virtual space on the display device if the display device is a head-mounted display device, but does not control the display device to display the image of the virtual space on the display device if the display device is not a head-mounted display device. The electronic device according to feature 1.

5. The aforementioned virtual space is a three-dimensional space that reproduces the real world. The electronic device according to feature 1.

6. The control means is capable of controlling the display of a list screen of one or more images, and selects the list screen to display from a plurality of list screens, including a first list screen that shows images that trigger the display of video in the virtual space and images that do not trigger the display of video in the virtual space, and a second list screen that shows only images that trigger the display of video in the virtual space. The electronic device according to feature 1.

7. The electronic device according to claim 6, wherein, when the second list screen displays images of real space and images of virtual space, the control means determines the order in which the plurality of images are arranged on the second list screen such that the images of real space are placed before the images of virtual space.

8. The second list screen shows only images captured from real space. The electronic device according to feature 6.

9. If an image containing a person is selected by the user, the control means does not control the display device to display the image of the virtual space. The electronic device according to feature 1.

10. If the user selects an image that does not have the required information attached, the control means does not control the display device to display the image of the virtual space. The electronic device according to feature 1.

11. If the location information acquired by the acquisition means indicates a location outside the predetermined area, the control means does not control the display device to display the image of the virtual space. The electronic device according to feature 1.

12. If the user selects an image that is not an image of the real world, the control means will not control the display device to display the image of the virtual world. The electronic device according to feature 1.

13. If a video is selected by the user, the control means does not control the display device to show the video of the virtual space. The electronic device according to feature 1.

14. The electronic device according to claim 1, wherein when a video is selected by the user, the acquisition means acquires location information corresponding to the thumbnail of the video, location information corresponding to the first frame of the video, or location information of a feature object shown in the video, as location information representing the video.

15. When a video consisting of multiple scenes is selected by the user, the acquisition means acquires multiple location information corresponding to each of the multiple scenes. When the acquisition means acquires the plurality of location pieces, the control means controls the display device to display an image of the virtual space with the viewpoint corresponding to one of the plurality of location pieces. The electronic device according to feature 1.

16. When the acquisition means acquires the plurality of location information, the control means switches the viewpoint between the plurality of locations corresponding to each of the plurality of location information in response to the first user operation. The electronic device according to feature 15.

17. When the acquisition means acquires the plurality of location information, the control means controls the display device to display the first item together with the image of the virtual space, with the viewpoint being one of the locations corresponding to the plurality of location information. The first user operation includes a user operation that specifies the first item. The electronic device according to feature 16.

18. The control means is The display device is controlled to display a list screen of one or more images, or a viewing screen of an image selected by the user. In response to a second user operation while the list screen or browsing screen is displayed, the display on the display device is controlled to switch from the display of the list screen or browsing screen to the display of the image of the virtual space. The electronic device according to feature 1.

19. The aforementioned browsing screen includes the second item, The second user operation includes a user operation that specifies the second item. The electronic device according to feature 18.

20. The control means, under predetermined circumstances, does not control the display device to show the image of the virtual space even if the second user operation is performed, and controls the browsing screen not to display the second item, or to display it in an invalid state. The electronic device according to feature 19.

21. The aforementioned list screen includes a third item associated with the image in the aforementioned list screen, The second user operation includes a user operation that specifies the third item. The electronic device according to feature 18.

22. The control means, under predetermined circumstances, does not control the display of the image of the virtual space on the display device even if the second user operation is performed, and controls the third item not to be displayed on the list screen, or to be displayed in an invalid state. The electronic device according to feature 21.

23. After the display on the display device switches from the list screen or the browsing screen to the display of the virtual space video, the control means controls the display on the display device to return from the display of the virtual space video to the screen that was on before the switch to the display of the virtual space video, in response to a third user operation. The electronic device according to feature 18.

24. The control means controls the display of an item representing at least one of the one or more images along with the video of the virtual space. The electronic device according to feature 1.

25. The control means controls the display device to display, along with the image of the virtual space, an advertisement associated with an area including the location indicated by the location information or an area including the viewpoint. The electronic device according to feature 1.

26. A step to obtain location information associated with one or more images selected by the user, If the location information is acquired in the acquisition step, a control step is performed to control the display device to display an image of a virtual space with the location corresponding to the location information as the viewpoint. A method for controlling electronic equipment, characterized by having the following features.

27. A program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 25.

Images