Information processing device, head-mounted display device, electronic apparatus, information processing method, and program

Abstract

To enable a user to easily find a desired setting item of an electronic apparatus and to change setting of the electronic apparatus with a simple operation.SOLUTION: An information processing device includes: first acquisition means for acquiring information about a setting item of an electronic apparatus; second acquisition means for acquiring information about a region of an operation screen of the electronic apparatus in a three-dimensional space which is a field of view of a user; display control means for displaying, outside the region of the operation screen in the three-dimensional space, an object indicating the setting item of the electronic apparatus, and changing the object to be displayed according to the region of the operation screen in the three-dimensional space; and transmission means for, in response to an operation by the user of moving the object to the region of the operation screen, transmitting, to the electronic apparatus, information about the setting item corresponding to the object having been moved.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, a head-mounted display device, an electronic device, an information processing method, and a program. [Background technology]

[0002] In recent years, with the increasing sophistication of electronic devices (e.g., smartphones and mirrorless cameras), the number of configurable items has increased, making the operation of setting the graphical user interface (GUI) of the electronic device more complicated. When a user wants to change the displayed items according to their purpose or preference, they can customize the screen by operating the electronic device itself or connecting the electronic device to a personal computer (PC).

[0003] Meanwhile, in recent years, mixed reality (MR) technology has become widespread. For example, a head mounted display (HMD) can use mixed reality technology to display a computer-generated virtual image superimposed on a real image captured by a camera to a user wearing the HMD. This allows the HMD to display a larger amount of information than in the real space. For example, Patent Document 1 describes a method of displaying a virtual item button related to a selected job on a transparent display section of an HMD and enabling the virtual item button to be operated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-180776 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a user wishes to change the settings of an item other than the specific job that has been selected, it may be difficult for the user to find the desired item in a complex hierarchical structure.

[0006] The present invention has an object to enable a user to easily find a desired setting item of an electronic device and to change the settings of the electronic device with a simple operation. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized in having a first acquisition means for acquiring information on setting items of an electronic device, a second acquisition means for acquiring information on an area of ​​an operation screen of the electronic device in a three-dimensional space that is a user's field of view, a display control means for displaying an object indicating a setting item of the electronic device outside the area of ​​the operation screen in the three-dimensional space and changing the object to be displayed in accordance with the area of ​​the operation screen in the three-dimensional space, and a transmission means for transmitting, in accordance with a user's operation to move the object into the area of ​​the operation screen, information on a setting item corresponding to the moved object to the electronic device. It is characterized by: Effect of the Invention

[0008] According to the present invention, a user can easily find a desired setting item of an electronic device and can change the settings of the electronic device with a simple operation. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of the hardware configuration of an HMD and a camera. [Diagram 2] FIG. 2 is a block diagram showing an example of the functional configuration of an HMD and a camera. [Diagram 3] FIG. 2 is a diagram showing an example of a virtual space generated by an HMD. [Figure 4] FIG. 11 is a diagram illustrating an example of setting item information. [Diagram 5] 5 is a flowchart showing an example of processing of the HMD according to the first embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of a user operation in a virtual space. [Figure 7]13A to 13C are diagrams illustrating other examples of user operations in the virtual space. [Figure 8] 13 is a flowchart illustrating an example of a display position determination process for a setting item. [Figure 9] FIG. 13 is a diagram showing an example of a display of setting items in a virtual space. [Figure 10] FIG. 13 is a diagram showing another example of display of setting items in the virtual space. [Figure 11] 10 is a flowchart illustrating an example of a setting process for a camera. [Figure 12] 10 is a flowchart showing an example of processing of an HMD according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiment. The configuration of the embodiment may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). The present invention may also be realized by appropriately combining parts of each embodiment. In each embodiment, the same configuration is described with the same reference symbol.

[0011] <Embodiment 1> Fig. 1(A) is a diagram showing an example of the hardware configuration of an HMD 101 as an information processing device. Fig. 1(B) is a diagram showing an example of the hardware configuration of a camera 120 as an electronic device. Fig. 2 is a block diagram showing an example of the functional configuration of the HMD 101 and the camera 120. Fig. 3 is a diagram showing an example of a virtual space generated by the HMD 101. Note that, although the following describes a case where the information processing device is the HMD 101 (included in the HMD 101), the information processing device may be configured as a separate device connected to the HMD 101.

[0012] With reference to Fig. 3, an example will be described in which an object (virtual object) indicating a setting item of the camera 120 is displayed in a virtual space so that the user can easily find and change the setting of a desired item. Fig. 3 shows an example of a virtual space generated by the HMD 101. The user can change the setting of the real camera 120 through the HMD 101 worn by the user.

[0013] Fig. 3 shows a virtual space 302 displayed on the display unit 109 of the HMD 101 worn by a user 301. The display unit 109 displays the camera 120, the left hand 303 of the user 301, and the right hand 304 of the user 301. The camera 120 has an operation screen 305. Fig. 3 shows an example in which the user 301 holds the camera 120 with the left hand 303 and operates an object placed in the virtual space 302 with the right hand 304.

[0014] The objects placed in the virtual space 302 are virtual objects generated by computer graphics (CG) technology, and indicate setting items 306 and the like that can be displayed on an operation screen 305 of the camera 120. The display position of the object is determined, for example, so that the object does not overlap the area of ​​the operation screen 305.

[0015] The HMD 101 is not limited to a helmet type, and may be, for example, a glasses type. The HMD 101 may be a video see-through type or an optical see-through type. Hereinafter, the HMD 101 will be described as a video see-through type head-mounted display device. When the HMD 101 is an optical see-through type HMD, the real space seen by the user is not an image captured by the imaging unit 108, but an actual field of view seen through the lens of the HMD 101.

[0016] The HMD 101 determines the display position of an object corresponding to each setting item based on, for example, the size and position of the operation screen 305 displayed on the display unit 109. The user 301 can change the position of the object by moving the object through a predetermined operation. When the object is moved through a predetermined operation by the user 301, the HMD 101 transmits information on the setting item corresponding to the moved object to the camera 120.

[0017] The information on the setting item may include information on the position on the operation screen 305 after movement, in addition to the setting value. The user can move the object indicating the setting item to a desired position on the operation screen 305. Based on the information received from the HMD 101, the camera 120 displays the setting item at the position specified by the user on the actual operation screen 305. This allows the user to easily customize the screen of the camera 120.

[0018] The predetermined operation may be any operation capable of moving an object of a setting item, such as drag and drop, or tapping the position of the object before and after the movement, etc. The predetermined operation may also include an operation by voice.

[0019] HMD 101 displays on display unit 109 not only objects of setting items related to the screen configuration currently displayed on display unit 117 of camera 120, but also objects of setting items unrelated to the current display. User 301 can easily find the settings of camera 120 without searching for a desired setting item from a complex hierarchical structure, and can change the settings with a simple operation. Note that electronic devices that are targets for setting changes are not limited to camera 120, but also include various devices having an operation screen, such as smartphones and printers.

[0020] The hardware configuration of the HMD 101 and the camera 120 will be described with reference to Figs. 1(A) and 1(B). Fig. 1(A) is a diagram showing an example of the hardware configuration of the HMD 101. The HMD 101 generates an image of an MR space by superimposing a virtual object generated by CG technology on an image of a real space captured by an imaging device (imaging unit 108) mounted on the HMD 101, and displays the image on the display unit 109. The HMD 101 may also display an image of a VR space generated by arranging a virtual object in a virtual space generated by CG technology on the display unit 109. The MR technology and the VR technology are well-known technologies, and detailed descriptions thereof will be omitted.

[0021] The CPU 102 is a system control unit that controls the entire HMD 101. The CPU 102 executes programs to realize various processes executed by the HMD 101. The ROM 103 is a read-only memory that stores programs and parameters that do not change, such as basic programs and initial data. The RAM 104 is a memory for temporarily storing calculation results in information processing or image processing, input information, and the like.

[0022] The recording unit 105 is a device capable of writing and reading various information, and includes a hard disk and a memory card that are built into or externally attached to the HMD 101. The recording unit 105 may also include a memory card, a flexible disk, an IC card, etc. that are detachable from the HMD 101.

[0023] The recording unit 105 records various programs that realize the processing of the HMD 101. The CPU 102 can realize the processing of the HMD 101 by loading the programs read from the recording unit 105 into the RAM 104 and executing the programs. The programs executed by the CPU 102 may be stored in the ROM 103.

[0024] The recording unit 105 records information on operations input by the user via the input detection unit 106 and information on the position and orientation of the user detected by an acceleration sensor or the like included in the HMD 101. The recording unit 105 records information on the setting items of the camera 120.

[0025] The input detection unit 106 receives an input from a user via an operation unit 110. The input from the operation unit 110 includes, for example, an input by a hand movement recognized by hand tracking.

[0026] The communication I / F 107 is a communication interface capable of transmitting and receiving data to and from an operating device for operating the HMD 101 and a cloud, etc. When the information processing device according to the present invention is configured as a separate entity from the HMD, the communication I / F 107 receives, via a network, voice data acquired from a microphone of the HMD, position and orientation information detected by the HMD, images captured by the HMD, etc. The connection form of the input / output by the communication I / F 107 includes a local connection by USB or Bluetooth (registered trademark), etc., and an Internet connection by Ethernet or Wi-Fi.

[0027] The imaging unit 108 is an imaging device mounted on the HMD 101. The actual image captured by the imaging unit 108 is used for tracking the movement of the user's hands and joints through image recognition processing, and for detecting a plane on which virtual objects such as buttons and icons are superimposed and displayed. The image captured by the imaging unit 108 is also used for recognizing the operation screen of the camera 120.

[0028] The display unit 109 is an electronic display device such as a liquid crystal display device. The display unit 109 receives user instructions recognized by hand tracking. The display unit 109 displays images of a VR world or the like based on the user's instructions and the information stored in the recording unit 105.

[0029] The operation unit 110 includes an operation device that enables pointing operations and input of various commands by hand movements, and acquires operation instructions and commands from a user. Specifically, the operation unit 110 acquires input information by hand movements when the user operates an object of a setting item in a virtual space. Information on the user's operation on the object is sent to the input detection unit 106. The operation unit 110 is not limited to a function of acquiring input information by hand movements recognized by hand tracking, and may have a function of acquiring input information by a controller including a mouse or a keyboard, and input information by voice.

[0030] 1A shows an example in which the information processing device according to the present invention is configured integrally with the HMD 101, and therefore the HMD 101 includes an operation unit 110 and a display unit 109 as internal components. When the information processing device according to the present invention is configured by a PC or the like that is separate from the HMD, the information processing device does not need to include the operation unit 110 and the display unit 109. In this case, the information processing device acquires information on a user's operation from the HMD, which is an external device, and transmits data of a generated virtual space to the HMD.

[0031] 1(B) is a diagram showing an example of the hardware configuration of camera 120. CPU 111 is a system control unit that controls the entire camera 120. CPU 111 executes programs to realize various processes executed by camera 120. ROM 112 is a read-only memory that stores unchanged programs and parameters, such as basic programs and initial data. RAM 113 is a memory for temporarily storing calculation results in information processing or image processing, input information, and the like.

[0032] The recording unit 114 is a device capable of writing and reading various information, and includes a hard disk and a memory card that are built into or externally attached to the camera 120. The recording unit 114 may also include a memory card, a flexible disk, and an IC card that are detachable from the camera 120. It may also include a.

[0033] Recording unit 114 records various programs for implementing the processing of camera 120. CPU 111 can implement the processing of camera 120 by loading the programs read from recording unit 114 into RAM 113 and executing the programs. The programs executed by CPU 111 may be stored in ROM 112. Recording unit 114 also records data used by the programs executed by CPU 111.

[0034] The input detection unit 115 receives an input from a user via the operation unit 118. The input from the operation unit 118 includes, for example, an input from a touch panel or button operation.

[0035] The communication I / F 116 is a communication interface capable of transmitting and receiving data to and from the HMD 101 and the cloud. The connection form of the input / output by the communication I / F 116 includes a local connection by USB or Bluetooth, etc., and an internet connection by Ethernet or Wi-Fi. The HMD 101 and the camera 120 are connected via the communication I / F 116. 107 and the communication I / F 116 .

[0036] The display unit 117 is an electronic display device such as a liquid crystal display device. The display unit 117 receives instructions from a user through a touch operation on the display. The display unit 117 displays a GUI of an operation screen based on the instructions from the user and the information stored in the recording unit 114.

[0037] The operation unit 118 acquires operation instructions from the user for the camera 120, such as button operations and operations on a touch panel. The operation unit 118 acquires input information when the user operates the camera 120 to change the settings of setting items. The operation information of the user for the camera 120 is sent to the input detection unit 115 and the HMD 101.

[0038] 2 is a block diagram showing an example of the functional configuration of the HMD 101 and the camera 120. The HMD 101 has a setting information acquisition unit 201, a screen area acquisition unit 202, a display position determination unit 203, an image generation unit 204, an operation acquisition unit 205, an item position acquisition unit 206, and a setting information transmission unit 207. The camera 120 has a setting information reception unit 208 and a setting value reflection unit 209. The camera 120 holds setting item information 210 in the recording unit 114 or the like.

[0039] The setting information acquisition unit 201 acquires setting item information 210 held by the camera 120 to be operated. Fig. 4 shows an example of the setting item information 210. The setting item information 210 includes the model number, screen ID, setting item, setting value, and display status of the camera 120. The display status is a flag (Y: displayed / N: hidden) indicating whether or not the item is displayed on the operation screen of the camera 120.

[0040] Note that the setting information acquisition unit 201 is not limited to acquiring setting item information 210 held by the camera 120, and may acquire setting item information of the camera 120 stored in advance in the recording unit 105. By the recording unit 105 holding setting item information for each model number of various cameras 120, the setting information acquisition unit 201 can acquire model number information from the camera 120 and acquire setting item information corresponding to that camera 120 from the recording unit 105.

[0041] The screen area acquisition unit 202 acquires the area of ​​the operation screen of the camera 120 operated by the user. Specifically, the screen area acquisition unit 202 acquires information indicating the position and size of the area of ​​the operation screen of the camera 120 using a known technique such as image recognition. Note that acquisition of the position of the area of ​​the operation screen is not limited to image recognition, and a known technique such as SLAM (Simultaneous Localization and Mapping) may also be used.

[0042] The display position determination unit 203 performs display control to change the object of the setting item to be displayed according to the area of ​​the operation screen in the virtual space. That is, the display position determination unit 203 determines which object of the setting item is to be displayed in the virtual space.

[0043] Further, the display position determination unit 203 determines the display position of the object for each setting item. The display position determination unit 203 determines the display position of the object for each setting item based on, for example, the position and size of the area of ​​the operation screen of the camera 120 displayed on the display unit 109 of the HMD 101 worn by the user.

[0044] The object of a setting item displayed on the display unit 109 is, for example, an icon indicating the setting item. The object of a setting item may be associated with information on the setting value of the setting item. The object of a setting item may also include an object indicating an explanation of the setting item.

[0045] The image generating unit 204 generates an image of a virtual space in which an object indicating a setting item is superimposed on an image of a real space captured by the imaging unit 108 of the HMD 101 at a three-dimensional position determined by the display position determining unit 203 .

[0046] The operation acquisition unit 205 acquires a user operation on an object indicating a setting item arranged in a virtual space. The image generation unit 204 generates an image in which the display position of the object indicating the setting item is moved based on the user operation acquired by the operation acquisition unit 205. The user can move the object indicating the setting item to a desired position on the operation screen.

[0047] The item position acquisition unit 206 acquires the display position of the object indicating the setting item after it has been moved. The setting information transmission unit 207 determines whether or not the display position of the object indicating the moved setting item is included in the area of ​​the operation screen of the camera 120. If the display position of the object is included in the area of ​​the operation screen of the camera 120, the item position acquisition unit 206 transmits information on the setting item corresponding to the object to the camera 120.

[0048] The setting information receiving unit 208 of the camera 120 receives information on the setting item corresponding to the object moved to the area of ​​the operation screen from the setting information transmitting unit 207 of the HMD 101. The received information on the setting item (such as the setting value and the display status) is stored in the setting item information 210.

[0049] The setting value reflecting unit 209 reflects the settings based on the information of the setting items received from the setting information receiving unit 208 in the camera 120. The setting value reflecting unit 209 changes the settings of the camera 120 according to the setting values ​​included in the information of the setting items. Furthermore, the setting value reflecting unit 209 displays the setting items on the actual operation screen of the camera 120 at the position to which an object has been moved by a user operation.

[0050] Fig. 5 is a flowchart showing an example of processing of the HMD 101 according to the first embodiment. The processing shown in Fig. 5 is realized by the CPU 102 executing a program corresponding to the processing of each step. The timing at which the processing of Fig. 5 is executed is not limited. The processing of Fig. 5 may be executed at a specified time interval, or may be executed when a user operation is detected, for example.

[0051] In step S501, the setting information acquisition unit 201 of the HMD 101 acquires information on setting items that can be displayed on the operation screen 305 of the camera 120 from the setting item information 210. The setting information acquisition unit 201 records the acquired setting item information in the recording unit 105. By acquiring the setting item information from the camera 120 once and recording it in the recording unit 105, the setting information acquisition unit 201 can acquire the setting item information stored in the recording unit 105 in subsequent processing. Also, if the setting item information for each model number of the camera 120 is stored in advance in the recording unit 105, The setting information acquisition unit 201 may acquire model number information from the camera 120, and acquire corresponding setting item information from the recording unit 105. The camera 120 may transmit model number information of the camera 120 to the HMD 101 instead of the setting item information.

[0052] In step S502, the screen area acquisition unit 202 acquires information on the area of ​​the operation screen of the camera 120 in the virtual space (three-dimensional space) that is the field of view of the user. Specifically, the screen area acquisition unit 202 acquires information indicating the position and size of the area of ​​the operation screen. The screen area acquisition unit 202 can acquire the information on the area of ​​the operation screen by detecting the operation screen of the camera 120 from an image captured by the imaging unit 108, for example.

[0053] In step S503, the display position determination unit 203 determines the display position of an object indicating a setting item based on the information of the setting item acquired in step S501 and the information of the area of ​​the operation screen of the camera 120 acquired in step S502. For example, the display position determination unit 203 determines the display position of the object so that the object of each setting item is placed outside the area of ​​the operation screen. Note that, for items currently displayed on the operation screen of the camera 120, the display position determination unit 203 may superimpose and display the corresponding object within the area of ​​the operation screen 305. Details of the display position determination process by the display position determination unit 203 will be described later using the flowchart of FIG. 6.

[0054] In step S504, the image generating unit 204 generates an image in which an object indicating the setting item is superimposed on the image of real space acquired by the imaging unit 108 at the display position determined in step S503.

[0055] The image generating unit 204 may obtain an image captured by the imaging unit 108 as an image of the operation screen, or may obtain an image displayed on the display unit 117 of the camera 120 from the camera 120 via the communication I / F 116 and the communication I / F 107. The image of the operation screen may be obtained from an image previously stored in the recording unit 105 of the HMD 101. In this case, the HMD 101 can obtain an image of the corresponding operation screen from the recording unit 105 by obtaining information on the screen ID of the screen currently displayed on the operation screen from the camera 120. The image generating unit 204 may synthesize the obtained image of the operation screen with the area of ​​the operation screen in the image of real space captured by the imaging unit 108.

[0056] In step S505, the operation acquisition unit 205 acquires a user operation on an object indicating a setting item in a virtual space. The user operation is, for example, an operation by a hand motion recognized by hand tracking, but is not limited thereto. The user operation may be an operation by a line of sight or a voice. The user operation may also be an operation by any combination of an operation by a hand motion, an operation by a line of sight, and an operation by a voice.

[0057] The process of acquiring a user operation in step S505 will be described with reference to Fig. 6 and Fig. 7. The operation acquisition unit 205 can acquire a user operation of moving an object indicating a setting item, for example, by a hand movement recognized by hand tracking. Fig. 6 and Fig. 7 are diagrams showing an example of a user operation in a virtual space. In the description of Fig. 6 and Fig. 7, an object indicating a setting item is simply described as a "setting item."

[0058] FIG. 6 shows an example of a user operation by an action of moving one of the setting items with the hand. The image of the virtual space 601 displays the camera 120, a number of setting items, text explaining the setting items, and the user's hand. Hand 604 represents the user's hand before moving the setting item 602 (Fricker OFF). Hand 605 represents the user's hand after moving the setting item 602 into the area of ​​the operation screen. Text 603 indicates an explanatory text for the setting item 602. For example, when the user's hand overlaps with the setting item, the text 603 is displayed for a predetermined time. It may be arranged so that

[0059] The user selects a setting item 602 displayed in a virtual space 601 with a hand 604, and moves it to a desired position within the area of ​​the operation screen 305 of the camera 120. The setting item 602 is placed at the position moved by the user (the position of a setting item 606).

[0060] The setting information transmission unit 207 of the HMD 101 transmits information of the setting item corresponding to the setting item 602 to the camera 120 in response to a user operation to move the setting item 602 to an area of ​​the operation screen 305. The setting information reception unit 208 of the camera 120 receives the information of the setting item 602 from the setting information transmission unit 207. The setting value reflection unit 209 reflects the setting of the setting item 602 in real time on the camera 120 based on the information received by the setting information reception unit 208. That is, the setting value reflection unit 209 displays the setting item 602 at a position 606 moved by the user operation on the operation screen 305 of the real camera 120. The setting value reflection unit 209 also applies the Flicker OFF setting to the camera 120.

[0061] When the moved setting item 606 is displayed in the area of ​​the operation screen 305, the image generation unit 204 of the HMD 101 hides the setting item 602 and text 603 that were displayed outside the area of ​​the operation screen 305. Since the area of ​​the operation screen 305 has limited space, the image generation unit 204 performs control so as not to display the explanation of the setting item within the area of ​​the operation screen 305.

[0062] 7 shows an example of a user operation using voice (user 301's voice) and a motion of surrounding and moving multiple setting items with the hand. The image of virtual space 701 displays camera 120, multiple setting items, text explaining the setting items, and the hand of user 301. Hand 705 represents the hand of user 301 before moving setting items 602, 702 to 704. Hand 709 represents the hand of the user after moving setting items 602, 702 to 704 to the area of ​​operation screen 305. User 301 can select multiple setting items by uttering voice 706 saying "multiple selection" and performing a motion of surrounding setting items 602, 702 to 704 displayed in virtual space 701 with hand 705.

[0063] The image of virtual space 701 may highlight the selected setting items by changing the display mode, such as by emitting light, or may display a locus 707 encircling the setting items with a hand 705. After selecting the multiple setting items 602, 702 to 704, the user moves to a desired position on operation screen 305 of camera 120. The setting items 602, 702 to 704 are placed at position 708 to which they were moved by the user.

[0064] When the setting items 602, 702 to 704 are moved to the area of ​​the operation screen 305, the setting information receiving unit 208 of the camera 120 receives information on the setting items 602, 702 to 704 from the setting information transmitting unit 207 of the HMD 101. The setting value reflecting unit 209 reflects the settings of the setting items 602, 702 to 704 in real time on the camera 120 based on the information received by the setting information receiving unit 208. That is, the setting value reflecting unit 209 displays the setting items 602, 702 to 704 at the position 708 moved by the user's operation on the operation screen 305 of the real camera 120. The setting value reflecting unit 209 also applies the settings of each setting item to the camera 120.

[0065] In Fig. 7, the operation of moving the setting item by combining the operation by the hand movement and the voice is described, but the user operation is not limited to this. The user operation may be an operation of selecting a setting item by eye gaze input. The user operation may also be an operation of selecting a setting item or changing the operation content by a hand gesture.

[0066] In step S506 in FIG. 5, the item position acquisition unit 206 acquires the display position of the object indicating the setting item moved by the user operation in step S505.

[0067] In step S507, item position acquisition unit 206 determines whether or not an object of the setting item moved by a user operation exists within the area of ​​the operation screen of camera 120. If the object of the moved setting item exists within the area of ​​the operation screen of camera 120, processing proceeds to step S508. If the object of the moved setting item does not exist within the area of ​​the operation screen of camera 120, processing proceeds to step S509.

[0068] In step S508, setting information transmission unit 207 transmits to camera 120 information on the setting item that has been moved to the area of ​​the operation screen of camera 120. The setting item information includes a display item name and a setting value of the setting item. Note that the setting value may be a value that has been set in advance as the value of the setting item, or may be a value that the user has specified by voice or the like. Also, the setting value may be set by an actual operation on camera 120 after the object of the setting item has been moved to the area of ​​the operation screen.

[0069] In step S509, setting information transmission unit 207 does not transmit information about the setting item moved by the user operation to camera 120 because the object of the setting item does not exist within the area of ​​the operation screen of camera 120.

[0070] In step S510, the CPU 102 of the HMD 101 determines whether or not the user has completed the setting of the setting items. For example, when the user presses a button for instructing completion of the setting of the setting items, the CPU 102 can determine that the setting of the setting items has been completed. The button for instructing completion of the setting may be provided in the HMD 101, or the camera 120 may be provided. In addition, the CPU 102 may determine that the setting of the setting items has been completed when an operation for instructing completion of the setting of the setting items by a gesture or a voice is received from the user. Furthermore, the CPU 102 may determine that the setting of the setting items has been completed when the user performs an action of removing the HMD 101. When the setting of the setting items has been completed, the process shown in FIG. 5 ends. When the setting of the setting items has not been completed, the process returns to step S505.

[0071] Fig. 8 is a flowchart showing an example of the display position determination process executed by the display position determination unit 203 in step S503 in Fig. 5. The display position determination unit 203 determines the display position of the object of the setting item based on, for example, the position and size of the operation screen of the camera 120 within the viewing angle of the HMD 101 and the size of the object of the setting item to be superimposed and displayed.

[0072] In step S801, the display position determination unit 203 acquires the position of the operation screen of the camera 120 within the viewing angle of the HMD 101 worn by the user in two-dimensional coordinates (x, y). The display position determination unit 203 acquires, as the position of the operation screen, for example, the coordinates of the center position or the position of any of the four corners of the operation screen.

[0073] In step S802, the display position determination unit 203 acquires the size of the operation screen of the camera 120 within the viewing angle of the HMD 101 worn by the user. The display position determination unit 203 acquires, for example, the vertical and horizontal lengths of the operation screen as the size of the operation screen.

[0074] In step S803, the display position determination unit 203 acquires the size (display size) of the object of the setting item to be displayed in the virtual space. The size of the object of each setting item is a size set in advance, and may be different for each setting item or may be the same.

[0075] In step S804, the display position determination unit 203 determines whether each object can be placed outside the area of ​​the operation screen within the viewing angle based on the position of the operation screen, the size of the operation screen, and the size of the object of the setting item acquired in steps S801 to S803. That is, the display position determination unit 203 determines whether or not all of the objects of the setting items to be displayed can be placed within the viewing angle so as not to overlap with the area of ​​the operation screen. If all of the objects of the setting items can be placed outside the area of ​​the operation screen within the viewing angle, the process proceeds to step S805. If some of the objects of the setting items cannot be placed outside the area of ​​the operation screen within the viewing angle, the process proceeds to step S806.

[0076] In step S805, the display position determination unit 203 determines the display position of the object of the setting item within the viewing angle so as not to overlap with the area of ​​the operation screen. Also, the display position determination unit 203 determines the display position so that the objects of the setting items do not overlap with each other.

[0077] Furthermore, the display position determination unit 203 may determine the display position of the object of the setting item so that the user can recognize an operation on the camera 120 in real space. The display position determination unit 203 determines the display position of the object of the setting item to a position that does not obstruct the visibility of the operation on the camera 120. For example, the display position determination unit 203 determines the display position of the object of the setting item to a position that does not overlap with the user's hand in real space.

[0078] An example of a process for determining the display positions of objects of setting items will be described with reference to Fig. 9 and Fig. 10. Fig. 9 and Fig. 10 show an example in which the number and display positions of objects of setting items arranged in a virtual space are changed depending on the difference in the size of the operation screen 305 of the camera 120 within the viewing angle of the HMD 101.

[0079] 9 is a diagram showing a display example of objects of setting items in an image of a virtual space 901 when a user brings the camera 120 closer to the HMD 101. An operation screen 305 of the camera 120 in the image of the virtual space 901 is displayed larger within the viewing angle of the HMD 101 than in the example of FIG.

[0080] In the image of the virtual space 901, the area for displaying objects of the setting items is limited, so the display position determination unit 203 determines the setting items to be displayed within the viewing angle to be fewer setting items 902 to 909 than in the example of Fig. 10. When the user moves the camera 120 closer to the HMD 101, the display position determination unit 203 can determine the setting items to be displayed based on, for example, a preset priority.

[0081] The display position determination unit 203 may set a higher priority for a setting item that is used more frequently. The display position determination unit 203 may also set a priority such that an item related to a setting item operated by a user is preferentially displayed. An item related to a setting item may be another setting item having the same screen ID. Furthermore, the relationship between screen IDs may be defined in advance, and an item related to a setting item may be another setting item having the same screen ID and a setting item associated with a related screen ID.

[0082] The display position determination unit 203 determines the display position so that the objects of the setting items do not overlap each other. In addition, the display position determination unit 203 may determine the display position so that the higher the priority of the setting item, the closer the setting item is to the area of ​​the operation screen 305.

[0083] The user can move the objects of the setting items 902 to 909 into the area of ​​the operation screen 305 by using hand movements recognized by hand tracking. Text 910 is an explanation for the setting item 902, and is displayed following the display position of the object of the setting item 902. The text displayed in the virtual space 901 (the explanation for the setting item) ) may be displayed when the object of the setting item is displayed outside the area of ​​the operation screen 305, or may be displayed while the user is operating the setting item.

[0084] 10 is a diagram showing an example of a display of objects of setting items in an image of a virtual space 1001 when a user moves the camera 120 away from the HMD 101. An operation screen 305 of the camera 120 in the image of the virtual space 1001 is displayed smaller within the viewing angle of the HMD 101 than in the example of FIG.

[0085] In the image of the virtual space 1001, the area for displaying objects of the setting items is larger than in the example of Figure 9, so the display position determination unit 203 determines that the setting items to be displayed within the viewing angle are setting items 1002 to 1011 in addition to setting items 902 to 909.

[0086] The display sizes of objects of setting items displayed in the image of virtual space 901 in Fig. 9 and the image of virtual space 1001 in Fig. 10 may be different sizes for each setting item or may be the same size. Furthermore, the display position determination unit 203 may determine the size of the objects of the setting items based on the priority of each setting item.

[0087] The display position determination unit 203 may determine an object of a setting item to be displayed within the viewing angle of the virtual space based on the priority of each setting item. For example, the display position determination unit 203 determines a setting item having a priority higher than a threshold value as a display target.

[0088] Furthermore, the display position determination unit 203 may determine the display position of the object of each setting item based on the priority of each setting item. For example, the display position determination unit 203 determines the display position such that the object of a setting item with a higher priority is displayed at a position closer to the operation area of ​​the camera 120.

[0089] Furthermore, the display position determination unit 203 may determine the position in the depth direction of the object of the setting item based on the priority. That is, the display position determination unit 203 determines the display position such that the object of the setting item with a higher priority is displayed closer to the user (toward the HMD 101).

[0090] The priority of the setting item may be a preset priority, may be set according to the frequency of use of the setting item by the user, or may be set according to the relevance of the setting item to the setting item being operated by the user.

[0091] In step S806 of FIG. 8, the display position determination unit 203 determines whether the display size of the object of the setting item acquired in step S803 can be reduced. The display position determination unit 203 can determine whether the object can be reduced based on, for example, the visibility of the object or the readability of the characters included in the object. Furthermore, the display position determination unit 203 determines that the display size of the object of at least one of the multiple setting items is reducible if it is reducible. If the display size is not reducible, the process proceeds to step S807. If the display size is reducible, the process proceeds to step S808.

[0092] In step S807, the display position determination unit 203 determines the display positions of objects of some setting items to be outside the viewing angle. In this case, the display position determination unit 203 determines the setting items to be displayed within the viewing angle in descending order of priority of the setting items, and determines the display positions of objects of the remaining setting items to be outside the viewing angle.

[0093] In step S808, the display position determination unit 203 determines the display size of the object of the setting item. The display position determination unit 203 may reduce the display size of the objects of some of the setting items among the multiple setting items, and the process returns to step S804.

[0094] Fig. 11 is a flowchart showing an example of a setting process of the camera 120. The process shown in Fig. 11 is realized by the CPU 111 executing a program corresponding to the process of each step. The process in Fig. 11 is started by the HMD 101 transmitting information on setting items to the camera 120.

[0095] In step S1101, the setting information receiving unit 208 of the camera 120 receives the setting item information transmitted from the setting information transmitting unit 207 of the HMD 101 via the communication I / F 116. The setting item information includes information on the display item name, the setting value, and the display position on the operation screen.

[0096] In step S1102, the setting information receiving unit 208 updates the setting item information 210 for the received setting item. The setting information receiving unit 208 updates the setting value of the setting item information 210 to the setting value received from the HMD 101, and updates the display status to "Y".

[0097] In step S1103, setting information receiving unit 208 updates setting item information 210 for setting items that have not been received from setting information transmitting unit 207. Setting information receiving unit 208 updates the display status of setting item information 210 to "N" for setting items that have not been received.

[0098] In step S1104, the setting value reflecting unit 209 reflects the setting item information 210 to the camera 120. Specifically, the setting value reflecting unit 209 displays the setting items whose display status is set to "Y" in the setting item information 210 on the operation screen of the real camera 120 at the position to which the user has moved the object of the setting item. Furthermore, the setting value reflecting unit 209 reflects the setting values ​​of the setting items whose display status is set to "Y" in the setting item information 210 on the camera 120. On the other hand, the setting value reflecting unit 209 does not display the setting items whose display status is set to "N" in the setting item information 210 on the operation screen of the real camera 120.

[0099] In the above-described first embodiment, the HMD 101 changes the object of the setting item displayed in the virtual space according to the area of ​​the operation screen of the camera 120 in the virtual space. The object indicating the setting item of the camera 120 is displayed outside the area of ​​the operation screen, so that the user can easily find the desired setting item. Furthermore, when the user moves the object of the setting item into the area of ​​the operation screen, the setting for the setting item is reflected in the camera 120. The user can easily find the desired setting item and change the setting of the camera 120 with a simple operation.

[0100] Furthermore, when the user moves an object of a setting item to a desired position on the operation screen, the camera 120 also changes the display position of the setting item on the actual operation screen. The user can change the arrangement of the setting items on the operation screen by moving an object that is arranged outside the area of ​​the operation screen. This improves visibility when customizing the operation screen (setting the screen arrangement), and allows the user to customize the operation screen with simple operations.

[0101] In the above-described first embodiment, when an object of a setting item is moved from outside the area of ​​the operation screen to inside the area, the HMD 101 transmits information of the setting item to the camera 120. However, this is not limited to this. When an object of a setting item is moved from inside the area of ​​the operation screen to outside the area, the HMD 101 transmits information of the setting item corresponding to the object moved outside the area of ​​the operation screen. Information on the setting item may be transmitted to camera 120. In this case, the information on the setting item includes information that the setting of the setting item has been cancelled. Furthermore, camera 120 hides the setting item that has been moved outside the area of ​​the operation screen, and cancels the setting of the setting item.

[0102] <Embodiment 2> In the first embodiment, the operation screen of the camera 120 operated by the user is displayed at the same size on the display unit 109 of the HMD 101. In contrast to this, in the second embodiment, the operation screen is displayed in an enlarged manner. By displaying the operation screen in an enlarged manner, the visibility of the operation screen is improved, and the user can easily check the setting status of the camera 120.

[0103] Fig. 12 is a flowchart showing an example of processing of the HMD 101 according to embodiment 2. Processing that is the same as the processing in embodiment 1 shown in Fig. 5 is given the same reference numerals and detailed description thereof will be omitted.

[0104] The processes in steps S501 to S505 are the same as those in Fig. 5. In step S1201, the operation acquisition unit 205 determines whether or not an operation for enlarging the operation screen in step S505 (hereinafter also referred to as an enlargement operation) has been performed. If an enlargement operation has been performed, the process proceeds to step S1202. If an enlargement operation has not been performed, the process proceeds to step S506.

[0105] The enlarged display operation may be an operation on a virtual object for enlarging and displaying the operation screen. The enlarged display operation may also be an operation by voice. The enlarged display operation may also be an operation on a button of camera 120 to which a function of enlarging and displaying is assigned. The enlarged display operation may include an operation for specifying a magnification factor when enlarging and displaying the operation screen.

[0106] In step S1202, the image generating unit 204 generates an image in which the operation screen of the camera 120 is enlarged. Specifically, the image generating unit 204 generates a virtual operation screen enlarged at a magnification ratio specified by the user or a predetermined magnification ratio, and displays the virtual operation screen in a superimposed manner on the real image acquired by the imaging unit 108. The processes in steps S506 to S510 are the same as those in FIG. 5.

[0107] In the above-described second embodiment, the user can reflect settings for the camera 120 in a state in which the operation screen of the camera 120 in the virtual space is enlarged. Also, similar to the first embodiment, the HMD 101 can display objects of setting items in the virtual space according to the area of ​​the operation screen of the camera 120. Also, when the object of the setting item is moved to the area of ​​the operation screen of the camera 120 by a user operation, the setting for the setting item is reflected in the camera 120. The user can change the settings of the camera 120 with a simple operation, reducing the effort of searching for a desired setting item.

[0108] The above-described embodiments are all illustrative and not restrictive of the present invention, and the present invention can be implemented in various other modified and altered forms.

[0109] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0110] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A first acquisition means for acquiring information on setting items of an electronic device; A second acquisition means for acquiring information on an area of ​​an operation screen of the electronic device in a three-dimensional space that is within a user's field of view; a display control means for displaying an object indicating a setting item of the electronic device outside an area of ​​the operation screen in the three-dimensional space, and changing the object to be displayed according to an area of ​​the operation screen in the three-dimensional space; a transmission means for transmitting, to the electronic device, information on a setting item corresponding to the moved object in response to an operation by the user of moving the object to an area of ​​the operation screen; 13. An information processing device comprising: (Configuration 2) The display control means determines a display position of the object based on an area of ​​the operation screen and information on a setting item of the electronic device. 2. The information processing device according to configuration 1. (Configuration 3) The display control means determines a display position of the object based on a size and a position of an area of ​​the operation screen. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The display control means determines a display position of the object at a position that does not obstruct visual recognition of an operation on the electronic device. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The display control means determines a display position of the object so as not to overlap with an area of ​​the operation screen. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The display control means determines the object to be displayed based on a priority level preset for the setting item of the electronic device, a priority level according to a frequency of use of the setting item of the electronic device, or a priority level according to a relevance to the setting item being operated by the user. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The display control means determines the display position or size of the object based on a priority previously set for the setting item of the electronic device, a priority according to a frequency of use of the setting item of the electronic device, or a priority according to relevance to the setting item being operated by the user. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The display control means controls so as to enlarge and display the area of ​​the operation screen. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The user's operation includes at least one of hand movement, voice, gaze, and gesture. 9. The information processing device according to any one of configurations 1 to 8. (Configuration 10) An information processing device according to any one of configurations 1 to 9, a display means for displaying the object; A head-mounted display device comprising: (Configuration 11) A transmitter for transmitting information of the electronic device to the information processing device according to any one of configurations 1 to 9. Step by step, a receiving means for receiving, from the information processing device, information on a setting item corresponding to the object moved to an area of ​​the operation screen by an operation of the user; a setting means for reflecting settings based on the received information on the setting items in the electronic device; 1. An electronic device comprising: (Configuration 12) The electronic device according to configuration 11, characterized in that the electronic device is a camera, a smartphone, or a printer equipped with the operation screen. (method) acquiring information on setting items of the electronic device; acquiring information on an area of ​​an operation screen of the electronic device in a three-dimensional space that is within a user's field of view; displaying an object indicating a setting item of the electronic device outside an area of ​​the operation screen in the three-dimensional space, and changing the object to be displayed according to an area of ​​the operation screen in the three-dimensional space; transmitting, to the electronic device, information on a setting item corresponding to the moved object in response to an operation by the user of moving the object to an area of ​​the operation screen; 13. An information processing method comprising: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 9. [Explanation of symbols]

[0111] 101: HMD (information processing device), 102: CPU, 107: communication I / F, 109: display unit, 201: setting information acquisition unit, 202: screen area acquisition unit, 203: display position determination unit, 204: image generation unit, 205: operation acquisition unit, 206: item position acquisition unit, 207: setting information transmission unit 120: Camera (electronic device), 210: Setting item information

Claims

1. An information processing device included in a head-mounted display device (HMD) worn by a user, A first acquisition means for acquiring information on the settings of an electronic device, A second acquisition means for acquiring information on the area of ​​the operation screen of the electronic device in the three-dimensional space that constitutes the field of view of the user wearing the HMD, A display control means that displays objects indicating the setting items of the electronic device outside the area of ​​the operation screen in the three-dimensional space, and changes the displayed objects according to the area of ​​the operation screen in the three-dimensional space, A transmission means transmits information of setting items corresponding to the object to the electronic device in response to the user's operation to move the object, which is displayed outside the area of ​​the operation screen, to the area of ​​the operation screen in the three-dimensional space. An information processing device characterized by having the following features.

2. The display control means determines the display position of the object based on the area of ​​the operation screen and the information of the setting items of the electronic device. The information processing apparatus according to feature 1.

3. The display control means determines the display position of the object based on the size and position of the area on the operation screen. The information processing apparatus according to feature 1.

4. The display control means determines the display position of the object at a location that does not obstruct the view of the operation of the electronic device. The information processing apparatus according to feature 1.

5. The display control means determines the display position of the object so as not to overlap with the area of ​​the operation screen. The information processing apparatus according to feature 1.

6. The display control means determines the priority of the setting items of the electronic device based on a preset priority, the frequency of use of the setting items of the electronic device, or the setting item being operated by the user. Based on the priority given according to their relevance, the objects to be displayed are determined. The information processing apparatus according to feature 1.

7. The display control means determines the display position or size of the object based on a predetermined priority for the setting items of the electronic device, a priority according to the frequency of use of the setting items of the electronic device, or a priority according to the relevance to the setting item being operated by the user. The information processing apparatus according to feature 1.

8. The display control means controls the operation screen to be displayed in an enlarged area. The information processing apparatus according to feature 1.

9. The user's actions include at least one of the following: hand movements, voice, gaze, and gestures. The information processing apparatus according to feature 1.

10. The second acquisition means acquires information indicating the position and size of the area of ​​the operation screen of the electronic device in the three-dimensional space. The information processing apparatus according to feature 1.

11. The invention further comprises a determination means for determining whether the display position of the object after it has been moved by the user's operation is included within the area of ​​the operation screen, The information processing apparatus according to claim 1, characterized in that the transmission means transmits information of setting items corresponding to the object to the electronic device when it is determined that the display position of the object is included within the area of ​​the operation screen.

12. Information of a setting item corresponding to the object transmitted by the transmission means includes at least one of the setting value of the setting item corresponding to the object and the display position of the setting item in the area of ​​the operation screen. The information processing apparatus according to feature 1.

13. An information processing device according to any one of claims 1 to 12, Display means for displaying the object and A head-mounted display device characterized by having the following features.

14. A transmission means for transmitting information of the electronic device to the information processing device according to any one of claims 1 to 12, A receiving means for receiving information from the information processing device about setting items corresponding to the object moved to the area of ​​the operation screen by the user's operation, Setting means for reflecting the settings based on the received setting item information to the electronic device. An electronic device characterized by having the following features.

15. The electronic device according to claim 14, characterized in that the electronic device is a camera, smartphone, or printer equipped with the operation screen.

16. An information processing method for controlling an information processing device included in a head-mounted display device (HMD) worn by a user, Steps to obtain information on the settings of an electronic device, The steps include acquiring information on the area of ​​the operating screen of the electronic device in a three-dimensional space that constitutes the field of view of the user wearing the HMD, The steps include: displaying objects indicating the setting items of the electronic device outside the area of ​​the operation screen in the three-dimensional space, and changing the displayed objects according to the area of ​​the operation screen in the three-dimensional space; The steps include: transmitting information of setting items corresponding to the object to the electronic device in response to a user operation that moves the object, which is displayed outside the area of ​​the operation screen, to the area of ​​the operation screen in the three-dimensional space; An information processing method characterized by having the following features.

17. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 12.

Images