Information processing apparatus, head-mounted display apparatus, electronic apparatus, information processing method, computer program product, and computer-readable storage medium

Abstract

The invention relates to an information processing apparatus, a head-mounted display apparatus, an electronic apparatus, an information processing method, a computer program product, and a computer readable storage medium. The information processing apparatus includes: a first acquisition unit configured to acquire information related to a setting item of an electronic apparatus; a second acquisition unit configured to acquire information related to an operation screen area of the electronic device in a three-dimensional space representing a field of view seen by the user; a display control unit configured to display an object representing a setting item of the electronic device outside an operation screen area in the three-dimensional space, and change the object being displayed according to the operation screen area in the three-dimensional space; and a transmission unit configured to transmit, to the electronic device, information related to the setting item corresponding to the moved object in response to an operation by a user that moves the object into the operation screen area.

Description

Technical Field

[0001] The present invention relates to an information processing device, a head mounted display device, an electronic device, an information processing method, a computer program product and a computer readable storage medium. Background Art

[0002] With the sophistication of electronic devices (such as smartphones and mirrorless cameras, etc.) in recent years, the number of configurable setting items has increased, and the configuration operation of the graphical user interface (GUI) of the electronic device has become more complicated. A user who wishes to change display items according to purpose or preference can customize the screen by operating the electronic device itself or by a personal computer (PC) connected to the electronic device.

[0003] On the other hand, mixed reality (MR) technology has gained popularity in recent years. For example, a head-mounted display (HMD) uses MR technology to present a computer-generated virtual image on a real image captured by a camera to a user wearing the HMD. Therefore, the HMD can provide a larger amount of information than the real space image.

[0004] For example, Japanese Patent Laid-Open No. 2018-180776 describes a method by which virtual item buttons related to a selected job are displayed in a transmissive display unit of an HMD so that these virtual item buttons can be operated.

[0005] However, in situations where a user wishes to change settings for items other than the particular job selected, locating the desired item in the complex hierarchical structure is sometimes challenging. Summary of the invention

[0006] The present invention provides a technology that enables a user to easily locate a desired setting item of an electronic device and to change the setting of the electronic device with a simple operation.

[0007] According to the present invention, the information processing device includes: a first acquisition unit, which is configured to acquire information related to the setting items of the electronic device; a second acquisition unit, which is configured to acquire information related to the operation screen area of ​​the electronic device in the three-dimensional space representing the field of view seen by the user; a display control unit, which is configured to display the object representing the setting items of the electronic device outside the operation screen area in the three-dimensional space, and change the object being displayed according to the operation screen area in the three-dimensional space; and a sending unit, which is configured to send information related to the setting item corresponding to the moved object to the electronic device in response to the user's operation of moving the object to the operation screen area.

[0008] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A is a diagram illustrating a hardware configuration example of an HMD;

[0010] Figure 1B is a diagram illustrating an example of a hardware configuration of a camera;

[0011] Figure 2 is a block diagram illustrating a functional configuration example of an HMD and a camera;

[0012] Figure 3 is a diagram illustrating an example of a virtual space generated by an HMD;

[0013] Figure 4 is a diagram illustrating an example of setting item information;

[0014] Figure 5 is a flowchart illustrating an example of processing in the HMD according to Embodiment 1;

[0015] Figure 6 is a diagram illustrating an example of a user operation in a virtual space;

[0016] Figure 7 is a diagram illustrating another example of user operations in a virtual space;

[0017] Figure 8 is a flowchart illustrating an example of a process for determining a display position of a setting item;

[0018] Fig. 9 is a diagram illustrating a display example of setting items in a virtual space;

[0019] Fig.10 is a diagram illustrating another display example of setting items in a virtual space;

[0020] Fig.11 is a flowchart illustrating an example of setting processing of a camera; and

[0021] Fig.12 is a flowchart illustrating an example of processing in the HMD according to Embodiment 2. DETAILED DESCRIPTION

[0022] Embodiments according to the present invention will be described below with reference to the accompanying drawings. The present invention is not limited to the following embodiments. According to the specifications of the equipment to which the present invention is applied or according to various conditions (such as conditions and environment of use, etc.), the configuration of the embodiments can be appropriately modified or changed. The present invention can be implemented by combining parts of various embodiments. In the description of the embodiments, the same elements are given the same reference numerals.

[0023] Example 1

[0024] Figure 1A is a diagram illustrating a hardware configuration example of the HMD 101 as an information processing device. Figure 1B is a diagram illustrating a hardware configuration example of the camera 120 as an electronic device. Figure 2 1 is a block diagram illustrating a functional configuration example of the HMD 101 and the camera 120 . Figure 3 is a diagram illustrating an example of a virtual space generated by the HMD 101. Although the information processing apparatus is the HMD 101 (or is included in the HMD 101) in the examples described below, the information processing apparatus may be configured as other devices connected to the HMD 101.

[0025] Reference below Figure 3 An example is described in which objects (virtual objects) representing setting items of the camera 120 are displayed in a virtual space to allow the user to easily locate a desired item and change the setting. Figure 3 1 shows an example of a virtual space generated by the HMD 101. The user can change the setting of the reality camera 120 through the HMD 101 worn by the user.

[0026] Figure 3 A virtual space 302 is shown displayed in the display unit 109 of the HMD 101 worn by the user 301. The display unit 109 shows the camera 120, the left hand 303 of the user 301, and the right hand 304 of the user 301. The camera 120 has an operation screen 305. Figure 3 An example is shown in which a user 301 holds the camera 120 with a left hand 303 and operates an object placed in a virtual space 302 with a right hand 304 .

[0027] The object placed in the virtual space 302 is a virtual object generated by computer graphics (CG) technology, and represents a setting item 306 that can be displayed in the operation screen 305 of the camera 120. The display position of the object is determined in such a manner that the object does not overlap a specific area such as the operation screen 305, for example.

[0028] The HMD 101 is not limited to a helmet type, and may be, for example, a glasses type. The HMD 101 may be a video see-through type HMD or an optical see-through type HMD. Hereinafter, the HMD 101 will be described as a head-mounted display device of a video see-through type. In the case where the HMD 101 is an optical see-through type HMD, the real space seen by the user is not an image captured by the camera unit 108, but an actual field of view seen through the lens of the HMD 101.

[0029] The HMD 101 determines the display position of the object corresponding to each setting item based on the size and position of the operation screen 305 displayed in the display unit 109, for example. The user 301 can move the object and change the position of the object by a predetermined operation. When the user 301 moves the object by a predetermined operation, the HMD 101 transmits information about the setting item corresponding to the moved object to the camera 120.

[0030] The setting item information may include the setting value, and in addition may include information about the position after the movement in the operation screen 305. The user can move the object representing the setting item to a desired position in the operation screen 305. Based on the information received from the HMD 101, the camera 120 also displays the setting item at the position designated by the user in the actual operation screen 305. Therefore, the user can easily customize the screen of the camera 120.

[0031] The predetermined operation may be any action that enables the object representing the setting item to be moved, such as dragging and dropping or tapping the position of the object before and after the movement, etc. The predetermined operation may also include a voice operation.

[0032] The HMD 101 displays in its display unit 109 not only objects representing setting items associated with the screen configuration currently shown in the display unit 117 of the camera 120, but also objects representing setting items not associated with the current display. The user 301 can easily locate the settings of the camera 120 without having to search for a desired setting item in a complicated hierarchical structure, which allows the user to change the settings with a simple operation. The electronic device whose settings are to be changed is not limited to the camera 120, but also includes various devices having an operation screen such as a smartphone and a printer.

[0033] Reference Figure 1A and Figure 1B The hardware configuration of the HMD 101 and the camera 120 will be described. Figure 1A109. The HMD 101 generates an image of an MR space and displays the generated image in the display unit 109. The image is obtained by displaying a virtual object generated by CG on an image of a real space captured by a camera (camera unit 108) installed in the HMD 101. The HMD 101 can also display an image of a VR space generated by placing a virtual object in a virtual space generated by CG in the display unit 109. MR and VR are well-known technologies, and therefore will not be described in detail.

[0034] The CPU 102 as a system controller controls the overall operation of the HMD 101. The CPU 102 executes a program to realize various types of processing performed by the HMD 101. The ROM 103 is a read-only memory for storing programs and parameters that remain unchanged, such as a basic program and initial data. The RAM 104 is a memory for temporarily storing operation results and input information during information processing or image processing.

[0035] The recording unit 105 is a device capable of writing and reading various types of information, and includes a hard disk and a memory card built in or external to the HMD 101. The recording unit 105 may also include a memory card, a floppy disk, an IC card, and the like that can be attached to and removed from the HMD 101.

[0036] The recording unit 105 records various programs for realizing processing in the HMD 101. The CPU 102 can realize processing in the HMD 101 by reading out the program from the recording unit 105 and expanding and executing the program in the RAM 104. The program executed by the CPU 102 may be stored in the ROM 103.

[0037] The recording unit 105 can record information about a user operation input via the input detection unit 106 and information about the position and posture of the user detected by an acceleration sensor or the like equipped in the HMD 101. The recording unit 105 records setting item information of the camera 120.

[0038] The input detection unit 106 receives input from the user via the operation unit 110. The input via the operation unit 110 includes, for example, input of hand movement recognized by using hand tracking.

[0039] Reference numeral 107 denotes a communication interface (I / F) capable of transmitting and receiving data to and from an operating device or a cloud for operating the HMD 101. In the case where the information processing device according to the present invention is configured separately from the HMD, the communication I / F 107 receives audio data acquired by a microphone in the HMD, position and posture information detected by the HMD, and an image captured by the HMD via a network. Connections for input and output via the communication I / F 107 include those via USB or Bluetooth. A local connection and an Internet connection via Ethernet or Wi-Fi.

[0040] The camera unit 108 is a camera device installed in the HMD 101. The real-time action video captured by the camera unit 108 is used to track the user's hand or joint movement through image recognition processing, or to detect a plane for superimposing a virtual object such as a button or icon. The image captured by the camera unit 108 is also used to recognize the operation screen of the camera 120.

[0041] The display unit 109 is an electronic display device such as a liquid crystal display device, etc. The display unit 109 receives a user instruction recognized by hand tracking, and displays an image such as a VR world based on the instruction from the user and the information stored in the recording unit 105.

[0042] The operation unit 110 includes an operation device capable of inputting pointing operations and various commands, etc., by hand movement, that is, the operation unit 110 obtains operation instructions and commands from the user. Specifically, the operation unit 110 obtains information input by hand movement when the user operates an object representing a setting item in a virtual space. Information related to the operation performed by the user on the object is sent to the input detection unit 106. The function of the operation unit 110 is not limited to obtaining information input by hand movement recognized by hand tracking. The operation unit may have a function of obtaining information input via a controller including a mouse or a keyboard, and voice input information.

[0043] Figure 1A An example of a configuration in which the information processing device according to the present invention is integrated with the HMD 101 is illustrated, and thus the HMD 101 includes the operation unit 110 and the display unit 109 as an internal configuration. In the case where the information processing device according to the present invention is constituted by a PC or the like separate from the HMD 101, the information processing device does not need to include the operation unit 110 and the display unit 109. In this case, the information processing device acquires information related to the user operation from the HMD as an external device, and transmits the data of the generated virtual space to the HMD.

[0044] Figure 1B1 is a diagram illustrating an example of a hardware configuration of a camera 120. A CPU 111 as a system controller controls the overall operation of the camera 120. The CPU 111 executes a program to realize various types of processing performed by the camera 120. A ROM 112 is a read-only memory for storing programs and parameters that remain unchanged, such as a basic program and initial data. A RAM 113 is a memory for temporarily storing operation results and input information during information processing or image processing.

[0045] The recording unit 114 is a device capable of writing and reading various types of information, and includes a hard disk and a memory card built into the camera 120 or external to the camera 120. The recording unit 114 may also include a memory card, a floppy disk, an IC card, and the like that can be attached to and removed from the camera 120.

[0046] The recording unit 114 records various programs for realizing the processing in the camera 120. The CPU 111 can realize the processing in the camera 120 by reading out the program from the recording unit 114 and expanding and executing the program in the RAM 113. The program executed by the CPU 111 can be stored in the ROM 112. The recording unit 114 records data used in the program executed by the CPU 111.

[0047] The input detection unit 115 receives input from the user via the operation unit 118. The input via the operation unit 118 includes, for example, input by operating a touch screen or a button.

[0048] The communication I / F 116 is a communication interface capable of transmitting and receiving data to and from the HMD 101 and the cloud, etc. Connections for input and output via the communication I / F 116 include local connections via USB or Bluetooth, and Internet connections via Ethernet or Wi-Fi. The HMD 101 and the camera 120 communicate with each other via the communication I / F 107 and the communication I / F 116.

[0049] The display unit 117 is an electronic display device such as a liquid crystal display device, etc. The display unit 117 receives a user instruction given by a touch action on the display. The display unit 117 displays a GUI of an operation screen based on the instruction from the user and the information stored in the recording unit 114.

[0050] The operation unit 118 obtains an operation instruction given by the user to the camera 120, such as by operation of a button or a touch screen. The operation unit 118 obtains information input by the user operating the camera 120 to change the setting of some setting items. Information related to the user operation on the camera 120 is sent to the input detection unit 115 and the HMD 101.

[0051] Figure 2 1 is a block diagram illustrating a functional configuration example of the HMD 101 and the camera 120. The HMD 101 includes a setting information acquisition unit 201, a screen area acquisition unit 202, a display position determination unit 203, an image generation unit 204, an operation acquisition unit 205, an item position acquisition unit 206, and a setting information transmission unit 207. The camera 120 includes a setting information reception unit 208 and a setting value reflection unit 209. The camera 120 holds the setting item information 210 in the recording unit 114.

[0052] The setting information acquisition unit 201 acquires setting item information 210 held in the camera 120 as an operation target. Figure 4 An example of the setting item information 210 is shown. The setting item information 210 includes the model of the camera 120, the screen ID, the setting item, the setting value, and the display status. The display status refers to a flag indicating whether an item is displayed in the operation screen of the camera 120 (Y: displayed, N: not displayed).

[0053] The setting information acquisition unit 201 can acquire not only the setting item information 210 held in the camera 120 but also the setting item information of the camera 120 pre-stored in the recording unit 105. The recording unit 105 can hold setting item information for each model of various cameras 120. The setting information acquisition unit 201 can obtain the model information from the camera 120 and then acquire the setting item information corresponding to the camera 120 from the recording unit 105.

[0054] The screen area acquisition unit 202 acquires information related to the operation screen area of ​​the camera 120 operated by the user. Specifically, the screen area acquisition unit 202 uses image recognition or other known technologies to acquire information indicating the position and size of the operation screen area of ​​the camera 120. The position of the operation screen area can be obtained not only by image recognition but also by other known technologies such as Simultaneous Localization and Mapping (SLAM).

[0055] The display position determination unit 203 performs display control to change the object representing the setting item being displayed according to the operation screen area in the virtual space. That is, the display position determination unit 203 determines which object representing the setting item is to be displayed in the virtual space.

[0056] The display position determination unit 203 determines the display position of the object for each setting item. The display position determination unit 203 determines the display position of the object representing the setting item based on the position and size of the operation screen area of ​​the camera 120 being displayed in the display unit 109 of the HMD 101 worn by the user, for example.

[0057] The object representing the setting item displayed in the display unit 109 is, for example, an icon for indicating the setting item. The objects representing the setting item may each be associated with information related to the setting value of each setting item. Alternatively, the object representing the setting item may include an object for indicating a description of the setting item.

[0058] The image generation unit 204 generates an image of a virtual space in which an object representing a setting item is superimposed at the three-dimensional position determined by the display position determination unit 203 on the real space image captured by the imaging unit 108 of the HMD 101 .

[0059] The operation acquisition unit 205 acquires a user operation performed on an object representing a setting item placed in a virtual space. The image generation unit 204 generates an image after the object representing the setting item has been moved to another display position based on the user operation acquired by the operation acquisition unit 205. The user can move the object representing the setting item to a desired position in the operation screen.

[0060] The item position acquisition unit 206 obtains the display position of the object representing the setting item after being moved. The setting information transmission unit 207 determines whether the display position of the object representing the setting item after being moved is included in the operation screen area of ​​the camera 120. If the display position of the object is included in the operation screen area of ​​the camera 120, the item position acquisition unit 206 transmits information about the setting item corresponding to the object to the camera 120.

[0061] The setting information receiving unit 208 of the camera 120 receives information about the setting items corresponding to the object moved to the operation screen area from the setting information transmitting unit 207 of the HMD 101. The received information about the setting items (such as setting values ​​and display conditions) is stored as setting item information 210.

[0062] The setting value reflecting unit 209 reflects the information about the setting item received from the setting information receiving unit 208 in the setting of the camera 120. The setting value reflecting unit 209 changes the setting of the camera 120 based on the setting value included in the setting item information. The setting value reflecting unit 209 also displays the setting item at the position to which the object is moved by the user operation in the real operation screen of the camera 120.

[0063] Figure 5 : is a flowchart illustrating an example of processing in the HMD 101 according to Embodiment 1. Figure 5 The processing shown in is realized by CPU 102 executing a program corresponding to each step of the processing. Figure 5 The processing can be executed at any timing. Figure 5 The processing may be executed, for example, at specified time intervals or when a user operation is detected.

[0064] At step S501, the setting information acquisition unit 201 of the HMD 101 acquires information about setting items that can be displayed in the operation screen 305 of the camera 120 from the setting item information 210. The setting information acquisition unit 201 records the acquired setting item information in the recording unit 105. By once recording the setting item information acquired from the camera 120 in the recording unit 105, the setting information acquisition unit 201 can obtain the setting item information stored in the recording unit 105 from the next processing session. In the case where the setting item information for each model of various cameras 120 has been stored in the recording unit 105, the setting information acquisition unit 201 can obtain the information of the model from the camera 120 and then acquire the corresponding setting item information from the recording unit 105. The camera 120 only needs to send its model information instead of the setting item information to the HMD 101.

[0065] At step S502, the screen area acquisition unit 202 acquires information about the operation screen area of ​​the camera 120 in the virtual space (three-dimensional space) that will be the field of view seen by the user. Specifically, the screen area acquisition unit 202 acquires information indicating the position and size of the operation screen area. The screen area acquisition unit 202 can acquire information about the operation screen area by detecting the operation screen of the camera 120 based on the image captured by the imaging unit 108, for example.

[0066] At step S503, the display position determination unit 203 determines the display position of the object representing the setting item based on the setting item information acquired at step S501 and the information related to the operation screen area of ​​the camera 120 acquired at step S502. The display position determination unit 203 determines the display position of the object, for example, in such a manner that the object representing the setting item will be placed outside the operation screen area. Here, the display position determination unit 203 may determine the display position in such a manner that the object corresponding to the item currently being displayed in the operation screen of the camera 120 is displayed in a superimposed manner within the area of ​​the operation screen 305. This will be referred to later. Figure 6 The flowchart of FIG. 1 describes the display position determination processing performed by the display position determination unit 203 in more detail.

[0067] At step S504 , the image generation unit 204 generates an image in which the object representing the setting item is superimposed at the display position determined at step S503 on the real space image captured by the imaging unit 108 .

[0068] The image generation unit 204 may obtain an image captured by the imaging unit 108, or may obtain an image being displayed in the display unit 117 of the camera 120 from the camera 120 via the communication I / F 116 and the communication I / F 107. The image of the operation screen may be obtained from an image pre-stored in the recording unit 105 of the HMD 101. In this case, the HMD 101 may obtain the screen ID information of the screen currently displayed in the operation screen from the camera 120, thereby obtaining the image of the corresponding operation screen from the recording unit 105. The image generation unit 204 only needs to combine the obtained image of the operation screen with the operation screen area in the real space image captured by the imaging unit 108.

[0069] At step S505, the operation acquisition unit 205 obtains a user operation performed on an object representing a setting item in the virtual space. The user operation is, for example, but not limited to, a hand movement identified by hand tracking. The user operation may be an eye gaze or a voice operation. The user operation may also involve an appropriate combination of hand movement, eye gaze operation, and voice operation.

[0070] Reference Figure 6 and Figure 7 The process of acquiring the user operation at step S505 is described. The operation acquisition unit 205 can acquire a user operation such as a hand movement recognized by hand tracking to move an object representing a setting item. Figure 6 and Figure 7 is a diagram illustrating an example of a user operation in a virtual space. Figure 6 and Figure 7 In the description of , an object representing a setting item is simply referred to as a “setting item”.

[0071] Figure 6 An example of a user operation as a hand movement for moving one of the setting items is shown. An image of a virtual space 601 shows the camera 120, a plurality of setting items, text describing the setting items, and a user's hand. A hand 604 indicates a user's hand before moving a setting item 602 (flash off). A hand 605 indicates a user's hand after moving the setting item 602 to the operation screen area. Reference numeral 603 indicates text for describing the setting item 602. For example, when the user's hand is placed on the setting item, the text 603 may be displayed for a predetermined time.

[0072] The user selects a setting item 602 displayed in the virtual space 601 with a hand 604 and moves the setting item 602 to a desired position within the area of ​​the operation screen 305 of the camera 120. The setting item 602 is placed at the position to which the user moved it (at the position indicated by 606).

[0073] The setting information transmitting unit 207 of the HMD 101 transmits the setting item information corresponding to the setting item 602 to the camera 120 in response to the user operation of moving the setting item 602 to the area of ​​the operation screen 305. The setting information receiving unit 208 of the camera 120 receives the information about the setting item 602 from the setting information transmitting unit 207. The setting value reflecting unit 209 reflects the setting of the setting item 602 in the camera 120 in real time based on the information received by the setting information receiving unit 208. That is, the setting value reflecting unit 209 displays the setting item 602 at the position 606 to which the setting item 602 is moved by the user operation in the actual operation screen 305 of the camera 120. The setting value reflecting unit 209 applies the “flicker off” setting to the camera 120.

[0074] In the case where the setting item 606 after the movement is displayed in the area of ​​the operation screen 305, the image generation unit 204 of the HMD 101 stops displaying the setting item 602 and the text 603 shown outside the area of ​​the operation screen 305. Since the area of ​​the operation screen 305 has a limited space, the image generation unit 204 performs control so that the description of the setting item is not shown in the area of ​​the operation screen 305.

[0075] Figure 7 An example of a user operation that is a combination of a voice (of the user 301) and a gesture that surrounds and moves a plurality of setting items is shown. An image of a virtual space 701 shows the camera 120, a plurality of setting items, text describing the setting items, and the hand of the user 301. A hand 705 represents the hand of the user 301 before moving the setting items 602, 702 to 704. A hand 709 represents the hand of the user after moving the setting items 602, 702 to 704 to the area of ​​the operation screen 305. The user 301 utters a voice 706 of "select multiple items" and makes a gesture with the hand 705 that surrounds the setting items 602, 702 to 704 displayed in the virtual space 701. Thus, the plurality of setting items can be selected.

[0076] In the image of the virtual space 701, the selected plurality of setting items may be highlighted by a change in display style such as lighting, or a trajectory 707 of movement of a hand 705 around the plurality of setting items may be displayed. After selecting the plurality of setting items 602, 702 to 704, the user moves the setting items to desired positions in the operation screen 305 of the camera 120. The setting items 602, 702 to 704 are placed at positions 708 to which the user moved the setting items.

[0077] When the setting items 602, 702 to 704 are moved to the area of ​​the operation screen 305, the setting information receiving unit 208 of the camera 120 receives information about the setting items 602, 702 to 704 from the setting information transmitting unit 207 of the HMD 101. The setting value reflecting unit 209 reflects the settings of the setting items 602, 702 to 704 in the camera 120 in real time based on the information received by the setting information receiving unit 208. That is, the setting value reflecting unit 209 displays the setting items 602, 702 to 704 at the position 708 to which the setting items 602, 702 to 704 are moved by the user operation in the actual operation screen 305 of the camera 120. The setting value reflecting unit 209 applies each setting of the setting items to the camera 120.

[0078] Although Figure 7 An example of an operation of moving a setting item by a combination of a hand movement operation and a voice is illustrated, but the user operation is not limited to this example. The user operation may be an eye gaze input for selecting a setting item. The user operation may be a gesture for selecting a setting item or for changing the operation content.

[0079] exist Figure 5 At step S506 , the item position acquisition unit 206 acquires the display position of the object representing the setting item moved by the user operation at step S505 .

[0080] At step S507, the item position acquisition unit 206 determines whether the object representing the setting item moved by the user operation exists in the operation screen area of ​​the camera 120. In a case where the object representing the moved setting item exists in the operation screen area of ​​the camera 120, the process proceeds to step S508. In a case where the object representing the moved setting item does not exist in the operation screen area of ​​the camera 120, the process proceeds to step S509.

[0081] At step S508, the setting information transmission unit 207 transmits information related to the setting item moved to the operation screen area of ​​the camera 120 to the camera 120. The setting item information includes a display name and a setting value of the setting item. The setting value may be a preset value of the setting item, or may be a value specified by a user voice. Alternatively, after moving an object representing the setting item to the operation screen area, the setting value may be set by an actual action performed on the camera 120.

[0082] At step S509 , the setting information transmission unit 207 does not transmit information related to the setting item to the camera because the object representing the setting item moved by the user operation does not exist in the operation screen area of ​​the camera 120 .

[0083] At step S510, the CPU 102 of the HMD 101 determines whether the user has completed the configuration of the setting items. For example, the CPU 102 may determine that the configuration of the setting items is completed when the user presses a button for indicating that the configuration of the setting items is completed. The button for indicating that the configuration is completed may be mounted on the HMD 101 or on the camera 120. The CPU 102 may determine that the configuration of the setting items is completed when a user operation such as a gesture or voice indicating that the configuration of the setting items is completed is received. The CPU 102 may determine that the configuration of the setting items is completed when the user takes off the HMD 101. When the configuration of the setting items is completed, Figure 5 The processing shown ends. If the setting item configuration is not completed, the processing returns to step S505.

[0084] Figure 8 is exemplified in Figure 5 Flowchart of an example of a process for determining a display position performed by the display position determination unit 203 at step S503 of the HMD 101. The display position determination unit 203 determines the display position of an object representing a setting item based on, for example, the position and size of the operation screen of the camera 120 within the field of view of the HMD 101 and the size of the object representing the setting item to be superimposed.

[0085] At step S801, the display position determination unit 203 obtains two-dimensional coordinates (x, y) indicating the position of the operation screen of the camera 120 in the field of view of the HMD 101 worn by the user. The display position determination unit 203 obtains, for example, the coordinates of the center of the operation screen or the coordinates of one of the four corner vertices as the position of the operation screen.

[0086] At step S802, the display position determination unit 203 obtains the size of the operation screen of the camera 120 in the field of view of the HMD 101 worn by the user. The display position determination unit 203 obtains, for example, the vertical length and horizontal length of the operation screen as the size of the operation screen.

[0087] At step S803, the display position determination unit 203 obtains the size (display size) of the object representing the setting item to be displayed in the virtual space. The objects representing the setting items have respective predetermined sizes, which may be different from each other or may be the same.

[0088] At step S804, the display position determination unit 203 determines whether each object can be placed outside the operation screen area in the field of view based on the position and size of the operation screen obtained at steps S801 to S803 and the size of the object representing the setting item. That is, the display position determination unit 203 determines whether the objects representing the setting items to be displayed can all be placed in the field of view without overlapping the operation screen area. In the case where the objects representing the setting items can all be placed outside the operation screen area in the field of view, the process proceeds to step S805. In the case where some of the objects representing the setting items cannot be placed outside the operation screen area in the field of view, the process proceeds to step S806.

[0089] At step S805, the display position determination unit 203 determines the display position of the objects representing the setting items in the field of view so as not to overlap with the operation screen area. The display position determination unit 203 determines the display position so that the objects representing the setting items do not overlap with each other.

[0090] Alternatively, the display position determination unit 203 may determine the display position of the object representing the setting item in a manner that allows the user to perceive the operation performed on the camera 120 in the real space. The display position determination unit 203 determines the display position of the object representing the setting item in a manner that does not hinder the visibility of the operation performed on the camera 120. For example, the display position determination unit 203 determines the display position of the object representing the setting item in a manner that does not overlap with the user's hand in the real space.

[0091] Reference Fig. 9 and Fig.10 An example of a process for determining a display position of an object representing a setting item is described. Fig. 9 and Fig.10 An example is shown in which the number and display positions of objects representing setting items placed in the virtual space are changed according to the size of the operation screen 305 of the in-field camera 120 of the HMD 101 .

[0092] Fig. 91 is a diagram illustrating a display example of an object representing a setting item in an image of a virtual space 901 when the user brings the camera 120 closer to the HMD 101. The operation screen 305 of the camera 120 in the image of the virtual space 901 is displayed in a manner similar to that in FIG. Fig.10 The sizes shown in the example of are larger than the sizes shown in the field of view of the HMD 101.

[0093] In the image of the virtual space 901, the area for displaying the object representing the setting item is limited. Therefore, the display position determination unit 203 Fig.10 The setting items 902 to 909 with a smaller number than the setting items in the example of are determined as the setting items to be displayed within the field of view. When the user brings the camera 120 closer to the HMD 101, the display position determination unit 203 may determine which setting items to display based on a preset priority, for example.

[0094] The display position determination unit 203 may give a higher priority to the setting items that are used more frequently. Alternatively, the display position determination unit 203 may set the priority so that the items related to the setting items that the user is operating will be displayed with a higher priority. The items related to the setting items here may be other setting items in the screen with the same screen ID. Alternatively, the correlation between the screen IDs may be predefined, in which case the items related to the setting items here may be other setting items in the screen with the same screen ID, or may be corresponding setting items of the screen with the related screen ID.

[0095] The display position determination unit 203 determines the display position so that objects representing setting items do not overlap each other. The display position determination unit 203 may also determine the display position so that a setting item having a higher priority is placed closer to an area of ​​the operation screen 305 .

[0096] The user can move the objects representing the setting items 902 to 909 to the area of ​​the operation screen 305 by using the hand movement recognized by the hand tracking. Reference numeral 910 denotes text as a description of the setting item 902. The text is displayed so as to follow the display position of the object representing the setting item 902. The text (description of the setting item) displayed in the virtual space 901 may appear in a case where the object representing the setting item is being displayed outside the area of ​​the operation screen 305, or may appear during a period when the user is operating the setting item.

[0097] Fig.101001 is a diagram illustrating a display example of an object representing a setting item in an image of a virtual space 1001 in a case where the user moves the camera 120 away from the HMD 101. The operation screen 305 of the camera 120 in the image of the virtual space 1001 is displayed as Fig. 9 A smaller size than the size shown in the example is displayed in the field of view of the HMD 101.

[0098] In the image of the virtual space 1001, the area for displaying objects representing setting items is relatively small. Fig. 9 Therefore, the display position determination unit 203 selects setting items 1002 to 1011 in addition to setting items 902 to 909 as setting items to be displayed within the field of view.

[0099] exist Fig. 9 The objects representing the setting items and the Fig.10 The objects representing the setting items displayed in the image of the virtual space 1001 may each have a different size, or may all have the same size. Alternatively, the display position determination unit 203 may determine the size of the object for each setting item based on the priority of each setting item.

[0100] The display position determination unit 203 may select an object representing a setting item to be displayed in the field of view of the virtual space based on the priority of each setting item. For example, the display position determination unit 203 may determine the setting item as a display object if the priority of the setting item is higher than a threshold.

[0101] The display position determination unit 203 can determine the display position of the object for each setting item based on the priority of each setting item. For example, the display position determination unit 203 determines the display position in such a way that the object representing the setting item with a higher priority is displayed closer to the operation area of ​​the camera 120.

[0102] The display position determination unit 203 may also determine the position of the object representing the setting item in the depth direction based on the priority. That is, the display position determination unit 203 determines the display position in such a manner that the object representing the setting item with a higher priority is displayed more in the foreground (displayed closer to the HMD 101).

[0103] The priority of the setting item may be a preset priority, or may be set according to the frequency with which the user uses the setting item. The priority of the setting item may be set according to the correlation with the setting item being operated by the user.

[0104] exist Figure 8At step S806, the display position determination unit 203 determines whether the display size of the object representing the setting item obtained at step S803 can be reduced. The display position determination unit 203 can determine whether the size can be reduced based on, for example, the visibility of the object or the readability of letters or words contained in the object. When the display size of the object representing at least one setting item among the multiple setting items can be reduced, the display position determination unit 203 determines that the size of the setting item can be reduced. When the display size cannot be reduced, the process proceeds to step S807. When the display size can be reduced, the process proceeds to step S808.

[0105] At step S807, the display position determination unit 203 determines that the display positions of some of the objects representing the setting items are outside the field of view. In this case, the display position determination unit 203 determines which setting items are to be displayed within the field of view in descending order of priority of the setting items, and determines that the display positions of the objects representing the remaining setting items are outside the field of view.

[0106] At step S808, the display position determination unit 203 reduces the display size of the object representing the setting item, and the process returns to step S804. The display position determination unit 203 may reduce the display size of the object representing some of the plurality of setting items.

[0107] Fig.11 is a flowchart illustrating an example of setting processing of the camera 120 . Fig.11 The processing shown is realized by the CPU 111 executing a program corresponding to each step of the processing. The HMD 101 starts when it sends the setting item information to the camera 120. Fig.11 processing.

[0108] At step S1101, the setting information receiving unit 208 of the camera 120 receives setting item information transmitted from the setting information transmitting unit 207 of the HMD 101 via the communication I / F 116. The setting item information includes the name of a display item, a setting value, and a display position in the operation screen.

[0109] At step S1102, the setting information receiving unit 208 updates the setting item information 210 for the received setting item. The setting information receiving unit 208 updates the setting value of the setting item information 210 to the setting value received from the HMD 101, and updates the display status to "Y".

[0110] At step S1103, the setting information receiving unit 208 updates the setting item information 210 for the setting items other than the setting items received from the setting information transmitting unit 207. The setting information receiving unit 208 updates the display status of the setting item information 210 to "N" for the setting items other than the received setting items.

[0111] At step S1104, the setting value reflection unit 209 reflects the setting item information 210 in the camera 120. Specifically, the setting value reflection unit 209 displays an object representing a setting item whose display status is set to “Y” in the setting item information 210 at a position to which the object is moved by the user in the operation screen of the real camera 120. The setting value reflection unit 209 reflects the setting value of the setting item whose display status is set to “Y” in the setting item information 210 in the camera 120. On the other hand, the setting value reflection unit 209 does not display the setting item whose display status is set to “N” in the setting item information 210 in the operation screen of the real camera 120.

[0112] In the above-mentioned Embodiment 1, the HMD 101 changes the object representing the setting item displayed in the virtual space according to the operation screen area of ​​the camera 120 in the virtual space. The object representing the setting item of the camera 120 is displayed outside the operation screen area so that the user can easily locate the desired setting item. In the case where the user moves the object representing the setting item into the operation screen area, the setting of the setting item is reflected in the camera 120. Therefore, the user can easily locate the desired setting item and change the setting of the camera 120 with a simple operation.

[0113] When the user moves the object representing the setting item to the desired position in the operation screen, the camera 120 also changes the display position of the setting item in the actual operation screen. The user can change the layout of the setting items in the operation screen by moving the object placed outside the operation screen area. The visibility when customizing the operation screen (setting screen layout) is improved so that the user can customize the operation screen with simple operations.

[0114] In the above-mentioned Embodiment 1, the HMD 101 transmits the setting item information to the camera 120 when the object representing the setting item is moved from the outside of the operation screen area to the inside of the operation screen area. This need not necessarily be the case. When the object representing the setting item is moved from the inside of the operation screen area to the outside, the HMD 101 may transmit information related to the setting item corresponding to the object moved out of the operation screen area to the camera 120. In this case, the setting item information includes information indicating that the setting of the setting item has been canceled. The camera 120 stops displaying the setting item that has been moved out of the operation screen area, and cancels the setting of the setting item.

[0115] Example 2

[0116] In Embodiment 1, the operation screen of the camera 120 operated by the user is displayed in equal size in the display unit 109 of the HMD 101. In Embodiment 2, the operation screen is displayed in a larger size. The improved visibility of the enlarged operation screen allows the user to check the setting status of the camera 120 more easily.

[0117] Fig.12 1 is a flowchart illustrating an example of processing in the HMD 101 according to Embodiment 2. Figure 5 The same processing steps as those of the illustrated embodiment 1 are given the same reference numerals, and detailed description thereof will be omitted.

[0118] Processing steps S501 to S505 and Figure 5 These processing steps are the same. In step S1201, the operation acquisition unit 205 determines whether an operation for displaying the operation screen in a larger size (hereinafter also referred to as a zoom-in action) has been performed in step S505. If there is a zoom-in action, the process proceeds to step S1202. If there is no zoom-in action, the process proceeds to step S506.

[0119] The zoom action may be an operation performed on the virtual object to display the operation screen in a larger size. The zoom action may be a voice operation. The zoom action may be an operation performed on a button of the camera 120 assigned with a zoom function, etc. The zoom action may include an operation for specifying a magnification when displaying the operation screen in a larger size.

[0120] At step S1202, the image generation unit 204 generates an image showing the operation screen of the camera 120 in a larger size. Specifically, the image generation unit 204 generates a virtual operation screen enlarged at a magnification specified by the user or at a predetermined magnification, and displays the virtual operation screen on the real image obtained by the imaging unit 108. Processing steps S506 to S510 are similar to Figure 5 These processing steps are the same.

[0121] In the above-described Embodiment 2, the user can reflect the setting in the camera 120 in a state where the operation screen of the camera 120 in the virtual space is being displayed in a larger size. Similar to Embodiment 1, the HMD 101 can display an object representing a setting item in the virtual space according to the operation screen area of ​​the camera 120. In the case where the object representing the setting item is moved to the operation screen area of ​​the camera 120 by the user operation, the setting of the setting item is reflected in the camera 120. This saves the trouble of locating the desired setting item and allows the user to change the setting of the camera 120 with a simple operation.

[0122] The above embodiments are described for the purpose of illustrating rather than limiting the present invention; various other changes and modifications are possible when implementing the present invention.

[0123] According to the present invention, a user can easily locate a desired setting item of an electronic device and change the setting of the electronic device with a simple operation.

[0124] The various controls described above may or may not be performed by a single hardware (e.g., a processor or circuit). Multiple hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share processing to control the entire device.

[0125] The above-mentioned processor is the most general processor, including both general-purpose processors and special-purpose processors. General-purpose processors include, for example, central processing units (CPUs), microprocessing units (MPUs) and digital signal processors (DSPs). Special-purpose processors include, for example, graphics processing units (GPUs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc. Programmable logic devices include, for example, field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc.

[0126] Other embodiments

[0127] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0128] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An information processing device, comprising: a first acquisition unit configured to acquire information related to a setting item of the electronic device; a second acquisition unit configured to acquire information related to an operation screen area of ​​the electronic device in a three-dimensional space representing a field of view seen by a user; a display control unit configured to display an object representing a setting item of the electronic device outside the operation screen area in the three-dimensional space, and to change the object being displayed according to the operation screen area in the three-dimensional space; as well as A transmitting unit is configured to transmit information on a setting item corresponding to the moved object to the electronic device in response to the user's operation of moving the object into the operation screen area.

2. The information processing device according to claim 1, wherein: The display control unit determines a display position of the object based on the operation screen area and information about a setting item of the electronic device.

3. The information processing device according to claim 1 or 2, wherein: The display control unit determines a display position of the object based on a size and a position of the operation screen area.

4. The information processing device according to claim 1 or 2, wherein: The display control unit determines a display position of the object in a manner not to hinder visibility of an operation performed on the electronic device.

5. The information processing device according to claim 1 or 2, wherein: The display control unit determines a display position of the object in such a manner that the object does not overlap the operation screen area.

6. The information processing device according to claim 1 or 2, wherein: The display control unit determines the object to be displayed based on a priority preset for each setting item of the electronic device, or a priority corresponding to a frequency of use of each setting item of the electronic device, or a priority corresponding to a relevance to a setting item being operated by the user.

7. The information processing device according to claim 1 or 2, wherein: The display control unit determines the display position of the object or the size of the object based on a priority preset for each setting item of the electronic device, or a priority corresponding to the frequency of use of each setting item of the electronic device, or a priority corresponding to the relevance to the setting item being operated by the user.

8. The information processing device according to claim 1 or 2, wherein: The display control unit performs control for displaying the operation screen area in a larger size.

9. The information processing device according to claim 1 or 2, wherein: The user operation includes an operation performed using at least one of hand movement, voice, eye gaze, and gesture.

10. A head mounted display device, comprising: The information processing device according to any one of claims 1 to 9; as well as A display unit is configured to display the object.

11. An electronic device comprising: a sending unit configured to send information related to the electronic device to the information processing device according to any one of claims 1 to 9; a receiving unit configured to receive information on a setting item corresponding to the object moved to the operation screen area by a user's operation; as well as A setting unit is configured to reflect settings based on the received information related to the setting item in the electronic device.

12. The electronic device according to claim 11, wherein: The electronic device is one of a camera, a smartphone, and a printer including an operation screen.

13. An information processing method comprising the following steps: acquiring information related to setting items of the electronic device; Acquiring information related to an operation screen area of ​​the electronic device in a three-dimensional space representing a field of view seen by a user; displaying an object representing a setting item of the electronic device outside the operation screen area in the three-dimensional space, and changing the object being displayed according to the operation screen area in the three-dimensional space; as well as In response to the user's operation of moving the object into the operation screen area, information on a setting item corresponding to the moved object is transmitted to the electronic device.

14. A computer program product comprising a program for causing a computer to execute each step of the information processing method according to claim 13. 15 . A computer-readable storage medium storing a program for causing a computer to execute each step of the information processing method according to claim 13 .

Images