Operation method and program

Abstract

To provide an operation method and a program for improving operability of an operation performed with an information processing apparatus.SOLUTION: There is provided an operation method for a display system 1 comprising: an HMD 100 that includes an image display unit 20 for allowing visual recognition of an outside scene, having a DP outside camera 61 mounted therein, and displaying an image superimposed on the outside scene, and a connection device 10 connected with the image display unit 20 and a control device 300, wherein the image display unit 20 is mounted on the head of a user U; and the control device 300 that has a touch panel 350 for receiving an operation. The method includes: displaying an operation image for receiving the user U's operation on the touch panel 350; displaying, in an image display area of the image display unit 20, an adjustment image for adjusting a picked-up image picked up by the DP outside camera 61; changing a display position of the adjustment image with an operation for the operation image; and causing the DP outside camera 61 to execute imaging with an operation on the operation image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an operating method and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a display device that includes a display unit that is attached to the head of a user and displays on the display unit an image captured by an imaging unit mounted on the display unit (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP2005-38321A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when an information processing device is connected to a display device and receives operations from the user through the information processing device, the display unit is attached to the user's head, which reduces the visibility of the information processing device and reduces the operability of the information processing device. [Means for solving the problem]

[0005] One aspect for solving the above problem is an operation method for a display system including a display device worn on a user's head, the display device including a display unit capable of viewing an external view and displaying an image superimposed on the external view, and an imaging unit mounted on the display unit, the operation image for accepting operation by the user being displayed on the operation surface, an adjustment image for adjusting the imaging unit or an image captured by the imaging unit being displayed in a display area of ​​the image on the display unit, the display of the adjustment image being changed according to an operation on the accepted operation image, and the imaging unit being caused to perform imaging according to an operation on the accepted operation image.

[0006] One aspect for solving the above problem is a program to be executed by a computer including a display device worn on a user's head, the display device including a display unit capable of viewing an external view and displaying an image superimposed on the external view, and an imaging unit mounted on the display unit, and an information processing device having an operation surface for accepting operations, the program displaying an operation image for accepting operations by the user on the operation surface, displaying an adjustment image for adjusting the imaging unit or an image captured by the imaging unit in a display area of ​​the image on the display unit, changing the display of the adjustment image according to the operation on the accepted operation image, and causing the imaging unit to perform imaging according to the operation on the accepted operation image. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a display system. [Diagram 2] FIG. 2 is a plan view of a main portion showing the configuration of an optical system of an image display unit. [Diagram 3] FIG. 1 is a block diagram of a display system. [Figure 4] FIG. 2 is a block diagram of a control device and a main control unit. [Diagram 5] FIG. 2 is a diagram showing a virtual joystick controller in a first display form. [Figure 6] FIG. 13 shows a virtual joystick controller in a second display form. [Figure 7] FIG. 13 shows a virtual joystick controller in a third display form. [Figure 8] FIG. 13 is a diagram showing a state in which a virtual joystick controller is displayed at the position touched by a user's thumb. [Figure 9] 11 is an explanatory diagram for explaining a case where a display image displayed on an image display unit is operated by a virtual joystick controller. FIG. [Figure 10] 11 is an explanatory diagram for explaining a case where a display image displayed on an image display unit is operated by a virtual joystick controller. FIG. [Figure 11] An explanatory diagram for explaining how to operate settings related to image capture by the DP outer camera using a virtual joystick controller. [Figure 12] An explanatory diagram for explaining how to operate settings related to image capture by the DP outer camera using a virtual joystick controller. [Figure 13] 4 is a flowchart showing the operation of a CO control unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] [1. Display system configuration] Hereinafter, embodiments to which the present invention is applied will be described with reference to the drawings. FIG. 1 is a diagram showing a schematic configuration of a display system 1. As shown in FIG. The display system 1 includes an HMD 100 and a control device 300. The HMD 100 is a head-mounted display device that includes an image display unit 20 that is mounted on the head of a user U and allows the user U to view images and videos, and is an example of a display device of the present invention. HMD is an abbreviation for Head Mounted Display. The control device 300 is an example of an information processing device of the present invention. The image display unit 20 is also an example of a display unit of the present invention.

[0009] The HMD 100 has a connection device 10 connected to an image display unit 20. The connection device 10 functions as an interface that connects the HMD 100 to a device different from the HMD 100. In the display system 1, a control device 300 is connected to the connection device 10. In the following description and drawings, for convenience of explanation, the prefix DP is added to the names of some of the functional units constituting the HMD 100, and the prefix CO is added to the names of some of the functional units constituting the control device 300.

[0010] The control device 300 is a portable terminal device, such as a smartphone, that includes a display screen for displaying characters and images, and a touch panel 350 that functions as an operation unit for detecting touch operations and pressing operations. The touch panel 350 is composed of a display panel and a touch sensor. LCD is an abbreviation for Liquid Crystal Display. The control device 300 may be a desktop personal computer, a notebook personal computer, a tablet personal computer, or the like.

[0011] The connection device 10 includes a box-shaped case and a connector 11A and a connector 11D. The image display unit 20 is connected to the connector 11A via a connection cable 40, and the control device 300 is connected to the connector 11D via a USB cable 46. This allows the image display unit 20 and the control device 300 to be connected so as to be able to transmit and receive data to each other. For example, the control device 300 outputs video data for the image display unit 20 to display an image and audio data to the image display unit 20. In addition, the image display unit 20 transmits detection data of various sensors included in the image display unit 20 to the control device 300 as described later. The control device 300 may be capable of supplying power to the image display unit 20. USB is an abbreviation for Universal Serial Bus.

[0012] The configuration in which the connection device 10 and the control device 300 are connected using the USB cable 46 is merely an example, and the specific connection form between the connection device 10 and the control device 300 is not limited. For example, they may be connected by wire using other types of cables, or may be connected via wireless communication. For example, in a configuration in which the USB cable 46 is connected to the connector 11D of the USB-TypeC standard, the USB cable 46 can supply a direct current of 20 volts, and can transmit video data of the HDMI standard as a function of an alternative mode of the USB-TypeC. HDMI and MHL are registered trademarks.

[0013] The image display unit 20 includes a main body having a right holding portion 21, a left holding portion 23, and a front frame 27, and is provided with a right display portion 22, a left display portion 24, a right light guide plate 26, and a left light guide plate .

[0014] The right holding part 21 and the left holding part 23 extend rearward from both ends of the front frame 27, and hold the image display unit 20 on the head of the user U. The right holding part 21 is connected to an end ER of the front frame 27 located on the right side of the user U, and the left holding part 23 is connected to an end EL located on the left side of the user U.

[0015] The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is located in front of the right eye of the user U when the image display unit 20 is worn, allowing the right eye to view an image. The left light guide plate 28 is located in front of the left eye of the user U when the image display unit 20 is worn, allowing the left eye to view an image. The right light guide plate 26 and the left light guide plate 28 are optical units formed of a light-transmitting resin or the like, and guide image light output by the right display unit 22 and the left display unit 24 to the eyes of the user U. The right light guide plate 26 and the left light guide plate 28 are, for example, prisms.

[0016] The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other, and this connection position corresponds to the space between the eyebrows of the user U when the user U wears the image display unit 20. The front frame 27 may be provided with a nose pad that comes into contact with the nose of the user U when the image display unit 20 is worn, or may be configured such that a belt is connected to the right holding portion 21 and the left holding portion 23 to hold the image display unit 20 on the head of the user U.

[0017] The right display unit 22 and the left display unit 24 are each a module in which an optical unit and peripheral circuits are unitized. The right display unit 22 displays images using a right light guide plate 26, and the left display unit 24 displays images using a left light guide plate 28. The right display unit 22 is provided in the right holding unit 21, and the left display unit 24 is provided in the left holding unit 23.

[0018] The image light guided by the right light guide plate 26 and the external light transmitted through the right light guide plate 26 are incident on the right eye of the user U. Similarly, the image light guided by the left light guide plate 28 and the external light transmitted through the left light guide plate 28 are incident on the left eye. The image light from the right light guide plate 26 and the left light guide plate 28 and the external light transmitted through the right light guide plate 26 and the left light guide plate 28 are incident on the user U's eye. That is, the user U can view the external scene while wearing the image display unit 20 on his / her head. As a result, the user U views the image displayed by the image display unit 20 superimposed on the external scene transmitted through the right light guide plate 26 and the left light guide plate 28.

[0019] A DP illuminance sensor 65 is disposed on the front frame 27. The DP illuminance sensor 65 is a sensor that receives external light from the front of the user U wearing the image display unit 20. The DP illuminance sensor 65 can detect the illuminance and light amount of external light that passes through the right light guide plate 26 and the left light guide plate 28 and enters the eyes of the user U.

[0020] The DP outer camera 61 corresponds to the imaging section of the present invention. The DP outer camera 61 is provided in the front frame 27 at a position that does not block external light passing through the right light guide plate 26 and the left light guide plate 28. The DP outer camera 61 is a digital camera equipped with an imaging element such as a CCD or a CMOS, an imaging lens, and the like, and may be a monocular camera or a stereo camera. The angle of view of the DP outer camera 61 includes at least a part of the range of the external scene that the user U wearing the image display unit 20 visually recognizes through the right light guide plate 26 and the left light guide plate 28. The DP outer camera 61 may be a wide-angle camera, and may be capable of capturing an image of the entire external scene that the user U wearing the image display unit 20 visually recognizes. CCD stands for Charge Coupled Device, and CMOS stands for Complementary Metal Oxide Semiconductor. An LED indicator 67 that lights up when the DP outer camera 61 is in operation is disposed on the front frame 27 .

[0021] The front frame 27 is provided with a distance sensor 64 that detects the distance to a measurement object located in a preset measurement direction. The distance sensor 64 is, for example, a light reflection type distance sensor using an LED or a laser diode, an infrared depth sensor, an ultrasonic distance sensor, or a laser range scanner. The distance sensor 64 may be a distance detection unit that combines image detection and sound detection, or a device that detects the distance by processing images obtained by stereo photography using a camera. The measurement direction of the distance sensor 64 is, for example, the direction of the outside scene visually recognized by the user U through the right light guide plate 26 and the left light guide plate 28.

[0022] The right display unit 22 and the left display unit 24 are each connected to the connection device 10 by a connection cable 40. The connection cable 40 has an audio connector 36. A headset 30 having a right earphone 32 and a left earphone 34 constituting stereo headphones, and a microphone 63 is connected to the audio connector 36. The right earphone 32 and the left earphone 34 output audio based on an audio signal output by the connection device 10. The microphone 63 collects audio and outputs an audio signal to the connection device 10.

[0023] [2. Optical system configuration of the image display unit] Fig. 2 is a plan view of the main part showing the configuration of the optical system of the image display unit 20. For the purpose of explanation, Fig. 2 illustrates the left eye LE and right eye RE of a user U. The right display section 22 and the left display section 24 are configured, for example, symmetrically. As a configuration for allowing the right eye RE to view an image, the right display unit 22 includes an OLED unit 221 that emits image light, and a right optical system 251 that guides the image light L emitted by the OLED unit 221 to the right light guide plate 26. OLED is an abbreviation for Organic Light Emitting Diode.

[0024] The OLED unit 221 has an OLED panel 223 and an OLED drive circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-luminous display panel in which light-emitting elements that emit, for example, R, G, and B color lights are arranged. The OLED drive circuit 225 drives the OLED panel 223 under the control of the DP control unit 120. The OLED drive circuit 225 is mounted on, for example, a substrate (not shown) that is fixed to the rear surface of the OLED panel 223, and the temperature sensor 217 shown in FIG. 3 is mounted on this substrate.

[0025] The right optical system 251 converts the image light L emitted from the OLED panel 223 into a parallel light beam by a collimator lens, and causes the image light L to enter the right light guide plate 26. Inside the right light guide plate 26, the image light L is reflected by a plurality of reflecting surfaces, reflected by a half mirror 261 located in front of the right eye RE, and emitted from the right light guide plate 26 toward the right eye RE.

[0026] As a configuration for allowing the left eye LE to view an image, the right display unit 22 includes an OLED unit 241 that emits image light, and a left optical system 252 that guides the image light L emitted by the OLED unit 241 to the left light guide plate .

[0027] The OLED unit 241 has an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. The OLED panel 243 is a self-luminous display panel in which light-emitting elements that emit, for example, R, G, and B color lights are arranged. The OLED drive circuit 245 drives the OLED panel 243 under the control of the DP control unit 120. The OLED drive circuit 245 is mounted on a substrate (not shown) that is fixed to the rear surface of the OLED panel 243, for example, and a temperature sensor 239 shown in FIG. 3 is mounted on this substrate.

[0028] The left optical system 252 converts the image light L emitted from the OLED panel 243 into a parallel light beam by a collimator lens, and causes the image light L to enter the left light guide plate 28. Inside the left light guide plate 28, the image light L is reflected by a plurality of reflecting surfaces, reflected by a half mirror 261 located in front of the left eye LE, and emitted from the left light guide plate 28 toward the left eye LE.

[0029] The HMD100 functions as a transmissive display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 are incident on the right eye RE of the user U. The image light L reflected by the half mirror 281 and the external light OL transmitted through the half mirror 281 are incident on the left eye LE. The HMD100 superimposes the image light L of an image processed inside and the external light OL and causes them to enter the eye of the user U. Therefore, the user U can see the outside scene through the right light guide plate 26 and the left light guide plate 28, and an image by the image light L is viewed superimposed on this outside scene. The half mirrors 261 and 281 are image extraction units that reflect the image light output by the right display unit 22 and the left display unit 24, respectively, and extract the image, and constitute the display unit.

[0030] [3. HMD control system] FIG. 3 is a block diagram of the display system 1, and in particular shows the configuration of the HMD 100 in detail. In the image display unit 20, the right display unit 22 has a right display board 210. On the right display board 210, a right I / F unit 211 connected to the connection cable 40, a receiving unit 213 that receives data input from the connection device 10 via the right I / F unit 211, and an EEPROM 215 are mounted. The right I / F unit 211 connects the receiving unit 213, the EEPROM 215, the temperature sensor 217, the DP outer camera 61, the distance sensor 64, the DP illuminance sensor 65, and the LED indicator 67 to the connection device 10. The receiving unit 213 connects the OLED unit 221 to the connection device 10.

[0031] The left display unit 24 has a left display unit board 230. On the left display unit board 230, a left I / F unit 231 connected to the connection cable 40 and a receiving unit 233 that receives data input from the connection device 10 via the left I / F unit 231 are mounted. On the left display unit board 230, a DP 6-axis sensor 235 and a DP magnetic sensor 237 are mounted. The left I / F unit 231 connects the receiving unit 233, the DP 6-axis sensor 235, the DP magnetic sensor 237, and the temperature sensor 239 to the connection device 10. The receiving unit 233 connects the OLED unit 241 to the connection device 10.

[0032] In the description and drawings of this embodiment, I / F is an abbreviation for interface. EEPROM is an abbreviation for Electrically Erasable Programmable Read-Only Memory. The receiving unit 213 and the receiving unit 233 may be written as Rx213 and Rx233, respectively.

[0033] The EEPROM 215 stores various data in a non-volatile manner. For example, the EEPROM 215 stores data related to the light emission characteristics and display characteristics of the OLED units 221 and 241 included in the image display unit 20, data related to the characteristics of the sensors included in the right display unit 22 or the left display unit 24, etc. Specifically, the EEPROM 215 stores parameters related to gamma correction of the OLED units 221 and 241, data for compensating for the detection values ​​of the temperature sensors 217 and 239, etc. in a manner readable by the DP control unit 120.

[0034] The DP outer camera 61 captures an image according to a signal input via the right I / F unit 211, and outputs the captured image to the right I / F unit 211. The DP illuminance sensor 65 receives external light, and outputs a detection value corresponding to the amount or intensity of received light. The LED indicator 67 lights up according to a control signal or drive current input via the right I / F unit 211. The temperature sensor 217 detects the temperature of the OLED unit 221, and outputs a voltage value or a resistance value corresponding to the detected temperature as a detection value.

[0035] The distance sensor 64 outputs a signal indicating the result of the distance detection to the connection device 10 via the right I / F unit 211.

[0036] The receiving unit 213 receives image data for display transmitted from the connection device 10 via the right I / F unit 211, and outputs the image data to the OLED unit 221. The OLED unit 221 displays an image based on the image data transmitted by the connection device 10. The receiving unit 233 receives image data for display transmitted from the connection device 10 via the left I / F unit 231, and outputs the image data to the OLED unit 241. The OLED units 221 and 241 display images based on the image data transmitted by the connection device 10.

[0037] The DP six-axis sensor 235 is a motion sensor including a three-axis acceleration sensor and a three-axis gyro sensor. The DP magnetic sensor 237 is, for example, a three-axis geomagnetic sensor. The DP six-axis sensor 235 and the DP magnetic sensor 237 may be an IMU in which the above-mentioned sensors are modularized, or may be a module in which the DP six-axis sensor 235 and the DP magnetic sensor 237 are integrated. IMU is an abbreviation of Inertial Measurement Unit. The temperature sensor 239 detects the temperature of the OLED unit 241. The DP six-axis sensor 235, the DP magnetic sensor 237, and the temperature sensor 239 each output a detection value to the connection device 10.

[0038] Each part of the image display unit 20 operates with power supplied from the connection device 10 via the connection cable 40. The image display unit 20 includes a power supply unit 229 in the right display unit 22 and a power supply unit 249 in the left display unit 24. The power supply unit 229 distributes and supplies the power supplied by the connection device 10 via the connection cable 40 to each part of the right display unit 22, including the right display unit board 210. The power supply unit 249 distributes and supplies the power supplied by the connection device 10 via the connection cable 40 to each part of the left display unit 24, including the left display unit board 230. The power supply units 229, 249 may include a conversion circuit that converts voltage, or the like.

[0039] The connection device 10 includes an I / F unit 110, a DP control unit 120, a sensor control unit 122, a main control unit 123, a power control unit 126, a non-volatile storage unit 130, an operation unit 140, a connection unit 145, and an audio processing unit 147. Details of the main control unit 123 will be described with reference to FIG.

[0040] The I / F unit 110 includes a connector 11D and an interface circuit that executes a communication protocol conforming to various communication standards via the connector 11D. The I / F unit 110 is, for example, an interface board on which the connector 11D and the interface circuit are mounted. The I / F unit 110 may include an interface for a memory card to which an external storage device or storage medium can be connected, or the I / F unit 110 may be configured as a wireless communication interface.

[0041] The DP control unit 120 includes a processor such as a CPU or a microcomputer, and the processor executes a program to control each part of the connection device 10. The DP control unit 120 may include a RAM that constitutes the work area of ​​the processor. RAM is an abbreviation for Random Access Memory.

[0042] The DP control unit 120 is connected to a non-volatile storage unit 130, an operation unit 140, a connection unit 145, and an audio processing unit 147. The non-volatile storage unit 130 is a ROM that non-volatilely stores programs and data executed by the DP control unit 120. ROM is an abbreviation for Read Only Memory.

[0043] The sensor control unit 122 operates each sensor included in the image display unit 20. Here, each sensor refers to each of the DP outer camera 61, the distance sensor 64, the DP illuminance sensor 65, the temperature sensor 217, the DP 6-axis sensor 235, the DP magnetic sensor 237, and the temperature sensor 239. Each sensor includes at least one or more of the DP outer camera 61, the DP illuminance sensor 65, the DP 6-axis sensor 235, and the DP magnetic sensor 237. The sensor control unit 122 sets and initializes the sampling period of each sensor according to the control of the DP control unit 120, and executes energization of each sensor, transmission of control data, acquisition of detection values, and the like in accordance with the sampling period of each sensor.

[0044] The sensor control unit 122 outputs detection data indicating the detection values ​​and detection results of each sensor to the I / F unit 110 at a preset timing. The sensor control unit 122 may include an A / D converter that converts analog signals into digital data. In this case, the sensor control unit 122 converts the analog signals of the detection values ​​and detection results acquired from the sensors of the image display unit 20 into detection data and outputs it. The sensor control unit 122 may acquire digital data of the detection values ​​and detection results from the sensors of the image display unit 20, convert the data format, adjust the output timing, etc., and output the detection data to the I / F unit 110.

[0045] By operation of the sensor control unit 122 , the control device 300 connected to the I / F unit 110 can obtain detection values ​​of each sensor of the HMD 100 and captured images of the DP outer camera 61 .

[0046] The sensor control unit 122 may output the result of arithmetic processing based on the detection values ​​of each of the above-mentioned sensors as detection data. For example, the sensor control unit 122 may integrally process the detection values ​​and detection results of a plurality of sensors and function as a so-called sensor fusion processing unit. In this case, the sensor control unit 122 may generate detection data of a virtual sensor not included in each sensor of the image display unit 20 by sensor fusion. For example, the sensor control unit 122 may output trajectory data indicating the trajectory of the movement of the image display unit 20, coordinate data indicating the position of the image display unit 20 in a three-dimensional space, and direction data indicating the direction of the image display unit 20 as detection data. Here, the coordinate data may be data indicating relative coordinates based on the position of the connection device 10, or may be data indicating a position relative to a reference position set in a space in which the image display unit 20 exists. The direction data may be data indicating a direction based on the position or direction of the connection device 10, or may be data indicating a direction relative to a reference position set in a space in which the image display unit 20 exists. The sensor control unit 122 executes a communication protocol between the device connected to the connector 11D via the USB cable 46, and outputs detection data.

[0047] The sensor control unit 122 and the main control unit 123 may be realized by a processor executing a program, and by cooperation between software and hardware. That is, the sensor control unit 122 and the main control unit 123 are configured by a processor, and execute the above-mentioned operations by executing a program. In this example, the sensor control unit 122 and the main control unit 123 may be realized by a processor configuring the DP control unit 120 executing a program. In other words, the processor may execute a program to function as the DP control unit 120, the main control unit 123, and the sensor control unit 122. Here, the processor can be rephrased as a computer. The sensor control unit 122 and the main control unit 123 may be provided with a work memory for performing data processing, and may perform processing using the memory of the DP control unit 120.

[0048] Furthermore, the main control unit 123 and the sensor control unit 122 may be configured with programmed hardware such as a DSP or FPGA. The sensor control unit 122 and the main control unit 123 may be integrated to configure an SoC-FPGA. DSP is an abbreviation for Digital Signal Processor, FPGA is an abbreviation for Field Programmable Gate Array, and SoC is an abbreviation for System-on-a-Chip.

[0049] The power supply control unit 126 is a circuit that is connected to the connector 11D and supplies power to each unit of the connection device 10 and the image display unit 20 based on the power supplied from the connector 11D.

[0050] The operation unit 140 detects the operation of a switch or the like provided in the connection device 10, and outputs data indicating the operation content to the DP control unit 120.

[0051] The audio processing unit 147 generates an audio signal in accordance with the audio data input from the DP control unit 120, and outputs it to the connection unit 145. This audio signal is output from the connection unit 145 to the right earphone 32 and the left earphone 34 via the audio connector 36. The audio processing unit 147 generates audio data of the audio collected by the microphone 63, and outputs it to the DP control unit 120. The audio data output by the audio processing unit 147 may be processed by the sensor control unit 122, similar to the detection data of the sensor provided in the image display unit 20.

[0052] [4. Control device configuration] FIG. 4 is a block diagram of the control device 300 and the main control unit 123. First, the control device 300 will be described. The control device 300 includes a CO control unit 310. The CO control unit 310 includes a processor 311, a memory 312, and a non-volatile memory 313. The processor 311 is composed of a CPU, a microcomputer, a DSP, etc., and controls each unit of the control device 300 by executing a program. The memory 312 forms a work area for the processor 311. The non-volatile memory 313 is composed of a semiconductor memory device, etc., and stores in a non-volatile manner the programs executed by the processor 311 and various data processed by the processor 311. For example, the non-volatile memory 313 stores an operating system as a basic control program executed by the processor 311, application programs that run on the operating system, etc. The non-volatile memory 313 stores data that is processed when the application program is executed and data of the processing result. The operating system is hereinafter abbreviated as OS. The CO control unit 310 may be an SoC that integrates a processor 311, a memory 312, and a non-volatile memory 313.

[0053] The CO control unit 310 is connected to a GNSS 321, a CO camera 322, a CO six-axis sensor 323, a CO magnetic sensor 324, a CO illuminance sensor 325, a vibrator 326, an audio output unit 327, a CO display unit 330, and a CO input unit 335. The GNSS 321 performs positioning using a satellite positioning system, and outputs the position of the control device 300 to the CO control unit 310. GNSS is an abbreviation for Global Navigation Satellite System.

[0054] CO camera 322 is a digital camera provided on the main body of control device 300, and is disposed, for example, adjacent to touch panel 350 and captures an image in the direction facing touch panel 350. CO camera 322 captures images according to the control of CO control unit 310, and outputs the captured images to CO control unit 310.

[0055] The CO six-axis sensor 323 is a motion sensor equipped with a three-axis acceleration sensor and a three-axis gyro sensor, and outputs detection data indicating a detection value to the CO control unit 310. The CO magnetic sensor 324 is, for example, a three-axis geomagnetic sensor, and outputs detection data indicating a detection value to the CO control unit 310. The CO six-axis sensor 323 and the CO magnetic sensor 324 may be an IMU in which the above-mentioned sensors are modularized, or may be a module in which the CO six-axis sensor 323 and the CO magnetic sensor 324 are integrated. The CO illuminance sensor 325 receives external light and outputs to the CO control unit 310 detection data indicating a detection value corresponding to the amount or intensity of received light.

[0056] The vibrator 326 generates vibrations according to the control of the CO control unit 310, and vibrates a part or the whole of the main body of the control device 300. The vibrator 326 is configured to include, for example, an eccentric weight and a motor. The audio output unit 327 includes a speaker and outputs audio from the speaker under the control of the CO control unit 310. The audio output unit 327 may include an amplifier that amplifies the audio signal output by the CO control unit 310 and outputs the amplified audio signal to the speaker. When the CO control unit 310 is configured to output digital audio data, the audio output unit 327 may include a D / A converter that converts the digital audio data into an analog audio signal.

[0057] CO display unit 330 has touch panel 350, and displays characters and images on touch panel 350 under the control of CO control unit 310. Touch panel 350 is an example of an operation surface of the present invention. The CO input unit 335 detects an operation on the switch 337, and outputs operation data indicating the detected operation to the CO control unit 310. The switch 337 is, for example, a hardware switch such as a power switch or a volume adjustment switch of the control device 300. The switch 337 may be a contact or non-contact sensor, for example, a fingerprint sensor embedded in the touch panel 350. The switch 337 may also be a software switch formed using a part or all of the touch panel 350.

[0058] To the CO control unit 310, a battery 341, a communication unit 342, and an I / F unit 343 are connected. The battery 341 is a secondary battery built into the main body of the control device 300, and supplies power to each unit of the control device 300. The battery 341 may include a control circuit (not shown) that controls the output of power and the charging of the secondary battery.

[0059] The communication unit 342 supports wireless communication protocols such as Bluetooth and Wi-Fi, and performs wireless communication with devices external to the display system 1. Bluetooth and Wi-Fi are registered trademarks. The communication unit 342 may be configured to perform mobile data communication using a mobile communication network such as LTE or a fifth generation mobile communication system. LTE is a registered trademark.

[0060] The I / F unit 343 includes a connector (not shown) to which a data communication cable is connected, and an interface circuit that executes a communication protocol conforming to various communication standards via the connector. For example, the I / F unit 343 includes a connector and an interface circuit conforming to the USB standard, and transmits and receives data via the USB cable 46.

[0061] In this embodiment, the control device 300 transmits video data to the HMD 100 via the I / F unit 343, and receives sensor detection data from the HMD 100. The control device 300 supplies power to the HMD 100 via the I / F unit 343.

[0062] The I / F unit 343 of this embodiment includes a USB interface, and a configuration in which the control device 300 transmits and receives data to and from the HMD 100 using a USB cable 46 connected to the I / F unit 343 is illustrated as an example. The control device 300 may perform wireless data communication with the HMD 100 via the communication unit 342, for example.

[0063] The main control unit 123 includes a non-volatile memory 124 and a processor 125 . The non-volatile memory 124 is composed of a semiconductor memory device or the like, and stores the programs executed by the processor 125 and various data processed by the processor 125 in a non-volatile manner. The processor 125 is composed of a CPU, a microcomputer, a DSP, etc., and controls the connection device 10 by executing a program.

[0064] The main control unit 123 includes a display control unit 125a as a functional block.

[0065] The display control unit 125a executes various processes for displaying an image on the image display unit 20 based on the display data input to the I / F unit 110. The display data includes, for example, video data. In this embodiment, the video data is transmitted in an alternative mode of USB-TypeC through the connector 11D configured as a USB-TypeC connector. The display control unit 125a executes various processes such as frame extraction, resolution conversion, scaling, intermediate frame generation, and frame rate conversion. The display control unit 125a outputs video data corresponding to the OLED units 221 and 241 to the connection unit 145. The video data input to the connection unit 145 is transmitted from the connector 11A to the right I / F unit 211 and the left I / F unit 231 as a video signal 201. The display control unit 125a causes the right display unit 22 and the left display unit 24 to display an image according to the display data transmitted by the control device 300. The area where an image is displayed by the right display unit 22 and the left display unit 24 of the image display unit 20 is referred to as the display area 200. The display area 200 is an area where the user U can view the image light L emitted by the OLED panels 223 and 243 and incident on the right eye and left eye of the user U by the right light guide plate 26 and the left light guide plate 28. As shown in FIG. 9, the display area 200 is a three-dimensional area having three axial directions, namely, up / down, left / right, and front / rear. Hereinafter, the upward direction of the display area 200 as viewed from the user U is referred to as the +X direction, the downward direction as the -X direction, the leftward direction as the -Y direction, and the rightward direction as the +Y direction. In addition, the forward direction, which is the depth direction of the display area 200, is referred to as the +Z direction, and the backward direction as the -Z direction.

[0066] 5 to 8 are diagrams showing a virtual joystick controller 500 displayed on the touch panel 350. The virtual joystick controller 500 will be described with reference to FIGS. A virtual joystick controller 500 is displayed on the touch panel 350, which is the operation surface, under the control of the CO control unit 310. The virtual joystick controller 500 is an operation image that accepts an operation by the user U, and is an operation image that can accept operations equivalent to those that an actual joystick controller can accept. In particular, the virtual joystick controller 500 can accept directional input. The virtual joystick controller 500 has a plurality of display forms, and can change the acceptable direction depending on the display form. Specifically, the virtual joystick controller 500 has a manipulator image 501 whose display position on the touch panel 350 changes according to the operation by the user U, and has three display forms by setting a limit on the direction in which the display position of this manipulator image 501 can be changed.

[0067] The virtual joystick controller 500 has three display modes: a first display mode, a second display mode, and a third display mode. Depending on the display image that the main control unit 123 causes to be displayed in the display area 200, the CO control unit 310 causes the virtual joystick controller 500 to be displayed in any one of the first display mode, the second display mode, and the third display mode. Specifically, when the display image displayed in the display area 200 is an image that accepts input in two directions, such as up and down and left and right, the CO control unit 310 causes the touch panel 350 to display the virtual joystick controller 500 in the first display form or the second display form. When the display image displayed in the display area 200 is an image that accepts input in all directions, the CO control unit 310 causes the touch panel 350 to display the virtual joystick controller 500 in the third display form.

[0068] Fig. 5 is a diagram showing virtual joystick controller 500 in the first display form. In particular, (A) shown in Fig. 5 shows a case where control image 501 is located in the center of slider bar 503A, and (B) shows a case where control image 501 is located above slider bar 503A. The virtual joystick controller 500 in the first display form includes an operator image 501 operated by the user U, and a slider bar 503A indicating the movement range of the operator image 501. The operator image 501 is an image whose display position is changed in response to the operation of the user U. The shape of the operator image 501 in this embodiment is a perfect circle, but is not limited to a perfect circle, and may be any shape that is easy for the user U to operate. The slider bar 503A has a vertically elongated shape extending in the vertical direction, which is the longitudinal direction of the touch panel 350, and indicates the movement range in which the operator image 501 can be moved by the operation of the user U. That is, when the operator image 501 is moved to the upper limit of the slider bar 503A, the operator image 501 cannot be moved upward any further, and when the operator image 501 is moved to the lower limit of the slider bar 503A, the operator image 501 cannot be moved downward any further. Furthermore, when a tablet PC is used as the control device 300, the proportion of the operator image 501 on the touch panel 350 of the tablet PC is small, whereas when a smartphone is used as the control device 300, the proportion of the operator image 501 on the touch panel 350 of the smartphone is large.

[0069] The virtual joystick controller 500 in the first display mode is a mode in which the control image 501 can be moved only in the up and down direction, which is the extension direction of the slider bar 503A. The up and down direction corresponds to the first direction of the present invention. The CO control unit 310 acquires information on the display image displayed in the display area 200 by the image display unit 20 from the main control unit 123, and determines the display mode of the virtual joystick controller 500 based on the acquired information. For example, when the display image is an image that accepts an input in the up and down direction, executes a process corresponding to the accepted input, or changes the display mode of the display image, the CO control unit 310 causes the virtual joystick controller 500 in the first display mode to be displayed on the touch panel 350.

[0070] Fig. 6 is a diagram showing virtual joystick controller 500 in the first display form. In particular, (A) shown in Fig. 6 shows a case where control image 501 is located in the center of slider bar 503B, and (B) shows a case where control image 501 is located to the right of slider bar 503B. The virtual joystick controller 500 in the second display form includes a manipulator image 501 operated by the user U, and a slider bar 503B indicating a range within which the manipulator image 501 can be moved. The shape of the manipulator image 501 in the second display form is also a perfect circle, but is not limited to a perfect circle, and may be any shape that is easy for the user U to operate. The slider bar 503B has a horizontally elongated shape extending in the left-right direction, which is the short side direction of the touch panel 350, and indicates a movement range within which the manipulator image 501 can be moved by the operation of the user U. That is, when the manipulator image 501 is moved to the left end of the slider bar 503B, the manipulator image 501 cannot be moved any further to the left, and when the manipulator image 501 is moved to the right end of the slider bar 503B, the manipulator image 501 cannot be moved any further to the right.

[0071] The virtual joystick controller 500 in the second display mode is a mode in which the control image 501 can be moved only in the left-right direction, which is the extension direction of the slider bar 503B. The left-right direction corresponds to the second direction of the present invention. The CO control unit 310 acquires information on the display image displayed in the display area 200 by the image display unit 20 from the main control unit 123, and determines the display mode of the virtual joystick controller 500 based on the acquired information. For example, when the display image is an image that accepts input in the left-right direction, executes processing corresponding to the accepted input, or changes the display mode of the display image, the CO control unit 310 causes the virtual joystick controller 500 in the second display mode to be displayed on the touch panel 350.

[0072] Furthermore, the CO control unit 310 may determine the attitude of the control device 300 based on the detection data input from the CO six-axis sensor 323, and change the display of the virtual joystick controller 500 according to the determined attitude. For example, assume that the user U holds the control device 300 in his / her hand, rotates the control device 300, and changes the way the user holds the control device 300 so that the short side direction of the touch panel 350 is parallel to the vertical direction. In this case, the CO control unit 310 changes the display of the virtual joystick controller 500 in the first display mode so that the extension direction of the slider bar 503A is the short side direction of the touch panel 350. The CO control unit 310 also changes the display of the virtual joystick controller 500 in the second display mode so that the extension direction of the slider bar 503B is the long side direction of the touch panel 350.

[0073] 5 shows a case where slider bar 503A extends in the longitudinal direction of touch panel 350, and FIG. 6 shows a case where slider bar 503B extends in the lateral direction of touch panel 350, but the extension directions of slider bar 503A and slider bar 503B are not limited to the longitudinal and lateral directions of touch panel 350. For example, slider bar 503A and slider bar 503B may be displayed so as to be parallel to the diagonal direction of touch panel 350.

[0074] FIG. 7 is a diagram showing the configuration of a virtual joystick controller 500 in the third display form. The virtual joystick controller 500 in the third display form includes a control image 501 operated by a user U, and a range image 505 indicating the range within which the control image 501 can be moved. The shapes of the control image 501 and the range image 505 are perfect circles, and the radius of the range image 505 is larger than the radius of the control image 501.

[0075] Unlike the virtual joystick controller 500 in the first and second display forms, the virtual joystick controller 500 in the third display form has no restrictions on the direction in which the control image 501 can be moved, and it can be moved in all directions 360° within the range of the range image 505. 7A shows the display position of the control image 501 before receiving an operation from the user U, with the control image 501 located at the center of the range image 505. Also, FIG. 7B shows the display position of the control image 501 before receiving an operation from the user U, with the control image 501 located at the left side of the range image 505. The CO control unit 310 specifies the direction input by the user U based on the display position of the control image 501 before receiving the operation and the display position of the control image 501 after receiving the operation. That is, the CO control unit 310 specifies the direction input by the user U, by specifying the direction in which the control image 501 has moved from the reference position, based on the display position of the control image 501 before receiving the operation as a reference position.

[0076] Furthermore, in the virtual joystick controller 500 in the third display form, the operations on the control image 501 include operations of rotating the control image 501 clockwise or counterclockwise, rotating counterclockwise, tapping, and long pressing. For these operations as well, the CO control unit 310 identifies the direction input by the user U based on the display position of the control image 501 before receiving the operation and the display position of the control image 501 after receiving the operation.

[0077] The CO control unit 310 acquires information about the display image that the image display unit 20 has caused to be displayed in the display area 200 from the main control unit 123, and determines the display mode of the virtual joystick controller 500 based on the acquired information. For example, when the display image is an image that receives input in four directions, up, down, left, and right, and executes processing corresponding to the received input or changes the display mode of the display image, the CO control unit 310 causes the virtual joystick controller 500 in the third display mode to be displayed on the touch panel 350. By operating the control image 501, the user U can, for example, move the display position of a capture range image 710 shown in Fig. 9 described later in the display area 200 in the ±X directions, ±Y directions, and ±Z directions.

[0078] In the case of the virtual joystick controller 500 in the third display form, the user U touches the display position of the virtual joystick controller 500 with a finger and moves the touched finger in the desired direction of input without removing the finger from the touch panel 350. This reduces the number of times the user U checks the touch panel 350 compared to a case where the user must check the key position every time an operation is performed, such as with a cross key. This improves operability for the user U who wears the image display unit 20 on his / her head and visually recognizes the touch panel 350 through the image display unit 20.

[0079] FIG. 8 is a diagram showing a state in which a user U holds the control device 300 in one hand, and in particular, a diagram showing a state in which a virtual joystick controller 500 is displayed at a position touched by the user U's thumb. 5 to 7 show a case where the virtual joystick controller 500 is displayed in the center of the screen of the touch panel 350. Fig. 8 shows a case where the virtual joystick controller 500 is displayed based on the position on the touch panel 350 where the finger of the user U, which is the pointing object, first touches. The CO control unit 310 displays the virtual joystick controller 500 centered on the coordinate position of the touch panel 350 indicated by the signal input from the touch panel 350. For example, the virtual joystick controller 500 is displayed so that the center of the control image 501 is displayed at the coordinate position of the touch panel 350.

[0080] The size of the user U's hand varies from person to person, and when the virtual joystick controller 500 is displayed in the center of the screen of the touch panel 350, it may be difficult to operate it with one hand. For this reason, the virtual joystick controller 500 is displayed based on the position of the touch panel 350 where the touch of the user U's finger is detected, thereby improving the operability of the virtual joystick controller 500 with one hand. In addition, the user U touches the touch panel 350 with his / her finger, and moves the finger in the direction in which he / she wants to input without removing it from the touch panel 350. Since the virtual joystick controller 500 is displayed at the position touched by the user U, the user U can operate it by blind touch without looking at the touch panel 350.

[0081] 5 to 8, in addition to the virtual joystick controller 500, a back button 520 and an OK button 530 are displayed on the touch panel 350. The back button 520 is a button that accepts an operation to return to the screen immediately before the currently displayed screen. The OK button is a button that accepts an operation to confirm the display of a display image changed by operating the virtual joystick controller 500 or the settings of the DP outer camera 61.

[0082] The objects that can be operated by the virtual joystick controller 500 having the first, second and third display forms as described above include the display image displayed in the display area 200 by the image display unit 20 and settings related to imaging by the DP outer camera 61. When the CO control unit 310 receives a preset operation of the virtual joystick controller 500, the CO control unit 310 switches the operation target to the display image or the DP outer camera 61. For example, when the CO control unit 310 receives an operation of tapping the control image 501 of the virtual joystick controller 500 once, the CO control unit 310 sets the operation target of the virtual joystick controller 500 to the display image. Also, when the CO control unit 310 receives an operation of tapping the control image 501 twice, the CO control unit 310 sets the operation target of the virtual joystick controller 500 to the DP outer camera 61. In this case, the CO control unit 310 outputs the operation information received by the virtual joystick controller 500 to the main control unit 123.

[0083] Furthermore, the CO control unit 310 may change the operation target of the virtual joystick controller 500 based on an application program executed by the main control unit 123 or a display image displayed on the image display unit 20. For example, when the image display unit 20 displays a display image in the display area 200, the CO control unit 310 changes the operation target of the virtual joystick controller 500 to the display image. Furthermore, when the CO control unit 310 receives a notification from the main control unit 123 that a camera application has been started, the CO control unit 310 changes the operation target of the virtual joystick controller 500 to the DP outer camera 61.

[0084] 9 and 10 are explanatory diagrams for explaining a case where the display image displayed on the image display unit 20 is operated by the virtual joystick controller 500. FIG. In the following, a case will be described in which the user U operates the virtual joystick controller 500 to move a capture range image 710, which is an example of a display image, to a range in which the user U wants to capture an image with the DP outer camera 61. The capture range image 710 is an image showing a range to be captured as a capture image among the captured images captured by the DP outer camera 61. Note that in this embodiment, a case will be described in which the capture range image 710 is used to change a range to be captured as a capture image among the captured images of the DP outer camera 61, but in reality, the imaging range of the DP outer camera 61 may be changed by adjusting the angle of view of the DP outer camera 61 by operating the virtual joystick controller 500. 9 and 10, a user U wearing the image display unit 20 on his / her head can see a desk 610 and a bottle 620 as objects in real space. The user U can also see a capture range image 710 as a display image displayed by the image display unit 20. The capture range image 710 shown by the dashed lines in Figs. 9 and 10 is a hemispherical image, and is a three-dimensional image having areas in the directions of the X-, Y-, and Z-axes, which are the three axes of the display area 200.

[0085] In addition to the capture range image 710, a first icon 810 and a second icon 830 are displayed in the display area 200. The first icon 810 is an icon indicating that the operation target of the virtual joystick controller 500 is the capture range image 710, which is a display image. The second icon 830 is an icon indicating that the operation target of the virtual joystick controller 500 is the DP outer camera 61. In Figs. 9 and 10, the display area of ​​the first icon 810 is hatched to indicate that the first icon 810 is selected and is lit or flashing.

[0086] In addition to the virtual joystick controller 500, the back button 520, and the OK button 530, a guidance display 540 is displayed on the touch panel 350 of the control device 300. The guidance display 540 displays the operation content by the virtual joystick controller 500. For example, when the operation target of the virtual joystick controller 500 is a display image, the guidance display 540 displays the display position or the display size as the operation content. The display position indicates that the operation of the virtual joystick controller 500 is an operation for changing the display position of the capture range image 710. The display size indicates that the operation of the virtual joystick controller 500 is an operation for enlarging or reducing the size, which is the display size of the capture range image 710.

[0087] The operation content of the virtual joystick controller 500 can be switched, for example, by a touch operation on the guidance display 540. When a touch operation on the guidance display 540 is detected while "display position" is displayed on the guidance display 540 of the touch panel 350, the CO control unit 310 changes the operation content of the virtual joystick controller 500 to "display size." At this time, the display of the guidance display 540 is also changed to the display size. Furthermore, when a touch operation on the guidance display 540 is detected while "display size" is displayed on the guidance display 540 of the touch panel 350, the CO control unit 310 changes the operation content of the virtual joystick controller 500 to "display position." At this time, the display of the guidance display 540 is also changed to the display position.

[0088] In the display example of the display area 200 shown in Fig. 9, the capture range image 710 is displayed at a position outside the bottle 620, and even if an image of the range indicated by the capture range image 710 shown in Fig. 9 is captured, it is not possible to capture an entire image of the bottle 620. For this reason, the user U operates the control image 501 of the virtual joystick controller 500 to move the capture range image 710 in the ±X, ±Y, and ±Z directions, and changes the display position so that the capture range image 710 covers the entire bottle 620 to be imaged, as shown in Fig. 10.

[0089] For example, assume that the user U moves the control image 501 of the virtual joystick controller 500 upward. When the CO control unit 310 detects an operation to move the control image 501 upward, it outputs an operation signal corresponding to the detected operation to the main control unit 123. As a result, the capture range image 710 moves in the +X direction, which is the upward direction of the display region 200. It is assumed that the user U moves the control image 501 of the virtual joystick controller 500 downward. When the CO control unit 310 detects an operation to move the control image 501 downward, it outputs an operation signal corresponding to the detected operation to the main control unit 123. As a result, the capture range image 710 moves in the −X direction, which is the downward direction of the display region 200. Similarly, when the CO control unit 310 detects an operation to move the control image 501 leftward or rightward, it outputs an operation signal corresponding to the detected operation to the main control unit 123. As a result, the capture range image 710 moves in the −Y direction, which is the leftward direction of the display area 200, or in the +Y direction, which is the rightward direction.

[0090] Also assume that the user U rotates the control image 501 of the virtual joystick controller 500 to the right. When the CO control unit 310 detects an operation of rotating the control image 501 to the right, it outputs an operation signal corresponding to the detected operation to the main control unit 123. As a result, the capture range image 710 moves in the +Z direction, which is the forward direction of the display region 200. Similarly, assume that the user U rotates the control image 501 of the virtual joystick controller 500 to the left. When the CO control unit 310 detects an operation to rotate the control image 501 to the left, it outputs an operation signal corresponding to the detected operation to the main control unit 123. As a result, the capture range image 710 moves in the −Z direction, which is the rear direction of the display region 200.

[0091] In addition, the CO control unit 310 may determine the attitude of the control device 300 based on the detection data input from the CO six-axis sensor 323, and move the capture range image 710 in the +Z direction or the -Z direction according to the determined attitude. For example, assume that a user U holding the control device 300 in his / her hand operates the control device 300 so as to tilt the upper side of the longitudinal direction of the touch panel 350 in the opposite direction to the direction of the user U. The CO control unit 310 detects the tilt, which is the attitude of the control device 300, based on the detection data input from the CO 6-axis sensor 323, and outputs an operation signal corresponding to the detected tilt to the main control unit 123. As a result, the capture range image 710 moves in the −Z direction, which is the rear direction of the display area 200. Also assume that the user U operates the control device 300 so as to tilt the upper side of the longitudinal direction of the touch panel 350 toward the user U. The CO control unit 310 detects the tilt, which is the attitude of the control device 300, based on the detection data input from the CO six-axis sensor 323, and outputs an operation signal corresponding to the detected tilt to the main control unit 123. As a result, the capture range image 710 moves in the +Z direction, which is the forward direction of the display area 200. When the user U has completed changing the display position of the capture area image 710, he or she presses the OK button 530 to input that the change in the display position of the capture area image 710 has been completed.

[0092] Next, when the user U has completed changing the display position of the capture range image 710 , he or she inputs a predetermined operation, such as tapping the control image 501 twice, to switch the operation target of the virtual joystick controller 500 to the DP outer camera 61 . The user U inputs a predetermined operation using the virtual joystick controller 500 to cause the DP outer camera 61 to capture an image. The predetermined operation is, for example, an operation of pressing and holding the control image 501. When the CO control unit 310 detects an operation of pressing and holding the control image 501 while the operation target is switched to the DP outer camera 61, the CO control unit 310 outputs an image capture instruction to the main control unit 123. When the image capture instruction is input from the CO control unit 310, the main control unit 123 causes the DP outer camera 61 to capture an image. In addition, the main control unit 123 cuts out an image of an area corresponding to the capture range image 710 from the captured image of the DP outer camera 61, and stores the cut-out image in the non-volatile storage unit 130. As a result, an image of the bottle 620 is saved in the non-volatile storage unit 130. The acquisition range of the captured image is changed by cutting out an image of an area corresponding to the capture range image 710 from the captured image of the DP outer camera 61.

[0093] Furthermore, when changing the display size of the capture range image 710 by operating the virtual joystick controller 500, the user U first touches the guidance display 540 and changes the display content of the guidance display 540 to "display size". Next, the user U inputs a predetermined operation using the virtual joystick controller 500 to change the display size of the capture range image 710. The predetermined operation is, for example, an operation of moving the control image 501 in the vertical direction. When the CO control unit 310 detects an operation to move the control image 501 upward, it outputs an operation signal corresponding to the detected operation to the main control unit 123. This causes the display size of the capture range image 710 to be enlarged. When the CO control unit 310 detects an operation to move the control image 501 downward, it outputs an operation signal corresponding to the detected operation to the main control unit 123. This causes the display size of the capture range image 710 to be reduced.

[0094] Furthermore, when the DP outer camera 61 is caused to capture an image, the capture range image 710 may be displayed only on the left display unit 24, and the DP outer camera 61 may be caused to capture an image in a state in which the display position of the capture range image 710 is adjusted by operating the virtual joystick controller 500. The main control unit 123 stores the image captured by the DP outer camera 61 at this time in the non-volatile storage unit 130 as an image for the left eye. Similarly, when the DP outer camera 61 is caused to capture an image, the capture range image 710 may be displayed only on the right display unit 22, and the DP outer camera 61 may be caused to capture an image in a state in which the display position of the capture range image 710 is adjusted by operating the virtual joystick controller 500. The main control unit 123 stores the image captured by the DP outer camera 61 at this time in the non-volatile storage unit 130 as an image for the right eye.

[0095] 11 and 12 are explanatory diagrams for explaining the case where the DP outer camera 61 is operated by the virtual joystick controller 500. FIG. Next, the operation of the DP outer camera 61 by the virtual joystick controller 500 will be described with reference to FIGS. The following describes a case where the user U adjusts the imaging range of the DP outer camera 61 and the focus of the DP outer camera 61 by operating the virtual joystick controller 500. 11 and 12, a user U wearing the image display unit 20 visually recognizes a baby, who is an image capture subject 650, as an object in real space. In addition, a capture range image 730 is visually recognized as a display image displayed by the image display unit 20. The capture range image 730 is also an image that indicates the range of the captured image of the DP outer camera 61 to be captured as a capture image. The image indicated by the dashed line in Fig. 11 is the capture range image 730. The capture range image 730 shown in Fig. 11 and Fig. 12 is a rectangular image, and is a two-dimensional image having areas in the up-down and left-right directions of the display area 200, which are the X-axis and Y-axis directions. Furthermore, when the operation target of the virtual joystick controller 500 is the DP outer camera 61, the second icon 830 in the display area 200 is turned on or blinking. In Fig. 11 and Fig. 12, the display area of ​​the second icon 830 is hatched to show that the second icon 830 is selected and is blinking or turned on.

[0096] When the operation target of the virtual joystick controller 500 is the DP outer camera 61, the guide display 540 displayed on the touch panel 350 displays the capture range or focus adjustment. Even when the operation target of the virtual joystick controller 500 is the DP outer camera 61, the operation content of the virtual joystick controller 500 can be switched by a touch operation on the guidance display 540. When a touch operation on the guidance display 540 is detected while "Capture range" is displayed on the guidance display 540 of the touch panel 350, the CO control unit 310 changes the operation content of the virtual joystick controller 500 to "Focus adjustment." At this time, the display of the guidance display 540 is also changed to focus adjustment. Furthermore, when a touch operation on the guidance display 540 is detected while "Focus Adjustment" is displayed on the guidance display 540 of the touch panel 350, the CO control unit 310 changes the operation content of the virtual joystick controller 500 to "Capture Range." At this time, the display of the guidance display 540 is also changed to the capture range.

[0097] As shown in FIG. 11 , the user U moves the control image 501 of the virtual joystick controller 500 up and down or left and right to change the display position of the capture range image 730 so that the subject 650 is within the capture range image 730. Next, the user U touches the guidance display 540 to change the operation content of the virtual joystick controller 500 to focus adjustment.

[0098] FIG. 12 is a diagram showing an image displayed in the display area 200 when the operation of the virtual joystick controller 500 is changed to focus adjustment. When the operation of the virtual joystick controller 500 is changed to focus adjustment, a plurality of focus points 750 are displayed in the display area 200. The focus points 750 are adjustment images of the present invention, and are images for focus adjustment. When the main control unit 123 displays a plurality of focus points 750 in the display area 200, it displays any one of the focus points 750 in a blinking state. The user U operates the virtual joystick controller 500 to change the blinking focus point 750. By changing the blinking focus point 750, the position at which the DP outer camera 61 focuses can be changed. After changing the focus point 750, the user U presses the OK button 530 to input that the adjustment of the focus point 750 has been completed.

[0099] Next, the user U presses and holds the control image 501 of the virtual joystick controller 500 to cause the DP outer camera 61 to capture an image. When the CO control unit 310 detects an operation of pressing and holding the control image 501, it outputs an image capture instruction to the main control unit 123. When the image capture instruction is input from the CO control unit 310, the main control unit 123 causes the DP outer camera 61 to capture an image. When the main control unit 123 causes the DP outer camera 61 to capture an image, it cuts out an image of an area corresponding to the capture range image 730 from the captured image of the DP outer camera 61, and stores the cut-out image in the non-volatile storage unit 130. As a result, an image of the baby of the person to be imaged 650 is saved in the non-volatile storage unit 130. The acquisition range of the captured image is changed by cutting out an image of an area corresponding to the capture range image 730 from the captured image of the DP outer camera 61.

[0100] Furthermore, the capture range image 730 and the focus point 750 may be displayed only on the left display section 24 of the image display unit 20, and a captured image for the left eye as seen from the viewpoint of the left eye of the user U may be generated. The main control unit 123 first displays the capture range image 730 only on the left display unit 24, and the CO control unit 310 outputs operation information received by operating the virtual joystick controller 500 to the main control unit 123. The main control unit 123 changes the display position of the capture range image 730 according to the input operation information. Next, the main control unit 123 displays the focus point 750 only on the left display unit 24, and the CO control unit 310 outputs operation information received by operating the virtual joystick controller 500 to the main control unit 123. The main control unit 123 changes the position on which the DP outer camera 61 focuses based on the input operation information. Next, when the CO control unit 310 receives an image capture instruction by operating the virtual joystick controller 500, it outputs the received information to the main control unit 123. The main control unit 123 causes the DP outer camera 61 to capture an image, and causes the captured image captured by the DP outer camera 61 to be stored in the non-volatile storage unit 130 as an image for the left eye. Therefore, a captured image for the right eye adjusted for the user U wearing the image display unit 20 can be obtained.

[0101] Similarly, the capture range image 730 and the focus point 750 are displayed only on the right display section 22 of the image display section 20, and a captured image for the left eye as seen from the viewpoint of the user U's right eye is generated. The main control unit 123 first displays the capture range image 730 only on the right display unit 22, and the CO control unit 310 outputs operation information received by operating the virtual joystick controller 500 to the main control unit 123. The main control unit 123 changes the display position of the capture range image 730 according to the input operation information. Next, the main control unit 123 displays the focus point 750 only on the right display unit 22, and the CO control unit 310 outputs operation information received by operating the virtual joystick controller 500 to the main control unit 123. The main control unit 123 changes the position on which the DP outer camera 61 focuses based on the input operation information. Next, when the CO control unit 310 receives an image capture instruction by operating the virtual joystick controller 500, it outputs the received information to the main control unit 123. The main control unit 123 causes the DP outer camera 61 to capture an image, and causes the captured image captured by the DP outer camera 61 to be stored in the non-volatile storage unit 130 as an image for the right eye. Therefore, a captured image for the right eye adjusted for the user U wearing the image display unit 20 can be obtained.

[0102] [5. Operation] FIG. 13 is a flowchart showing the operation of the CO control unit. The operation of the CO control unit 310 will be described with reference to the flowchart shown in Fig. 13. Below, a case will be described in which a capture range is set by the capture range image 710 described with reference to Figs. 9 and 10, and an image of a range corresponding to the capture range image 710 is captured among images captured by the DP outer camera 61. First, the CO control unit 310 determines whether or not a display image has been displayed in the display area 200 by the image display unit 20 (step S1). For example, the CO control unit 310 determines whether or not an operation requesting the display of the capture range image 710 has been received by operating the touch panel 350. When an operation requesting the display of the capture range image 710 has been received, the CO control unit 310 determines that the capture range image 710, which is a display image, has been displayed in the display area 200 (step S1 / YES). When the capture range image 710 is not displayed in the display area 200 (step S1 / NO), the CO control unit 310 waits until the capture range image 710 is displayed.

[0103] When the capture range image 710 is displayed in the display area 200 (step S1 / YES), the CO control unit 310 causes the virtual joystick controller 500 to be displayed on the touch panel 350 (step S2). Next, the CO control unit 310 determines whether or not a change in the display position is set as the operation content of the virtual joystick controller 500 (step S3). The CO control unit 310 determines that a change in the display position is set when the guidance display 540 displayed on the touch panel 350 is touched and the display of the guidance display 540 is changed to "display position."

[0104] When the operation of the virtual joystick controller 500 is set to change the display position (step S3 / YES), the CO control unit 310 judges whether or not the operation of the virtual joystick controller 500 has been accepted (step S4). When the CO control unit 310 has not accepted the operation of the virtual joystick controller 500 (step S4 / NO), the CO control unit 310 proceeds to the judgment of step S6. When the CO control unit 310 has accepted the operation of the virtual joystick controller 500 (step S4 / YES), the CO control unit 310 changes the display position of the capture range image 710 according to the accepted operation (step S5). The CO control unit 310 outputs the operation information of the virtual joystick controller 500 to the main control unit 123. The main control unit 123 changes the display position of the capture range image 710 according to the operation information input from the CO control unit 310.

[0105] Next, the CO control unit 310 judges whether or not pressing of the OK button 530 has been accepted (step S6). If pressing of the OK button 530 has not been accepted (step S6 / NO), the CO control unit 310 returns to the judgment of step S4. If pressing of the OK button 530 has been accepted (step S6 / YES), the CO control unit 310 proceeds to the judgment of step S7.

[0106] If the determination in step S3 is a negative determination, or if the operation of the OK button 530 is accepted in step S6, the CO control unit 310 determines whether or not an operation to change the display size of the capture range image 710 is set as the operation of the virtual joystick controller 500 (step S7). If the guidance display 540 displayed on the touch panel 350 is touched and the display of the guidance display 540 is changed to "display size", the CO control unit 310 determines that a change in the display size is set.

[0107] When the operation of the virtual joystick controller 500 is set to change the display size (step S7 / YES), the CO control unit 310 judges whether or not the operation of the virtual joystick controller 500 has been accepted (step S8). When the CO control unit 310 has not accepted the operation of the virtual joystick controller 500 (step S8 / NO), the CO control unit 310 proceeds to the judgment of step S10. When the CO control unit 310 has accepted the operation of the virtual joystick controller 500 (step S10 / YES), the CO control unit 310 changes the display size of the capture range image 710 according to the accepted operation. The CO control unit 310 outputs the operation information of the virtual joystick controller 500 to the main control unit 123. The main control unit 123 changes the display size of the capture range image 710 according to the operation information input from the CO control unit 310.

[0108] Next, the CO control unit 310 judges whether or not pressing of the OK button 530 has been accepted (step S10). If pressing of the OK button 530 has not been accepted (step S10 / NO), the CO control unit 310 returns to the judgment of step S7. If pressing of the OK button 530 has been accepted (step S10 / YES), the CO control unit 310 proceeds to the judgment of step S11.

[0109] The CO control unit 310 determines whether or not an operation to switch the operation target of the virtual joystick controller 500 to the DP outer camera 61 has been accepted (step S11). For example, when the control image 501 is tapped twice, the CO control unit 310 switches the operation target of the virtual joystick controller 500 to the DP outer camera 61.

[0110] When the CO control unit 310 has not received an operation to switch the operation target of the virtual joystick controller 500 to the DP outer camera 61 (step S11 / NO), the process returns to the determination in step S3. When the CO control unit 310 has received an operation to switch to the DP outer camera 61 (step S11 / YES), the CO control unit 310 determines whether or not an operation corresponding to an image capture instruction has been received by operating the virtual joystick controller 500 (step S12).

[0111] When the CO control unit 310 has not received an operation corresponding to the imaging instruction (step S11 / NO), the process returns to the determination in step S3. When the CO control unit 310 has received an operation corresponding to the imaging instruction (step S11 / YES), the CO control unit 310 outputs an imaging instruction to the main control unit 123 (step S12). When the main control unit 123 receives an imaging instruction from the CO control unit 310, the main control unit 123 causes the DP outer camera 61 to capture an image.

[0112] When a captured image is input from the DP outer camera 61, the main control unit 123 cuts out an image of an area corresponding to the capture range image 710 from the input captured image (step S13). After that, the main control unit 123 stores the cut-out image in the non-volatile storage unit 130 (step S14). The main control unit 123 also controls the image display unit 20 to display the cut-out image in the display area 200 (step S14).

[0113] [6. Effects] As described above, the display system 1 of this embodiment includes the HMD 100 and the control device 300 connected to the HMD 100. The HMD 100 is equipped with an image display unit 20 that allows the user U to view an external scene and displays an image superimposed on the external scene, and a DP outer camera 61 mounted on the image display unit 20, and is worn on the head of a user U. The control device 300 has a touch panel 350 that accepts operations. A virtual joystick controller 500, which is an operation image that accepts operations by the user U, is displayed on the touch panel 350 of the control device 300, and the DP outer camera 61 or an adjustment image that adjusts the captured image captured by the DP outer camera 61 is displayed in the image display area 200 of the image display unit 20. The display system 1 changes the display of the adjusted image in response to the received operation on the virtual joystick controller 500, and causes the DP outer camera 61 to capture an image in response to the received operation on the virtual joystick controller 500.

[0114] Therefore, by operating the virtual joystick controller 500 displayed on the touch panel 350, the DP outer camera 61 or the captured image captured by the DP outer camera 61 can be adjusted. The virtual joystick controller 500 displayed on the touch panel 350 can accept operations corresponding to the joystick controller. For this reason, the user U touches the display position of the virtual joystick controller 500 with his / her finger, and moves the touched finger in the direction of the desired input without removing the finger from the touch panel 350. For this reason, compared with the case where the position of the key is checked every time an operation is performed, as with a cross key, the number of times the user U checks the touch panel 350 can be reduced. Therefore, the operability for the user U who wears the image display unit 20 on his / her head and visually checks the touch panel 350 through the image display unit 20 can be improved.

[0115] The adjustment image displayed in the display area 200 by the image display unit 20 is an image that adjusts the acquisition range of the captured image of the DP outer camera 61, and by operating the accepted operation image, the display position or size of the adjustment image displayed in the display area is changed to change the display of the adjustment image, thereby changing the acquisition range of the captured image. Therefore, the acquisition range of the image captured by the DP outer camera 61 can be changed by operating the virtual joystick controller 500. Therefore, the user U can change the acquisition range of the image captured by the DP outer camera 61 mounted on the image display unit 20 without moving the head of the user U. In addition, the user U can change the acquisition range of the image captured by the DP outer camera 61 by intuitive operations such as operating a joystick controller, thereby improving operability.

[0116] The adjustment image displayed in the display area by the image display unit 20 is a focus point 750 for focus adjustment having a plurality of focus positions, and the display of the adjustment image is changed by selecting the focus position for the DP outer camera 61 by operating the received operation image, thereby changing the position at which the DP outer camera 61 is focused. Therefore, the focal position of the DP outer camera 61 can be changed by operating the virtual joystick controller 500. Therefore, the user U can change the focal position of the DP outer camera 61 mounted on the image display unit 20 without moving the user U's head.

[0117] The control device 300 detects the position on the touch panel 350 that is touched by the finger of the user U, which is an indicator. The control device 300 displays the virtual joystick controller 500 based on the position on the touch panel 350 where the touch of the finger of the user U is detected. Therefore, the virtual joystick controller 500 can be displayed at a position on the touch panel 350 that is easy for the user U to operate, thereby improving operability.

[0118] The virtual joystick controller 500 has an operator image 501 whose display position changes in response to an operation. The virtual joystick controller 500 has a first display mode, a second display mode, and a third display mode as display modes. The first display mode is a mode in which the movement direction of the control image 501 is restricted to the up and down directions. The second display mode is a mode in which the movement direction of the control image 501 is restricted to the left and right direction. The third display mode is a mode in which the operator image 501 can be moved in all directions on the touch panel 350. The control device 300 causes the touch panel 350 to display a virtual joystick controller 500 in any one of the first display form, the second display form, and the third display form, corresponding to the capture range image 710 displayed in the display area 200 by the image display unit 20. Therefore, it is possible to display the virtual joystick controller 500 capable of inputting a direction that can be accepted by the capture range image 710 displayed in the display area 200.

[0119] [7. Modifications] In the above embodiment, a case has been described in which the display image to be operated by the virtual joystick controller 500 is the capture range image 710 or 730 indicating the capture range of a captured image. In this modified example, a case will be described in which the display image is associated with a real object in the real space and is displayed superimposed on the real object, and an operation to change the display position of this display image is performed by the virtual joystick controller 500.

[0120] The main control unit 123 causes the DP outer camera 61 to capture an image and detects a registered object that has been registered in advance from the captured image. When a registered object is detected from the captured image, the main control unit 123 causes the image display unit 20 to display a display image associated with the registered object. When the display image associated with the real object is displayed by the image display unit 20, the user U presses and holds the control image 501 of the virtual joystick controller 500 for a preset time or longer. When the CO control unit 310 accepts the operation of pressing and holding the control image 501, it permits the change of the display position of the display image associated with the real object. Next, the user U moves to a position where he / she wants to display the display image while pressing and holding the control image 501. When the movement of the control device 300 is detected by the sensor data of the CO 6-axis sensor 323 while the control image 501 is being pressed and held, the CO control unit 310 detects the amount of movement of the control device 300 by the sensor data of the CO 6-axis sensor 323.

[0121] When the user moves the display image to a position where the user wants it to be displayed, the user ends the long press operation of the control image 501. When the long press of the control image 501 is no longer detected, the CO control unit 310 moves the display position of the display image by the amount of movement detected from the sensor data of the CO 6-axis sensor 323.

[0122] In other words, the modified example detects a predetermined registered object contained in the external scene from an image captured by the DP outer camera 61, and when a registered object is detected, the image display unit 20 displays a display image corresponding to the registered object at a position corresponding to the registered object, and when a predetermined operation on the operation image is received, the amount of movement of the control device 300 is detected, and the display position of the display image is changed in accordance with the detected amount of movement of the control device 300. Therefore, the display position of the display image associated with the registered object can be adjusted by manipulating the possible virtual joystick controller 500.

[0123] The present invention is not limited to the configurations described in the above embodiments, and can be embodied in various forms without departing from the spirit and scope of the present invention. For example, the display system 1 is exemplified as having an HMD 100, which is a head-mounted display device, but the present invention is not limited thereto, and various display devices can be adopted. For example, instead of the image display unit 20, other types of image display units, such as an image display unit worn like a hat, may be adopted, and it is sufficient to have a display unit that displays an image corresponding to the left eye of the user U and a display unit that displays an image corresponding to the right eye of the user U. In addition, the display device may be configured as a head-mounted display mounted on a vehicle such as an automobile or an airplane. In addition, it may be configured as a head-mounted display built into a body protector such as a helmet. In this case, a part that positions the position relative to the body of the user U and a part that is positioned relative to the part can be the wearing part.

[0124] The HMD 100 is an example of a display device to which the present invention is applied, and is not limited to the configuration shown in FIG. 3. For example, in the above embodiment, the image display unit 20 and the connection device 10 are separated from each other, but the connection device 10 and the image display unit 20 may be integrally configured and attached to the head of the user U. The optical system of the image display unit 20 may be configured arbitrarily, and may use, for example, an optical member that is positioned in front of the eyes of the user U and overlaps a part or all of the field of vision of the user U. Alternatively, a scanning optical system that scans laser light or the like to produce image light may be adopted. Alternatively, the optical system may not be limited to one that guides image light inside the optical member, but may only have a function of refracting and / or reflecting the image light toward the eyes of the user U. The display device may be a liquid crystal monitor or a liquid crystal television that displays an image on a liquid crystal display panel. A display device equipped with a plasma display panel or an organic EL display panel may also be used. The display device may also be a projector that projects image light onto a screen or the like.

[0125] Also, for example, in the HMD 100 shown in FIG. 3, the connection device 10 may be configured using a USB-TypeC connector, a USB-TypeC controller, and a USB hub. In this case, the DP outer camera 61 and other sensors may be connected to the USB hub. Also, as a controller for controlling the display of the right display unit 22 and the left display unit 24 in the image display unit 20, an FPGA that outputs display data to the right display unit 22 and the left display unit 24 may be disposed in either the right display unit 22 or the left display unit 24. In this case, the connection device 10 may include a bridge controller that connects the USB-TypeC controller and the FPGA. Also, in the image display unit 20, the DP 6-axis sensor 235, the DP magnetic sensor 237, the EEPROM 215, and the like may be mounted on the same board as the FPGA. The arrangement of the other sensors can also be changed as appropriate. For example, the distance sensor 64 and the DP illuminance sensor 65 may be disposed at positions suitable for measurement or detection and connected to the FPGA and the USB-TypeC controller. Furthermore, there is no limitation on the specific specifications of the display device including the OLED units 221 and 241. For example, the OLED units 221 and 241 may have a common configuration.

[0126] At least some of the functional blocks shown in Figures 3 and 4 may be realized by hardware, or may be realized by a combination of hardware and software, and are not limited to the configuration in which independent hardware resources are arranged as shown in the figures. In addition, the program executed by the processor 311 may be configured to acquire a program stored in an external device via the communication unit 342 or the I / F unit 343 and execute the program. [Explanation of symbols]

[0127] 1...display system, 11A...connector, 11D...connector, 20...image display unit, 21...right holding unit, 22...right display unit, 23...left holding unit, 24...left display unit, 26...right light guide plate, 27...front frame, 28...left light guide plate, 30...headset, 32...right earphone, 34...left earphone, 36...audio connector, 40...connection cable, 46...USB cable, 61...DP outer camera, 63...microphone, 64...distance sensor, 65...DP illuminance sensor, 67...LED indicator, 100...HMD, 110...I / F unit, 120...DP control unit, 1 22...sensor control unit, 123...main control unit, 124...non-volatile memory, 125...processor, 125a...display control unit, 126...power control unit, 130...non-volatile storage unit, 140...operation unit, 145...connection unit, 147...audio processing unit, 201...video signal, 210...right display unit board, 211...right I / F unit, 213...receiving unit, 215...EEPROM, 217...temperature sensor, 221...OLED unit, 223...OLED panel, 225...OLED driving circuit, 229...power supply unit, 230...left display unit board, 231...left I / F unit, 233...receiving unit, 235 ...DP 6-axis sensor, 237...DP magnetic sensor, 239...temperature sensor, 241...OLED unit, 243...OLED panel, 245...OLED driving circuit, 249...power supply unit, 251...right optical system, 252...left optical system, 261...half mirror, 281...half mirror, 300...control device, 310...CO control unit, 311...processor, 312...memory, 313...non-volatile memory, 321...GNSS, 322...CO camera, 323...CO 6-axis sensor, 324...CO magnetic sensor, 325...CO illuminance sensor, 326...vibrator, 327...audio output section, 330...CO display section, 335...CO input section, 337...switch, 341...battery, 342...communication section, 343...I / F section, 350...touch panel, 500...virtual joystick controller, 501...operation image, 503A, 503B...slider bar, 505...range image, 520...back button, 530...OK button, 540...guidance display, 610...desk, 620...bottle, 650...subject, 710, 730...capture range image, 750...focus point, 810...first icon, 830...second icon.

Claims

1. A method for operating a display system including a display device worn on a user's head, the display device including a display unit capable of visually recognizing an external scene and displaying an image superimposed on the external scene, and an imaging unit mounted on the display unit, and an information processing device having an operation surface for receiving operations, the method comprising the steps of: displaying an operation image on the operation surface for receiving an operation by the user; displaying an adjustment image for adjusting the imaging unit or an image captured by the imaging unit in a display area of ​​the image on the display unit; changing a display of the adjustment image according to an operation on the accepted operation image; causing the imaging unit to perform imaging in response to an operation on the accepted operation image; Detecting a preset registered object included in the external scene from the captured image captured by the imaging unit; When the registered object is detected, the display unit displays a display image corresponding to the registered object at a position corresponding to the registered object; When a preset operation on the operation image is received, a movement amount of the information processing device is detected; A method of operation in which a display position of the display image is changed in response to the detected amount of movement of the information processing device.

2. the operation image has an operator whose display position on the operation surface is changed by an operation; 2. The operating method according to claim 1, further comprising the step of identifying the input from the user based on a display position of the operator before receiving the operation and a display position of the operator after receiving the operation.

3. the adjustment image displayed in the display area by the display unit is an image for adjusting an acquisition range of the captured image of the imaging unit, An operating method according to claim 1 or 2, further comprising changing the display position or size of the adjustment image displayed in the display area by operating the received operation image, thereby changing the display of the adjustment image and changing the acquisition range of the captured image.

4. the adjustment image displayed in the display area by the display unit is an image for focus adjustment having a plurality of focus positions, The operating method according to claim 1 , further comprising the step of changing the display of the adjustment image by selecting a focal position for the imaging unit through an operation on the received operation image, thereby changing a position for the imaging unit to focus.

5. Detecting a position on the operation surface that is touched by a pointer; The operating method according to claim 1 , further comprising displaying the operation image based on the detected position of the operation surface.

6. the operation image has an operator whose display position is changed in response to an operation; a first display mode in which a moving direction of the operator is limited to a first direction; a second display mode in which a moving direction of the operator is limited to a second direction different from the first direction; a third display mode in which the operator is movable in all directions on the operation surface; An operating method according to any one of claims 1 to 5, further comprising displaying an operation image in any one of the first display form, the second display form, and the third display form on the operating surface in response to the adjustment image displayed in the display area by the display unit.

7. A method for operating a display system including a display device worn on a user's head, the display device including a display unit capable of visually recognizing an external scene and displaying an image superimposed on the external scene, and an imaging unit mounted on the display unit, and an information processing device having an operation surface for receiving operations, the method comprising the steps of: displaying an operation image on the operation surface for receiving an operation by the user; displaying an adjustment image for adjusting the imaging unit or an image captured by the imaging unit in a display area of ​​the image on the display unit; changing a display of the adjustment image according to an operation on the accepted operation image; causing the imaging unit to perform imaging in response to an operation on the accepted operation image; The display unit includes a left display unit that allows the user to view an image with the left eye and a right display unit that allows the user to view an image with the right eye, receiving an operation on the operation image while the adjustment image is displayed on the left display portion, and changing the display of the adjustment image based on the received operation; accepting an operation on the operation image, and causing the imaging unit to perform imaging based on the accepted operation to obtain a captured image for a left eye; receiving an operation on the operation image while the adjustment image is displayed on the right display portion, and changing the display of the adjustment image based on the received operation; an operation method for receiving an operation on the operation image, and causing the imaging unit to perform imaging based on the received operation to obtain a captured image for a right eye;

8. A program to be executed by a computer equipped in an information processing device connected to a display device attached to a user's head and having an operation surface for receiving operations, displaying an operation image on the operation surface for receiving an operation by the user; displaying an image pickup unit mounted on the display unit or an adjustment image for adjusting an image picked up by the image pickup unit in a display area of ​​the image on the display unit; changing the display of the adjustment image according to the received operation on the operation image; causing the imaging unit to perform imaging in response to an operation on the accepted operation image; detecting a preset registered object included in an external scene visually recognized by the user wearing the display unit from a captured image captured by the imaging unit; When the registered object is detected, the display unit displays a display image corresponding to the registered object at a position corresponding to the registered object; detecting an amount of movement of the information processing device when a preset operation on the operation image is received; A program that changes a display position of the display image in response to the detected amount of movement of the information processing device.

9. A program to be executed by a computer equipped with an information processing device having an operation surface that accepts operations, the information processing device being connected to a display device attached to a head of a user, the display unit including a left display unit that allows a user to view an image with the left eye of the user and a right display unit that allows the user to view an image with the right eye of the user, the program comprising: displaying an operation image on the operation surface for receiving an operation by the user; displaying an image pickup unit mounted on the display unit or an adjustment image for adjusting an image picked up by the image pickup unit in a display area of ​​the image on the display unit; changing the display of the adjustment image according to the received operation on the operation image; causing the imaging unit to perform imaging in response to an operation on the accepted operation image; receiving an operation on the operation image while the adjustment image is displayed on the left display portion, and changing the display of the adjustment image based on the received operation; accepting an operation on the operation image, and causing the imaging unit to perform imaging based on the accepted operation to obtain a captured image for a left eye; receiving an operation on the operation image while the adjustment image is displayed on the right display portion, and changing the display of the adjustment image based on the received operation; A program that receives an operation on the operation image, and causes the imaging unit to capture an image based on the received operation to obtain a captured image for the right eye.

Images