Information processing system, display unit, and program

JP2023124095A5Inactive Publication Date: 2024-11-29SEIKO EPSON CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2022027686
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2022-02-25
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing head-mounted display systems require users to move their heads to gaze at objects, compromising operability and convenience.

Method used

A head-mounted display system that includes a transmissive display unit and an eye movement detection system to identify objects in real space, allowing the display of captured images from multiple cameras based on the user's gaze, enabling direct viewing and improved operability.

Benefits of technology

Enhances user operability by allowing direct viewing of real space while displaying relevant images without requiring head movement, improving convenience and visual recognition of objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To allow a user of a display unit to easily watch a video related to what the user gazes at.SOLUTION: An information processing system 1 comprises: an HMD 100 having a transmission type display that is mounted on the head of a user U, and a detection unit that detects the movement of the eyes of the user U; and a management server 300 having a specification unit that, based on the movement of the eyes of the user U, specifies an object in a real space that the user U gazes at through the display, and an imaging control unit that causes a first imaging apparatus, of imaging apparatuses 201, which is at a position where it can pick up an image of a first portion of the object to pick up an image of a first imaging range including the first portion. The HMD 100 causes the display to display a first picked-up image picked up by the first imaging apparatus.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an information processing system, a display device, and a program.

Background Art

[0002] Conventionally, a technique for displaying a captured image of a camera on a display device is known. Patent Document 1 discloses a system that displays a video captured by a camera at a position away from a user using a transmissive head-mounted display device. This system selects a captured image based on the movement of the user's head from captured images captured by a plurality of cameras, and displays it using the head-mounted display device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The configuration disclosed in Patent Document 1 selects a video based on the movement of the user's head. Therefore, when the user wants to display a video related to what the user is gazing at, the user has to move the head in the direction of gazing. Accordingly, an improvement in operability has been demanded.

Means for Solving the Problems

[0005] One aspect of the present disclosure is an information processing system comprising: a display device having a transparent display unit worn on the user's head and a detection unit for detecting the user's eye movements; an identification unit for identifying an object in real space that the user is looking at through the display unit based on the user's eye movements; and an imaging control unit for causing a first imaging device, positioned in a location capable of imaging a first portion of the object, to image a first imaging range including the first portion, wherein the display device causes a first image captured by the first imaging device to be displayed on the display unit.

[0006] Another aspect of the present disclosure is a display device comprising: a transparent display unit worn on the head of a user; a detection unit for detecting the eye movements of the user; and a display control unit that, based on the eye movements of the user detected by the detection unit, identifies an object in real space that the user is looking at through the display unit, and when a first image is acquired that captures a first imaging range including a first portion of the object, the display control unit displays the first image on the display unit.

[0007] A further aspect of the present disclosure is a program executable by a computer that controls a display device having a transparent display unit worn on the head of a user, the program causing the computer to function as a display control unit that, based on the user's eye movements, identifies a real-space object that the user is looking at through the display unit, and when a first image is acquired that captures a first imaging range including a first portion of the object, the first image is displayed on the display unit. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing the configuration of the information processing system according to the first embodiment. [Figure 2] External view of the HMD. [Figure 3] Block diagram of the HMD. [Figure 4] Block diagram of an information processing system. [Figure 5] A flowchart illustrating the operation of an information processing system. [Figure 6] A sequence diagram illustrating the operation of an information processing system. [Figure 7] A sequence diagram illustrating the operation of an information processing system. [Figure 8] A diagram showing an example of display on an HMD. [Figure 9] A diagram showing an example of display on an HMD. [Figure 10] Block diagram of the HMD of the second embodiment. [Modes for carrying out the invention]

[0009] [1. First Embodiment] [1-1. Configuration of the Information Processing System] Figure 1 is a diagram showing the configuration of the information processing system 1 of the first embodiment. Information processing system 1 provides video to the HMD 100 used by user U. The HMD 100 is a head-mounted display device worn on user U's head. By wearing the HMD 100, user U can view video with both hands free. HMD stands for Head Mounted Display.

[0010] The HMD100 includes a communication device 141, which will be described later with reference to Figure 4, and connects to the communication network N via the communication device 141. The communication network N may be a LAN (Local Area Network) or a WAN (Wide Area Network). It may also be a global network consisting of dedicated lines, public telephone networks, the Internet, etc. The communication network N may also include equipment such as wireless base stations, servers, gateways, wireless access points, and routers.

[0011] The information processing system 1 includes an imaging unit 200. The imaging unit 200 includes one or more imaging devices 201. There is no limit to the number of imaging devices 201 included in the imaging unit 200. In this embodiment, the imaging unit 200 is exemplified as having an imaging device 201 that includes at least a first camera 210, a second camera 220, a third camera 230, a fourth camera 240, and a fifth camera 250. In the following description, the first camera 210, the second camera 220, the third camera 230, the fourth camera 240, the fifth camera 250, and other imaging devices will be referred to as imaging device 201 when they are not distinguished.

[0012] Each imaging device 201 is connected to a communication network N. One of the imaging devices 201, including the first camera 210, second camera 220, third camera 230, fourth camera 240, and fifth camera 250, corresponds to an example of the first imaging device, and the other corresponds to an example of the second imaging device. The video captured by the first imaging device corresponds to the first captured image, and the video captured by the second imaging device corresponds to the second captured image.

[0013] The information processing system 1 includes a management server 300. The management server 300 is connected to a communication network N and communicates with the imaging device 201 and the HMD 100 via the communication network N. The management server 300 acquires the images captured by the imaging device 201 and transmits them to the HMD 100. The management server 300 corresponds to an example of an information processing device.

[0014] The imaging device 201 only needs to have the function of performing imaging. There are no restrictions on the location or method of installation of the imaging device 201, but it is preferable that each imaging device 201 be installed at a distance from each other.

[0015] In this embodiment, an example in which the imaging device 201 is arranged so as to be able to image a specific area WS will be described. The area WS is a geographically delimited area. The user U wears the HMD 100 to monitor, observe, manage, etc. the work in the area WS. The area WS is, for example, a work area, a ground, court or field where a sports game is held, an event venue where an official or entertainment event is held, a park or entertainment facility, an area set in an urban area, etc. The area WS may be outdoors or indoors. Examples of the work area include a construction site including a building site, a port where cargo handling work is performed, a warehouse where cargo sorting is performed, a parking lot, a farm, a manufacturing factory, a repair or maintenance factory, but it may also be a place where other work is performed.

[0016] The first camera 210, the second camera 220, the third camera 230, the fourth camera 240, and the fifth camera 250 may be configured to image with a fixed angle of view, but it is preferable that they are configured to be able to adjust the operating conditions. The adjustment of the operating conditions includes adjustment of the angle of view or adjustment of the imaging conditions. The adjustment of the angle of view is, for example, moving the angle of view in at least one of the directions of pan, tilt, and roll. Also, the adjustment of the imaging conditions is at least one of changing the zoom ratio and focus adjustment.

[0017] FIG. 1 shows, as an example, the first camera 210 installed on the wall surface W and the second camera 220 movable along the guide rail G. The first camera 210 is a camera fixed to the wall surface W, and for example, the angle of view and focus adjustment are possible. The wall surface W is not limited to the wall of a building, and may be the wall surface of a heavy machine or a vehicle. Focus adjustment is an example of adjustment of the focal length.

[0018] The guide rail G is a rail fixed to the side, ceiling, or top surface of a building, heavy machinery, vehicle, etc. The guide rail G may be straight or curved. The second camera 220 has a body 290 that fits onto the guide rail G. The body 290 is equipped with a traveling device having a motor and rollers. The second camera 220 can move along the guide rail G in directions M1 and M2 by operating the traveling device. The second camera 220 can travel along the guide rail G according to instructions, for example, when the direction and amount of movement are specified. The second camera 220 can adjust its field of view.

[0019] The specific configurations of the third camera 230, the fourth camera 240, the fifth camera 250, and the other imaging devices 201 are arbitrary. For example, they may be configured to be fixed to a wall surface, as with the first camera 210, or they may be configured to be movable along a fixed guide rail, as with the second camera 220.

[0020] The imaging unit 200 preferably includes an imaging device 201 installed in a position capable of imaging the entire area WS. That is, it is preferable that most of the area WS can be imaged by any of the imaging devices 201.

[0021] Furthermore, the third camera 230, the fourth camera 240, and the fifth camera 250 may be configured to move autonomously in multiple directions. Specifically, they may be flying vehicles equipped with propellers, etc., or driving vehicles equipped with tires and steering mechanisms. These may be so-called drones. They may also fly or drive autonomously. However, fixed cameras like the first camera 210, or cameras that move along fixed rails like the second camera 220, do not pose a concern for getting too close to people or work vehicles. For this reason, in places where people are densely packed or where machinery is operating, a configuration like the first camera 210 or the second camera 220 is preferable.

[0022] The HMD100 is a transmissive display device that transmits ambient light. Therefore, while wearing the HMD100, user U can see the outside world through the HMD100. In other words, user U can directly perceive real space with their own eyes. This state of perception is called direct viewing. Real space refers to the actual three-dimensional space.

[0023] The information processing system 1 transmits the image captured by the imaging unit 200 to the HMD 100, allowing the user U to view the image captured by the imaging unit 200. The management server 300 transmits the image captured by the imaging unit 200 of an object located in the direction that the user U is looking to the HMD 100.

[0024] Each imaging device 201 can be installed at a location separate from the HMD 100. The imaging device 201 is configured as a separate unit from the HMD 100. User U can approach the location where the imaging device 201 is installed, but if user U is away from the imaging device 201, the imaging device 201 will image the object from a different position and / or direction than user U. Therefore, if user U can directly view the object, user U can view the image captured from a different position or direction than user U through the HMD 100.

[0025] [1-2. HMD Configuration] Figure 2 is an external view of the HMD100, and Figure 3 is a block diagram of the HMD100. The configuration of the HMD100 will be explained with reference to these figures.

[0026] The HMD100 comprises a display unit 20 worn on the user U's head and a controller 10 connected to the display unit 20. The controller 10 may be integrated with the display unit 20. The controller 10 functions as a control device for the user U to operate the HMD100.

[0027] The display unit 20, when worn on the user U's head, allows the user U to view a virtual image. The display unit 20 is an optically transmitted display device that allows the user to view the virtual image while simultaneously directly viewing the external scenery. The external scenery refers to the view outside of user U wearing the HMD 100, and refers to the real-world scenery that is visible to the naked eye even when user U is not wearing the display unit 20. User U can also be considered the operator of the HMD 100. In the following explanation, the virtual image that the display unit 20 allows user U to view will be referred to as the displayed image for convenience. When the HMD 100 emits image light based on image data from the display unit 20, it is expressed as displaying an image. The term "image" is not limited to still images, but also includes moving images or videos.

[0028] The controller 10 has a box-shaped main body 11. The main body 11 is equipped with various switches and the like as operators that receive operations from the user U. The display unit 20 has a spectacle shape in this embodiment. The display unit 20 has a main body composed of a right holding part 21, a left holding part 23, and a front frame 27. The main body of the display unit 20 is provided with a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28.

[0029] The right holding portion 21 and the left holding portion 23 extend rearward from both ends of the front frame 27 and hold the display unit 20 on the user U's head. Of the ends of the front frame 27, the end located on the right side of the user U when the display unit 20 is attached is designated as end ER, and the end located on the left side is designated as end EL.

[0030] The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. When the display unit 20 is mounted, the right light guide plate 26 is positioned in front of the user U's right eye RE. The left light guide plate 28 is positioned in front of the user U's left eye LE.

[0031] The right display unit 22 and the left display unit 24 are modules that integrate an optical unit and peripheral circuits. The right display unit 22 is attached to the right holding unit 21, and the left display unit 24 is attached to the left holding unit 23. The right display unit 22 and the left display unit 24 are equipped with, for example, an organic EL (Electro-Luminescence) display device or a liquid crystal display device, and emit image light based on image data.

[0032] The right light guide plate 26 and the left light guide plate 28 are optical components formed from a light-transmitting resin or the like. For example, the right light guide plate 26 and the left light guide plate 28 are prisms. The right light guide plate 26 guides the image light output by the right display unit 22 toward the user U's right eye RE. The left light guide plate 28 guides the image light output by the left display unit 24 toward the user U's left eye LE. As a result, image light enters both of the user U's eyes, allowing the user U to view the image.

[0033] In user U's right eye RE, image light guided by the right light guide plate 26 and ambient light transmitted through the right light guide plate 26 enter the eye. In user U's left eye LE, image light guided by the left light guide plate 28 and ambient light transmitted through the left light guide plate 28 enter the eye. In other words, the HMD100 superimposes the image light corresponding to the image processed internally with ambient light and directs it into user U's eye. User U can see the outside world by passing through the right light guide plate 26 and the left light guide plate 28. Furthermore, user U can see the image created by the image light superimposed on the outside world.

[0034] An illuminance sensor 65 is positioned on the front frame 27. The illuminance sensor 65 receives ambient light from in front of the user U to whom the display unit 20 is attached. The external camera 61 is equipped with an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor). The external camera 61 is a digital camera that captures an image range including the area in front of the user U. The external camera 61 may be a monocular camera or a stereo camera.

[0035] An LED (Light Emitting Diode) indicator 67 is positioned on the front frame 27. The LED indicator 67 is located near the outer camera 61 at the end ER and lights up while the outer camera 61 is operating to indicate that imaging is in progress.

[0036] A distance sensor 64 is provided on the front frame 27. The distance sensor 64 detects the distance from the display unit 20 to the object to be measured located in front of the user U. The object to be measured is a real object or structure located in real space. The distance sensor 64 is, for example, a light reflection type distance sensor, an ultrasonic type distance sensor, or a laser range scanner.

[0037] The controller 10 and the display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to the connector 42 of the main unit 11. The connection cable 40 is equipped with an audio connector 46. The headset 30 is connected to the audio connector 46. The headset 30 has a right earphone 32 and a left earphone 34 that constitute stereo headphones, and a microphone 63.

[0038] The right earphone 32 is worn on the user U's right ear. The left earphone 34 is worn on the user U's left ear. The right earphone 32 and the left earphone 34 are either in-ear type earphones or canal type earphones. The right earphone 32 and the left earphone 34 may also be over-ear headphones that contact the auricle via earmuffs. The right earphone 32 and the left earphone 34 output sound based on the audio signal output by the audio interface 181, which will be described later.

[0039] Microphone 63 collects sound and outputs the audio signal to the audio interface 181. Microphone 63 may be a mono microphone or a stereo microphone. Microphone 63 may be a directional microphone or an omnidirectional microphone.

[0040] The controller 10 includes a wheel operation unit 12, a center key 13, an operation pad 14, up / down keys 15, an LED display unit 17, and a power switch 18. These can be considered operable parts operated by the user U. These operable parts are located on the surface of the main unit 11. These operable parts are operated, for example, by the fingers of the user U.

[0041] An LED display unit 17 is installed on the main unit 11. The LED display unit 17 is an LED indicator that shows the operating status of the HMD 100. The LED display unit 17 is covered by a light-transmitting part. The cover of the LED display unit 17 forms part of the surface of the main unit 11. When the LED display unit 17 emits light, this light passes through the light-transmitting part. This makes characters, symbols, patterns, etc. formed on the light-transmitting part visible. A touch sensor that detects contact of the user U's fingers is placed on top of the light-transmitting part of the LED display unit 17. The combination of the LED display unit 17 and the touch sensor functions as a software key.

[0042] Power switch 18 is a switch that turns the HMD100's power on and off. The main unit 11 is equipped with a USB connector 19 as an interface for connecting the controller 10 to an external device. USB stands for Universal Serial Bus.

[0043] Figure 3 is a block diagram showing the configuration of each part that makes up the HMD100. The HMD100 includes a control unit 120, a communication device 141, a GNSS 142, a display interface 143, and an audio interface 144. These are housed, for example, in the main body 11 of the controller 10.

[0044] The communication device 141 is a device including a transmitter and a receiver connected to the communication network N. The communication device 141 communicates with the management server 300 via the communication network N. The communication device 141 performs wireless communication using cellular communication methods such as 3G communication, 4G communication, LTE (Long Term Evolution) communication, and 5G communication. The communication device 141 may also be a communication device that performs short-range wireless communication. For example, the communication device 141 may be configured to perform wireless communication compliant with standards such as Bluetooth and Wi-Fi. Bluetooth is a registered trademark. Wi-Fi is a registered trademark.

[0045] The GNSS (Global Navigation Satellite System) 142 is a device that detects the current position of the HMD 100. The GNSS 142 includes, for example, a GNSS receiver that receives positioning signals transmitted from multiple positioning satellites.

[0046] The display interface 143 includes, for example, a connector 42, and connects the controller 10 and the display unit 20 by a connecting cable 40. The audio interface 144 includes, for example, an audio connector 46, and connects the right earphone 32, the left earphone 34, and the microphone 63 to the controller 10.

[0047] The control unit 120 includes a processor 121 and a memory 130. The processor 121 is composed of, for example, a CPU (Central Processing Unit), an MPU (Micro-processing unit), or other integrated circuits. The processor 121 controls each part of the HMD 100 by executing a program.

[0048] Memory 130 is a storage device that non-volatilely stores programs and data executed by the processor 121. Memory 130 is composed of a magnetic storage device, a semiconductor storage element such as flash ROM (Read Only Memory), or other types of non-volatile storage devices. Memory 130 may also include RAM (Random Access Memory) which constitutes the work area of ​​the processor 121. Memory 130 stores data processed by the processor 121 and control programs 131 executed by the processor 121.

[0049] The display unit 20 includes a right eye display unit 151, a left eye display unit 152, a gaze detection device 153, and a motion sensor 154. Furthermore, as described above, the display unit 20 also includes an external camera 61, a distance sensor 64, an illuminance sensor 65, and an LED indicator 67.

[0050] The right eye display unit 151 is a display unit that displays an image to the user U's right eye RE. Specifically, the right eye display unit 151 includes a right display unit 22 and a right light guide plate 26. The right eye display unit 151 may also include other peripheral circuits and optical components. The left eye display unit 152 is a display unit that displays an image to the user U's left eye LE. Specifically, the left eye display unit 152 includes a left display unit 24 and a left light guide plate 28. The left eye display unit 152 may also include other peripheral circuits and optical components.

[0051] The gaze detection device 153 detects the eye movements of the user U. The gaze detection device 153 determines the direction of gaze by detecting eye movements. The gaze detection device 153 includes a sensor or camera for detecting eye movements. The gaze detection device 153 corresponds to an example of a detection unit.

[0052] The gaze detection device 153 specifically includes a camera that images at least one of the user U's right eye RE and left eye LE. This camera is positioned, for example, on the front frame 27 in the direction that the user U's face is facing when the user U is wearing the display unit 20, and it images the user U's eyes. The display unit 20 includes, for example, a pair of left and right cameras. These pairs of cameras each image the user U's right eye RE and the user U's left eye LE, respectively.

[0053] The gaze detection device 153 may be configured to include one camera and capture images of either the user U's right eye RE or left eye LE. Furthermore, the position in which the camera of the gaze detection device 153 is installed is not limited. For example, the display unit 20 may be configured to capture images of either the right eye RE or the left eye LE from outside the right light guide plate 26 or the left light guide plate 28. Additionally, the gaze detection device 153 may have a configuration different from the camera for detecting the user U's eye movements.

[0054] The motion sensor 154 detects the movement of the display unit 20. The motion sensor 154 can also be called a motion sensor or inertial sensor. The motion sensor 154 includes, for example, at least one of a 3-axis accelerometer, a 3-axis gyroscope, or a magnetic sensor. The motion sensor 154 may also be an IMU (Inertial Measurement Unit) in which multiple sensors are modularized.

[0055] The processor 121 functions as a communication control unit 122, a display control unit 123, and a detection control unit 124 by reading and executing the control program 131 stored in the memory 130.

[0056] The communication control unit 122 controls the communication device 141 to send and receive data and various signals with the management server 300.

[0057] The display control unit 123 controls the right eye display unit 151 and the left eye display unit 152 to display images based on data received from the management server 300 via the communication device 141. When the display control unit 123 receives data from the management server 300 specifying the display mode of the image, it displays the image according to this data.

[0058] The detection control unit 124 acquires detection values ​​from the gaze detection device 153, motion sensor 154, external camera 61, distance sensor 64, and illuminance sensor 65. At least a portion of the detection values ​​acquired by the detection control unit 124, or data generated by the detection control unit 124 based on at least a portion of the acquired detection values, is transmitted to the management server 300 by the communication control unit 122.

[0059] The detection control unit 124 calculates the gaze direction of at least one of the right eye RE and left eye LE based on the detection result of the gaze detection device 153, and identifies the user U's point of fixation based on the calculated gaze direction. The user U's point of fixation is the position in real space that the user U is fixating on. In addition to the gaze direction, the detection control unit 124 may also identify the point of fixation by determining the fixation distance. The fixation distance refers to the distance to the position that the user U is fixating on. The position of the user U's point of fixation can be expressed, for example, as a relative three-dimensional position with respect to the position of the display unit 20.

[0060] Furthermore, the detection control unit 124 acquires location information indicating the current position of the HMD 100 detected by the GNSS 142.

[0061] The detection control unit 124 generates posture information indicating the orientation of the display unit 20 based on the detection value of the motion sensor 154. The posture information indicates the orientation of the head of the user U wearing the display unit 20.

[0062] The detection control unit 124 generates gaze point information, which includes information indicating the gaze point, location information, and orientation information. The gaze point information is transmitted to the management server 300 by the communication control unit 122.

[0063] Furthermore, the detection control unit 124 may perform SLAM (Simultaneous Localization and Mapping) processing using at least one of the external camera 61 and the distance sensor 64. The detection control unit 124 creates an environmental map of objects around the user U wearing the display unit 20 and identifies the user U's own position on the environmental map. The distance sensor 64 is a sensor for SLAM processing and specifically constitutes a LiDAR (Light Detection and Ranging) system. Alternatively, the detection control unit 124 may generate data for the management server 300 to perform SLAM processing. The results of the SLAM processing performed by the detection control unit 124, or the SLAM processing data generated by the detection control unit 124, may be transmitted to the management server 300 by the communication control unit 122.

[0064] [1-3. Configuration of the imaging device] Figure 4 is a block diagram of the information processing system 1, showing in detail the configuration of the imaging unit 200 and the management server 300. First, the configuration of the imaging unit 200 will be explained.

[0065] Figure 4 shows the configuration of the first camera 210 and the second camera 220 as part of the imaging device 201. The third camera 230, the fourth camera 240, the fifth camera 250, and other imaging devices 201 can be configured in the same way as the first camera 210 and the second camera 220.

[0066] The first camera 210 includes a first camera control unit 211, a first imaging unit 212, a first field of view adjustment unit 213, a focus adjustment mechanism 215, and a first camera communication device 216. The first camera control unit 211 includes a processor such as a microcontroller, MPU, or CPU, and controls various parts of the first camera 210. The first camera control unit 211 may be a processor that executes programs, or it may be programmed hardware.

[0067] The first imaging unit 212 includes an image sensor such as a CMOS or CCD, and an imaging optical unit including an imaging lens, and performs imaging according to the control of the first camera control unit 211 and outputs an image. The first imaging unit 212 may perform imaging using visible light or using light outside the visible range. For example, the first imaging unit 212 may perform imaging using infrared light. The first imaging unit 212 may also include a depth sensor. In this case, the first imaging unit 212 outputs the image captured using visible light and / or light outside the visible range in association with depth data detected by the depth sensor.

[0068] The first field of view adjustment unit 213 adjusts the imaging range, i.e., the field of view, captured by the first imaging unit 212. Specifically, the first field of view adjustment unit 213 has a drive unit that displaces the field of view in at least one of the directions of pan, tilt, or roll. Examples of the drive unit include an actuator or a motor. The first field of view adjustment unit 213 includes, for example, an actuator 214. The first field of view adjustment unit 213 rotates or moves the imaging optical unit of the first imaging unit 212 by operating the actuator 214 according to the control of the first camera control unit 211. The first field of view adjustment unit 213 may also be an image processing device that adjusts the field of view by performing image processing, including cropping and rotation, on the image captured by the first imaging unit 212.

[0069] The focus adjustment mechanism 215 includes a drive unit (not shown) that operates the imaging optical unit of the first imaging unit 212. Examples of the drive unit include an actuator or a motor. The focus adjustment mechanism 215 performs focus adjustment of the first imaging unit 212 in accordance with the control of the first camera control unit 211.

[0070] The first camera communication device 216 is a device that includes a transmitter and a receiver connected to the communication network N. The first camera communication device 216 communicates with the management server 300 via the communication network N. The communication device 141 performs wireless communication using a cellular communication method such as 3G, 4G, LTE, and 5G. The first camera communication device 216 may also be a communication device that performs short-range wireless communication. For example, the first camera communication device 216 may be configured to perform wireless communication compliant with standards such as Bluetooth and Wi-Fi.

[0071] The first camera control unit 211 controls the first imaging unit 212 to perform imaging and generates video data or still image data based on the image output by the first imaging unit 212. The first imaging unit 212 controls the first camera communication device 216 to transmit the video data or still image data to the management server 300. If the first imaging unit 212 is equipped with a depth sensor, the first camera control unit 211 associates depth data with the video data or still image data and transmits it to the management server 300.

[0072] The first camera control unit 211 controls the first field of view adjustment unit 213 and the focus adjustment mechanism 215 to adjust the field of view and focus of the first camera 210. The first camera control unit 211 receives data from the management server 300 via the first camera communication device 216 that instructs the adjustment of the field of view and / or focus adjustment. Based on the received data, the first camera control unit 211 controls the first field of view adjustment unit 213 and the focus adjustment mechanism 215.

[0073] The second camera 220 includes a second camera control unit 221, a second imaging unit 222, a second field of view adjustment unit 223, a movement unit 225, and a second camera communication device 227. The second camera control unit 221 includes a processor such as a microcontroller, MPU, or CPU, and controls various parts of the second camera 220. The second camera control unit 221 may be a processor that executes programs, or it may be programmed hardware.

[0074] The second imaging unit 222 includes an image sensor such as a CMOS or CCD, and an imaging optical unit including an imaging lens, and performs imaging according to the control of the second camera control unit 221 and outputs an image. The second imaging unit 222 may perform imaging using visible light or using light outside the visible range. For example, the second imaging unit 222 may perform imaging using infrared light. The second imaging unit 222 may also include a depth sensor. In this case, the second imaging unit 222 outputs the image captured using visible light and / or light outside the visible range in association with depth data detected by the depth sensor.

[0075] The second field of view adjustment unit 223 adjusts the imaging range, i.e., the field of view, captured by the second imaging unit 222. Specifically, the second field of view adjustment unit 223 has a drive unit that displaces the field of view in at least one of the following directions: pan, tilt, or roll. Examples of the drive unit include an actuator or a motor. The second field of view adjustment unit 223 includes, for example, an actuator 224. The second field of view adjustment unit 223 rotates or moves the imaging optical unit of the second imaging unit 222 by operating the actuator 224 according to the control of the second camera control unit 221. The second field of view adjustment unit 223 may also be an image processing device that adjusts the field of view by performing image processing, including cropping and rotation, on the image captured by the second imaging unit 222.

[0076] The moving unit 225 includes a drive motor 226 housed in the main body 290, and rollers (not shown) driven by the drive motor 226. The rollers of the moving unit 225 are fitted onto the guide rail G. The moving unit 225 moves the second camera 220 along the guide rail G by operating the drive motor 226 according to the control of the second camera control unit 221. The moving unit 225 also switches the direction of movement of the second camera 220 between movement direction M1 and movement direction M2 by switching the rotation direction of the drive motor 226.

[0077] The second camera communication device 227 is a device that includes a transmitter and a receiver connected to the communication network N. The second camera communication device 227 communicates with the management server 300 via the communication network N. The communication device 141 performs wireless communication using a cellular communication method such as 3G, 4G, LTE, and 5G. The second camera communication device 227 may also be a communication device that performs short-range wireless communication. For example, the second camera communication device 227 may be configured to perform wireless communication compliant with standards such as Bluetooth and Wi-Fi.

[0078] The second camera control unit 221 controls the second imaging unit 222 to perform imaging and generates video data based on the images captured by the second imaging unit 222. The second imaging unit 222 controls the second camera communication device 227 to transmit the video data to the management server 300.

[0079] The second camera control unit 221 controls the second imaging unit 222 to perform imaging and generates video data or still image data based on the image output by the second imaging unit 222. The second imaging unit 222 controls the first camera communication device 216 to transmit the video data or still image data to the management server 300. If the second imaging unit 222 is equipped with a depth sensor, the second camera control unit 221 associates depth data with the video data or still image data and transmits it to the management server 300.

[0080] The second camera 220 may be equipped with a focus adjustment mechanism similar to that of the focus adjustment mechanism 215. In this case, the second camera control unit 221 controls the focus adjustment mechanism to perform focus adjustment of the second imaging unit 222.

[0081] The third camera 230, the fourth camera 240, the fifth camera 250, and other imaging devices 201 are configured in the same manner as the first camera 210 and the second camera 220.

[0082] [1-4. Configuration of the Management Server] The management server 300 comprises a server control unit 310 and a server communication device 350. The server control unit 310 comprises a processor 320 and memory 330. The processor 320 is composed of, for example, a CPU, MPU, or other integrated circuit. The processor 320 controls the various parts of the HMD 100 by executing programs.

[0083] Memory 330 is a non-volatile storage device that stores programs and data executed by the processor 320. Memory 130 is composed of a magnetic storage device, a semiconductor storage element such as flash ROM, or other types of non-volatile storage devices. Memory 330 may also include RAM that constitutes the work area of ​​the processor 320. Memory 330 stores data processed by the processor 320 and control programs 331 executed by the processor 320.

[0084] The server communication device 350 is a device including a transmitter and receiver connected to the communication network N. The server communication device 350 communicates with the HMD 100 and the imaging device 201 via the communication network N. The server communication device 350 performs wireless communication using cellular communication methods such as 3G, 4G, LTE, and 5G. The communication device 141 may be a communication device that performs short-range wireless communication. For example, the communication device 141 may be configured to perform wireless communication compliant with standards such as Bluetooth and Wi-Fi. The server communication device 350 corresponds to an example of a communication unit.

[0085] The processor 320 functions as a communication control unit 321, a position management unit 322, a specific unit 323, and a video management unit 324 by reading and executing a control program 331 stored in the memory 330.

[0086] The communication control unit 321 transmits and receives various data and signals to and from the HMD 100 via the server communication device 350. The communication control unit 321 also transmits and receives various data and signals to and from the first camera 210, second camera 220, third camera 230, fourth camera 240, and fifth camera 250 via the server communication device 350.

[0087] The position management unit 322 detects the position of an object in the area where the imaging unit 200 is installed. For example, the position management unit 322 identifies the positions of the first camera 210, the second camera 220, the third camera 230, the fourth camera 240, the fifth camera 250, and other imaging devices 201 in area WS. These positions may be determined in advance. The position management unit 322 generates map data 332 that three-dimensionally maps the positions of the imaging devices 201 in area WS and stores it in memory 330. The map data 332 may be created in advance by another device and stored in memory 330.

[0088] The position management unit 322 identifies the position and shape of objects located in area WS by analyzing video or still images captured by the imaging device 201. The position management unit 322 reflects the identified position and shape of objects in map data 332. To identify the position and shape of objects in area WS, the position management unit 322 performs, for example, SLAM processing. In SLAM processing, for example, at least one of still image data, video data, and depth data received from the imaging device 201 is used. In addition to the imaging device 201, the information processing system 1 may also include a LiDAR system including a laser range scanner. In this case, the position management unit 322 may use LiDAR to identify the position and shape of objects in area WS.

[0089] The position management unit 322 identifies the position of user U and reflects the identified position in the map data 332. More preferably, the position management unit 322 identifies the orientation of the display unit 20 and reflects the identified orientation in the position management unit 322. For example, the position management unit 322 uses position information transmitted by the HMD 100 to identify the position of user U and the orientation of the display unit 20. The position management unit 322 may also use images captured by the external camera 61 of the HMD 100 to identify the position of user U and the orientation of the display unit 20. Alternatively, the position management unit 322 may use attitude information generated by the HMD 100 based on the distance measured by the distance sensor 64 and / or the detection results of the motion sensor 154 to identify the position of user U and the orientation of the display unit 20.

[0090] The map data 332 includes information that identifies the position and orientation of at least the imaging device 201, the display unit 20, and other objects in area WS. For example, the map data 332 includes information indicating the imaging direction and position of the first camera 210, the second camera 220, the third camera 230, the fourth camera 240, and the fifth camera 250. Also, for example, the position management unit 322 includes information indicating the orientation and position of the display unit 20. Also, for example, the position management unit 322 includes information indicating the orientation, shape, and position of objects other than the imaging unit 200 and the user U. The information included in the map data 332 is, for example, information that represents the three-dimensional position of an object in area WS as coordinates in a common coordinate system. Also, the information included in the map data 332 is, for example, vector information representing the orientation of an object in area WS.

[0091] The position management unit 322 repeatedly identifies the position and shape of objects in area WS at predetermined intervals and updates the map data 332 based on the results. This enables the position management unit 322 to track the position of specific objects in area WS.

[0092] The identification unit 323 identifies the object that user U is fixated on based on the map data 332. Based on the fixation point information transmitted by the HMD 100, the identification unit 323 identifies the position of the fixation point in the coordinate system indicated by the map data 332. The fixation point information includes the position information, orientation information, and information indicating the fixation point of the HMD 100. From this information, the identification unit 323 calculates the coordinates of the fixation point. Furthermore, the identification unit 323 identifies an object located at a position overlapping with the fixation point as the object that user U is fixated on. The identification unit 323 may also identify the position on the object that user U is fixated on.

[0093] Each time the identification unit 323 receives gaze point information from the HMD 100, it performs a process to map the gaze point to the coordinates of the map data 332. This allows the identification unit 323 to track the movement of the user U's gaze point.

[0094] The video management unit 324 performs processing related to imaging by the imaging unit 200 and display by the HMD 100. The video management unit 324 corresponds to an example of an imaging control unit and a video control unit. The video management unit 324 selects an imaging device 201 from among the imaging devices 201 included in the imaging unit 200 that is suitable for imaging the target object. For example, the video management unit 324 selects an imaging device 201 that is close to the target object.

[0095] The video management unit 324 adjusts the field of view of the selected imaging device 201 so that the object is included within its field of view. For example, the video management unit 324 transmits data to at least one of the first camera 210, second camera 220, third camera 230, fourth camera 240, and fifth camera 250 indicating the direction in which to move the field of view and the amount of adjustment to move the field of view. The adjustment amount is, for example, the angle of movement of the field of view or the amount of movement of the drive unit. The video management unit 324 may also transmit data instructing adjustments to the gain, white balance, etc., of the imaging data of each imaging device 201.

[0096] The video management unit 324 calculates the distance between the object and the imaging device 201, which has a focus adjustment function, based on the map data 332, and transmits data to instruct focus adjustment in accordance with the calculated distance.

[0097] The camera function data 333 stored in memory 330 includes information for each imaging device 201 indicating whether or not there is a field of view adjustment function, a focus adjustment function, a movement function, and the content of these functions. By referring to the camera function data 333, the video management unit 324 can transmit data that matches the function of the imaging device 201 and perform the adjustments appropriately.

[0098] The video management unit 324 transmits the video captured by the selected imaging device 201 to the HMD 100. The video management unit 324 transmits data specifying the display state on the HMD 100 to the HMD 100. For example, the video management unit 324 selects either display setting data for direct viewing, which specifies a display state suitable for when the user U can directly view the object, or display setting data for camera images, which specifies a display state suitable for when the user U cannot directly view the object. These display setting data are stored in the memory 330 as display setting data 334, for example.

[0099] [1-5. Operation of the Information Processing System] Figure 5 is a flowchart showing the operation of the information processing system 1. Figures 6 and 7 are sequence diagrams showing the operation of the information processing system 1, illustrating the operation of the HMD 100 and the management server 300. The operation of the information processing system 1 will be explained below with reference to these figures.

[0100] During the operation described below, the position management unit 322 detects the position and orientation of objects in area WS as described above and performs a process to update the map data 332 at predetermined intervals. This process is not shown in the flowchart.

[0101] The operation shown in Figure 5 includes the operations of the management server 300 and the HMD 100. Step S11 is performed by the management server 300 and the HMD 100. Steps S12 to S21 and S23 to S24 are performed by the management server 300, and step S22 is performed by the HMD 100.

[0102] In the operation shown in Figure 5, the information processing system 1 performs position detection processing using the HMD 100 and the management server 300 (step S11).

[0103] The position detection process in step S11 is shown in detail in Figure 6. The communication control unit 321 transmits a gaze point detection instruction to the HMD 100 (step SA11). The gaze point detection instruction is data that instructs the HMD to perform gaze point detection.

[0104] The communication control unit 122 receives a gaze point detection instruction from the management server 300 (step SB11). The detection control unit 124 detects the direction of the user U's gaze based on the detection results of the gaze detection device 153, etc. (step SB12), and identifies the point of the user U's gaze based on the direction of the gaze (step SB13). The detection control unit 124 generates position information and attitude information based on the detection results of the GNSS 142 and the motion sensor 154 (step SB14).

[0105] Here, the detection control unit 124 determines whether or not the user U has performed an operation to determine the object (step SB15). Examples of operations to determine the object include (1) to (4) below. (1) Stare at an object in real space for a specified period of time or longer. (2) The decision motion is performed while the subject is fixed on an object in real space. (3) Stare at the object shown in the image displayed by the HMD100 for a specified period of time or longer. (4) The user makes a decision motion while fixating on the object shown in the image displayed by the HMD100.

[0106] Here, the predetermined time is a pre-set time. The HMD100 stores the set value of the predetermined time in the memory 130. The decision motion is a pre-set operation for setting the target object, and multiple decision motions may be set. The type and content of the decision motion are not limited. For example, the decision motion may be an operation on the wheel operation unit 12, the center key 13, the operation pad 14, the power switch 18, etc. of the controller 10. Alternatively, the decision motion may be an operation to move the controller 10 or the display unit 20 in a predetermined operation pattern. Alternatively, the decision motion may be an operation to vibrate the controller 10 or the display unit 20 in a predetermined vibration pattern. Alternatively, the decision motion may be a gesture operation in which the user U moves their body in a predetermined operation pattern. Preferably, the gesture operation is a movement that can be detected by the detection control unit 124 from the image captured by the external camera 61.

[0107] If the detection control unit 124 determines that an object determination operation has been performed (step SB15; YES), it adds an object determination flag to the information indicating the point of focus identified in step SB13 (step SB16) and proceeds to step SB17. If the detection control unit 124 determines that an object determination operation has not been performed (step SB1; NO), it skips step SB16 and proceeds to step SB17.

[0108] In step SB17, the detection control unit 124 generates gaze point information including the processing results of steps SB13 and SB14 and transmits it to the management server 300 via the communication device 141 (step SB17). After that, the control unit 120 returns to the operation shown in Figure 5.

[0109] The communication control unit 321 receives the gaze point information transmitted by the HMD 100 (step SA12). The identification unit 323 determines whether the gaze point information includes an object determination flag (step SA13). If the gaze point information includes an object determination flag (step SA13; YES), the identification unit 323 identifies the object based on the gaze point information and map data 332 (step SA14). In step SA14, if the object has already been identified, the identification unit 323 changes the object. After that, the server control unit 310 proceeds to step SA15. If the gaze point information does not include an object determination flag (step SA13; NO), the server control unit 310 skips step SA14 and proceeds to step SA15.

[0110] In step SA15, the position management unit 322 identifies the position of the object based on the gaze point information (step SA15). In step SA15, the position management unit 322 may also refer to the map data 332. The position management unit 322 identifies the position and orientation of the display unit 20 based on the gaze point information (step SA16). The position management unit 322 updates the map data 332 based on the processing result of step SA16 (step SA17). After that, the server control unit 310 returns to the operation shown in Figure 5.

[0111] Figure 6 shows an example in which the detection control unit 124 generates and transmits gaze point information triggered by the position management unit 322 transmitting a gaze point detection instruction. This is just one example; for example, the detection control unit 124 may be configured to generate gaze point information at predetermined intervals during the operation of the information processing system 1 or according to user U's input. In this case, the management server 300 can receive gaze point information at predetermined time intervals.

[0112] As shown in Figure 5, the video management unit 324 determines the work mode of the HMD 100 (step S12). Multiple work modes are pre-set for the HMD 100. The display state on the HMD 100 changes depending on the work mode. In this embodiment, two types of work modes are used: a camera-utilizing mode and a mode that prioritizes the user U directly viewing the object. In this embodiment, the camera-utilizing mode is called the first mode, and the mode that prioritizes the user U directly viewing the object is called the second mode. The selection of the work mode is performed by operating the HMD 100.

[0113] Figure 7 shows the operation of the HMD100 and the management server 300 when setting and changing the work mode. In Figure 7, steps SA21 to SA22 show the operation of the management server 300, and steps SB21 to SB22 show the operation of the HMD 100.

[0114] The detection control unit 124 detects that user U has performed an operation to set the work mode (step SB21). The operation to set the work mode is a pre-set operation, and multiple operations may be set. The type and content of the operation are not limited. For example, the operation to set the work mode may be an operation on the wheel operation unit 12, the center key 13, the operation pad 14, the power switch 18, etc. of the controller 10. Alternatively, the operation to set the work mode may be an operation to move the controller 10 or the display unit 20 in a predetermined operation pattern. Alternatively, the operation to set the work mode may be an operation to vibrate the controller 10 or the display unit 20 in a predetermined vibration pattern. Alternatively, the operation to set the work mode may be a gesture operation in which user U moves their body in a predetermined operation pattern. Preferably, the gesture operation is a movement that the detection control unit 124 can detect from the image captured by the external camera 61.

[0115] The communication control unit 122 transmits the work mode setting information to the management server 300 (step SB22). The work mode setting information includes information indicating the work mode set by user U in step SB21.

[0116] The communication control unit 321 receives the work mode setting information transmitted by the HMD 100 (step SA21). The video management unit 324 sets or changes the work mode of the HMD 100 based on the work mode setting information (step SA22).

[0117] The operation shown in Figure 7 is performed intermittently during the operation of the information processing system 1. For example, when user U performs an operation to set the work mode, this operation triggers the HMD100 to start step SB21 in Figure 7 as an interrupt process.

[0118] Returning to Figure 5, the video management unit 324 determines the work mode set based on the operation of the HMD 100 in step S12. The video management unit 324 determines whether the work mode is the first mode, i.e., the camera usage mode (step S13).

[0119] If the system determines that the work mode is not a camera-using mode (step S13; NO), the video management unit 324 determines whether the user U can directly view the object (step S14). In step S14, the video management unit 324 determines, for example, whether there is an obstruction between the position of the display unit 20 and the position of the object that would obstruct the line of sight, based on the map data 332. If an obstruction exists, the video management unit 324 determines that the user U cannot directly view the object; if no obstruction exists, it determines that the user U can directly view the object.

[0120] If user U determines that the object can be viewed directly (step S14; YES), the video management unit 324 obtains display setting data for direct viewing from the display setting data 334 (step S15). The video management unit 324 calculates the distance from each imaging device 201 to the object based on the map data 332 (step S16). The video management unit 324 selects a main camera from among the multiple imaging devices 201 (step S17). For example, the video management unit 324 selects the imaging device 201 with the shortest distance to the object as the main camera. The video management unit 324 may also select an imaging device 201 that can capture the object near the center of the field of view as the main camera.

[0121] The video management unit 324 further selects a predetermined number of imaging devices 201 (step S18). The number of imaging devices 201 selected by the video management unit 324 in step S18 is the number determined by the display setting data acquired in step S15. The images captured by the imaging devices 201 selected in step S18 are transmitted to the HMD 100 and displayed on the HMD 100. For example, the video management unit 324 selects a predetermined number of imaging devices 201 in order of the shortest distance from each imaging device 201 to the object. Alternatively, the video management unit 324 may select imaging devices 201 that can capture the object at a position close to the center of the field of view.

[0122] In step S18, the video management unit 324 selects the imaging devices 201 such that each imaging device 201 images different parts of the object. That is, the video management unit 324 makes the selection in step S18 such that the main camera selected in step S17 and the imaging devices 201 selected in step S18 are a combination that images different parts of the object. When multiple imaging devices 201 image an object, if the angles at which the imaging devices 201 face the object are different, then these multiple imaging devices can be said to be imaging different parts of the object. For example, if at least two imaging devices 201 are selected in steps S17 and S18, the above condition is met if the angles at which the two imaging devices 201 face the object are greater than or equal to a set value. According to this condition, if the part of the object imaged by one imaging device 201 is called the first part, then the other imaging devices 201 will image a second part that is different from the first part. In other words, one imaging device 201 selected by the video management unit 324 is positioned to capture a first portion of the object, and another imaging device 201 is positioned to capture a second portion of the object. When the images captured by these imaging devices 201 are displayed on the HMD 100, the user U can view images capturing two different parts of the object. Therefore, the imaging devices 201 can be used to effectively assist the user U in viewing the object. Subsequently, the server control unit 310 proceeds to step S23.

[0123] If the system determines that the work mode is a camera-using mode (step S13; NO), or if it determines that user U cannot directly look at the object (step S14; NO), the video management unit 324 proceeds to step S19. In step S19, the video management unit 324 obtains display setting data for camera use from the display setting data 334 (step S19). The video management unit 324 calculates the distance from each imaging device 201 to the object based on the map data 332 (step S20). The video management unit 324 selects a predetermined number of imaging devices 201 (step S21). The number of imaging devices 201 selected by the video management unit 324 in step S21 is the number determined by the display setting data obtained in step S19. The selection in step S21 is performed in the same way as in step S18. Next, the video management unit 324 ranks the imaging devices 201 selected in step S21 based on their distance to the object (step S22). Specifically, the video management unit 324 determines the ranking of the imaging devices 201 in order of proximity to the object. After that, the server control unit 310 proceeds to step S23.

[0124] In step S23, the video management unit 324 adjusts the field of view of the imaging device 201 selected in steps S17, S18, and S21 (step S19). Specifically, the video management unit 324 transmits data to the imaging device 201 specifying the direction and amount of adjustment for changing the field of view. If the selected imaging device 201 is configured to be movable, the video management unit 324 transmits data to the imaging device 201 specifying the direction and amount of movement.

[0125] The video management unit 324 determines the display mode on the HMD 100 based on the display setting data acquired in step S15 or step S19 (step S24). Specifically, the video management unit 324 determines the arrangement of the images captured by the imaging device 201 selected in steps S17, S18, and S21, and the display size of the images, according to the display setting data.

[0126] The video management unit 324 starts acquiring video footage captured by the imaging device 201 selected in steps S17, S18, and S21 (step S25). Also in step S25, the video management unit 324 starts transmitting the acquired video to the HMD 100. Specifically, the video management unit 324 synthesizes display data for displaying the video from the selected imaging device 201 in the display mode determined in step S24, and transmits the synthesized display data to the HMD 100.

[0127] The HMD100 receives video transmitted by the management server 300 via the communication control unit 122, and the display control unit 123 controls the display unit 20 based on the received video to display the video on the display unit 20 (step S26).

[0128] In step S25, the video management unit 324 may transmit to the HMD 100 the video from the imaging device 201 and data specifying the display mode of the video. In this case, in step S26, the HMD 100 arranges the multiple videos received from the management server 300 according to the display mode determined by the video management unit 324 and displays them on the display unit 20.

[0129] Figures 8 and 9 show examples of displays in the HMD 100. In Figures 8 and 9, the symbol VA indicates the field of view of the user U when the user U is wearing the display unit 20. As described above, the display unit 20 includes a right-eye display unit 151 that displays an image to the user U's right eye RE, and a left-eye display unit 152 that displays an image to the user U's left eye LE, and the right-eye display unit 151 and the left-eye display unit 152 display images independently. The image displayed on the right-eye display unit 151 and the image displayed on the left-eye display unit 152 may be the same or different. For example, the display control unit 123 can make the user U perceive a three-dimensional image by providing parallax between the image displayed on the right-eye display unit 151 and the image displayed on the left-eye display unit 152.

[0130] Therefore, the display examples in Figures 8 and 9 are not necessarily identical to the images displayed on the right eye display unit 151 and the left eye display unit 152. The display examples in Figures 8 and 9 are schematic examples illustrating the state as seen by user U.

[0131] Figure 8 shows an example of the display when user U can directly view the object OB. In the field of view VA, a predetermined area including the center of user U's field of view VA is protected as the central region CA. The image captured by the imaging device 201 is not displayed in the central region CA. This ensures that user U has a range to directly view the object OB.

[0132] Around the central region CA, camera image regions 401 are arranged to display images captured by the imaging device 201. In the example in Figure 8, four camera image regions 401A, 401B, 401C, and 401D are arranged. Each camera image region 401 displays an image captured by one imaging device 201. At least two camera image regions 401 are arranged in the field of view VA. As a result, the field of view VA displays an image of the area including a first part of the object OB, and an image of the area including a second part of the object OB that is different from the first part. In the display configuration of Figure 8, the camera image regions 401 are arranged to avoid the central region CA.

[0133] When user U can directly view the object OB, the image displayed in the camera image area 401 may be an image with transparency of 0% or more. In this case, user U can directly view real space even at positions overlapping with the camera image area 401, which has the advantage of not giving user U the impression that their field of view is narrowed.

[0134] Figure 8 shows an example where user U directly views the suspended load during crane operation. In this example, if the object OB moves due to the crane operation, the object OB may be hidden in the shadow of obstacle HZ. In that case, user U will no longer be able to directly view the object OB. When user U is unable to directly view the object OB, the display on the HMD100 is switched to the state shown in Figure 9, for example.

[0135] Figure 9 shows an example of display when user U can directly view the object OB. In the field of view VA, the central region CA contains a camera image region 401E displaying the image captured by the main camera. The image in the camera image region 401E may have transparency of 0% or more. In this case, user U can directly view real space even in positions overlapping with the camera image region 401, and there is an advantage that it does not give user U the impression that their field of view is narrowed. At least one camera image region 401 is arranged around the camera image region 401E, displaying the image captured by imaging devices 201 other than the main camera. The image displayed in the camera image region 401E and the images displayed around the camera image region 401E are images that capture imaging ranges that include different parts of the object OB. Even when user U cannot directly view the object OB, they can view the object OB from multiple directions.

[0136] The size and position of the central region CA in Figures 8 and 9, and the size, number, and position of the camera image region 401 are determined by the display setting data selected in step S15. Memory 330 may contain three or more display setting data 334. In this case, the display setting data may be selectable by user U.

[0137] [1-6. Operation of the First Embodiment] As described above, the information processing system 1 according to the embodiment of this disclosure includes an HMD 100 having a transparent display unit 20 worn on the head of user U and a gaze detection device 153 that detects the eye movements of user U. The information processing system 1 also includes a management server 300 having a identification unit 323 that identifies an object in real space that user U is looking at through the display unit 20 based on the eye movements of user U, and an image management unit 324 that causes a first imaging device, which is positioned in a location capable of imaging a first portion of the object among the imaging devices 201, to image a first imaging range including the first portion. The HMD 100 displays the first image captured by the first imaging device on the display unit 20. With this configuration, based on the user U's gaze, the object that user U is fixated on can be identified, and the object can be captured by the imaging device 201. Therefore, as user U fixates on the object, the captured image of the object is displayed on the display unit 20, thereby improving usability.

[0138] In the information processing system 1, the video management unit 324 causes the second imaging device of the imaging device 201, which is positioned to capture a second portion of the object that is different from the first portion, to capture a second imaging range that includes the second portion. The HMD 100 displays the second image captured by the second imaging device on the display unit 20. With this configuration, multiple images capturing different portions of the object are displayed on the display unit 20. Therefore, the user U can view multiple images of the object that the user U is focusing on through the display unit 20.

[0139] In the information processing system 1, the video management unit 324 acquires a first distance from the first imaging device to the object and a second distance from the second imaging device to the object, and selects either the first or second imaging device by comparing the first and second distances. The HMD 100 displays the image captured by the imaging device 201 selected by the video management unit 324 on the display unit 20. With this configuration, the imaging device 201 located in a position suitable for imaging the object is selected, and the image captured by the selected imaging device 201 is displayed on the display unit 20. As a result, the user U can clearly see the object using the imaging device 201.

[0140] In the information processing system 1, the video management unit 324 controls the first imaging device to change its focal length based on the distance from the first imaging device to the object. With this configuration, the focal length of the imaging device 201 is adjusted to match the object, so that a high-quality image can be displayed by the display unit 20.

[0141] In the information processing system 1, the gaze detection device 153 detects the movement of the user U's right eye RE and left eye LE, and the identification unit 323 identifies the user U's point of fixation based on the movement of the right eye RE and left eye LE. The gaze detection device 153 identifies an object in real space that is located at a position that overlaps with the point of fixation for a predetermined time or longer as the target object. With this configuration, the object that the user U is fixating on can be reliably identified based on the movement of the user U's right eye RE and left eye LE. As a result, when the user U fixates on the object with both eyes, the captured image of the object is displayed on the display unit 20.

[0142] In the information processing system 1, the video management unit 324 controls the first imaging device to change the first imaging range in accordance with the movement of the object. As a result, even when the object is moving, the image captured of the object can be displayed on the display unit 20.

[0143] In the information processing system 1, the management server 300 includes a video management unit 324. The video management unit 324 determines whether the user U can see the object through the display unit 20 based on the positional relationship between the HMD 100 and the object, and switches the display mode on the display unit 20 based on the determination result. As a result, the display mode of the display unit 20 is switched when the user U cannot directly see the object. Therefore, the display mode is appropriately switched without the user U having to determine whether or not the object can be seen and operate the display unit 20, thus further improving convenience.

[0144] In the information processing system 1, the display unit 20 displays the first captured image in a position that avoids the user U viewing the object through the display unit 20. Therefore, the image captured by the imaging device 201 can be displayed without interfering with the user U's direct viewing of the object.

[0145] The HMD100 includes a transparent display unit 20 worn on the user U's head and a gaze detection device 153 that detects the user U's eye movements. The HMD100 also includes a display control unit 123. The display control unit 123 displays a first captured image on the display unit 20. That is, based on the eye movements of the user U detected by the gaze detection device 153, an object in real space that the user U is fixated on through the display unit 20 is identified, and when a first captured image is acquired that captures a first imaging range including a first part of the object, the first captured image is displayed. With this configuration, based on the user U's gaze, the object that the user U is fixated on is identified, and the image of the object captured by the imaging device 201 is displayed on the display unit 20. Therefore, by fixing on the object, the user U can use the imaging device 201 to visually identify the object, thereby improving operability.

[0146] The control program 131 is a program that can be executed by the processor 121, which controls the HMD 100 having a transparent display unit 20 that is worn on the head of the user U. The control program 131 causes the processor 121 to function as a display control unit 123. Based on the user U's eye movements, the display control unit 123 identifies an object in real space that the user U is fixating on through the display unit 20, and when it acquires a first image captured by capturing a first imaging range including a first part of the object, it displays the first image captured on the display unit 20. By executing this program, the processor identifies an object that the user U is fixating on based on the user U's gaze, and an image of the object captured by the imaging device 201 is displayed on the display unit 20. Therefore, the user U can visually identify the object using the imaging device 201 by fixating on the object, thereby improving operability.

[0147] [2. Second Embodiment] Figure 10 is a block diagram of the HMD101 of the second embodiment. The information processing system 1 of the second embodiment does not include the management server 300. In the information processing system 1 of the second embodiment, the configuration of the imaging unit 200 is the same as that of the first embodiment.

[0148] The HMD101 shown in Figure 10 is a modified version of the HMD100, with the control unit 120 being replaced by a control unit 120A. In the HMD101, components common to both the HMD100 and the HMD100 are denoted by the same reference numerals and their descriptions are omitted.

[0149] The control unit 120A includes a processor 121A and a memory 130A. The hardware configuration of the processor 121A is the same as that of the processor 121. The hardware configuration of the memory 130A is the same as that of the memory 130.

[0150] Memory 130A stores the control program 131A executed by processor 121A. Furthermore, memory 130A stores map data 332, camera function data 333, and display setting data 334. These data are stored by the management server 300 of the first embodiment.

[0151] The processor 121A functions as the communication control unit 122A, the display control unit 123A, and the detection control unit 124A by executing the control program 131A. The processor 121A also functions as the position management unit 322A, the identification unit 323A, and the video management unit 324A.

[0152] The position management unit 322A performs the same functions as the position management unit 322 of the management server 300 in the first embodiment. The identification unit 323A functions the same as the identification unit 323 of the management server 300, and the video management unit 324A functions the same as the video management unit 324. The communication control unit 122A functions the same as the communication control unit 122. Furthermore, the communication control unit 122A communicates with the imaging device 201 via the communication device 141 during the operation of the position management unit 322A, the identification unit 323A, and the video management unit 324A.

[0153] The HMD101 shown in Figure 10 performs the operations shown in Figures 5, 6, and 7 in order to display the image from the imaging device 201 on the HMD100. In other words, the HMD101 functions as both a display device and an information processing device. The information processing system 1 equipped with the HMD101 operates and produces the same effects as in the first embodiment.

[0154] [3. Other Embodiments] The present invention is not limited to the configuration of the above embodiments, and can be implemented in various forms without departing from its spirit.

[0155] The specific processes used by the detection control unit 124 to detect the direction of gaze, identify the point of focus, and generate posture information can be modified as appropriate. For example, the detection control unit 124 may use a magnetic sensor or the like included in the motion sensor 154 to detect the direction of gaze and generate posture information.

[0156] The various processes performed by the HMD100 may be executed using an external computer instead of the controller 10. That is, the various functional units of the processor 121 shown in Figure 3 may be provided by a computer connected to the display unit 20, and the various numerical values ​​and information stored in the memory 130 may be stored by the computer connected to the display unit 20. In this case, the HMD100 can transmit detection data from the various sensors of the display unit 20 and the various sensors of the controller 10 to the computer and perform processing based on the data input from the computer. For example, a smartphone or a tablet computer can be used as this type of computer.

[0157] Furthermore, although the above embodiment illustrates a configuration in which the controller 10 is wiredly connected to the display unit 20, the invention is not limited to this configuration, and the display unit 20 may be wirelessly connected to the controller 10. Also, the controller 10 may be implemented using multiple devices. Alternatively, instead of the controller 10, a wearable device that can be attached to the user's body, clothing, or accessories worn by the user may be used. In this case, the wearable device may be, for example, a watch-type device or a pen-type device.

[0158] Furthermore, in the above embodiment, a configuration in which the display unit 20 and the controller 10 are separate and connected via a connecting cable 40 was described as an example. However, the invention is not limited to this configuration, and it is also possible for the controller 10 and the display unit 20 to be integrated and worn on the user's head.

[0159] Furthermore, the display unit 20 is not limited to being directly attached to the user U's head. For example, the user U may wear a helmet or hat on their head, and the display unit 20 may be attached to this helmet or hat.

[0160] The optical system of the display unit 20 can employ various configurations as long as the image light is directed to the user's eyes by the right light guide plate 26 and the left light guide plate 28. Furthermore, diffraction gratings, prisms, or holographic display units may be used as the right light guide plate 26 and the left light guide plate 28. The configurations of the right eye display unit 151 and the left eye display unit 152 are not limited to organic EL devices or liquid crystal panels; digital micromirror devices or configurations applying LCOS (Liquid Crystal on Silicon) technology may also be used. Additionally, the right eye display unit 151 and the left eye display unit 152 may utilize self-emissive display elements such as LED arrays, laser arrays, or quantum dot light-emitting elements. Furthermore, the right eye display unit 151 and the left eye display unit 152 may employ, for example, a laser scanning method combining a laser light source and a laser scanner.

[0161] At least some of the functional blocks shown in Figures 3 and 4 may be implemented in hardware, or they may be implemented through the collaboration of hardware and software, and are not limited to a configuration in which independent hardware resources are arranged as shown in the figures.

[0162] Furthermore, the processing units in the flowcharts shown in Figures 5, 6, and 7 are divided according to their main processing content to facilitate understanding of the processing performed by the control unit 120. The method of dividing and naming the processing units in each flowchart does not limit the embodiments. Also, the processing order in the flowcharts described above is not limited to the examples shown.

[0163] Furthermore, the control program 131 executed by the processor 121 may be stored in an external device or apparatus and retrieved via a communication device 141 or the like. It can also be recorded on a recording medium that is readable by a computer. Magnetic, optical, or semiconductor memory devices can be used as recording media. Specifically, portable or fixed recording media such as flexible disks, various optical disks, magneto-optical disks, flash memory, and card-type recording media can be used. The recording medium may also be a non-volatile storage device such as RAM, ROM, or HDD, which are internal storage devices of the image display device. The same applies to the control program 331 executed by the processor 320. [Explanation of Symbols]

[0164] 1…Information processing system, 10…Controller, 20…Display unit, 61…External camera, 64…Distance sensor, 65…Illuminance sensor, 67…LED indicator, 100…HMD (Display device), 101…HMD (Display device, Information processing device), 120, 120A…Control unit, 121, 121A…Processor, 122, 122A…Communication control unit, 124, 124A…Detection control unit, 130, 130A…Memory, 131, 131A…Control program, 141…Communication device, 142…GNSS, 144…Voice interface, 151…Right eye display unit, 152…Left eye display unit, 153…Eye gaze detection device (Detection unit), 154…Motion sensor, 18 1...Audio interface, 200...Imaging unit, 201...Imaging device, 210...First camera, 220...Second camera, 230...Third camera, 240...Fourth camera, 250...Fifth camera, 290...Main unit, 300...Management server (information processing device), 310...Server control unit, 320...Processor, 321...Communication control unit, 322, 322A...Location management unit, 323, 323A...Specification unit, 324, 324A...Video management unit (Imaging control unit, Video control unit), 330...Memory, 331...Control program, 332...Map data, 333...Camera function data, 350...Server communication device (Communication unit), G...Guide rail, OB...Target object, U...User.

Claims

1. A transparent display unit that is attached to the user's head; A detection unit that detects eye movement of the user; A display device having A communication unit that communicates with one or more imaging devices; an identification unit that identifies an object in real space that the user is gazing at through the display unit based on an eye movement of the user; an imaging control unit that causes a first imaging device, among the one or more imaging devices, that is located at a position capable of imaging a first portion of the object to image a first imaging range including the first portion; and an information processing device having The display device displays a first captured image captured by the first imaging device on the display unit.

2. the imaging control unit causes a second imaging device, among the one or more imaging devices, that is located at a position capable of imaging a second portion of the object different from the first portion to image a second imaging range including the second portion; The information processing system according to claim 1 , wherein the display device causes a second captured image captured by the second imaging device to be displayed on the display unit.

3. The imaging control unit is acquiring a first distance from the first imaging device to the object and a second distance from the second imaging device to the object; selecting one of the first imaging device and the second imaging device by comparing the first distance with the second distance; The information processing system according to claim 2 , wherein the display device displays, on the display unit, an image captured by either the first imaging device or the second imaging device selected by the imaging control unit.

4. The information processing system according to claim 1 , wherein the imaging control unit controls the first imaging device to change a focal length based on a distance from the first imaging device to the object.

5. The detection unit detects a movement of a right eye and a movement of a left eye of the user, 5. The information processing system according to claim 1, wherein the identification unit identifies the user's point of gaze based on the movement of the right eye and the movement of the left eye, and identifies an object in real space that overlaps with the point of gaze for a predetermined period of time or more as the target object.

6. The information processing system according to claim 1 , wherein the imaging control unit controls the first imaging device to change the first imaging range in accordance with a movement of the object.

7. The information processing system according to any one of claims 1 to 6, wherein the information processing device is provided with an image control unit that determines whether the user can see the object through the display unit based on a positional relationship between the display device and the object, and switches the display mode on the display unit based on the determination result.

8. The information processing system according to claim 1 , wherein the display unit displays the first captured image in a manner that avoids a position where the user views the object through the display unit.

9. A transparent display unit that is attached to the user's head; A detection unit that detects eye movement of the user; a display control unit that, when an object in real space that the user is gazing at through the display unit is identified based on the eye movement of the user detected by the detection unit, and a first captured image that captures a first imaging range including a first portion of the object is acquired, displays the first captured image on the display unit.

10. A program executable by a computer to control a display device having a transmissive display unit mounted on a user's head, The computer, a program that functions as a display control unit that, when an object in real space that the user is gazing at through the display unit is identified based on the movement of the user's eyes, and a first captured image that captures a first imaging range that includes a first portion of the object is acquired, displays the first captured image on the display unit.