CONTROL DEVICE, DISPLAY CONTROL SYSTEM, AND CONTROL METHOD

Abstract

To reduce the burden on a user for sharing video images from various viewpoints in real time.SOLUTION: A display device is provided, comprising image capturing means, detection means for detecting an object of interest from a video image captured by the image capturing means, acquisition means for acquiring a video image related to the object of interest from an external device, and display control means configured to display the video image acquired from the external device according to a detection result of the detection means.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to control Device, display control system ,oh Yo and Regarding the method. [Background technology]

[0002] In recent years, technologies such as AR (Augumented Reality) and MR (Mixed Reality) have been used in head-mounted displays that can be worn on the user's head, as well as camera-equipped smartphones and tablet devices. In particular, head-mounted displays (HMDs) can provide useful information according to the usage scenario by placing a display in front of the user's eyes, giving the user a deeper sense of immersion.

[0003] A user wearing an HMD can share images captured by multiple imaging devices. For example, Patent Document 1 discloses a technology that displays images from an external imaging device on the user's own display device by specifying location information and camera specification information for specifying a specific imaging device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-141272 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the operation of designating a camera device for sharing video while enjoying an event or the like is cumbersome, and this may distract the user from enjoying the event.

[0006] An object of the present invention is to provide a display device that reduces the burden on a user when sharing video images from various viewpoints in real time. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention control The device is Filming method A control device for controlling a head-mounted display device having , a detection means for detecting a subject of interest from the image captured by the imaging means; An acquisition means for acquiring an image related to the target subject from an external device; When the image captured by the image capturing means does not include the target subject or a front of the target subject, The image acquired from the external device The head mounted display device View Control Display control means; The present invention is characterized by having do. Effect of the Invention

[0008] According to the present invention, it is possible to reduce the burden on users who share video from various viewpoints in real time. [Brief description of the drawings]

[0009] [Figure 1] Appearance of the HMD. [Diagram 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of an HMD and a shared server. [Diagram 3] 11 is a flowchart illustrating a video sharing process. [Figure 4] FIG. 13 is a diagram illustrating a video sharing setting screen. [Diagram 5] FIG. 4 is a diagram illustrating an example of a setting screen for a target subject. [Figure 6] 11 is a flowchart illustrating a video display control process. [Figure 7] FIG. 13 is a diagram showing an example of a display of an enlarged image of an image captured by an HMD. [Figure 8] FIG. 11 is a diagram showing a display example of a video acquired from a shared server. [Figure 9]11A and 11B are diagrams showing examples of video display when a subject of interest is not detected in a see-through area. [Figure 10] FIG. 4 is a diagram for explaining gaze transition information. [Figure 11] 11A and 11B are diagrams showing examples of notification of a change in a subject of interest. [Figure 12] FIG. 13 is a diagram showing an example of a virtual monitor display. [Figure 13] FIG. 13 is a diagram for explaining estimation of an overlooked period. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Embodiment 1> The HMD (head mounted display device) according to this embodiment is, for example, an optical see-through type HMD that uses a transparent display. In an optical see-through type HMD, the user can see both the image display and the incident light from the outside world. That is, the user can see various information displayed on the display of the HMD while watching an event such as a concert or an athletic meet through the see-through.

[0011] The HMD in this embodiment is assumed to be an optical see-through type HMD, but may be a video see-through type HMD that uses an opaque display. A video see-through type HMD converts the outside world into a video image and electronically combines it with an image of the virtual world to display it.

[0012] The HMD can display, for example, information about a subject that the user is paying attention to (hereinafter also referred to as a subject of interest) on the display. The information about the subject of interest is, for example, an image captured by an imaging device attached to another HMD. By transmitting and receiving images captured by the HMDs to each other using a communication function, users wearing the HMDs can share part of their field of view in real time.

[0013] The HMD may transmit and receive video via a shared server. The shared server is a computer capable of communicating with multiple HMDs, and corresponds to an information processing device. The display control system according to the present invention includes a shared server and multiple HMDs, and realizes sharing of video between the HMDs via the shared server.

[0014] The HMD acquires images of a subject of interest set by the user from another HMD, and displays the acquired images on the display at a position and size that does not interfere with viewing the subject through the see-through system.

[0015] A preferred embodiment of the present invention will be described below with reference to the drawings. An HMD as a display device according to this embodiment will be described with reference to FIG. 1. FIG. 1 illustrates an external view of an HMD 1. Hereinafter, the HMD 1 will be described as an optical see-through type HMD, but it may be a video see-through type HMD using an opaque display. In a video see-through type HMD, incident light from the front is visualized and displayed.

[0016] The HMD 1 has a frame and two display units, 11a and 11b. The frame has a rim 15 and temples 16a and 16b joined to both sides of the rim 15. The display units 11a and 11b are joined to one surface of the rim 15.

[0017] The image projection device 12a and the light guide device 13a guide light from a display element (not shown) of the display unit 11a to the right eye of a wearer wearing the HMD 1. Similarly, the image projection device 12b and the light guide device The light device 13b guides light from a display element (not shown) of the display unit 11b to the left eye of a wearer wearing the HMD 1.

[0018] The wearer can simultaneously perceive the images displayed on the display units 11a and 11b and the light incident from the front of the HMD 1. The HMD 1 also has imaging devices 14a and 14b. The imaging devices 14a and 14b capture images of the surroundings including the front of the HMD 1.

[0019] (Hardware configuration) The hardware configuration of the HMD 1 and the shared server 2 included in the display control system will be described with reference to Fig. 2. The shared server 2 (information processing device) is an example of an external device for controlling transmission and reception of video when video is shared among a plurality of HMDs 1. The HMD 1 may share video with other HMDs as external devices by directly transmitting and receiving video.

[0020] The HMD 1 includes a calculation device 101, an image capture device 102, a primary storage device 103, a detection device 104, a communication device 105, a display unit 106, and an operation unit 107. The components included in the HMD 1 are connected to each other via a bus 108.

[0021] The arithmetic device 101 is, for example, a CPU (Central Processing Unit), and is responsible for the overall control of the HMD 1. The photographing device 102 corresponds to the image capturing devices 14a and 14b shown in Fig. 1, and captures an image of the field of view of a user wearing the HMD 1. The photographing device 102 has an image capturing element such as a lens, a CCD or a CMOS sensor, and generates image data of an image captured by the subject.

[0022] The primary storage device 103 is, for example, a dynamic random access memory (DRAM). The primary storage device 103 temporarily stores images captured by the image capturing device 102 and images acquired from the external shared server 2.

[0023] The detection device 104 analyzes the video stored in the primary storage device 103, and determines, for example, whether or not a subject of interest is included in the video. The subject of interest is a subject that is set in advance by the user. The subject of interest may also be changed by a user operation during shooting. The detection device 104 is also capable of detecting the orientation, size, etc. of the subject of interest.

[0024] The communication device 105 is capable of communicating with other devices using communication standards such as Wi-Fi and / or BLE (Bluetooth (registered trademark) Low Energy). For example, the communication device 105 can communicate with an external shared server 2 via the communication device 205. The communication device 105 transfers video captured by the image capturing device 102 to the shared server 2.

[0025] The arithmetic device 101 acquires a video related to the subject of interest from the external shared server 2 via the communication device 105 in accordance with the detection result of the detection device 104. For example, the arithmetic device 101 acquires the video from the shared server 2 when the subject of interest is not detected from the video captured by the HMD 1. The communication device 105 acquires the video from the shared server 2 when the subject of interest facing forward is not detected or when the size (area) of the detected subject of interest is equal to or smaller than a predetermined threshold.

[0026] The display unit 106 displays the image captured by the image capturing device 102 and the image acquired from the external shared server 2. The display unit 106 corresponds to the display unit 11a and the display unit 11b in FIG. 1. The operation unit 107 receives operation instructions from the user through button operations, voice operations, etc. Accept the indication.

[0027] The shared server 2 has a calculation device 201, a primary storage device 202, a secondary storage device 203, a detection device 204, and a communication device 205. The components of the shared server 2 are connected to each other via a bus 208. The calculation device 201 is, for example, a CPU, and is responsible for the overall control of the shared server 2.

[0028] The primary storage device 202 is, for example, a DRAM, and temporarily stores the video received from the HMD 1 and the video read from the secondary storage device 203. The secondary storage device 203 is, for example, a flash memory, and records the video received from the HMD 1, etc.

[0029] The detection device 204 analyzes the video read from the secondary storage device 203 to the primary storage device 202 and detects the subject. The detection device 204 acquires information about the subject, such as the orientation and size of the subject. The detection device 204 assigns the same tag to the same subject among the detected subjects and records them in the secondary storage device 203. The detection device 204 also records the detected subject and the video including the subject in association with each other in the secondary storage device 203.

[0030] The communication device 205 can communicate with other devices using a communication standard such as Wi-Fi and / or BLE. For example, the communication device 205 can communicate with the HMD 1 via the communication device 105. The communication device 205 transmits a video in which a subject of interest in the HMD 1 is detected to the HMD 1.

[0031] (Video sharing processing) The video sharing process will be described with reference to Fig. 3. The video sharing process is a control flow of the HMD 1 when a user wearing the HMD 1 shares a video with another user wearing the HMD 1 in real time while watching an event or the like.

[0032] In S301, the arithmetic device 101 sets whether or not to share video with other HMDs 1, based on a user operation on the operation unit 107. Note that the user may set video sharing in advance before watching an event or the like, or may change the sharing setting during watching.

[0033] A specific method for setting up video sharing will be described with reference to Fig. 4. Fig. 4 is a diagram illustrating an example of a video sharing setting screen. The video sharing setting screen is displayed on the display unit 106 of the HMD 1. The user can operate the video sharing setting screen via the operation unit 107.

[0034] In the example of Fig. 4, the user can set whether or not to share video with HMD1 worn by other users by setting video sharing setting 401 to ON. If individual setting 402 is set to OFF, the user can set video sharing collectively with other HMD1 capable of video sharing. An HMD1 capable of video sharing is an HMD1 that can communicate with the HMD1 worn by the user directly or via a shared server 2.

[0035] When individual setting 402 is turned ON, the user can individually set video sharing with the HMD 1 of the user displayed in the shared user list 403. The shared user list 403 displays HMD 1 that can share video with the HMD 1 worn by the user at an event venue or the like.

[0036] In S302, the arithmetic device 101 sets a target object based on a user operation on the operation unit 107. The target object is an object that the user wearing the HMD 1 wants to mainly view at an event or the like. For example, the target object may be students competing in an athletic meet (user In a concert, the subject of interest may be a favorite singer, in a car race, the car of a team that the user is rooting for, etc. The subject of interest may be set in advance before the event is viewed, or may be changed during the event, etc.

[0037] Setting of a subject of interest will be described with reference to Fig. 5. Fig. 5 is a diagram illustrating a setting screen for a subject of interest. The arithmetic device 101 displays a subject image set in advance on the display unit 106, and registers a subject selected by the user as a subject of interest. The subject of interest is not limited to a person, and may be an animal or a car.

[0038] The preset subject image, i.e., the subject image displayed as a selection candidate on the setting screen for the subject of interest, may be, for example, a subject detected from the video of the HMD 1. The arithmetic device 101 may also receive information on a subject detected from the video of another HMD 1 or the video recorded in the shared server 2, and display the received subject image as a selection candidate.

[0039] In S303, when the user starts watching the event, the arithmetic device 101 starts shooting with the shooting device 102. For example, the arithmetic device 101 can determine that watching the event has started when the user turns on the power of the HMD 1 or performs an operation to start shooting.

[0040] The arithmetic device 101 transfers the captured video to the shared server 2 at any time via the communication device 105. The arithmetic device 101 may transmit information about the subject of interest set in S302 to the shared server 2 together with the video.

[0041] In the shared server 2, the detection device 204 detects the subject in the video transferred from the HMD 1. The detection device 204 also recognizes the orientation and size of the subject detected in the video. The detection device 204 stores information about the subject, including image data of the subject, orientation and size of the subject, in the secondary storage device 203. The information about the subject is recorded in the secondary storage device 203 with a tag for identifying the subject added. The shared server 2 similarly stores videos captured by other HMDs 1 in the secondary storage device 203.

[0042] The same tag is added to the same subject detected by multiple HMDs 1. Whether or not the subject is the same can be determined by using a known technique such as Re-ID (Re-identification), which is a technique for matching the same person.

[0043] In S304, the arithmetic device 101 acquires an image related to the target subject set in S302 from the shared server 2. The image related to the target subject is an image in which the target subject appears or an image in which the front of the target subject appears.

[0044] The arithmetic device 201 of the shared server 2 acquires an image of a target subject from the images stored in the primary storage device 202 or the secondary storage device 203. The arithmetic device 201 transmits the acquired image to the HMD 1 via the communication device 205.

[0045] When there are multiple images in which the target subject is captured, the arithmetic unit 201 preferentially acquires an image in which the target subject is captured from the front (for example, the face in the case of a person) in a larger size. In addition, the arithmetic unit 201 takes into consideration the real-time nature of the images and acquires the image most recently saved (for example, within one second). The image to be acquired may be a stored image.

[0046] The shared server 2 may transmit the video captured by another HMD 1 that is set for video sharing to the HMD 1 regardless of whether the target subject is captured in the video. In this case, whether the target subject is captured in the video is notified to the other HMD 1 by the shared server 2. This can be determined on the HMD 1 side when it receives the captured image.

[0047] In S305, the arithmetic device 101 of the HMD 1 displays the video acquired from the shared server 2 on the display unit 106 based on the detection result of the subject of interest. That is, the arithmetic device 101 displays the video acquired from the shared server 2 depending on whether or not the subject of interest has been detected and the state of the detected subject of interest. A display control process for the video acquired from the shared server 2 will be described with reference to Fig. 6. Fig. 6 is a flowchart illustrating the video display control process.

[0048] In S601, the arithmetic unit 101 reads out the video captured by the image capture device 102 from the primary storage device 103, and determines whether the detection device 104 has detected a subject of interest in the video. If the angle of view of the video captured by the image capture device 102 is wider than the user's viewing angle, the detection device 104 may perform detection processing by limiting it to a field of view within the capture area.

[0049] If the detection device 104 detects a subject of interest in the captured video, the process proceeds to S602. If the detection device 104 does not detect a subject of interest in the video, the process proceeds to S608. Note that the arithmetic device 101 may proceed to the process of S608 if a subject of interest has not been detected continuously for a predetermined period of time or more.

[0050] In S602, the arithmetic unit 101 determines whether the detection unit 104 has detected the front of the subject of interest. If the detection unit 104 has detected the front of the subject, the process proceeds to S603. If the detection unit 104 has not detected the front of the subject, the process proceeds to S605.

[0051] In S603, the arithmetic device 101 determines whether or not the subject of interest captured by the image capturing device 102 is equal to or larger than a predetermined size. If the subject of interest is equal to or larger than a predetermined size, the user wearing the HMD 1 can adequately recognize the subject of interest with the naked eye (see-through). In a situation in which the user can recognize the subject of interest, the arithmetic device 101 does not display the video etc. acquired from the shared server 2, and the process shown in FIG. 6 ends. The process proceeds to the end determination of S306 in FIG. 3.

[0052] If the target subject is smaller than the predetermined size in S603, the process proceeds to S604. Note that if the target subject is smaller than the predetermined size, the calculation device 101 may proceed to S605 and display the image acquired from the shared server 2 on the display unit 106, similar to the case where the detection device 104 did not detect the front of the subject in S602.

[0053] In S604, the arithmetic device 101 cuts out an area in which the subject of interest is captured from the video captured by the image capturing device 102, and displays an enlarged image obtained by enlarging the cut-out image on the display unit 106. The arithmetic device 101 displays the enlarged image at a position that does not overlap with the subject of interest detected in the video captured by the image capturing device 102, for example.

[0054] Fig. 7 is a diagram showing a display example of an enlarged image of an image captured by the HMD 1. The dotted frame in Fig. 7 indicates a display area 700 that is visible in a see-through manner through the frame of the display unit 106 of the HMD 1. A rectangular frame displayed in the upper right corner of the display area 700 indicates an enlarged image display area 702 that displays an enlarged image of the target subject displayed in the display area 700. The target subject is displayed enlarged to a size that fits the target subject within the enlarged image display area 702. The process proceeds to the end determination of S306 in Fig. 3.

[0055] In S605, since the front of the target subject is not detected in the image captured by the image capturing device 102, the calculation device 101 determines whether or not there is any image in which the target subject is captured in the image acquired from the shared server 2.

[0056] If there is an image including the target object, the process proceeds to S606. If there is no image including the target object, the arithmetic device 101 does not display the image acquired from the shared server 2, and the process shown in Fig. 6 ends. The process proceeds to the end determination of S306 in Fig. 3.

[0057] In S606, the arithmetic device 101 displays the video acquired from the shared server 2 on the display unit 106. The arithmetic device 101 displays the video acquired from the shared server 2 in a position that does not overlap with a subject of interest in an area that can be seen through the see-through, such as a peripheral area of ​​the display unit 106.

[0058] Fig. 8 is a diagram showing an example of the display of a video acquired from the shared server 2. As in Fig. 7, the dotted frame in Fig. 8 indicates a display area 800 that can be seen through the frame of the display unit 106 of the HMD 1. In the display area 800, a subject of interest 801 is photographed facing backwards. A rectangular frame displayed in the upper right corner of the display area 800 indicates a shared video display area 802 that displays a video of a subject of interest 803 acquired from the shared server 2.

[0059] As in S604, the arithmetic device 101 may enlarge an area in which the subject of interest 803 appears in the video acquired from the shared server 2 and display it in the shared video display area 802. The arithmetic device 101 may also move the display position of the shared video display area 802 up, down, left, or right according to the detected position of the subject of interest 801 so that the shared video display area 802 does not overlap with the subject of interest 801 in the area visible in the see-through mode. Furthermore, the arithmetic device 101 may determine or change the display size of the shared video display area 802 according to the size of the subject of interest 801 detected in the see-through area.

[0060] In S607, since the target subject is not detected in the video captured by the image capturing device 102, the arithmetic device 101 determines whether or not the video acquired from the shared server 2 contains a video in which the target subject is captured.

[0061] If there is an image in which the target object is captured, the process proceeds to S608. If there is no image in which the target object is captured, the arithmetic device 101 does not display the image acquired from the shared server 2, and the process shown in Fig. 6 ends. The process proceeds to the end determination of S306 in Fig. 3.

[0062] In S608, the arithmetic device 101 displays the image acquired from the shared server 2 in the center of the display unit 106. Fig. 9 is a diagram showing an example of an image display when a subject of interest is not detected in an area visible in a see-through manner. As in Fig. 7, the dotted frame in Fig. 9 indicates a display area 900 visible in a see-through manner through the frame on the display unit 106 of the HMD 1.

[0063] A rectangular frame displayed in the center of display area 900 indicates a shared video display area 902 that displays a video image showing target subject 901 obtained from shared server 2. As in S604, calculation device 101 may enlarge the area showing target subject 901 in the video image obtained from shared server 2 or cut out a part of it and display it in shared video display area 902.

[0064] Moreover, the display position of the shared video display area 902 is not limited to the center, and may be a position closer to the right or left side of the display area 900. Furthermore, when a subject of interest is detected in the see-through display area 900, the arithmetic device 101 may reduce the shared video display area 902 and move it to a position that does not overlap with the subject of interest.

[0065] Returning to FIG. 3, in S306, the arithmetic device 101 determines whether or not the shooting has ended. For example, the arithmetic device 101 can determine that the shooting has ended when it receives an operation to end the shooting or to turn off the power from the user via the operation unit 107. If the shooting has ended, the video sharing process shown in FIG. 3 ends. If the shooting has not ended, the process returns to S304.

[0066] The processes of S304 and S305 are repeated until the user finishes watching the event and issues an instruction to end shooting. This allows a user wearing an HMD 1 to share in real time the image of a subject of interest captured by another HMD 1 when the user is unable to fully recognize the subject of interest through the see-through.

[0067] In addition, input operations such as shooting instructions and sharing settings only need to be performed once before the event starts. By setting up video sharing in advance, the user is freed from the need to share video with other HMDs 1 while watching the event, allowing them to concentrate more on watching the event.

[0068] The video display process in FIG. 6 may be performed for each frame, or may be performed at intervals of one to several seconds. Furthermore, information about the subject, such as the subject's position or size, is not limited to being evaluated for one frame, and may be evaluated based on an average value obtained from multiple frames. This allows the arithmetic device 101 to smoothly switch the video display even for a target subject that moves rapidly within an event venue, such as a basketball. By smoothly switching the video display, the user can concentrate on watching the event.

[0069] In addition, the video shared via the shared server 2 has been described as being video shot by another HMD 1, but this is not limited to this. Video that can be shared by the HMD 1 may be video shot by a fixed camera installed at the event venue, or video shot by a professional cameraman using a handheld camera.

[0070] <Embodiment 2> In the first embodiment, the user sets the object of interest in advance before watching the event, or changes it during the event. In contrast, in the second embodiment, the object of interest is automatically switched based on the user's line of sight. The configurations of the HMD 1 and the shared server 2 are the same as those described in the first embodiment with reference to Figs. 1 and 2, and therefore will not be described.

[0071] When a user wants to change the object of interest while watching an event, it is troublesome for the user to reset the object of interest. In the second embodiment, the HMD1 detects the user's line of sight and switches the object of interest to the object the user's line of sight is directed at. Therefore, the operational burden on the user to switch the object of interest is reduced.

[0072] One method for detecting gaze is the common corneal reflex method. The corneal reflex method is a method in which infrared light is irradiated onto the user's eyeball, and the state of the eyeball (direction of gaze) is detected based on the difference in reflectance between the cornea and the pupil. From the detected gaze direction and the image captured by the HMD, it is possible to obtain information on the transition of the gaze, such as what point in the field of view the user was looking at at what time.

[0073] The gaze transition information will be described with reference to Fig. 10. Fig. 10 shows a scene in which three students are running in a footrace. The gaze positions of the user at time T0, time T1, and time T2 are represented by crosses. At time T0, the user is looking at student 1001. At time T1, the user's gaze moves to student 1002, and at time T3, to student 1003.

[0074] For example, when it is detected that the user's gaze is fixed on a specific object for a predetermined period of time or more, the arithmetic device 101 of the HMD 1 switches the target object to the object on which the user's gaze is fixed. Before switching the target object, the arithmetic device 101 notifies the user that the target object will be changed.

[0075] 11 is a diagram showing an example of a notice of a change in the subject of interest. The arithmetic device 101 displays a message 1101 notifying that the subject of interest has been changed, and a frame 1102 surrounding the changed subject of interest.

[0076] 11 shows an example of notifying the user of the change of the target object, but is not limited to the notification. The arithmetic device 101 may display a confirmation message so that the user can select whether or not to change the target object. If the user agrees to change the target object, the arithmetic device 101 switches the target object. If the user does not agree to change the target object, the arithmetic device 101 continues the sharing process of the video of the target object without changing it.

[0077] In the second embodiment, the HMD 1 detects the user's line of sight and changes the target object based on the user's line of sight, which allows the arithmetic device 101 to easily switch the target object without the user having to perform any additional operation.

[0078] <Embodiment 3> In the first embodiment, the display position and display size of the video acquired from the shared server 2 are determined based on the position and size of the subject of interest detected in the area visible through the see-through.

[0079] In contrast, the third embodiment is an embodiment in which a virtual monitor is displayed within the shooting range of the image capturing device 102, and an image acquired from the shared server 2 is projected onto the virtual monitor. The HMD 1 can display the virtual monitor at a fixed position in an event venue (real space), for example. The configurations of the HMD 1 and the shared server 2 are the same as those described in the first embodiment with reference to Figs. 1 and 2, and therefore description thereof will be omitted.

[0080] In a situation where the subject of interest moves around rapidly, the display position and size of the video acquired from the shared server 2 are frequently changed, which may interfere with the see-through viewing. In the third embodiment, the output destination of the video acquired from the shared server 2 is fixed to a virtual monitor installed at a fixed position in the real space, allowing the user to calmly watch the event without being distracted by the video display of the shared video.

[0081] Fig. 12 is a diagram showing a display example of a virtual monitor. In the example of Fig. 12, the computing device 101 displays a large virtual monitor placed in an open space (inside the track) in a schoolyard where a footrace is taking place. The display position of the virtual monitor is set in an open space for each event, for example, in a layout diagram of an event venue. When the HMD 1 detects that the display position of the virtual monitor has entered the user's viewing angle, it displays the virtual monitor on the display unit 106.

[0082] The virtual monitor may be directed in front of the user or may be displayed at an angle to the user, depending on the installation position. When the virtual monitor is displayed at an angle to the user, the computing device 101 performs projective transformation of the image displayed on the virtual monitor depending on the installation angle of the monitor with respect to the HMD 1 worn by the user. The user can simultaneously view the image of the subject of interest that is naturally displayed on the virtual monitor by the projective transformation and the see-through image.

[0083] <Embodiment 4> In each of the above-mentioned embodiments, the HMD 1 shares the video in real time. In contrast, the fourth embodiment relates to a method for estimating a period (scene) that the user missed in real time based on the detection result of the object of interest and the transition information of the user's line of sight, and generating a playback video to be played (replayed) later on the HMD 1. The HMD 1 shares the video in real time. When playing back the video, for any scenes that the user missed, the video acquired from the shared server 2 is played back. The configurations of the HMD 1 and the shared server 2 are the same as those described in the first embodiment with reference to Figs. 1 and 2, and therefore a description thereof will be omitted.

[0084] 13 is a diagram for explaining estimation of a missed period. In order to estimate a missed period, the calculation device 101 records information about a subject of interest at each time during the event viewing. The subject of interest may be set in advance by the user as described in the first embodiment, or may be set based on the user's line of sight information as described in the second embodiment.

[0085] The information recorded for each scene is A: time, B: detection result of the subject of interest, C: face detection result of the subject of interest, and D: size determination result of the subject of interest. B: detection result of the subject of interest is whether or not the subject of interest has been detected in an area visible through see-through. C: face detection result of the subject of interest is whether or not the face (front) of the subject of interest has been detected. D: size determination result of the subject of interest is whether or not the face of the subject of interest is a predetermined size or larger.

[0086] In the example of Figure 13, since the subject of interest was not detected in the period from time T5 to time T7, B: detection result of the subject of interest, C: face detection result of the subject of interest, and D: size determination result of the subject of interest are NG.

[0087] At time T8, the subject of interest is detected, but the face of the subject of interest (the front of the subject of interest) is not detected, so C: face detection result of the subject of interest and D: size determination result of the subject of interest are NG. At time T9, the face of the subject of interest is detected, but the face of the subject of interest is smaller than a predetermined size, so D: size determination result of the subject of interest is NG.

[0088] If any of B: detection result of the subject of interest, C: face detection result of the subject of interest, and D: size determination result of the subject of interest is NG, the arithmetic device 101 plays the video acquired from the shared server 2. In the example of Fig. 13, the arithmetic device 101 plays the video captured by the HMD 1 worn by the user during the period from time T0 to time T4 and time T10. Also, the arithmetic device 101 plays the video acquired from the shared server 2 during the period from time T5 to time T9.

[0089] As a result, the arithmetic device 101 can generate a digest video (playback video) by splicing together the video captured by the HMD 1 and the video acquired from the shared server 2 based on the detection result of the subject of interest. Therefore, even if a user wearing the HMD 1 misses the subject of interest in real time, the user can view a digest video including the video of the subject captured during the period in which the user missed the subject.

[0090] The above-described embodiments are merely illustrative of configuration examples of the present invention. The present invention is not limited to the above-described specific embodiments, and various combinations and modifications are possible within the scope of the technical concept.

[0091] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0092] 1: HMD, 101: computing device, 102: imaging device, 104: detection device, 105: communication device, 106: display unit

Claims

1. A control device for controlling a head-mounted display device having an image capturing means, a detection means for detecting a subject of interest from the image captured by the imaging means; An acquisition means for acquiring an image related to the target subject from an external device; a display control means for controlling the image acquired from the external device to be displayed on the head-mounted display device when the image captured by the image capturing means does not include the target subject or a front of the target subject; A control device comprising:

2. A control device for controlling a head-mounted display device having an imaging means, a detection means for detecting a subject of interest from the image captured by the imaging means; An acquisition means for acquiring an image related to the target subject from an external device; and a display control means for controlling the image acquired from the external device to be displayed on the head-mounted display device based on a detection result of the detection means, The display control means displays preferentially an image in which the front of the target subject is largely captured among images in which the target subject is captured. A control device comprising:

3. A control device for controlling a head-mounted display device having an imaging means, a detection means for detecting a subject of interest from the image captured by the imaging means; An acquisition means for acquiring an image related to the target subject from an external device; and a display control means for controlling the image acquired from the external device to be displayed on the head-mounted display device based on a detection result of the detection means, The display control means controls the display of the image acquired by the acquisition means when the target subject in the image captured by the image capture means becomes smaller than a predetermined size, or controls the display of an image obtained by cutting out an area in which the target subject appears from the image captured by the image capture means and enlarging the image to be displayed on the head-mounted display device. A control device comprising:

4. A control device for controlling a head-mounted display device having an imaging means, a detection means for detecting a subject of interest from the image captured by the imaging means; An acquisition means for acquiring an image related to the target subject from an external device; a display control means for displaying the image acquired from the external device based on the detection result of the detection means, The display control means determines a display size of the image acquired by the acquisition means based on a size of the target subject in the image captured by the imaging means. A control device comprising:

5. When the subject of interest in the image captured by the imaging means becomes smaller than a predetermined size, the display control means determines the size of an area for displaying the image captured by the acquisition means so that the image captured by the acquisition means does not overlap with the subject of interest, and controls to display the image captured by the imaging means and the image captured by the acquisition means together.

5. The control device according to claim 4.

6. A control device for controlling a head-mounted display device having an imaging means, comprising: a detection means for detecting a subject of interest from the image captured by the imaging means; An acquisition means for acquiring an image related to the target subject from an external device; a display control means for displaying the image acquired from the external device based on the detection result of the detection means, The display control means determines a display position of the image acquired by the acquisition means based on a position of the target subject in the image captured by the imaging means. A control device comprising:

7. When the subject of interest in the image captured by the imaging means becomes smaller than a predetermined size, the display control means determines a position of an area for displaying the image captured by the acquisition means so that the image captured by the acquisition means does not overlap with the subject of interest, and controls to display the image captured by the imaging means and the image captured by the acquisition means together.

7. The control device according to claim 6.

8. The target object is preset by a user.

8. The control device according to claim 1, wherein the control device is a control unit.

9. The detection means changes the target subject based on a gaze position of the user.

9. The control device according to claim 1, wherein the control device is a control unit.

10. The display control means displays a virtual monitor within a shooting range of the shooting means, and displays the image acquired by the acquisition means on the virtual monitor.

10. The control device according to claim 1 ,

11. The method further includes estimating a period during which the user missed the target object, The display control means generates a playback video of the target subject for a period including the missed period, using the video captured by the imaging means and the video captured during the missed period among the videos acquired from the external device.

11. The control device according to claim 1 .

12. The camera further includes a transmission unit for transmitting the image captured by the image capture unit to the external device.

12. The control device according to claim 1 ,

13. A display control system including an information processing device and the control device according to any one of claims 1 to 12, The information processing device includes: A receiving means for receiving images captured by the imaging means from a plurality of the control devices; an acquisition means for detecting a subject from the video received by the receiving means and acquiring information about the subject including a direction and a size of the subject; a transmission means for transmitting to the control device an image selected from the images in which the target object is detected based on information about the target object. A display control system comprising:

14. The computer A first acquisition step of acquiring an image captured by a head-mounted display device; a detection step of detecting a subject of interest from the image acquired in the first acquisition step; a second acquisition step of acquiring an image related to the subject of interest from an external device; a display control step of displaying the image acquired from the external device when the image acquired in the first acquisition step does not include the target subject or a front of the target subject; A control method comprising the steps of:

15. A computer comprising: A first acquisition step of acquiring an image captured by a head-mounted display device; a detection step of detecting a subject of interest from the image acquired in the first acquisition step; a second acquisition step of acquiring an image related to the subject of interest from an external device; a display control step of displaying the image acquired from the external device based on the detection result in the detection step, In the display control step, a video in which the front of the target subject is captured larger than the other video images in which the target subject is captured is preferentially displayed. A control method comprising:

16. A computer comprising: A first acquisition step of acquiring an image captured by a head-mounted display device; a detection step of detecting a subject of interest from the image acquired in the first acquisition step; a second acquisition step of acquiring an image related to the subject of interest from an external device; a display control step of displaying the image acquired from the external device based on the detection result in the detection step, In the display control step, when the target subject in the image acquired in the first acquisition step becomes smaller than a predetermined size, the image acquired in the second acquisition step is displayed, or an image obtained by cutting out an area in which the target subject appears from the image acquired in the first acquisition step is enlarged and displayed on the head-mounted display device. A control method comprising:

17. A computer comprising: A first acquisition step of acquiring an image captured by a head-mounted display device; a detection step of detecting a subject of interest from the image acquired in the first acquisition step; a second acquisition step of acquiring an image related to the subject of interest from an external device; a display control step of displaying the image acquired from the external device based on the detection result in the detection step, In the display control step, a display size of the image acquired in the second acquisition step is determined based on a size of the subject of interest in the image acquired in the first acquisition step. A control method comprising:

18. A computer comprising: A first acquisition step of acquiring an image captured by a head-mounted display device; a detection step of detecting a subject of interest from the image acquired in the first acquisition step; a second acquisition step of acquiring an image related to the subject of interest from an external device; a display control step of displaying the image acquired from the external device based on the detection result in the detection step, A control method characterized in that, in the display control step, a display position of the image acquired in the second acquisition step is determined based on the position of the subject of interest in the image acquired in the first acquisition step.

19. A program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 12.

20. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 12.

Images