Electronic device, method for controlling electronic device, and program

Abstract

This technology provides the ability to notify users of the presence of real-world objects without interfering with their interaction with computer graphics. [Solution] The electronic device of the present invention is characterized by having a detection means for detecting a predetermined situation in which an image of a virtual object is displayed on a head-mounted display device such that it obscures at least a part of a predetermined real object, and a control means for controlling the device to provide a predetermined notification when the detection means detects the predetermined situation.

Description

Technical Field

[0001] The present invention relates to an electronic device, a control method for an electronic device, and a program.

Background Art

[0002] In recent years, research has been conducted on Mixed Reality (MR) that presents information in a virtual space in real time overlaid on the real space to users. An image processing device used to generate an MR space generates a composite image in which an image of a virtual object (CG: Computer Graphics) is overlaid on the entire or part of a real video captured by an imaging device such as a video camera. As a device for displaying such a composite image, there is a head-mounted display device such as a head-mounted display.

[0003] When a CG is displayed so as to be overlaid on the real space for a user of a head-mounted display device, if the CG is displayed so as to block surrounding objects (real objects) in the real space, there is a risk that the user may come into contact (collide) with the real object blocked by the CG.

[0004] Patent Document 1 discloses a technique for notifying when there is an object within a predetermined range from a user of a head-mounted display device in an MR space.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the technology disclosed in Patent Document 1 notifies the user of the presence of a real object in the vicinity of the user, regardless of whether or not CG is displayed in the MR space. Therefore, it notifies the user even if the real object is visible to the user. Such notifications may interfere with user interaction with CG.

[0007] The present invention aims to provide a technology that can notify a user of the presence of a real-world object without interfering with user interaction with computer graphics. [Means for solving the problem]

[0008] The electronic device of the present invention is characterized by having a detection means for detecting a predetermined situation in which an image of a virtual object is displayed on a head-mounted display device such that it obscures at least a portion of a predetermined real object, and a control means for controlling the device to provide a predetermined notification when the detection means detects the predetermined situation. [Effects of the Invention]

[0009] According to the present invention, the presence of a real object can be notified to the user without interfering with the user's operation of the computer graphics. [Brief explanation of the drawing]

[0010] [Figure 1] Block diagram showing the hardware configuration of the display device in Embodiment 1. [Figure 2] This is a schematic diagram of the display device and hand in Embodiment 1. [Figure 3] This is a schematic diagram of the image displayed on the display device in Embodiment 1. [Figure 4] This is a schematic diagram of the image displayed on the display device in Embodiment 1. [Figure 5] This is a data list in Embodiment 1. [Figure 6] This is a flowchart showing the operation of the display device in Embodiment 1. [Figure 7] This is a schematic diagram of the image displayed on the display device in Embodiment 1. [Figure 8] This is a schematic diagram of the image displayed on the display device in Embodiment 1. [Figure 9] This is a flowchart showing the operation of the display device in Embodiment 2. [Modes for carrying out the invention]

[0011] <Embodiment 1> Embodiment 1 of the present invention will be described below with reference to the drawings. Figure 1 is a block diagram showing the hardware of a head-mounted display device as an example of a display device (electronic device) in Embodiment 1.

[0012] The display device 150 in Figure 1 is a video see-through type display device and is a head-mounted display (HMD). However, the head-mounted display device (display device 150) is not limited to such an HMD, but may also be a hand-held display (HHD) or a display device such as a tablet or smartphone. In other words, the head-mounted display device can be any display that the user can place in front of their eyes and observe the displayed image on.

[0013] The imaging unit 101 is a camera provided in the display device 150. The imaging unit 101 acquires an image representing the real space by imaging the area in front of the user in front of the display device 150. The imaging unit 101 is, for example, a twin-lens camera having a camera positioned to correspond to the user's left eye and a camera positioned to correspond to the user's right eye. The imaging unit 101 may acquire an image having two image regions with parallax (stereo image) using two optical systems facing the same direction.

[0014] The control unit 102 controls each unit connected to the internal bus 108. The control unit 102 controls the entire system of the display device 150 by reading and executing the program stored in the ROM 107. However, the location where the control unit 109 is provided is not limited to the inside of the display device 150. For example, the control unit 102 may be one component in an information processing device such as a computer, or may be the information processing device (electronic device) itself.

[0015] Based on the information acquired by the imaging unit 101 and the distance measurement unit 104 described later, the control unit 102 acquires the position and orientation (position and orientation) of the real object. The control unit 102 determines (calculates) the position and orientation of the CG based on the information acquired by the imaging unit 101 or the distance measurement unit 104. For example, the control unit 102 controls the position and orientation of a GUI (Graphical User Interface) such as a pointer displayed on the display unit 105 based on the information about the position and orientation of the user's hand. Note that the positions (coordinates) of the CG (GUI) and the real object are represented by three-dimensional coordinate information according to the three-axis orthogonal coordinate system of the X-axis, Y-axis, and Z-axis. The orientation of the CG corresponds to the direction of the GUI in the three-dimensional virtual space. However, the three-dimensional coordinate information is not limited to the information according to the three-axis orthogonal coordinate system, and may be, for example, three-dimensional coordinate information according to the polar coordinate system.

[0016] The storage unit 103 includes a RAM (Random Access Memory) and the like, and can temporarily hold various information for operating the display device 150. The storage unit 103 operates as a work memory for the control unit 102 and the drawing unit 106. The storage unit 103 is also a memory that temporarily holds the captured image captured by the imaging unit 101.

[0017] The distance measurement unit 104 is a depth sensor that can measure the distance between the user (display device 150) and the real object existing around the user. However, the distance measurement unit 104 is deep It is not limited to the temperature sensor, and it is only necessary to be able to measure the distance between the user and the real object. The display device 150 may not include the distance measurement unit 104, and the control unit 102 may calculate the distance between the user and the real object from the image captured by the imaging unit 101.

[0018] The display unit 105 displays the composite image (MR space) generated by the drawing unit 106 to the user. The display unit 105 is arranged near both eyes of the user so as to be visible from both eyes of the user. The display unit 105 is a display element capable of binocular VR display. Although an EL (Electro Luminescence) panel, an LCD, etc. can be applied to the display element, it is not limited thereto.

[0019] The drawing unit 106 is an image processing device (GPU: Graphics Processing Unit). The drawing unit 106 draws so as to superimpose CG on the captured image stored in the storage unit 103, and generates a composite image (MR space). As the CG, graphics, characters, etc. for notification (for example, warning) can also be drawn. When the control unit 102 is provided in an information processing device outside the display device 150, the drawing unit 106 may also be provided in the information processing device.

[0020] The ROM 107 stores various programs such as programs related to the operation of the display device 150. By the control unit 102 reading and executing the programs stored in the ROM 107, various processes can be executed.

[0021] The internal bus 108 connects the imaging unit 101, the control unit 102, the storage unit 103, the distance measurement unit 104, the display unit 105, the drawing unit 106, and the ROM 107. Each unit connected to the internal bus 108 can transmit and receive data to and from each other via the internal bus 108.

[0022] Figure 2 is a schematic diagram showing a user wearing the display device 150 and the user's hand. In Figure 2, the user's hand 201, with the display device 150 attached to their head, is captured by the imaging unit 101. The control unit 30 detects the hand 201 from the captured image and can determine what kind of operation (e.g., hand gesture) the user is performing.

[0023] Figures 3 and 4 are schematic diagrams showing examples of images displayed on the display unit 105 when a user uses the display device 150. Figure 3 shows an example where only the captured image is displayed on the display unit 105. Real objects 301 to 304 are examples of real objects that exist around the user in real space. The dashed line 306 represents a virtual ray emitted from the hand 201. The circle 305, located at the end point of the dashed line 306 when the starting point of the dashed line 306 is the hand 201, represents a pointer used when the user operates the device. The circle 305 moves in conjunction with the movement of the hand 201. The control unit 102 can control the circle 305 to press GUI elements such as buttons displayed on the display unit 105 in accordance with the movement of the hand 201.

[0024] Figure 4 is an example of an image displayed on the display unit 105, which is an image in which computer graphics (CG) are superimposed on the captured image shown in Figure 3. Virtual objects 401 to 404 are CG. In the case of Figure 4, virtual object 401 is displayed on the display unit 105 in such a way that it obscures the real object 301 in Figure 3, and virtual object 402 is displayed in such a way that it obscures the real object 302.

[0025] Figure 5(a) is a data list relating to the CG displayed on the display unit 105, and Figure 5(b) is a data list relating to the real object displayed on the display unit 105. The control unit 102 determines whether or not to notify the user based on the data lists shown in Figures 5(a) and 5(b). Figure 5(a) shows point cloud information representing the position and shape of virtual objects 401 to 404 displayed on the display unit 105, using coordinates. The CG is, for example, stored in ROM by the control unit 102. The program stored in ROM 107 is read and the drawing unit 106 is controlled to display the image on the display unit 105. CG shape information is pre-stored in ROM 107, and the display unit 105 draws the CG using a point cloud based on the shape information. In the data list shown in Figure 5(a), virtual objects 401 to 404 are listed as CG1 to CG4. For example, suppose virtual object 401 is drawn at the coordinate (xc1, yc1, zc1). The point cloud of virtual object 401 is drawn at coordinates such as (xc11, yc11, zc11), (xc12, yc12, zc12), ... In this way, virtual object 401 is drawn at a specific location in the MR space. Here, the location of virtual object 401 may be the centroid of virtual object 401, or it may be the location of the single point (point cloud) closest to the user among the multiple points (point cloud) of virtual object 401. Virtual objects 402 and 404 are similarly drawn at the coordinates (xc2, yc2, zc2) and (xc4, yc4, zc4). The user can determine the shape of the CG from its point cloud information (coordinates of each vertex). Since each CG has different vertices depending on its shape, the number of coordinates included in the point cloud information also differs depending on the shape. Figure 5(a) shows an example where the point cloud information includes coordinates such as (xc11, yc11, zc11), but the point cloud information only needs to show which area the CG occupies in the 3D virtual space, and does not need to include coordinates. Also, the data list shown in Figure 5(a) is updated each time the CG moves relative to the user.

[0026] Figure 5(b) shows point cloud information representing the position and shape of real objects 301 to 304 in the captured image, displayed in coordinates. In the data list shown in Figure 5(b), real objects 301 to 304 are listed as OBJ1 to OBJ4. Figure 5(b) lists point cloud information including each vertex of real object 301, such as coordinates (xо11, yо11, zо11), coordinates (xо12, yо12, zо12), etc. The control unit 102 estimates the shape of real object 301 based on the coordinates of each vertex detected by the distance measurement unit 104. Then, based on the point cloud information of real object 301, the control unit 102 determines that the position of real object 301 is coordinates (xо1, yо1, zо1). In this way, the presence of real object 301 at a specific location in the MR space is detected. Here, the position of the real object 301 may be the estimated center of gravity of the real object 301, or it may be the position of the single point (point cloud) closest to the user among the multiple points (point cloud) of the real object 301. Similarly, the positions of real objects 302 to 304 are determined to be at coordinates (xо2, yо2, zо2) to (xо4, yо4, zо4). Furthermore, the data list shown in Figure 5(b) is updated each time the real object moves relative to the display device 150.

[0027] Figure 6 is a flowchart showing the operation of the display device 150 in Embodiment 1. For example, when the display device 150 is started up, the operation shown in Figure 6 begins.

[0028] In S601, the control unit 102 controls the imaging unit 101 to acquire an image (background image). The control unit 102 stores the acquired image in the storage unit 103 and proceeds to processing in S602.

[0029] In S602, the control unit 102 displays the captured image acquired in S601, or a composite image obtained by combining the captured image with computer graphics, on the display unit 105.

[0030] In S603, the control unit 102 detects the user's hand from the captured image acquired in S601. The control unit 102 determines the instruction indicated by the detected user's hand and proceeds to S604. The control unit 102 detects the user's hand from the captured image using image recognition and determines the movement of the hand. The method for determining the movement of the user's hand detected by image recognition is not particularly limited; for example, it may be a method of comparing the position of the user's hand in the current frame with the position of the user's hand in the previous frame.

[0031] In S604, the control unit 102 determines whether the user's instruction, as determined in S603, will activate the computer graphics (CG). If the control unit 102 determines that the user's instruction (hand movement) will activate the CG, it proceeds to S605; otherwise, it proceeds to S606.

[0032] In S605, the control unit 102 prepares to start the computer graphics (CG) based on the user's instructions determined in S603. This preparation includes, for example, reading CG data from the ROM 107.

[0033] In S606, the control unit 102 determines whether the user's instruction determined in S603 is to operate the computer graphics (CG). If the control unit 102 determines that the user's instruction is to operate the CG, it proceeds to S607; otherwise, it proceeds to S608.

[0034] In S607, the control unit 102 performs CG operation processing based on the user's instructions determined in S603. This CG operation processing includes preparing the CG for movement and performing corresponding processing for the CG. For example, in response to the user's instruction to operate the CG's decision button, the decision function assigned to the decision button is executed. Preparation for CG movement includes, for example, processing to calculate the amount and direction of movement of the CG based on the user's instructions.

[0035] In S608, the control unit 102 controls the drawing unit 106 to update the CG drawing. If preparation for CG activation was performed in S605, the control unit 102 controls the drawing unit 106 to draw the CG so that it is newly superimposed on the captured image. If preparation for CG movement was performed in S607, the already drawn CG is moved. If there is already a drawn CG, the drawn CG may be deleted (cleared) once and then drawn again. If neither preparation for CG activation nor movement has been performed, the control unit 102 does not need to control the drawing unit 106 and does not need to update the CG drawing. Each time the CG drawing is updated, the control unit 102 acquires the position and point cloud information of the CG.

[0036] The drawing position of the computer graphics (CG) in S608 is not particularly limited. For example, if preparation for starting the CG is performed in S605, the CG may be drawn at a predetermined default position, or at the position where the CG was last finished. The position of the drawn CG may be updated based on the operation process in S607.

[0037] In S608, the drawing unit 106 also draws GUI elements such as pointers (circle 305). The position where the drawing unit 106 draws the pointer may be, for example, 10 cm away from the user's hand (or fingers, arm) in the direction indicated by the hand, or it may be at the intersection of the direction indicated by the hand and an object. The distance from the hand to the pointer (ray length) may be changed according to the user's instructions.

[0038] In S609, the control unit 102 controls the distance measurement unit 104 to perform distance measurement. The control unit 102 detects real objects from the image captured by the imaging unit 101 and, based on the measurement results of the distance measurement unit 104, acquires the position and point cloud information of the real objects included in the image captured by the imaging unit 101. In S609, the control unit 102 acquires information on all real objects included in the image.

[0039] In S610, the control unit 102 updates the data list as illustrated in Figure 5. The information in the data list for virtual objects is updated based on the CG position and point cloud information acquired in S608. The data list for real objects is updated based on the real object position and point cloud information acquired in S609. Here, the CG data list is updated in S608 when the shape or position of the CG changes, and the real object data list is updated in S609 when the position of the real object changes. Furthermore, the point cloud information only needs to be updated when it changes; it is not always necessary to update it.

[0040] In S611, the control unit 102 performs object detection (object identification) to detect (identify) hazardous materials and determines whether or not there are hazardous materials (predetermined real-world objects) around the user. If the control unit 102 determines that there are hazardous materials, it proceeds to S612; otherwise, it proceeds to S613. Note that the predetermined real-world objects are not limited to hazardous materials.

[0041] In S611, the control unit 102, when it determines that there are real objects around the user, marks the real objects that are hazardous materials in the data list of surrounding real objects. Based on the positions of real objects acquired sequentially over time, the control unit 102 can determine whether the user is moving or whether the real objects are moving. The control unit 102 determines that moving real objects are hazardous materials and marks the corresponding real objects in the data list. Real objects that are not marked may be determined not to be hazardous materials. The control unit 102 may also determine the type of real object detected by analyzing the captured images. For example, the control unit 102 may determine that real object 301 shown in Figures 3 and 4 is a table, real object 302 is a sofa, real object 303 is a television, and real object 304 is a tree. The control unit 102 may also determine whether the real object is a hazardous material by considering the type of real object. The control unit 102 may determine whether the real object is a hazardous material based solely on the type of real object, or it may also consider other information to determine whether the real object is a hazardous material. For example, when classifying hazardous materials based on whether or not they are moving, items that do not pose a danger even if a user collides with them may be excluded. Alternatively, the size of a real-world object may be considered when determining whether or not it is a hazardous material. For example, small real-world objects may be judged as not being hazardous materials.

[0042] In S612, the control unit 102 determines whether or not the hazardous material is hidden behind the CG. If the control unit 102 determines that the hazardous material is hidden behind the CG, it proceeds to S613; otherwise, it proceeds to S613. The control unit 102 determines whether or not the hazardous material is hidden behind the CG by comparing the position and point cloud information of the hazardous material with the position and point cloud information of the CG. However, the control unit 102 may determine that the hazardous material is hidden behind the CG even if it is not completely hidden. In the display unit 105, the control unit 102 may also determine that the hazardous material is hidden behind the CG if the proportion of the hazardous material hidden by the CG is greater than or equal to a threshold value. For example, if approximately 90% of the hazardous material is hidden by the CG in terms of the display area ratio on the display unit 105, it may be determined that the hazardous material is hidden behind the CG. The threshold value may be arbitrarily set by the user, or it may be changed depending on the type of hazardous material determined in S611. The control unit 102 may determine that a hazardous material is not hidden by the CG if it is outside the range of a predetermined area, even if the hazardous material is hidden by the CG. For example, the control unit 102 may determine (detect) that a hazardous material is outside the range of a predetermined area if it is not in the vicinity of the CG or if it is far from the user.

[0043] Furthermore, in S612, the control unit 102 may predict (detect) the future movement of the real object based on the position of the real object over a predetermined period of time. If the future position of the real object is predicted to be a position where it will be hidden by the computer graphics (i.e., if movement of the real object that will be hidden by the computer graphics is detected), the control unit 102 may also determine that the real object is hidden by the computer graphics.

[0044] In S613, the control unit 102 controls the drawing unit 106 to provide the user with a predetermined notification (warning notification) that alerts them to the real object that was determined to be hidden in S612. The warning notification is, for example, a notification using a warning image such as a predetermined image or text displayed on the display unit 105. The control unit 102 controls the drawing unit 106 to draw the warning image so that it is superimposed on the captured image. The format of the warning notification is not particularly limited; for example, it can be any format that conveys danger to the user. The predetermined notification is not limited to a notification using a warning image displayed on the display unit 105, but may also be a notification using, for example, a predetermined sound output. Furthermore, the control unit 102 also controls the hidden danger If the system determines that a dangerous object is something the user should keep away from, it may provide a warning notification, such as by increasing the transparency of the CG, so that the user can become aware of this.

[0045] In S614, the control unit 102 determines whether a user operation (termination operation) has been performed to terminate the operation shown in Figure 6. The termination operation is, for example, a user operation to turn off the power of the display device 150. The termination operation may or may not be a gesture operation. If the control unit 102 determines that a termination operation has been performed, it terminates the operation shown in Figure 6; otherwise, it proceeds to S601.

[0046] Furthermore, a predetermined range set by the user may be used as the target area, and the detection of real objects to be registered in the data list or marked as hazardous materials may be limited to that target area. The target area is, for example, a range close to the user, such as an area 10m in front of the user, 10m wide, and 10m high. The target area may also be a range displayed at the edge of the display unit 105 that is difficult for the user to recognize. The target area is not limited to a range close to the user, and may also be, for example, a range near the CG, a range hidden by the CG displayed on the display unit 105, and a range including the hidden range and its surroundings. The target area is not limited to a range based on the user, and may also be, for example, a pre-set hazardous area.

[0047] Figure 7 is an example of an image displayed on the display unit 105, which is an image in which a warning image is superimposed on the composite image shown in Figure 4. In the state shown in Figure 7, the real object 301 is hidden by the virtual object 401, and the user cannot see the real object 301. The control unit 102 marks the real object 301 in S611 and determines in S612 that the real object 301 is hidden behind the virtual object 401. Therefore, the control unit 102 controls the drawing unit 106 to draw a composite image on the display unit 105 in which the warning image 701, which is a warning notification, is superimposed on the captured image. From the user's perspective, part of the real object 302 is also hidden by the virtual object 402. However, the control unit 102 does not issue a warning notification in S612 because it has not determined that the real object 302 is hidden behind the virtual object 402.

[0048] Figures 8(a) and 8(b) show examples of images displayed on the display unit 105. In the state shown in Figure 8(a), the real object 801 displayed on the display unit 105 is not hidden behind the CG, so the control unit 102 does not issue a warning notification. In the state shown in Figure 8(b), part of the real object 801 is hidden by the virtual object 802, and a warning 803 is displayed on the display unit 105.

[0049] As described above, the control unit 102 may determine the type of real object in S611. In that case, the control unit 102 may change the threshold used in S612 (the minimum percentage of the area hidden by the CG that determines that the hazardous material is hidden behind the CG) according to the type of hazardous material. In the state shown in Figure 8(b), the control unit 102 has determined that the real object 801 is a saw. The control unit 102 has also determined that the saw is extremely dangerous to the user and has set the threshold lower (smaller) than usual. Based on the changed threshold, the control unit 102 has determined that the real object 801 is hidden behind the virtual object 802. The threshold is the ratio of the area of ​​the part hidden by the CG to the total display area of ​​the real object. For normal hazardous materials, the threshold may be set to 90%, and for extremely dangerous hazardous materials, the threshold may be set to 60%.

[0050] As described above, in the display device according to Embodiment 1, a predetermined notification is given to the user when a predetermined situation is detected in which a predetermined real object (hazardous material) is hidden in the CG. If such a predetermined situation is not detected, the predetermined notification is not given. In this way, the presence of a predetermined real object hidden in the CG can be notified to the user without interfering with user operation of the CG.

[0051] <Embodiment 2> Embodiment 2 of the present invention will now be described. In Embodiment 1, the system determined whether or not there were any hazardous materials among all real-world objects (all real-world objects around the user) captured by the imaging unit 101. Embodiment 2 describes a method in which real-world objects existing in the area hidden by CG are extracted first and then the system operates. Figure 9 is a flowchart detailing the notification process in Embodiment 2.

[0052] In steps S901 to S908, the control unit 102 performs the same processing as in steps S601 to S608 in Figure 6.

[0053] In S909, the control unit 102 extracts the area behind the CG in the captured image (the area hidden by the CG in real space). The area behind the CG in the captured image will henceforth be referred to as the extracted area. For example, the control unit 102 does not cut out the extracted area from the captured image, but rather acquires the position information of the extracted area. The extracted area is not limited to the area hidden by the CG, but may be a wider area, for example, about 10% of the display area ratio on the display unit 105, beyond the area hidden by the CG.

[0054] In S910, the control unit 102 controls the distance measurement unit 104 to perform distance measurement and acquires the position and point cloud information of only the real objects included in the extracted region acquired in S909. For example, if information is acquired only for real objects whose entirety is contained within the extracted region, then information of real objects that are determined to be hidden in S612 of Embodiment 1 (Figure 6) is acquired. Assume that the extracted region is a wide area from the range where 10% of the display area ratio on the display unit 105 is hidden by the CG. In that case, information of real objects where 90% or more of the display area ratio on the display unit 105 is hidden by the CG is acquired. In other words, information of real objects that are determined to be hidden when a 90% threshold is used in S612 is acquired.

[0055] In S911 and S912, the control unit 102 performs the same processing as in S610 and S611 in Figure 6.

[0056] In S913 and S914, the control unit 102 performs the same processing as in S613 and S614 in Figure 6.

[0057] As described above, the display device according to Embodiment 2, like Embodiment 1, can notify the user of the presence of real objects without interfering with user operation of the computer graphics (CG). Furthermore, in the display device according to Embodiment 2, a predetermined real object (hazardous material) is detected only within the range hidden by the CG in the real space. This reduces the processing load and processing time of the display device 150.

[0058] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0059] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Yes, there are. Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0060] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0061] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0062] This embodiment includes the following configurations, methods, and programs. (Composition 1) A detection means for detecting a predetermined situation in which an image of a virtual object is displayed on a head-mounted display device in such a way that it obscures at least a portion of a predetermined real object, A control means that controls the detection means to issue a predetermined notification when the predetermined situation is detected by the detection means. An electronic device characterized by having the following features. (Configuration 2) The system further comprises object detection means for detecting the predetermined real-world object. The electronic device according to configuration 1, characterized by the features described above. (Composition 3) The object detection means detects the predetermined real object from a predetermined range in real space. The electronic device according to configuration 2, characterized by the features described above. (Composition 4) The predetermined range is the vicinity of the virtual object. The electronic device according to configuration 3, characterized by the features described above. (Composition 5) The predetermined range is the range that is hidden by the virtual object. The electronic device according to configuration 3 or 4, characterized by the above. (Composition 6) The predetermined range includes the range hidden by the virtual object and the range surrounding the hidden range. The electronic device according to configuration 3 or 4, characterized by the above. (Composition 7) The detection means further detects the movement of the predetermined real object that results in the predetermined situation, The control means is also controlled to make the predetermined notification when the movement of the predetermined real object that results in the predetermined situation is detected. An electronic device according to any one of configurations 1 to 6 characterized by the above. (Composition 8) The aforementioned predetermined situation is a situation in which the image of the virtual object is displayed on a head-mounted display device such that the proportion of the portion of the predetermined real object hidden by the virtual object relative to the whole of the real object is equal to or greater than a threshold. An electronic device according to any one of configurations 1 to 7, characterized by the features described herein. (Composition 9) The control means uses a value corresponding to the type of predetermined real object hidden by the virtual object as the threshold value. The electronic device according to configuration 8, characterized by the above. (Composition 10) The electronic device according to any one of configurations 1 to 9, characterized in that the control means controls the predetermined notification to be made by displaying predetermined text. (Composition 11) The control means controls the system to provide the predetermined notification by displaying a predetermined image. An electronic device according to any one of configurations 1 to 9 characterized by the above. (Composition 12) The control means controls the system to provide the predetermined notification by outputting a predetermined sound. An electronic device according to any one of configurations 1 to 9 characterized by the above. (Composition 13) The control means controls the virtual object to perform the predetermined notification by increasing its transparency. An electronic device according to any one of configurations 1 to 9 characterized by the above. (method) A detection step for detecting a predetermined situation in which an image of a virtual object is displayed on a head-mounted display device in such a way that it obscures at least a portion of a predetermined real object, A control step which controls the system to issue a predetermined notification when the predetermined situation is detected in the detection step. A method for controlling electronic equipment, characterized by having the following features. (program) A program for causing a computer to function as one of the electronic devices described in any one of configurations 1 to 13. [Explanation of symbols]

[0063] 150: Display device 102: Control unit

Claims

1. A detection means for detecting a predetermined situation in which an image of a virtual object is displayed on a head-mounted display device in such a way that it obscures at least a portion of a predetermined real object, A control means that controls the detection means to issue a predetermined notification when the predetermined situation is detected by the detection means. An electronic device characterized by having the following features.

2. The system further comprises object detection means for detecting the predetermined real-world object. The electronic device according to feature 1.

3. The object detection means detects the predetermined real object from a predetermined range in real space. The electronic device according to feature 2.

4. The predetermined range is the vicinity of the virtual object. The electronic device according to feature 3.

5. The predetermined range is the range that is hidden by the virtual object. The electronic device according to feature 3.

6. The predetermined range includes the range hidden by the virtual object and the range surrounding the hidden range. The electronic device according to feature 3.

7. The detection means further detects the movement of the predetermined real object that results in the predetermined situation, The control means is also controlled to make the predetermined notification when the movement of the predetermined real object that results in the predetermined situation is detected. The electronic device according to feature 1.

8. The aforementioned predetermined situation is a situation in which the image of the virtual object is displayed on a head-mounted display device such that the proportion of the portion of the predetermined real object hidden by the virtual object relative to the whole of the real object is equal to or greater than a threshold. The electronic device according to feature 1.

9. The control means uses a value corresponding to the type of predetermined real object hidden by the virtual object as the threshold value. The electronic device according to feature 8.

10. The electronic device according to claim 1, characterized in that the control means controls the predetermined notification to be made by displaying predetermined text.

11. The control means controls the system to provide the predetermined notification by displaying a predetermined image. The electronic device according to feature 1.

12. The control means controls the system to provide the predetermined notification by outputting a predetermined sound. The electronic device according to feature 1.

13. The control means controls the virtual object to perform the predetermined notification by increasing its transparency. The electronic device according to feature 1.

14. A detection step for detecting a predetermined situation in which an image of a virtual object is displayed on a head-mounted display device in such a way that it obscures at least a portion of a predetermined real object, A control step which controls the system to issue a predetermined notification when the predetermined situation is detected in the detection step. A method for controlling electronic equipment, characterized by having the following features.

15. A program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 13.

Images