Display program and electronic device

Abstract

To improve visibility of a virtual object.SOLUTION: An electronic device has a detection unit that detects a three-dimensional real space, a display unit that displays a virtual object in the three-dimensional virtual space on a display screen, and a display control unit that changes a display mode of the virtual object based on a three-dimensional position of an object existing in the three-dimensional real space detected by the detection unit and a three-dimensional position of the virtual object in the three-dimensional virtual space.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a display program and an electronic device. [Background technology]

[0002] A goggle display system has been disclosed in which an image of a virtual space displayed on a liquid crystal display is reflected by a half mirror to superimpose the image of the virtual space on the real space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-86130 Summary of the Invention

[0004] The electronic device of the first disclosed technology has a detection unit that detects a three-dimensional real space, a display unit that displays a virtual object in a three-dimensional virtual space on a display screen, and a display control unit that changes the display mode of the virtual object based on the three-dimensional position of an object existing in the three-dimensional real space detected by the detection unit and the three-dimensional position of the virtual object in the three-dimensional virtual space.

[0005] The display program of the second disclosed technology causes a processor to execute a detection process for detecting a three-dimensional real space, and a display control process for controlling a display screen for displaying a virtual object in a three-dimensional virtual space, and changing a display manner of the virtual object based on a three-dimensional position of an object existing in the three-dimensional real space detected by the detection process and a three-dimensional position of the virtual object in the three-dimensional virtual space. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is an explanatory diagram showing an example of a system configuration of an electronic device system. [Diagram 2]FIG. 2 is an explanatory diagram showing an example of three-dimensional position data and shape data of a hand. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a hardware configuration of the server. [Figure 4] FIG. 4 is a block diagram showing an example of a hardware configuration of the head mounted display. [Diagram 5] FIG. 5 is an explanatory diagram showing a first example of a change in the virtual object display mode. [Figure 6] FIG. 6 is an explanatory diagram showing a second example of a change in the virtual object display mode. [Figure 7] FIG. 7 is an explanatory diagram showing a third example of a change in the virtual object display mode. [Figure 8] FIG. 8 is a flowchart illustrating a first example of a display mode change processing procedure performed by a head mounted display. [Figure 9] FIG. 9 is a flowchart illustrating a second example of a display mode change processing procedure performed by the head mounted display. [Figure 10] FIG. 10 is an explanatory diagram illustrating an example of dividing a virtual object. [Figure 11] FIG. 11 is a flowchart illustrating a detailed example of the process of dividing the virtual object (step S901) illustrated in FIG. [Figure 12] FIG. 12 is an explanatory diagram showing an example of a division process of a virtual object when a layout algorithm using a dynamics system is applied. [Figure 13] FIG. 13 is an explanatory diagram showing a first example of a virtual object arrangement change. [Figure 14] FIG. 14 is an explanatory diagram showing a second example of a virtual object arrangement change. [Figure 15] FIG. 15 is a flowchart illustrating a first example of a procedure for processing a layout change by a head mounted display. [Figure 16] FIG. 16 is an explanatory diagram illustrating a first example of restricting operations on a virtual object. [Figure 17] FIG. 17 is an explanatory diagram illustrating a second example of restricting operations on a virtual object. [Figure 18]FIG. 18 is a flowchart illustrating an example of a virtual object operation suppression processing procedure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS EXAMPLES

[0007] <System configuration example of electronic equipment system> 1 is an explanatory diagram showing an example of a system configuration of an electronic device system. The electronic device system 100 includes a server 101, head mounted displays Da and Db, and a composite device 112. When there is no need to distinguish between the head mounted displays Da and Db, they are referred to as a head mounted display D.

[0008] The server 101 and the head mounted display D are communicably connected via a network 130 such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0009] The instructor environment 110 includes a head mounted display Da, a multifunction device 112, instructors S1 and S2, and a first work object TaR. The "R" at the end of the first work object TaR is a symbol added to emphasize that it is a real object. Other symbols also have the "R" at the end when emphasizing that they are real objects.

[0010] When there is no need to distinguish between instructors S1 and S2, they will be referred to as instructor S. Head mounted display Da is a display device worn by instructor S1. Instructor S2 does not wear head mounted display D. Head mounted display D has a camera 102 and a display 103. Note that it is sufficient that either instructor S1 or instructor S2 is present as an instructor in instructor environment 110. Camera 102 captures an image of the three-dimensional real space, and outputs image data of the three-dimensional real space (including objects and background within the three-dimensional real space) to head mounted display D. The space in which image data of the three-dimensional real space is displayed is referred to as a three-dimensional virtual space.

[0011] The multifunction device 112 has a TOF (Time Of Flight) camera and a hand tracker. The TOF camera captures images of the instructor S2 and the first work target TaR and measures the distance to the captured objects in the three-dimensional real space. The hand tracker detects the hand of the instructor S2 by infrared rays, recognizes the shape of the hand, and obtains the coordinate values ​​of feature points such as the joints and fingertips of the hand in the captured three-dimensional real space.

[0012] The TOF camera and the hand tracker may be provided separately. In this case, the server 101 recognizes the installation position of the hand tracker based on the coordinate origin of the three-dimensional real space captured by the TOF camera, and corrects the coordinate values ​​of the feature points of the hand, such as the joints and fingertips, detected by the hand tracker so that they match the coordinate values ​​in the three-dimensional real space captured by the TOF camera.

[0013] The first work object TaR is an actual device, such as a machine tool, for which the instructor S demonstrates the operation of the work instruction to the worker W. The virtual first work object TaV is a virtual three-dimensional model (virtual object) of the first work object TaR, which is generated by the head-mounted display Da and displayed on the display 103, and is visible to the instructor S1.

[0014] The "V" at the end of the virtual first work object TaV is a symbol added to emphasize that it is a virtual object. When emphasizing that it is a virtual object, other symbols are also added to the end with a "V". When there is no need to distinguish between the first work object TaR and the virtual first work object TaV, they are written as the first work object Ta.

[0015] In the worker environment 120, which is a three-dimensional real space, there are a head mounted display Db, a worker W, and a second work object TbR. The head mounted display Db is a display device worn by the worker W. The second work object TbR is an actual device operated by the worker W, and is, for example, the same type of machine tool as the first work object TaR.

[0016] When the first work object Ta and the second work object Tb are not differentiated, they are referred to as work object T. When the instructor S and the worker W are not differentiated, they are referred to as user S / W. The first work object TaR may be simplified in shape, etc., compared to the second work object TbR.

[0017] The camera 102 is an image sensor (RGB-D camera) that converts red (R), green (G), and blue (B) light in the visible light range into electrical signals and acquires the distance to the subject by, for example, a TOF method. The camera 102 acquires the three-dimensional position of the subject based on the planar position of the subject on the imaging surface and the distance to the subject.

[0018] Display 103 has a display screen. If display 103 is of a video transmission type (video see-through), then a moving image captured by or played back by camera 102 is displayed on the display screen. On the other hand, if display 103 is of an optical transmission type (optical see-through), then the user can view the real object in front of the user and can display virtual data superimposed on it.

[0019] 2 is an explanatory diagram showing an example of three-dimensional position data and shape data of a hand. (A) shows three-dimensional position data 201 of the hand H in a local three-dimensional coordinate system 200 in which an image of the user's hand H is captured. (B) shows shape data 202 of the hand H in the local three-dimensional coordinate system 200 in which an image of the user's hand H is captured.

[0020] The hand H of the instructor S is written as hand Ha, and the hand H of the worker W is written as hand Hb. When distinguishing between the left and right hand H, r and l are added to the end of the code. For example, the right hand of the instructor S is written as Har, and the left hand of the instructor S is written as Hal.

[0021] The three-dimensional position data 201 of the user's hand H is acquired from the camera 102, which is an RGB-D camera that can acquire distance information in addition to an image. The three-dimensional position data 201 of the user's hand H is a collection of feature points of the user's hand H. The feature points have three-dimensional coordinate values. The feature points are, for example, skeletal points that are the fingertips and joints of the user's hand H. The feature points may also be points that make up the outline of the user's hand H.

[0022] Shape data 202 of the hand H is generated from three-dimensional position data 201 indicating the three-dimensional position of the hand H. Specifically, for example, when the three-dimensional position data 201 is a collection of skeletal points of the hand H, the shape data 202 of the hand H is generated by connecting the skeletal points along the skeleton of the hand H and generating the contour of the hand H by providing a predetermined width to the skeletal points and the links between the skeletal points. Also, when the three-dimensional position data 201 is a collection of points constituting the contour of the hand H, the shape data 202 of the hand H is generated by connecting adjacent points. Note that the virtual data of the hand H may be configured as a link model in which the skeletal points are linked to each other without providing a predetermined width to the links between the skeletal points.

[0023] The head mounted display D can perform existing object recognition using machine learning or shape pattern matching from the feature points of an object. Alternatively, the head mounted display D may transmit the feature points of an object to the server 101, and the server 101 may perform object recognition from the feature points of the object and transmit the object recognition result to the head mounted display D.

[0024] <Hardware configuration example for Server 101> FIG. 3 is a block diagram showing an example of a hardware configuration of the server 101. The server 101 has a processor 301, a storage device 302, an input device 303, an output device 304, and a communication interface (communication IF) 305. The processor 301, the storage device 302, the input device 303, the output device 304, and the communication IF 305 are connected by a bus 306. The processor 301 controls the server 101. The storage device 302 serves as a working area for the processor 301. The storage device 302 is a non-transient or temporary recording medium that stores various programs and data. Examples of the storage device 302 include a read only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device 303 inputs data. Examples of the input device 303 include a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, and a sensor. The output device 304 outputs data. The output device 304 may be, for example, a display, a printer, or a speaker. The communication IF 305 is connected to the network 130 and transmits and receives data.

[0025] <Example of hardware configuration for head mounted display D> 4 is a block diagram showing an example of a hardware configuration of the head mounted display D. The head mounted display D includes a processor 301, a storage device 302, a communication IF 305, and a bus 306, as well as an operation device 403, a display 103, a camera 102, a microphone 404, and a speaker 405.

[0026] The operation device 403 accepts various inputs, such as starting, stopping, and pausing the recording of a moving image captured by the camera 102, and starting, stopping, and pausing the playback of the recorded moving image, by operation of the instructor S or the worker W. The operation device 403 may also accept recording of a still image captured by the camera 102, etc.

[0027] The microphone 404 converts sound around the head mounted display D into an electrical signal and inputs it into the head mounted display D. The speaker 405 converts the electrical signal related to sound into sound and outputs it to the outside of the head mounted display D.

[0028] Note that communication between the head mounted displays D may be performed directly or via the server 101.

[0029] The camera 102 functions as a detection unit that detects the three-dimensional real space. The display 103 functions as a display unit that displays a virtual object in the three-dimensional virtual space on a display screen. Functions that are realized by the processor 301 executing the program stored in the storage device 302 include a display control unit, a recognition unit, and a division unit.

[0030] The display control unit changes a display mode of the virtual object based on a three-dimensional position of an object existing in the three-dimensional real space and a three-dimensional position of the virtual object in the three-dimensional virtual space.

[0031] The recognition unit recognizes whether or not the object is a work target. Details of the display control by the display control unit and details of the recognition by the recognition unit will be described later with reference to Figs.

[0032] If the virtual object is configured to be divisible into a plurality of partial virtual objects, the dividing unit divides the virtual object into the plurality of partial virtual objects. Details of the division by the dividing unit will be described later with reference to Figs.

[0033] In the following drawings, in order to distinguish between real objects in the three-dimensional real space and objects in the three-dimensional virtual space that are captured by the camera 102 of the head mounted display D and displayed on the display 103, the symbols of the real objects are enclosed in parentheses to denote them as objects. For example, an object of the user's real right hand Hr is denoted as the right hand (Hr), an object of the real left hand Hl is denoted as the left hand (Hl), and an object of a real work target T is denoted as the work target (T). Note that objects that are not captured by the camera 102 but are generated by the head mounted display D and displayed on the display 103 are denoted without their symbols enclosed in parentheses.

[0034] <Change virtual object display mode> Fig. 5 is an explanatory diagram showing a virtual object display mode change example 1. Fig. 5 shows a state in which a user S / W wearing a head mounted display D captures an image of a work target T and its surroundings in a three-dimensional real space 500R in which the work target T exists, using a camera 102 of the head mounted display D. A three-dimensional captured image space 500I is a three-dimensional image of the three-dimensional real space 500R displayed on a display 103 of the head mounted display D by the image capture, and is defined by a world coordinate system 501.

[0035] The world coordinate system 501 is a coordinate system defined by the X-axis, Y-axis, and Z-axis that are orthogonal to each other. The intersection of the X-axis, Y-axis, and Z-axis is the origin. The terminal ends (ends with arrowheads) of the X-axis, Y-axis, and Z-axis are the positive directions, and the starting ends (ends without arrowheads) are the negative directions. For example, in the case of the Z-axis, the Z positive direction is the direction from the starting end of the Z-axis to the terminal end, and the Z negative direction is the direction from the terminal end of the Z-axis to the starting end. The same is true for the X-axis and Y-axis. When the positive and negative directions are not distinguished, they are simply referred to as the X direction, Y direction, and Z direction. In addition, the plane stretched by the X-axis and Y-axis is referred to as the XY plane. In the case of the X-axis and Z axis, it is referred to as the XZ plane, and in the case of the Y-axis and Z axis, it is referred to as the YZ plane.

[0036] The XY plane corresponds to the screen of the display 103. The Z axis corresponds to the depth of the screen of the display 103.

[0037] Virtual objects 502A and 502B are the same content but have different transparency. Specifically, for example, virtual object 502A has a lower transparency than virtual object 502B. When there is no need to distinguish between virtual objects 502A and 502B, they are simply referred to as virtual objects 502.

[0038] The virtual object 502 is electronic data of an operation manual including text and image data for the user S / W to operate the work target T. The virtual object 502 can turn pages, enlarge, reduce, move, and rotate within the three-dimensional imaging space 500I automatically or according to the detection result of the movement of the hand (H) of the user S / W.

[0039] The head mounted display D displays a virtual object 502A in the three-dimensional imaging space 500I by the hand (H) of the user S / W selecting an item (not shown) for displaying the virtual object 502 displayed on the display 103.

[0040] Furthermore, the head mounted display D may refer to related information that associates feature information such as the shape, color, and dimensions of the work target T with the virtual object 502, identify feature information of the work target (T) by existing object recognition, obtain the virtual object 502 related to the identified work target (T), and display it in the three-dimensional imaging space 500I. Note that the related information may be stored in the storage device 302 of the head mounted display D, or may be stored in the storage device 302 of the server 101.

[0041] As shown in FIG. 5(A), the virtual object 502 is displayed at a preset default position in the three-dimensional imaging space 500I. In FIG. 5(A), in the Z direction indicating the depth of the three-dimensional imaging space 500I as seen from the user S / W, the virtual object 502 is located in front of the work object (T) (more specifically, in front of the frontmost surface (503) of the work object (T)) and overlaps with the work object (T) in the XY plane. In this case, the user S / W cannot visually recognize the part of the work object (T) that overlaps with the virtual object 502 in the XY plane in the Z direction. Therefore, when the work object (T) is to be operated, the visibility of the work object (T) is reduced due to the presence of the virtual object 502, and the operability of the work object (T) is deteriorated.

[0042] Therefore, as shown in FIG. 5(B), when the head mounted display D detects the positional relationship between the virtual object 502A and the work object (T) as shown in FIG. 5(A), it increases the transparency of the virtual object 502A to the extent that the work object (T) can be seen, and converts it into a virtual object 502B.

[0043] The transparency at which the work target (T) can be seen is, for example, the transparency of the virtual object 502 within a range where the work target (T) is displayed without being hidden by the virtual object 502 in the overlapping portion between the virtual object 502A and the work target (T) and the virtual object 502 does not disappear. This allows the user S / W to see the portion of the work target (T) that overlaps with the virtual object 502B in the XY plane. Note that instead of increasing the transparency of the virtual object 502A to a level at which the work target (T) can be seen, the transparency of the work target (TA) may be decreased to a level at which the work target (T) can be seen.

[0044] In this way, the overlapping portion of the work object (T) hidden by the virtual object 502A can be easily visually recognized, thereby improving the operability of the work object (T). In addition, the relative position of the virtual object 502B in the three-dimensional imaging space 500I and the work object T in the three-dimensional real space 500R can be easily grasped. Therefore, when trying to operate the virtual object 502A, unintended interference between the hand H or a tool held by the hand H and the work object T can be suppressed, and injury to the hand H and damage to the tool and the work object T can be suppressed.

[0045] In FIG. 5, the head mounted display D increases the overall transparency of the virtual object 502A and converts it into the virtual object 502B, but it may also be possible to increase the transparency of only the portion of the virtual object 502A that overlaps with the work target object (T) in the XY plane and convert it into the virtual object 502B.

[0046] Furthermore, the head mounted display D may convert the color of the virtual object 502A instead of increasing the transparency of the entire virtual object 502A or the overlapping portion with the work target (T) in the XY plane. In this case, the user S / W cannot visually recognize the overlapping portion of the work target (T) due to the color-converted virtual object 502B, but can visually recognize that the virtual object 502B and the work target (T) overlap in the XY plane.

[0047] In the case of color conversion, the head mounted display D may blink the color of the color-converted overlapping portion of the virtual object 502B in order to draw the attention of the user S / W.

[0048] Furthermore, the head mounted display D may not display the virtual object 502A. Specifically, for example, the head mounted display D may delete the virtual object 502A from the three-dimensional imaging space 500I, and may set the transmittance at which the virtual object 502A disappears. In this case, only the portion of the virtual object 502A that overlaps with the work target object (T) on the XY plane may be set to the transmittance at which the virtual object 502A disappears.

[0049] Fig. 6 is an explanatory diagram showing a second modified example of the virtual object display mode. Unlike Fig. 5, in the Z direction indicating the depth of the three-dimensional imaging space 500I as seen from the user S / W, the virtual object 502A is located behind the work target (T) (more specifically, behind the frontmost surface (503) of the work target (T)) and overlaps with the work target (T) in the XY plane. In this case, the user S / W visually recognizes the part of the virtual object 502 that overlaps with the work target (TA) in the XY plane in the Z direction as being overlapped with the work target (TA).

[0050] For this reason, as shown in FIG. 6(B), when the head mounted display D detects the positional relationship between the virtual object 502A and the work target object (TA) as shown in FIG. 6(A), it increases the transparency of the virtual object 502A to the point where the virtual object 502A becomes invisible, and converts it into a virtual object 502B.

[0051] The transparency at which virtual object 502 cannot be seen is, for example, a transparency at which the content of virtual object 502A cannot be recognized in the overlapping portion between virtual object 502 and work target object (T). This makes it impossible for the user S / W to see the portion of virtual object 502 that overlaps with work target object (TB) on the XY plane. Note that instead of increasing the transparency of virtual object 502A to the extent that virtual object 502 cannot be seen, the transparency of work target object (T) may be decreased to the extent that virtual object 502 cannot be seen.

[0052] As a result, the user S / W cannot view the virtual object 502A or the contents of the virtual object 502A in the portion of the virtual object 502A that overlaps with the work object (T) on the XY plane, and therefore will not operate the virtual object 502A. As a result, unintended interference between the hand H or a tool held by the hand H and the work object T when attempting to operate the virtual object 502 can be suppressed, and injury to the hand H and damage to the tool and the work object T can be suppressed. In addition, the visibility of the work object (T) that is the object of operation can be improved.

[0053] Note that, instead of increasing the transparency of virtual object 502A to the extent that virtual object 502A is not visible, the transparency of virtual object 502A may be increased to the extent that virtual object 502 is visible. The transparency at the extent that virtual object 502A is visible is, for example, a transparency rate at which the content of virtual object 502A can be recognized in the overlapping portion between virtual object 502A and work target object (T). This allows the user S / W to clearly see both the work target object (TB) and the portion of virtual object 502 that overlaps with work target object (TB) on the XY plane.

[0054] In addition, in FIG. 6, head mounted display D converts virtual object 502A into virtual object 502B by increasing the overall transparency of virtual object 502A, but it is also possible to convert virtual object 502A into virtual object 502B by increasing the transparency of only the portion of virtual object 502A that overlaps with virtual object 502 in the XY plane.

[0055] Furthermore, the head mounted display D may convert the color of the work object (TA) instead of increasing the transparency of the entire work object (TA) or the overlapping portion with the virtual object 502 in the XY plane. In this case, the user S / W cannot see the virtual object 502 in the overlapping portion due to the color-converted work object (TB), but can see that they overlap in the XY plane.

[0056] In the case of color conversion, the head mounted display D may blink the color of the color-converted overlapping portion of the work object (TB) in order to draw the attention of the user S / W.

[0057] Fig. 7 is an explanatory diagram showing a third change example of a virtual object display mode. Fig. 7 shows an example of dividing a virtual object 502. The virtual object 502 has a data structure in which a plurality of blocks are combined, and can be divided in block units. In Fig. 7, the virtual object 502 is divided into two. One of the divided data is referred to as a divided virtual object 502d1, and the other is referred to as a divided virtual object 502d2. Note that, although Fig. 7 shows the division into two, it may be divided into three or more.

[0058] (A) shows the same state as (A) in Fig. 6. As shown in (B), when the head mounted display D detects the positional relationship between the virtual object 502 and the work target (TA) as shown in (A), it divides the virtual object 502 into divided virtual objects 502d1 and 502d2 in the XY plane direction and moves the divided virtual objects 502d1 and 502d2 in the XY plane direction so as not to overlap with the work target (T).

[0059] In this way, by dividing the virtual object 502, the user S / W can visually recognize the divided virtual objects 502d1 and 502d2 without being displayed overlapping with the work object (T), and can easily grasp the content of the virtual object 502. In addition, unintended interference between the hand H or a tool held by the hand H and the work object T when attempting to operate the virtual object 502 can be suppressed, and injury to the hand H and damage to the tool and the work object T can be suppressed. Furthermore, since the work object (T) is not displayed overlapping with the virtual object 502, the operability of the work object (T) can be improved.

[0060] In addition, in FIG. 7, when the virtual object 502 is located behind the work object (T) (more specifically, behind the frontmost surface (503) of the work object (T)) and overlaps with the work object (T) in the XY plane, the head mounted display D divides the virtual object 502. However, when the virtual object 502 is located in front of the work object (T) (more specifically, in front of the frontmost surface (503) of the work object (T)) and overlaps with the work object (T) in the XY plane, the head mounted display D may also divide the virtual object 502.

[0061] <Display mode change processing procedure> Fig. 8 is a flowchart showing a first example of a display mode change processing procedure by the head mounted display D. Fig. 9 is a flowchart showing a second example of a display mode change processing procedure by the head mounted display D. The head mounted display D acquires feature points of an object in the three-dimensional imaging space 500I in the local three-dimensional coordinate system 200 captured by the camera 102 (step S801).

[0062] The head mounted display D performs coordinate transformation of the feature points of the object in the three-dimensional imaging space 500I acquired in step S801 into feature points in the world coordinate system 501 (step S802). It is assumed that the transformation matrix from the coordinate points in the local three-dimensional coordinate system 200 to the coordinate points in the world coordinate system 501 is set in advance.

[0063] The head mounted display D acquires a virtual object (for example, the virtual object 502) in the world coordinate system 501 (step S803).

[0064] The head mounted display D determines whether or not the object in the three-dimensional imaging space 500I and the virtual object overlap in the XY plane direction of the world coordinate system 501 (step S804).

[0065] If the object in the three-dimensional imaging space 500I and the virtual object do not overlap in the XY plane direction (step S804: No), the head-mounted display D displays the object in the three-dimensional imaging space 500I and the virtual object in the three-dimensional imaging space 500I (step S805), and ends the series of processes.

[0066] On the other hand, when the object in the three-dimensional imaging space 500I and the virtual object overlap in the XY plane direction (step S804: Yes), the head mounted display D determines whether or not the virtual object is located in front of the object in the three-dimensional imaging space 500I in the Z direction (step S806). Specifically, for example, the head mounted display D determines whether or not the virtual object is located in front of the surface (for example, the surface (503) of the work target object (T)) on the front side in the Z direction of the object in the three-dimensional imaging space 500I.

[0067] If the virtual object is in front of the object in the three-dimensional imaging space 500I (step S806: Yes), the head mounted display D determines whether the object in the three-dimensional imaging space 500I is a work target (step S807). This determination process is executed by inputting image data captured by the camera 102 of the head mounted display D to a learning model.

[0068] This learning model is an identification model trained by a teacher data set, which is a set of a work object name and image data of the work object from multiple directions created based on CAD data, and is stored in the storage device 302. Note that there may be multiple types of teacher data set.

[0069] The user S / W specifies the name of the work object before performing the work. The camera 102 captures the image of the object to be worked on, and the processor 301 inputs the image data captured by the camera 102 to the learning model. The learning model identifies an object corresponding to the learned work object from among multiple objects included in the captured image data, and outputs the learned work object name together with the identified image data.

[0070] When the name of the work object specified by the user S / W before performing the work matches the name of the work object output from the learning model, the processor 301 determines that the object in the three-dimensional imaging space 500I is the work object. Note that when the captured image data includes multiple types of work objects, the learning model may output the corresponding multiple types of work object names.

[0071] Specifically, in this determination process, for example, the head mounted display D recognizes an object in the three-dimensional imaging space 500I by object recognition. When the recognized object and the hand (H) of the user S / W overlap on the XY plane of the three-dimensional imaging space 500I and a specific gesture of the hand (H) of the user S / W (for example, a gesture in which the hand (H) interferes with one or more vertices of the recognized object) is detected, the head mounted display D determines the recognized object as the work target (T).

[0072] In addition, the head-mounted display D may determine that the recognized object is the work target (T) when it detects a gesture in which the hand (H) of the user S / W draws a figure that includes the recognized object on the XY plane of the three-dimensional imaging space 500I, even if the hand (H) does not overlap with the recognized object.

[0073] If it is determined in step S807 that the object is not a work object (T) (step S807: No), the head-mounted display D displays the object in the three-dimensional imaging space 500I and the virtual object 502 in the three-dimensional imaging space 500I (step S805), and ends the series of processes.

[0074] On the other hand, in step S807, if it is determined that the object is a work target object (T) (step S807: Yes), the head mounted display D determines whether or not the virtual object 502 is divisible (step S808). If the virtual object 502 is composed of multiple blocks, the virtual object 502 is divisible. If the virtual object 502 is divisible (step S808: Yes), the process proceeds to FIG. 9.

[0075] Note that even if the virtual object 502 can be divided (step S808: Yes), when the head mounted display D detects a hand (H) gesture indicating a selection of a change in the display mode of the object / work target (T) in the three-dimensional imaging space 500I and the virtual object 502, the head mounted display D may execute the processing of step S809 without proceeding to the processing of FIG. 9.

[0076] In step S808, if the virtual object 502 cannot be divided (step S808: No), the head mounted display D calculates the overlap range in the XY plane between the object in the 3D imaging space 500I (step S806: No) / work object (T) (step S807: Yes) and the virtual object 502 (step S809).

[0077] As shown in FIG. 5 and FIG. 6, the head mounted display D changes the display mode of the overlapping range of either the object / work target (T) in the three-dimensional imaging space 500I or the virtual object 502 (step S810).

[0078] Specifically, for example, when the virtual object 502 is located in front of the object / work object (T) in the three-dimensional imaging space 500I (step S806: Yes), the head-mounted display D calculates the overlap range of the virtual object 502 where the virtual object 502 overlaps with the work object (T) in the XY plane direction.

[0079] In addition, if the virtual object 502 is not located in front of the object / work object (T) in the three-dimensional imaging space 500I (step S806: No), the head-mounted display D calculates the overlap range of the object / work object (T) in the three-dimensional imaging space 500I where the object / work object (T) in the three-dimensional imaging space 500I overlaps with the virtual object 502 in the XY plane direction.

[0080] Next, the head mounted display D changes the display mode of the overlapping range calculated in step S809 (step S810). Specifically, for example, when the overlapping range of the virtual object 502 is calculated in step S809, the head mounted display D changes the display mode of at least the overlapping range of the virtual object 502 among the virtual objects 502 as shown in FIG. 5 and FIG. 6.

[0081] Note that changes in the display mode include changes in transparency, color conversion, and blinking, as shown in FIG. 5 and FIG.

[0082] After that, the head mounted display D displays the object / work target (T) and the virtual object 502 in the three-dimensional imaging space 500I after the display mode change in step S810 (step S811), and ends the series of processes.

[0083] 9, when it is determined in step S808 that the virtual object 502 is divisible (step S808: Yes), the head mounted display D executes division processing of the virtual object 502 (step S901). Then, the head mounted display D displays the object / work target and the divided virtual object 502 in the three-dimensional imaging space 500I (step S902), and ends a series of processes. Here, the division processing of the virtual object (step S901) will be specifically described.

[0084] Fig. 10 is an explanatory diagram showing an example of division of a virtual object. In Fig. 10, a virtual object 1000 is composed of three blocks b1 to b3. The virtual object 1000 can be divided into the blocks b1 to b3. Note that the head mounted display D may divide the virtual object 1000 into a maximum number of blocks that is equal to or greater than two.

[0085] For example, virtual object 1000 can be divided into block b1 and a combined block of blocks b2 and b3, block b2 and a combined block of blocks b1 and b3, and block b3 and a combined block of blocks b1 and b2.

[0086] FIG. 11 is a flowchart showing a detailed example of the process procedure of dividing the virtual object shown in FIG. 9 (step S901). FIG. 12 is an explanatory diagram showing an example of dividing the virtual object 1000 when a layout algorithm using a dynamic system is applied. In FIG. 11, a graph layout algorithm modeled by a dynamic system is applied as an example of an algorithm for adjusting the position. Also, in FIG. 12, the virtual object 1000 is positioned on the back side in the Z direction relative to the front surface (503) of the work target object (T). The dividing process (step S901) of the virtual object shown in FIG. 11 will be described below with reference to FIG. 12.

[0087] 11, the head mounted display D determines a block that has the smallest overlapping range with the object / work target in the XY plane direction from the block set as an anchor block (step S1101). The block set is all blocks b1 to b3 included in the virtual object 1000.

[0088] An anchor block is a block that is a reference for other blocks in a block set. In the example of Fig. 12, block b1 (A) is the block with the smallest overlapping range with the work object (T). Therefore, the head mounted display D determines block b3 as the anchor block.

[0089] 11, the head mounted display D detects the arrangement direction of the other blocks relative to the anchor block in the XY plane direction (step S1102). The arrangement direction is the direction from the arrangement position of the anchor block to the arrangement position of the other blocks other than the anchor block.

[0090] Specifically, for example, the head mounted display D detects the direction indicated by the line segment connecting the center of gravity of the anchor block and the center of gravity of the other blocks. Then, the head mounted display D calculates the angular difference between the direction indicated by the line segment and the X-axis and the Y-axis, and detects the direction indicated by the axis with the smallest angular difference as the arrangement direction.

[0091] In the example of FIG. 12, (A) the arrangement direction of block b1 is the Y plus direction, the arrangement direction of block b2 is the X minus direction, and the arrangement direction of block b3 is the X plus direction.

[0092] 11, the head mounted display D moves the anchor block in a direction away from the representative point of the object / work object so that the object / work object and the anchor block do not overlap in the XY plane direction (step S1103). The representative point of the object / work object is, for example, a point on the surface (503) of the object / work object in the XY plane direction, such as the center of gravity. Note that the head mounted display D may determine the position indicated by the user's hand (H) as the representative point.

[0093] Then, the head mounted display D moves the anchor block in a direction away from the representative point of the object / work target. In the example of Fig. 12, (B) head mounted display D moves the anchor block b3 in the X plus direction away from the representative point P on the surface (503) so as not to overlap with the surface (503). After the movement, the position of the anchor block b3 is fixed.

[0094] 11, the head mounted display D places other blocks in the XY plane direction from the representative point of the object / work target in the placement direction (step S1104). Specifically, for example, in the example of Fig. 12, (B) head mounted display D places block b1 at a position in the Y plus direction from the representative point P, and places block b2 at a position in the X minus direction from the representative point P.

[0095] 11, the head mounted display D moves the other blocks in the arrangement direction until they do not overlap with the object / work target in the XY plane direction (step S1105), and proceeds to step S902. Specifically, in the example of Fig. 12, (C) head mounted display D moves block b2 in the negative X direction from the representative point P until they do not overlap with the work target (T). Also, (D) head mounted display D moves block b1 in the positive Y direction from the representative point P until they do not overlap with the work target (T).

[0096] In this way, according to the first embodiment, by changing the display mode of the virtual object, it is possible to display the virtual object 502 so as not to overlap with the work target object (T) in the three-dimensional imaging space 500I. Therefore, it is possible to improve the visibility of the work target object (T) and the virtual objects 502 and 1000. EXAMPLES

[0097] Next, a second embodiment will be described. In the first embodiment, an example was described in which the display mode of either the work object (T) or the virtual object 502, 1000 was changed depending on the Z-direction positional relationship between the work object (T) and the virtual object 502, 1000 that overlap in the XY plane direction. In the second embodiment, an example is described in which the position of the virtual object 502, 1000 in such a positional relationship is changed so that it does not overlap with the work object (T) in the XY plane direction. Note that in the second embodiment, the differences from the first embodiment will be mainly described, so that the same components as those in the first embodiment are denoted by the same reference numerals and their description will be omitted.

[0098] Fig. 13 is an explanatory diagram showing virtual object arrangement change example 1. Fig. 13 shows an example in which a virtual object 502 is moved in the XY plane direction.

[0099] (A) shows the same state as (A) in Fig. 7. As shown in (B), when the head mounted display D detects the positional relationship between the virtual object 502 and the work target (T) as shown in (A), it moves the virtual object 502 in the XY plane direction so that the virtual object 502 does not overlap with the work target (T).

[0100] Specifically, for example, the head mounted display D calculates the area of ​​the work target (T) in the XY plane and the area of ​​the virtual object 502 in the XY plane, and moves the virtual object 502 to the outside of the area of ​​the work target (T) in the XY plane. In addition, the head mounted display D may return the virtual object 502 to its original position from position (B) to position (A) after a certain period of time has elapsed.

[0101] The head mounted display D may detect the movement direction of the virtual object 502 in the XY plane from the gesture of the hand (H) of the user S / W, and move the virtual object 502 in the detected movement direction. For example, in the state (A), when the shape of the hand (H) is a first shape indicating overlap avoidance of the virtual object 502 in the XY plane, the head mounted display D may detect the first shape, and when detecting the movement of the hand (H) in the X plus direction while maintaining the first shape, may move the virtual object 502 in the X plus direction.

[0102] Similarly, in a state transitioned from (A) to (B), when the shape of the hand (H) is a second shape indicating the return of the virtual object 502 to its original position, the head mounted display D may detect the second shape, and when the head mounted display D detects movement of the hand (H) in the minus X direction while maintaining the second shape, it may move the virtual object 502 in the minus X direction to return the virtual object 502 to its original position as shown in (A).

[0103] Furthermore, the head mounted display D may move the virtual object 502 to the widest area among areas in the three-dimensional image space 500I in which the work object (T) is not displayed. The area in the three-dimensional image space 500I in which the work object (T) is displayed changes as the user S / W proceeds with the work, but if the virtual object 502 is placed in the widest area, the chances of the work object (T) and the virtual object 502 overlapping are reduced. As a result, the virtual object 502 does not need to be moved frequently, making it easier for the user to check the content of the virtual object 502.

[0104] Furthermore, when the work target object (T) and the virtual object 502 overlap, the head mounted display D may move the virtual object 502 in a direction that shortens the moving distance of the virtual object 502. As a result, the virtual object 502 does not move significantly, and the user can easily check the content of the virtual object 502.

[0105] Fig. 14 is an explanatory diagram showing virtual object arrangement change example 2. Fig. 14 shows an example in which a virtual object 502 is moved in the Z direction. (A) shows the same state as (A) in Fig. 13.

[0106] Specifically, for example, the head mounted display D converts the Z direction coordinate value of the virtual object 502 into a coordinate value in front of the Z direction coordinate value of the surface (503), thereby moving the virtual object 502 to a position in front of the surface (503). Furthermore, the head mounted display D may return the virtual object 502 to its original position, from position (B) to position (A), after a certain period of time has elapsed.

[0107] The head mounted display D may detect the movement direction of the virtual object 502 in the Z axis from the gesture of the hand (H) of the user S / W, and move the virtual object 502 in the detected movement direction. For example, in the state (A), when the shape of the hand (H) is a third shape indicating movement of the virtual object 502 in the Z direction, the head mounted display D may detect the third shape, and when it detects movement of the hand (H) in the negative Z direction while maintaining the third shape, it may move the virtual object 502 in the negative Z direction.

[0108] Similarly, in a state transitioned from (A) to (B), when the shape of the hand (H) is a second shape indicating the return of the virtual object 502 to its original position, the head mounted display D may detect the second shape, and when the head mounted display D detects movement of the hand (H) in the positive Z direction while maintaining the second shape, it may move the virtual object 502 in the positive Z direction to return the virtual object 502 to its original position as shown in (A).

[0109] In this way, by changing the placement position of the virtual object 502, the user S / W can visually recognize the virtual object 502 without it being displayed overlapping with the work target (T). Also, unintended interference between the hand H or a tool held by the hand H and the work target T when attempting to operate the virtual object 502 can be suppressed, and injury to the hand H and damage to the tool and the work target T can be suppressed.

[0110] <Relocation procedure> Fig. 15 is a flowchart showing a first example of a procedure for processing a position change by the head mounted display D. Steps S801 to S809 are the same as those in Fig. 8, and therefore the description thereof will be omitted.

[0111] After step S809, the head mounted display D moves the virtual object 502 to a position where it does not overlap with the physical object / work target object, as shown in FIG. 13 and FIG. 14 (step S1510).

[0112] Then, the head mounted display D displays the object / work target (T) and the virtual object 502 in the three-dimensional imaging space 500I after the movement in step S1510 (step S811), and ends the series of processes.

[0113] In this way, according to the second embodiment, by moving the virtual objects 502 and 1000, the virtual object 502 can be displayed so as not to overlap with the work target object (T) in the three-dimensional imaging space 500I. Therefore, the visibility of the work target object (T) and the virtual objects 502 and 1000 can be improved. EXAMPLES

[0114] Next, a third embodiment will be described. In the third embodiment, an example in which the operation of the virtual object 502 by the user S / W is suppressed will be described. Note that in the third embodiment, the differences from the first and second embodiments will be mainly described, so that the same components as those in the first and second embodiments will be given the same reference numerals and the description thereof will be omitted.

[0115] 16 is an explanatory diagram showing an example 1 of suppressing operation of a virtual object 502. In (A), the right hand (Hr) and right arm (Ar) of the user S / W, a tool (1600) held by the right hand (Hr), and a virtual object 502 are displayed in a three-dimensional imaging space 500I.

[0116] The head mounted display D acquires feature points of the object group by the camera 102 (or the composite device 112) and executes object recognition (object recognition may be executed by the server 101). As a result, the head mounted display D recognizes that the right hand (Hr) is in a shape holding a tool 1600, based on the three-dimensional position data 201 of the right hand (Hr). In addition, the head mounted display D recognizes that the tool 1600 is being held by the right hand Hr, based on the three-dimensional position data of the tool (1600).

[0117] The head mounted display D may recognize that the right hand (Hr) is holding the tool 1600 based only on the recognition result of the shape of the right hand (Hr). In this case, since object recognition of the tool (1600) is not required, it is possible to reduce the calculation load. Also, when an unlearned tool is being held, if the unlearned tool is recognized as not being a tool, it is possible to prevent the virtual object 502 from being hidden.

[0118] Furthermore, when the position of a right hand (Hr) in a shape gripping the tool 1600 overlaps with the position of the tool (1600), it may be recognized that the right hand Hr is gripping the tool 1600. In this case, gripping of the tool 1600 can be recognized with high accuracy. For example, when gripping of the tool 1600 is recognized only from the shape of the hand, if the tool 1600 is actually gripped in an unlearned gripping manner, and the gripping manner is recognized as not gripping of the tool 1600, it is possible to suppress the virtual object 502 from being hidden.

[0119] (B) When an object held in the hand is present in the object recognition result, the head mounted display D hides the virtual object 502. In the example of (B), a tool 1600 is held in the right hand Hr, so the head mounted display D hides the virtual object 502. Hiding refers to, for example, a process of making the transparency of the virtual object 502 100% or erasing the virtual object 502 itself.

[0120] In this way, by making the virtual object 502 invisible to the user S / W, the operation of the virtual object 502, such as turning a page with the hand H while holding the tool 1600, is suppressed. Therefore, it is possible to suppress a situation where the tool 1600 collides with the work target T in the three-dimensional real space 500R due to the operation of the hand H on the virtual object 502, resulting in damage to the tool 1600 or the work target T, or injury to the user S / W.

[0121] 16, the held object is a tool 1600, but the head mounted display D may hide the virtual object 502 regardless of the type of held object. Also, the head mounted display D may hide the virtual object 502 only when the held object is a specific object such as the tool 1600. What type of object corresponds to the specific object is set in advance in the head mounted display D (or the server 101).

[0122] Furthermore, the head mounted display D may repeatedly execute such object recognition, and when it becomes impossible to recognize that the tool 1600 is being held by the right hand Hr, may redisplay the virtual object 502 in the three-dimensional imaging space 500I.

[0123] Furthermore, when it can be recognized that the right hand Hr is holding the tool 1600, and the distance between the position of the right hand Hr or the position of the tool 1600 and the virtual object 502 is within a predetermined range, the virtual object 502 may be hidden. For example, if the user S / W tries to operate the virtual object 502 with his / her hand H and approaches the virtual object 502, the operation of the virtual object 502, such as turning a page, by the hand H is suppressed. This makes it possible to suppress the tool 1600 colliding with the work object T in the three-dimensional real space 500R, which may result in damage to the tool 1600 or the work object T, or injury to the user S / W.

[0124] Fig. 17 is an explanatory diagram showing an example 2 of suppressing operation of a virtual object 502. (A) shows the same state as (A) of Fig. 16. (B) of Fig. 17 differs from (B) of Fig. 16 in that when the head mounted display D recognizes that a tool 1600 is being held, it does not hide the virtual object 502 but displays a pop-up 1700 indicating that operation is not possible on or near the virtual object 502.

[0125] By displaying warning information such as the pop-up 1700 on or near the virtual object 502, the operation of the virtual object 502, such as turning a page with the hand H while holding the tool 1600, is suppressed. Therefore, it is possible to suppress a situation in which the tool 1600 collides with the work target T in the three-dimensional real space 500R due to an operation of the hand H on the virtual object 502, resulting in damage to the tool 1600 or the work target T, or injury to the user S / W.

[0126] Such warning information is not limited to the pop-up 1700. For example, the head mounted display D may convert the color of the virtual object 502 into a specific color indicating a warning, or may blink the virtual object 502. Also, the warning may be issued to the user S / W by voice.

[0127] Furthermore, the head mounted display D may repeatedly perform such object recognition, and when it becomes unable to recognize that the tool 1600 is being held by the right hand Hr, may erase warning information such as the pop-up 1700.

[0128] In the case of (B) in Fig. 17, the head mounted display D may make the virtual object 502 inoperable (for example, making it impossible to turn pages). Note that either the non-display mode of Fig. 16 (B) or the display mode of warning information of Fig. 17 (B) is set in advance in the head mounted display D. This setting can be changed by operating the user S / W.

[0129] <Procedure for suppressing operation of virtual object 502> 18 is a flowchart showing an example of a procedure for a process of suppressing an operation on a virtual object 502. The head mounted display D displays the virtual object 502 in the three-dimensional imaging space 500I in the world coordinate system 501 (step S1801).

[0130] The head mounted display D displays the virtual object 502 in the three-dimensional imaging space 500I (step S1802).

[0131] The head mounted display D executes object recognition (step S1803). Specifically, for example, the head mounted display D acquires feature points of the hand (H) holding the tool 1600 in the local three-dimensional coordinate system 200 by the camera 102 (or the composite device 112) and recognizes the shape of the hand (H).

[0132] The head mounted display D determines whether the shape of the hand (H) is a shape holding a tool 1600 or not based on the object recognition result in step S1803 (step S1804).

[0133] If it is determined that the tool 1600 is not being held (step S1804: No), the process returns to step S1802, whereby the display of the virtual object 502 and the object recognition are repeated.

[0134] On the other hand, if it is determined that the tool 1600 is being held (step S1804: Yes), the head mounted display D hides the virtual object 502 or displays warning information (step S1805), and returns to step S1803. This causes the object recognition to be repeated.

[0135] It should be noted that the head mounted display D ends the operation restricting process when an operation to instruct the end of the operation restricting process or an operation to turn off the power is performed by the user S / W.

[0136] In this way, according to the third embodiment, by detecting that the tool 1600 is being held, the operation of the virtual object 502 by the user S / W is suppressed in advance. Therefore, it is possible to suppress a situation in which the tool 1600 collides with the work target T in the three-dimensional real space 500R due to an operation of the hand H on the virtual object 502, resulting in damage to the tool 1600 or the work target T, or injury to the user S / W.

[0137] In the above-mentioned embodiments 1 to 3, if the display 103 is of an optically transmissive type (optically see-through) type, the virtual objects 502, 1000 are displayed in the three-dimensional imaging space 500I, but the objects in the three-dimensional real space 500R captured by the camera 102 are not displayed on the display 103.

[0138] However, the head mounted display D detects feature points of objects in the three-dimensional real space 500R captured by the camera 102, the distance to the objects, etc. Therefore, the display mode and arrangement position of the virtual object can be changed in the same way as in the first to third embodiments.

[0139] For example, in the example of Fig. 5(B), the work target (TB) is not displayed, but the user S / W visually recognizes the work target T. When the transmittance of the virtual object 502B in Fig. 5(B) is increased, the user S / W can visually recognize the work target T without it being hidden by the virtual object 502B. In the case of color conversion, the user S / W cannot visually recognize the work target T in the overlapping portion due to the color-converted virtual object 502B, but can visually recognize that the virtual object 502B and the work target T overlap on the XY plane.

[0140] 6, the work target (T) is not displayed on the display 103, and the user S / W can visually recognize the work target T. For this reason, the head mounted display D converts the Z coordinate value of the overlapping portion of the virtual object 502 with the work target T to the coordinate value of a position in front of the surface (503) of the non-displayed work target (T) detected as a feature point, and displays it with a higher transparency than the non-overlapping portion. This allows the user S / W to visually recognize the overlapping portion of the virtual object 502 with the work target T.

[0141] Furthermore, changing the display mode of a virtual object includes not only increasing the transparency of the virtual object in the above-described embodiment to a level where the virtual object becomes invisible, but also reducing the size of the virtual object, lowering the resolution of the content of the virtual object, and deleting the content of the virtual object. In this way, the user cannot see the virtual object, so that unintentional interference between the hand or a tool held in the hand and the work object when trying to operate the virtual object can be suppressed, and injury to the hand and damage to the tool and the work object T can be suppressed.

[0142] In the example of FIG. 7, it has been explained that in the Z direction, a portion of the virtual object 502 that overlaps with the work object (T) on the XY plane is displayed overlapping with the work object (T). However, even if a portion of the virtual object 502 that overlaps with the work object (T) is hidden by the work object (T) and not displayed, the virtual object 502 can be divided and displayed as shown in FIG. 7(B).

[0143] The present invention is not limited to the above, and may be implemented by any combination of the above. In addition, other embodiments that are conceivable within the scope of the technical concept of the present invention are also included in the scope of the present invention. [Explanation of symbols]

[0144] D, Da, Db head mounted display, HaV virtual hand, S, S1, S2 instructor, T, Ta, Tb work object, W worker, 100 electronic device system, 101 server, 102 camera, 103 display, 110 instructor environment, 112 multifunction device 120 worker environment, 501G first world coordinate system, 501L first local coordinate system

Claims

1. The processor: A detection process for detecting a three-dimensional real space; a display control process for controlling a display screen that displays a virtual object in a three-dimensional virtual space, and changing a display mode of the virtual object based on a three-dimensional position of an object existing in the three-dimensional real space detected by the detection process and a three-dimensional position of the virtual object in the three-dimensional virtual space; A display program that executes the

2. The display program according to claim 1 , the display control process causes the processor to display the three-dimensional real space detected by the detection process as the three-dimensional virtual space on the display screen; Display program.

3. 3. The display program according to claim 1, the display control process causes the processor to change a display mode of the virtual object when the virtual object overlaps with the object in a surface direction of the display screen. Display program.

4. The display program according to claim 3, In the display control process, the processor is caused to change color information of the virtual object. Display program.

5. The display program according to claim 3, In the display control process, the processor is caused to change a transmittance of a portion of the virtual object that overlaps with the virtual object in at least a surface direction of the display screen. Display program.

6. The display program according to claim 3, In the display control process, the processor is caused to hide at least a portion of the virtual object that overlaps with the virtual object in a surface direction of the display screen. Display program.

7. The display program according to claim 3, In the display control process, the processor causes a portion of the virtual object that overlaps at least in a surface direction of the display screen to blink. Display program.

8. The display program according to claim 3, The display control process causes the processor to display warning information indicating that the virtual object overlaps with the physical object in a surface direction of the display screen. Display program.

9. The display program according to claim 3, In the display control process, when the virtual object overlaps with the object in a depth direction of the display screen or is located deeper than the object, the processor changes a display mode of the virtual object. Display program.

10. The display program according to claim 3, the display control process causes the processor to change a display mode of the virtual object when the virtual object is located in front of the object in a depth direction of the display screen. Display program.

11. The display program according to claim 3, The processor, If the virtual object is configured to be divisible into the plurality of partial virtual objects, executing a division process to divide the virtual object into the plurality of partial virtual objects. Display program.

12. The display program according to claim 11, In the display control process, the processor is caused to display, among the plurality of partial virtual objects divided by the division process, the partial virtual object that overlaps with the object at least in the plane direction, at a position where the partial virtual object does not overlap with the object in the plane direction. Display program.

13. The display program according to claim 3, In the display control process, the processor is caused to display the virtual object at a position that does not overlap with the object in the planar direction. Display program.

14. The display program according to claim 3, In the display control process, the processor is caused to display the virtual object in front of the physical object. Display program.

15. The display program according to claim 3, The processor, Executing a recognition process to recognize whether the object is a work target or not; In the display control process, when the virtual object is located in front of the object in a depth direction of the display screen and when the object is recognized as the work target by the recognition process, the processor is caused to change a display mode of the virtual object. Display program.

16. The display program according to claim 1 , The processor, executing a recognition process for recognizing whether or not the hand present in the three-dimensional real space detected by the detection process is holding a specific subject present in the three-dimensional real space; the display control process causes the processor to change a display state of the virtual object based on a recognition result of the recognition process. Display program.

17. The display program according to claim 16, the display control process causes the processor to change a display mode of the virtual object based on a three-dimensional position of the hand and a three-dimensional position of the virtual object that exist in the three-dimensional real space; Display program.

18. The display program according to claim 16, In the recognition process, when a shape of the hand existing in the three-dimensional real space is a predetermined shape, or when a three-dimensional position of the hand and a three-dimensional position of the specific subject overlap in the three-dimensional real space, the processor is caused to recognize that the hand is holding the specific subject. Display program.

19. The display program according to claim 16, In the display control process, when the recognition process recognizes that the hand is holding the specific subject and that a distance between a three-dimensional position of the hand or the specific subject in the three-dimensional real space and a three-dimensional position of the virtual object in the three-dimensional virtual space is recognized to be within a predetermined range, the processor changes a display mode of the virtual object. Display program.

20. The display program according to claim 16, In the display control process, when the recognition process recognizes that the hand is holding the specific subject and that a distance between a three-dimensional position of the hand or the specific subject in the three-dimensional real space and a three-dimensional position of the virtual object in the three-dimensional virtual space is recognized to be within a predetermined range, the processor displays warning information indicating that the virtual object is inoperable. Display program.

21. A detection unit that detects a three-dimensional real space; a display unit that displays a virtual object in a three-dimensional virtual space on a display screen; a display control unit that changes a display mode of the virtual object based on a three-dimensional position of an object existing in the three-dimensional real space detected by the detection unit and a three-dimensional position of the virtual object in the three-dimensional virtual space; An electronic device having the

Images