AR display device and program

Abstract

To provide an AR display device that displays a virtual object in an area that can be recognized in a field of vision without interfering with a daily life.SOLUTION: An AR display device 1 includes: a non-gazing distribution determination unit 211 that generates a saliency map of an environmental image and determines, on the basis of the saliency map, a non-gazing distribution indicating a probability distribution of non-gazing; a time continuous distribution determination unit 213 that determines a time continuous distribution indicating a temporally continuous probability distribution from an AR display area at previous time; a display candidate distribution calculation unit 214 that calculates a product of the non-gazing distribution and the time continuous distribution as a display candidate distribution; a center coordinate determination unit 22 that determines, on the basis of the display candidate distribution, center coordinates of the AR display area; and a projection control unit 23 that controls a center of the AR display area to be located at center coordinates.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an AR (Augmented Reality) display device and a program. [Background technology]

[0002] In recent years, eyeglass-type devices have been developed and sold, and demand for them as wearable devices is increasing. There are various types of eyeglass-type devices, and some of them have AR functions (see, for example, Non-Patent Document 1).

[0003] Once AR display devices such as AR glasses become widespread, it is expected that they will be worn at all times. By wearing an AR display device, virtual objects will be displayed in everyday life, allowing seamless access to a variety of information. Compared to hardware-based display information presentation devices such as smartphones and TVs, AR display devices have the advantage of not displaying frames. Another advantage is that they can present information in a wider space relative to the field of view than smartphones.

[0004] When viewing video on an AR display device, the video can be viewed on a virtual display that does not exist in reality. There are three main methods for displaying objects on AR display devices. The first method places an object at an arbitrary location in space, and the object's coordinates are fixed at that location even if the user moves. This method recognizes the external world using a SLAM (Simultaneous Localization and Mapping) algorithm and continues to exist in that location, just like a real object. The second method fixes the object relative to the head coordinate system and continues to display it in a specific area of ​​the field of view. This is the simplest method, requiring no external information. The third method uses an algorithm called Tagalong, which moves an object that moves outside the display area so that it enters the display area (see, for example, Non-Patent Document 2).

[0005] On the other hand, an area where people are likely to gaze is called a saliency map, and methods for estimating a saliency map have been disclosed (see, for example, Non-Patent Documents 3 and 4). Also, research is being conducted to attempt to derive an information presentation area using a saliency map (see, for example, Non-Patent Document 5 and Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2020-42759 [Non-patent literature]

[0007] [Non-Patent Document 1] Microsoft, "Hololens 2," [online], [Retrieved January 28, 2022], Internet<https: / / www.microsoft.com / ja-jp / hololens / hardware> [Non-patent document 2] Microsoft, "Billboard and tag-along," [Retrieved January 28, 2022], Internet <URL:https: / / docs.microsoft.com / ja-jp / windows / mixed-reality / design / billboarding-and-tag-along> [Non-patent document 3] Laurent Itti, Christof Koch, and Ernst Niebur, “A model of saliency-based visual attention for rapid scene analysis,” IEEE Transactions on pattern analysis and machine intelligence, Vol. 20, No. 11, pp. 1254-1259, 1998 [Non-patent document 4] Ting Zhao and Xiangqian Wu, “Pyramid feature attention network for saliency detection,” In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 3085-3094, 2019 [Non-Patent Document 5] Takayuki Abe, Sho Takahashi, and Toru Hagiwara, “A calculation method of degree of data indication regions in first-person view videos for improvement transportation,” In 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE), pp. 558-559. IEEE, 2019 Summary of the Invention [Problem to be solved by the invention]

[0008] However, with the first AR display method described above, the object disappears from view when the user moves or faces a different direction. Therefore, it is not suitable for situations where there is continuity, such as video viewing, and it is desirable to extend the time the object is presented to the user. Furthermore, the second method described above may be a hindrance when viewing real objects in daily life. Furthermore, the third method has the problem of interrupting the recognition of real objects. These discussions are discussed in detail in Reference 1 below. [Reference 1] Hiroyuki Kawakita and three others, "Study on the Form of 2D Video in 3D Space," In HCG2018-I-1-1. HCG Symposium 2018, 2018.

[0009] In addition, the conventional method of deriving the information presentation area using a saliency map had the problem of not taking into account the movement of the first-person viewpoint targeting the image in the AR glasses.

[0010] The present invention was made in consideration of the above circumstances and aims to provide an AR display device and program that can display virtual objects in an area that is recognizable in the field of vision without interfering with daily life. [Means for solving the problem]

[0011] In order to solve the above problem, an AR display device according to one embodiment is an AR display device that changes an AR display area, which is a display area for an AR image, in accordance with an external world image, and includes: a non-gaze distribution determination unit that generates a saliency map of the external world image and determines a non-gaze distribution indicating a probability distribution of non-gaze based on the saliency map; a time continuous distribution determination unit that determines a time continuous distribution indicating a probability distribution that is temporally continuous from the AR display area at the previous time; a display candidate distribution calculation unit that calculates the product of the non-gaze distribution and the time continuous distribution as a display candidate distribution; a center coordinate determination unit that determines a center coordinate of the AR display distribution based on the display candidate distribution; and a projection control unit that controls the center of the AR display area to be located at the center coordinate.

[0012] Furthermore, in one embodiment, the time continuous distribution determination unit determines the time continuous distribution based on the central coordinates of the AR display area one time before in the external world image and a search range parameter that specifies the range of the time continuous distribution, and the search range parameter may be changeable based on a user instruction.

[0013] Furthermore, in one embodiment, the system may further include a low-load distribution determination unit that determines a low-load distribution that indicates a probability distribution of low viewing load in an external image, and the display candidate distribution calculation unit may calculate the product of the non-attention distribution, the time continuous distribution, and the low-load distribution as the display candidate distribution.

[0014] Furthermore, in one embodiment, the low load distribution determination unit determines the low load distribution based on a low load distribution parameter that specifies a range of the low load distribution, and the low load distribution parameter may be changeable based on a user instruction.

[0015] Furthermore, in one embodiment, the projection control unit may use the tilt of the AR display device measured by a tilt sensor to control the AR image to be kept horizontal with respect to the ground.

[0016] Furthermore, in one embodiment, the center coordinate determination unit may determine the center coordinate based on second center coordinates, which are the center coordinates of the second AR image, so that the AR display area does not overlap with the display area of ​​the second AR image.

[0017] Moreover, a program according to an embodiment causes a computer to function as the AR display device. [Effects of the Invention]

[0018] According to the present invention, it is possible to display a virtual object in an area that is not an obstacle to daily life and that can be recognized in the field of vision. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a block diagram showing an example of the configuration of an AR display device according to a first embodiment. FIG. [Figure 2] 3 is a block diagram showing an example of the configuration of a display candidate distribution determination unit in the AR display device according to the first embodiment. FIG. [Figure 3] 3 is a schematic diagram showing an example of saliency map estimation by a non-gaze distribution determination unit in the AR display device according to the first embodiment. FIG. [Figure 4] 3 is a schematic diagram showing an example of a low load distribution determined by a low load distribution determination unit in the AR display device according to the first embodiment. FIG. [Figure 5] 3 is a schematic diagram showing an example of a time continuous distribution determined by a time continuous distribution determination unit in the AR display device according to the first embodiment. FIG. [Figure 6] 4A to 4C are diagrams illustrating the processing of a display candidate distribution calculation unit in the AR display device according to the first embodiment. [Figure 7] 5 is a flowchart illustrating an operation of the AR display device according to the first embodiment when worn. [Figure 8] 3A to 3C are diagrams illustrating behavior of an AR image as seen by a user wearing the AR display device according to the first embodiment. [Figure 9] FIG. 10 is a block diagram showing an example of the configuration of an AR display device according to a second embodiment. [Figure 10] FIG. 10 is a block diagram showing an example configuration of a parameter change unit and a display candidate distribution determination unit in an AR display device according to a second embodiment. [Figure 11] 10 is a flowchart illustrating an operation of the AR display device according to the second embodiment when worn. [Figure 12] FIG. 10 is a block diagram showing an example of the configuration of an AR display device according to a third embodiment. [Figure 13] 10 is a flowchart illustrating an operation of the AR display device according to the third embodiment when worn. [Figure 14] FIG. 10 is a block diagram showing an example of the configuration of an AR display device according to a fourth embodiment. [Figure 15] FIG. 10 is a block diagram showing an example of the configuration of a display candidate distribution determination unit in an AR display device according to a fourth embodiment. [Figure 16] 10 is a flowchart illustrating an operation of the AR display device according to the fourth embodiment when worn. DETAILED DESCRIPTION OF THE INVENTION

[0020] In this specification, a glasses-type AR display device equipped with a camera and having an AR function for displaying virtual objects (AR images) according to the state of the outside world will be described in detail with reference to the drawings.

[0021] (First embodiment) An AR display device according to the first embodiment will be described below. Fig. 1 shows an example of the configuration of the AR display device according to the first embodiment. The AR display device 1 shown in Fig. 1 includes an outside world image acquisition unit 10, a control unit 20, a storage unit 30, and a display unit 40, and changes the display area of ​​the AR image (hereinafter referred to as "AR display area") according to the outside world image.

[0022] The outside world image acquisition unit 10 is an imaging device (small camera) that captures an image of the outside of the AR display device 1 to acquire the outside world image. The outside world image acquisition unit 10 outputs the outside world image to the control unit 20.

[0023] The control unit 20 may be configured with dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be configured with a processor, or may be configured with both. The control unit 20 includes a display candidate distribution determination unit 21, a center coordinate determination unit 22, and a projection control unit 23.

[0024] The display candidate distribution determination unit 21 determines a probability distribution of display candidate areas that are candidates for displaying AR video, and outputs the result to the center coordinate determination unit 22. The AR display device 1 displays AR video in a position that makes it easy to view the video, while avoiding areas that are likely to be gazed upon by people, such as people or text. Specifically, the AR video is displayed in an area that simultaneously meets three requirements: (a) a non-gazing area where the probability of the user gazing is below a threshold, (b) a low-load area within the user's field of view where the viewing load is below a threshold (i.e., easy to view), and (c) a time-continuous area that is continuous from the AR display area of ​​the previous time.

[0025] Fig. 2 is a block diagram showing an example of the configuration of the display candidate distribution determination unit 21. The display candidate distribution determination unit 21 shown in Fig. 2 includes a non-gaze distribution determination unit 211, a low-load distribution determination unit 212, a time-sequential distribution determination unit 213, and a display candidate distribution calculation unit 214. The display candidate distribution determination unit 21 may include only one of the low-load distribution determination unit 212 and the time-sequential distribution determination unit 213.

[0026] The non-gaze distribution determination unit 211 generates a saliency map of the external world image acquired by the external world image acquisition unit 10, determines a non-gaze distribution indicating the probability distribution of non-gaze based on the saliency map, and outputs it to the display candidate distribution calculation unit 214.

[0027] FIG. 3 is a schematic diagram showing an example of saliency map estimation using a convolutional neural network. The area indicated by diagonal lines is an area estimated to have high saliency. In this schematic diagram, the higher the saliency, the narrower the interval between the diagonal lines. In other words, this indicates that if person M is in front of you, there is a high possibility that you will gaze at person M. The height of saliency is defined for each pixel. The non-gaze distribution determination unit 211 subtracts the saliency value estimated for each pixel from the maximum value and uses this as the probability distribution of non-gaze. In the following, the center coordinate of the AR display area at a certain time t is defined as X t Let the non-attention distribution be p(X t It is written as |S￣).

[0028] The low load distribution determination unit 212 determines a low load distribution that indicates a probability distribution of low viewing load in the external world image (within the user's field of view) based on the height of the line of sight, and outputs the low load distribution to the display candidate distribution calculation unit 214. The probability distribution can be, for example, a two-dimensional uniform distribution or a normal distribution.

[0029] FIG. 4 is a schematic diagram showing an example of a low load distribution. The low load distribution can be modeled based on ergonomics. In the case of a PC monitor, as disclosed in Reference 2 below, it is known that it is preferable for work if the height of the top of the monitor matches the height of the eye line. Using such PC monitor placement guidelines, the low load distribution determination unit 212 defines the low load distribution based on the height of the eye line. The displayable area width differs for each device and also varies from person to person, so the low load distribution is a variable probability distribution. Therefore, the low load distribution determination unit 212 may externally acquire low load distribution parameters that specify the range of the low load distribution, and determine the low load distribution based on the low load distribution parameters. The low load distribution parameters are, for example, the downward angle based on the height of the eye line. The example shown in FIG. 4 shows the range L of the low load distribution within which the user's looking down angle falls within a predetermined range. Below, the low load distribution is defined as p(X t It is written as |H). [Reference 2] Microsoft, "Adjusting your work environment," [Retrieved January 14, 2022], Internet <URL:https: / / support.microsoft.com / ja-jp / topic / %E4%BD%9C%E6%A5%AD%E7%92%B0%E5%A2%83%E3%81%AE%E8%AA%BF%E6%95%B4-e9f3fc2c-e6fa-d27f-78c4-274b3669c425>

[0030] The time continuous distribution determination unit 213 determines a time continuous distribution that indicates a probability distribution that is continuous in time from the AR display area of ​​the previous time in the external world image, and outputs the time continuous distribution to the display candidate distribution calculation unit 214. The time continuous distribution can be, for example, a two-dimensional uniform distribution or a normal distribution.

[0031] 5 is a schematic diagram showing an example of a time continuous distribution. The time continuous distribution determination unit 213 receives the central coordinate X t-1 Get the center coordinate X t-1In the initial state, the time continuous distribution determination unit 213 acquires the initial value of the center coordinate from the storage unit 30 and determines the time continuous distribution. The time continuous distribution determination unit 213 acquires a search range parameter that specifies the range of the time continuous distribution from the outside, and determines the area around the center coordinate X t-1 The time-continuous distribution may be determined based on the search range parameter, for example, the radius of the circle. In the example shown in FIG. 5, the center coordinate X t-1 The circle with the center at and the radius specified by the search range parameter is shown as the range C of the time continuous distribution. Below, the time continuous distribution is defined as p(X t |X t-1 ) is written as

[0032] The display candidate distribution calculation unit 214 calculates the non-gaze distribution p(X t |Ŝ), the low load distribution p(X t |H), and the time continuous distribution p(X t |X t-1 ) is calculated as the display candidate distribution.

[0033] 6 is a diagram for explaining the processing by the display candidate distribution calculation unit 214. The central coordinate X given by three conditions t The probability distribution of p(X t |S￣,H,X t-1 ) is proportional to the product of three probability distributions, as shown in equation (1) using Bayes' theorem. Here, ∝ is the symbol that represents proportionality, and the central coordinate X t The prior distribution of is a uniform distribution. That is, the display candidate distribution calculation unit 214 calculates the product of each of the three probability distributions for each pixel, thereby obtaining a display candidate distribution that satisfies the three conditions.

[0034]

number

[0035] The center coordinate determination unit 22 determines the center coordinate X of the AR display area based on the display candidate distribution calculated by the display candidate distribution determination unit 21. t and outputs it to the projection control unit 23 and the storage unit 30. Since the display candidate area is given as a distribution, the center coordinate determination unit 22 can calculate the optimal coordinates by calculating the expected value. When the center coordinate determination unit 22 calculates the expected value as a weighted average of the display candidate distribution, the center coordinate X of the AR display area at time t is ^t is calculated using equation (2). Note that in equation (2), weighting is performed on a pixel-by-pixel basis after the product of the three distributions is calculated, but it is also possible to weight each of the three probability distributions separately.

[0036]

number

[0037] Similarly, when the maximum value of the display candidate distribution is used, the center coordinate determination unit 22 determines the center coordinate X ^t The center coordinate determination unit 22 determines the center coordinate X by using the formula (2) or (3). ^t and outputs the coordinates to the projection control unit 23 and the storage unit 30. Note that the center coordinate X ^t The calculation formula is not limited to formula (2) or formula (3).

[0038]

number

[0039] The projection control unit 23 acquires an AR image from the outside and controls the AR display area so that the center of the AR display area is positioned at the center coordinates determined by the center coordinate determination unit 22. Then, the projection control unit 23 outputs the AR image to the display unit 40.

[0040] The storage unit 30 stores the center coordinates determined by the center coordinate determination unit 22, as well as any information used in the operation of the AR display device 1. The storage unit 30 includes one or more memories, and may include, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like.

[0041] The display unit 40 includes an optical system and a display. The display unit 40 converts the AR image input from the projection control unit 23 into an optical signal and displays it on a display (glasses). The display unit 40 displays the AR image using any method, such as an optical see-through method or a video see-through method.

[0042] Next, with reference to FIG. 7, the operation of the AR display device 1 when worn will be described.

[0043] In step S101, the AR display device 1 starts the algorithm and reflects the initial values ​​of the center coordinates in the storage unit 30.

[0044] In step S102, the outside world image acquisition unit 10 acquires an outside world image.

[0045] In step S103, the display candidate distribution determination unit 21 determines the display candidate distribution.

[0046] In step S104, the center coordinate determination unit 22 determines the center coordinate of the AR image.

[0047] In step S105, the projection control unit 23 changes the display position of the AR image based on the center coordinates.

[0048] In step S106, the center coordinates are stored in the storage unit 30, and the process enters the next loop. Through this process, the AR display area continues to be updated in real time.

[0049] FIG. 8 shows the behavior of the AR image as seen by a user wearing the AR display device 1. The non-gaze distribution determining unit 211 has diagonally shaded the portion of the external image that is estimated to have high saliency (person M and its surroundings). In FIG. 8(a), the AR image P is displayed so as not to overlap with person M. Subsequently, as person M moves from FIG. 8(b) to FIG. 8(f), the AR image P moves in a direction that avoids person M (an area with a low value in the saliency map). Furthermore, the saliency map also changes as the direction in which the user is looking changes. In this way, the AR display device 1 can display the AR image while avoiding the target to which the user wants to focus their attention (person M in FIG. 8).

[0050] As can be seen from Fig. 8, the AR display device 1 makes it possible to dynamically display AR video in an unobtrusive location. Therefore, compared to existing display methods, it is possible to create a more comfortable video viewing environment when used in daily life. For example, it becomes possible to view AR video while cooking or operating a PC.

[0051] (Second embodiment) Next, an AR display device according to a second embodiment will be described. FIG. 9 is a block diagram showing an example of the configuration of the AR display device according to the second embodiment. The AR display device 2 shown in FIG. 9 includes an outside world image acquisition unit 10, a control unit 20a, a storage unit 30, and a display unit 40. The AR display device 2 of the second embodiment differs from the AR display device 1 of the first embodiment in that the control unit 20 is replaced with a control unit 20a. As the other configurations are the same as those of the first embodiment, the same reference numerals are used and descriptions thereof will be omitted where appropriate.

[0052] The control unit 20a includes a display candidate distribution determination unit 21, a center coordinate determination unit 22, a projection control unit 23, and a parameter change unit 24.

[0053] 10 is a block diagram showing an example of the configuration of the parameter modification unit 24 and the display candidate distribution determination unit 21 according to the second embodiment. The parameter modification unit 24 independently modifies the low-load distribution parameters and the search range parameters based on a user instruction. The parameter modification unit 24 then outputs the modified low-load distribution parameters to the low-load distribution determination unit 212, and outputs the modified search range parameters to the time-sequential distribution determination unit 213.

[0054] Next, with reference to FIG. 11, the operation of the AR display device 2 when worn will be described.

[0055] In step S201, the parameters are changed by the parameter change unit 24. The processing of step S201 can be performed at any timing. Steps S101 to S106 are the same as in the first embodiment, and therefore their explanation will be omitted. However, in step S103, the display candidate distribution is determined using the parameters changed in step S201.

[0056] Probability distribution p(X t |H) and probability distribution p(X t |X t-1 ) may differ depending on the user's preferred distribution. In this embodiment, the distribution p(X t |H) and the search range parameter to find the distribution p(X t |X t-1 ) can be changed. That is, by providing the parameter change unit 24, the AR display device 2 can change parameters in response to a user instruction for personalization.

[0057] (Third embodiment) Next, an AR display device according to a third embodiment will be described. FIG. 12 is a block diagram showing an example of the configuration of the AR display device according to the third embodiment. The AR display device 3 shown in FIG. 12 includes an outside world image acquisition unit 10, a control unit 20b, a storage unit 30, a display unit 40, and a tilt sensor 50. The AR display device 3 of the third embodiment differs from the AR display device 1 of the first embodiment in that it further includes a tilt sensor 50 and the control unit 20 is changed to a control unit 20b. Since the other configurations are the same as those of the first embodiment, the same reference numerals are used and descriptions thereof will be omitted as appropriate.

[0058] The control unit 20b includes a display candidate distribution determination unit 21, a center coordinate determination unit 22, and a projection control unit 23a.

[0059] The tilt sensor 50 is, for example, an acceleration sensor or a gyro sensor, and measures the tilt of the AR display device 3. Then, the tilt sensor 50 outputs the measured tilt to the projection control unit 23a.

[0060] The projection control unit 23a controls the AR image acquired from outside so that its center is positioned at the center coordinates determined by the center coordinate determination unit 22, and further controls the AR image so that it remains horizontal to the ground using the tilt measured by the tilt sensor 50. Furthermore, when the tilt measured by the tilt sensor 50 exceeds a threshold, the projection control unit 23a may control the AR image so that it remains horizontal to the ground by rotating it by 90 degrees.

[0061] Next, with reference to FIG. 13, the operation of the AR display device 3 when worn will be described.

[0062] In step S301, the tilt sensor 50 measures the tilt of the AR display device 3. Steps S101 to S106 are the same as those in the first embodiment, and therefore their explanations will be omitted. However, in step S105, the tilt measured in step S301 is also used to change the display position of the AR image.

[0063] Since the AR display device is fixed to the user's head when worn, the image projected by a conventional AR display device does not change relative to the user's head even if the user tilts their head. In contrast, the AR display device 3 is equipped with a tilt sensor 50, which enables the AR image to be displayed horizontally relative to the ground even if the user tilts their head.

[0064] (Fourth embodiment) Next, an AR display device according to a fourth embodiment will be described. FIG. 14 is a block diagram showing an example of the configuration of the AR display device according to the fourth embodiment. The AR display device 4 shown in FIG. 14 includes an outside world image acquisition unit 10, a control unit 20c, a storage unit 30, and a display unit 40. The AR display device 4 of the fourth embodiment differs from the AR display device 1 of the first embodiment in that the control unit 20 is replaced with a control unit 20c. As the other configurations are the same as those of the first embodiment, the same reference numerals are used and descriptions thereof will be omitted as appropriate.

[0065] In the AR display devices 1, 2, and 3, multiple AR images can be displayed using multiple image display areas. However, if multiple AR images are simply displayed using the above algorithm, they may be displayed overlapping each other. Therefore, in the AR display device 4, a distribution is created so as to exclude multiple display areas. In the following description, it is assumed that the AR display device 4 displays a first AR image and a second AR image on the display unit 40.

[0066] The control unit 20c includes a display candidate distribution determination unit 21a, a center coordinate determination unit 22a, a projection control unit 23b, and a second center coordinate determination unit 25.

[0067] The second center coordinate determination unit 25 determines the center coordinates of the second AR video (hereinafter referred to as "second center coordinates") according to a predetermined algorithm and outputs them to the display candidate distribution determination unit 21a and the projection control unit 23b. The method for determining the second center coordinates may be the same as the method for determining center coordinates in the AR display devices 1, 2, and 3. Furthermore, if the second center coordinate determination unit 25 has parameters related to the display area of ​​the second AR video, it also outputs the parameters to the display candidate distribution determination unit 21a and the projection control unit 23b.

[0068] 15 is a block diagram showing an example of the configuration of the display candidate distribution determination unit 21a. The display candidate distribution determination unit 21a shown in FIG. 15 includes a non-gaze distribution determination unit 211, a low-load distribution determination unit 212, a time-sequential distribution determination unit 213, and a display candidate distribution calculation unit 214a. As described above, the non-gaze distribution determination unit 211 determines the non-gaze distribution p(X t |Ŝ), and the low load distribution determination unit 212 determines the low load distribution p(X t |H), and the time continuous distribution determination unit 213 determines the time continuous distribution p(X t |X t-1 ) to determine

[0069] The display candidate distribution calculation unit 214a calculates the non-display distribution p(X t |Y t-1 Then, the display candidate distribution calculation unit 214a calculates the product of each of the four probability distributions for each pixel as shown in equation (4), thereby obtaining a display candidate distribution that satisfies the four conditions.

[0070]

number

[0071] The center coordinate determination unit 22a determines center coordinates based on the second center coordinates so that the AR display area does not overlap with the display area of ​​the second AR image, and outputs the determined center coordinates to the projection control unit 23b and the storage unit 30. For example, the center coordinate determination unit 22 determines the expected value or maximum value based on the weighted average of the display candidate distribution as the center coordinates.

[0072] The projection control unit 23b acquires the first AR image and the second AR image from the outside, acquires the first center coordinates from the center coordinate determination unit 22a, and acquires the second center coordinates from the second center coordinate determination unit 25. Then, the projection control unit 23b controls the projection so that the center of the first AR image is located at the first center coordinates, and so that the center of the second AR image is located at the second center coordinates.

[0073] Next, with reference to FIG. 16, the operation of the AR display device 4 when worn will be described.

[0074] In step S401, second center coordinates are determined by the second center coordinate determination unit 25. Steps S101 to S106 are the same as those in the first embodiment, and therefore their explanations will be omitted. However, in step S103, the display candidate distribution is determined using the second center coordinates determined in step S401.

[0075] As described above, in this embodiment, the center coordinates of the AR display area are determined so that they do not overlap with the display area of ​​the second AR image. With this configuration, even when multiple AR images are displayed, the present embodiment makes it possible to display them so that they do not overlap with each other.

[0076] (program) A computer capable of executing program instructions can be used to function as each of the above-described AR display devices 1, 2, 3, and 4. Here, the computer may be a general-purpose computer, a special-purpose computer, a workstation, a personal computer (PC), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing necessary tasks.

[0077] The computer includes a processor, a storage unit, an input unit, an output unit, and a communication interface. The processor may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an SoC (System on a Chip), and may be configured with multiple processors of the same or different types. The processor reads and executes a program from the storage unit to control the above components and perform various arithmetic processing. At least a portion of these processing operations may be implemented by hardware.

[0078] The program may be recorded on a computer-readable recording medium. Using such a recording medium, the program can be installed on a computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. Furthermore, the program may be downloaded from an external device via a network.

[0079] For example, a program for functioning as the AR display device 1 causes a computer to execute the following steps: generating a saliency map of an external image; determining a non-gaze distribution that indicates the probability distribution of non-gaze based on the saliency map; determining a time-continuous distribution that indicates a probability distribution that is continuous over time from the AR display area at the previous time; calculating the product of the non-gaze distribution and the time-continuous distribution as a display candidate distribution; determining the center coordinates of the AR display area based on the display candidate distribution; and controlling the center of the AR display area to be located at the center coordinates.

[0080] Furthermore, the above-described AR display devices 1, 2, 3, and 4 may be configured with one or more semiconductor chips. The semiconductor chip may be equipped with a CPU that executes a program that describes the processing content that realizes each function of the AR display devices 1, 2, 3, and 4.

[0081] The above-described embodiments have been described as typical examples, but it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and alterations are possible without departing from the scope of the claims. For example, multiple building blocks shown in the configuration diagrams of the embodiments can be integrated, or a single building block can be divided. Furthermore, multiple steps shown in the flowcharts of the embodiments can be integrated into one, or a single step can be divided. [Explanation of symbols]

[0082] 1,2,3,4 AR display device 10 External image acquisition department 20, 20a, 20b, 20c control unit 21,21a Display candidate distribution determining unit 22,22a Center coordinate determination part 23, 23a, 23b Projection control unit 24 Parameter change section 25 Second center coordinate determination section 30 Storage section 40 Display section 50 Tilt sensor 211 Non-gaze distribution determining unit 212 Low load distribution determining section 213 Time-Continuous Distribution Determination Unit 214,214a Display candidate distribution calculation unit

Claims

1. An AR display device that changes an AR display area, which is a display area of ​​an AR image, according to an external image, a non-gaze distribution determination unit that generates a saliency map of an external image and determines a non-gaze distribution indicating a probability distribution of non-gaze based on the saliency map; a time continuous distribution determination unit that determines a time continuous distribution that indicates a probability distribution that is continuous in time from the AR display area at a previous time in an outside world image; a display candidate distribution calculation unit that calculates a product of the non-gaze distribution and the time continuous distribution as a display candidate distribution; a center coordinate determination unit that determines center coordinates of the AR display area based on the display candidate distribution; a projection control unit that controls the center of the AR display area to be positioned at the center coordinates; An AR display device comprising:

2. 2. The AR display device according to claim 1, wherein the time continuous distribution determination unit determines the time continuous distribution based on the central coordinates of the AR display area one time before and a search range parameter that specifies the range of the time continuous distribution, and the search range parameter is changeable based on a user instruction.

3. a low-load distribution determination unit that determines a low-load distribution that indicates a probability distribution of low viewing loads in the outside world image; The AR display device according to claim 1 , wherein the display candidate distribution calculation unit calculates a product of the non-gaze distribution, the time continuous distribution, and the low load distribution as the display candidate distribution.

4. 4. The AR display device according to claim 3, wherein the low-load distribution determination unit determines the low-load distribution based on a low-load distribution parameter that specifies a range of the low-load distribution, and the low-load distribution parameter is changeable based on a user instruction.

5. The AR display device according to claim 1 , wherein the projection control unit controls the AR image to be kept horizontal with respect to the ground, using a tilt of the AR display device measured by a tilt sensor.

6. 6. The AR display device according to claim 1, wherein the center coordinate determination unit determines the center coordinate based on second center coordinates that are center coordinates of the second AR image so that the AR display area does not overlap with the display area of ​​the second AR image.

7. A program for causing a computer to function as the AR display device according to any one of claims 1 to 6.

Images