Information processing device and information processing method

Abstract

This information processing device worn by a user comprises a display processing device that displays an augmented reality object within the field of view, a walking detection sensor that detects the walking state of the user, and a processor. The processor determines whether or not the user is walking on the basis of sensor information from the walking detection sensor, and if determining that the user is walking, executes control of enlarging the augmented reality object and displaying the enlarged augmented reality object within the field of view.

Description

Information processing device and information processing method

[0001] The present invention relates to an information processing apparatus and an information processing method for displaying virtual objects together with objects in real space.

[0002] In recent years, Augmented Reality (AR) technology, which visually augments the real world by overlaying virtual objects (virtual objects) in a virtual space composed of digital information created using CG (Computer Graphics) or the like onto the real space, has become widely used, and information processing devices that allow users to easily view virtual objects while recognizing objects in real space have become widespread. Examples include head-mounted displays (HMDs) worn on the head and AR glasses, which are eyeglass-type digital devices as a form of HMD. HMDs and AR glasses display real-world physical objects and virtual objects on a display unit attached to the front of the head (so-called video see-through type), or display virtual objects on a display unit while allowing the user to view real objects in front of the user (so-called optical see-through type), allowing the user to experience the virtual objects as if they were actually present in the real world. Virtual objects are generated from digital information consisting of various types of display items such as text, icons, and moving images, and are visualized by being displayed in front of the user's eyes.

[0003] Here, when a user wearing an HMD or AR glasses is walking outdoors, it is desirable that the display items, which are virtual objects, can be easily viewed along with the field of view ahead. However, if the display items, which are virtual objects, have the effect of interfering with the user's overall field of view, the user may feel annoyed and experience discomfort when experiencing AR technology. In response to this issue, Patent Literature 1 describes, as a method for reducing the effect of the display items on the user's field of view, "an information processing device including a display control unit that, triggered by a change in the user's behavior, controls the presence or absence of the display items on the display unit based on a display order of the display items determined according to environmental information or user information."

[0004] Japanese Patent Application Laid-Open No. 2019-101743

[0005] Although Patent Document 1 discloses that the impact on the user's field of vision can be reduced by controlling whether or not display items, which are virtual objects, are displayed while the user is walking, there is a problem in that there is a possibility that display items, which are virtual objects, will not be displayed while the user is walking, making it difficult for the user to consistently use the AR technology.

[0006] In consideration of the above-mentioned problems, the present invention provides an information processing device and an information processing method that, when walking, displays display items within the field of view display range as virtual objects in an easy-to-view display mode together with the scenery ahead.

[0007] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0008] In other words, the present invention is an information processing device worn by a user, comprising a display processing device that displays an augmented reality object within a field of view display range, a walking detection sensor that detects the user's walking state, and a processor, wherein the processor determines whether the user is walking based on sensor information from the walking detection sensor, and when it determines that the user is walking, it executes control to enlarge the augmented reality object and display the enlarged augmented reality object within the field of view display range.

[0009] The present invention also provides an information processing device worn by a user, comprising a display processing device that displays an augmented reality object within a field of view display range, a walking detection sensor that detects the user's walking state, and a processor, wherein the field of view display range includes a central region that includes the center point of the field of view display range and a peripheral region located on the outer edge of the central region, and the processor determines whether the user is walking based on sensor information from the walking detection sensor, and, if it determines that the user is walking, executes display control to move the augmented reality object displayed in the central region to the peripheral region.

[0010] The present invention also provides an information processing method executed by an information processing device worn by a user, comprising: a walking determination step of determining whether the user is walking based on sensor information from a walking detection sensor that detects the user's walking state; an enlargement step of enlarging an augmented reality object compared to when the user is stationary when it is determined that the user is walking; and a display step of displaying the enlarged augmented reality object on a display processing device.

[0011] By using the technology of the present invention, it is possible to realize an information processing device and an information processing method that display items within the field of view display range as virtual objects in an easy-to-view display mode along with the scenery ahead. Furthermore, problems, configurations, and advantages other than those described above will become clear from the description of the following embodiments.

[0012] 1 is an example of a diagram for schematically illustrating the appearance of an embodiment of an information processing device according to the present embodiment. It is an example of a diagram for illustrating a forward scenery screen and a field of view display range when a user is stationary in the information processing device according to the embodiment described in FIG. 1. It is an example of a diagram for illustrating a case where all display items are enlarged when the user is walking in the information processing device according to the embodiment described in FIG. 1. It is an example of a diagram for illustrating a case where peripheral display items are enlarged when the user is walking in the information processing device according to the embodiment described in FIG. 1. It is an example of a diagram for illustrating a case where peripheral display items are enlarged when the user is walking, and display items in the central region are slightly enlarged. It is an example of a diagram for illustrating a case where display items that move to the periphery are enlarged when the user is walking diagonally left in the information processing device according to the embodiment described in FIG. 1. It is an example of a diagram for illustrating a case where display items that move to the periphery are enlarged when the user is walking diagonally right in the information processing device according to the embodiment described in FIG. 1. It is an example of a diagram for illustrating a case where the user turns their head to the left when walking forward in the information processing device according to the embodiment described in FIG. 1. It is another example of a diagram for illustrating a case where the user turns their head to the left when walking forward in the information processing device according to the embodiment described in FIG. 1. FIG. 1 is an example of a diagram illustrating a case where the user turns his head upward and left while walking diagonally right in the information processing device according to the embodiment described in FIG. 1. FIG. 2 is an example of a diagram illustrating a relationship between a forward scenery screen and a field of view display range when the user is stationary, and a central region, a first peripheral region, and a second peripheral region in the information processing device according to the embodiment described in FIG. 1. FIG. 3 is an example of a diagram illustrating a case where a display item in the central region is moved to the first peripheral region when the user is walking in the information processing device according to the embodiment described in FIG. 1. FIG. 4 is an example of a diagram illustrating a case where a display item in the central region is moved to the first peripheral region and enlarged when the user is walking in the information processing device according to the embodiment described in FIG. 1. FIG. 5 is an example of a diagram illustrating a relationship between walking speed and display size of display items in the information processing device according to the embodiment described in FIG. 1. FIG. 6 is an example of a diagram illustrating a relationship between walking speed and a destination of a display item in the central region in the information processing device according to the embodiment described in FIG. 1.1 is an example of a diagram illustrating a case where an operation menu is displayed in a stationary state in the information processing device according to the embodiment described in FIG. 1. FIG. 2 is an example of a flowchart (first half) illustrating the operation of the information processing device according to the embodiment. FIG. 3 is an example of a flowchart (second half) illustrating the operation of the information processing device according to the embodiment. FIG. 4 is a block diagram illustrating an example configuration of AR glasses as an example of the information processing device according to the embodiment. FIG. 5 is an example of a diagram illustrating a case where the screen of a smartphone or smartwatch is displayed as a display item when the user is stationary in the information processing device according to the embodiment described in FIG. 1. FIG. 6 is an example of a diagram illustrating a case where the screen of a smartphone or smartwatch is displayed as a display item when the user is walking in the information processing device according to the embodiment described in FIG.

[0013] The present invention can provide an information processing device and method that contributes to improving both the visibility of augmented reality objects while walking and the visibility of the scenery ahead. This is expected to lead to improvements in technology for labor-intensive industries, and is therefore expected to contribute to goal 8.2 of the Sustainable Development Goals (SDGs) advocated by the United Nations (increase economic productivity through diversification, technological improvement and innovation, particularly in industries that add value to goods and services and labor-intensive industries).

[0014] In addition, improving the visibility of displayed items is expected to support people with poor eyesight, and is therefore expected to contribute to Goal 3 of the United Nations' Sustainable Development Goals (SDGs), which aims to "ensure good health and well-being for all."

[0015] <Embodiment 1> Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. As a specific example of an information processing device according to this embodiment, a pair of eyeglass-type AR glasses, which is a form of HMD that is worn on a user's head and displays and recognizes information on real space and virtual space, will be described as an example. Fig. 1 is an example of a diagram that schematically illustrates the appearance of an embodiment of the information processing device according to this embodiment, and Figs. 2 and 3 are example diagrams that illustrate a display screen of a view ahead and within a field of view display range of the information processing device according to this embodiment.

[0016] In Figure 1, AR glasses 100 are worn on the head of a user 10, and the AR glasses 100 are equipped with a left eye gaze sensor 101 and a right eye gaze sensor 102 that detect the gaze of the left and right eyes of the user 10, a camera 103 that captures the outside world, and an acceleration sensor 104 that detects acceleration, which is the change in speed per unit time.

[0017] In the case of the optical see-through type AR glasses 100, the user 10 directly sees real objects in the field of view in front of the user, while in the case of the video see-through type, the camera 103 photographs real objects in the field of view in front of the user, and the image of the photographed real object is displayed on a display.

[0018] Additionally, the AR glasses 100 generate various types of display items such as text characters, icons, and images as augmented reality objects in addition to real objects, and display and arrange them within the field of view display range of the user 10. Figure 1 shows a case where display items 111, 112, and 113 represented by text characters and icons such as "Trains to Tokyo are delayed. Raining tonight," "** Teacher," and "Turn right 8m from the station" are virtually arranged in a real space consisting of real objects such as buildings, roads, and pedestrians.

[0019] The field of view display range is the range in which display items 111, 112, and 113 of real objects and virtual objects are displayed superimposed through the AR glasses 100.

[0020] The user's line of sight may be detected from the movement of the user's eyeballs, and the user's field of view may be used as the field of view display range. Also, the field of view display range may be an area that includes the field of view when the user looks at a distant object in front of them, as well as the field of view when the user moves their eyeballs up, down, left, and right.

[0021] The AR glasses 100 use a camera 103 provided in the AR glasses 100 to capture images of the user 10 in front of them and their left and right feet 11, 12, and capture the user's walking state from the feet of both feet that have been captured, thereby obtaining user walking information such as walking movements, walking direction, walking speed, as well as the relative positional relationship between the user's 10 feet and body and the AR glasses 100.

[0022] In addition, the AR glasses 100 measure acceleration in three dimensions using an acceleration sensor 104 provided in the AR glasses 100, and analyze and detect movement information such as the position, direction, tilt, and vibration of the user's head when wearing the AR glasses 100.

[0023] Furthermore, the AR glasses 100 can capture the user's walking state from the analyzed and detected movement information of the user's head, and obtain user walking information such as walking movement, walking direction, and walking speed.

[0024] Therefore, the AR glasses 100 are equipped with a walking detection sensor that detects walking information such as the walking movement, walking direction, and walking speed of the user wearing the AR glasses 100, using a camera 103 that photographs the user's feet and an acceleration sensor 104 that measures the three-dimensional acceleration of the user's head.

[0025] The AR glasses 100 also include a head movement detection sensor that detects the movement of the user's head wearing the AR glasses 100 using an acceleration sensor 104 that measures the acceleration of the user's head in three dimensions.

[0026] In addition, the AR glasses 100 can detect the line of sight of the user 10 using a left eye line of sight sensor 101 and a right eye line of sight sensor 102.

[0027] Figure 2 shows a schematic diagram of the scenery 201 in front of the user, the field of view display range 202 of the AR glasses 100, and the user 10 wearing the AR glasses 100 when the user 10 wearing the AR glasses 100 is in the state shown in Figure 1 and the camera 103 and acceleration sensor 104 determine that the user is not walking but is stationary.

[0028] In FIG. 2, the user 10 has stopped walking, so even if the display items 111, 112, and 113 are small in size, they are not difficult to read and can be easily viewed as display items 211, 212, and 213 within the field of view display range.

[0029] Furthermore, by moving the head up, down, left and right to move the field of view display range 202, the entire view of the scenery 201 ahead can be grasped.

[0030] FIG. 3 shows an example of a display when the user 10 starts walking from the stationary state shown in FIG.

[0031] When walking in the forward direction indicated by arrow 300, the user primarily views the scenery in field of view display range 202 in the direction the user is walking, while also keeping an eye on display items 211, 212, and 213. The smaller the display size of the display items in field of view display range 202, the more difficult it is to see or read them, and the worse the visibility of the display items in the field of view display range becomes.

[0032] Therefore, when it is determined that the user 10 is walking by a walking detection sensor that analyzes the output of the camera 103 and the acceleration sensor 104, the display items 211, 212, and 213, which are virtual objects, are greatly enlarged and displayed as display items 311, 312, and 313, as shown in Fig. 3. This allows the user to easily see and read the enlarged display items while walking, and the visibility of the display items, which are virtual objects, can be significantly improved.

[0033] The enlargement of a display item may be achieved by enlarging the display item itself, or by enlarging the characters or icons within the display item. If enlarging a display item would obstruct too much of the field of view, the number of characters may be limited to reduce the amount of information displayed, or the character spacing may be narrowed.

[0034] <Second Embodiment> A second embodiment will be described with reference to Fig. 4. Elements having similar features to those in Fig. 2 and Fig. 3 are given the same numbers, and detailed descriptions thereof will be omitted.

[0035] When the user is walking, in consideration of the safety ahead in the walking direction and at their feet, the user's gaze is directed primarily to the central region 250 of the field of view display range 202 along which the user is walking, avoiding concentration on the periphery. Therefore, within the field of view display range 202, the central region 250 is relatively easy for the user to see, but the display items in the peripheral region 260 are more difficult to see than the display items in the central region 250.

[0036] 2 in the central region 250 of the field of view display range 202 is left as is, and the display item 211 in FIG. 2 in the peripheral region 260 within the field of view display range 202 is enlarged and displayed as display item 311, as in FIG. 3. This makes it possible to improve the visibility of the display items in the peripheral region 260, which becomes more difficult for the user to see as compared to the central region 250 of the field of view display range 202 as the user walks, and makes it possible to view the display items in the peripheral region 260 without any difference from the display items in the central region 250.

[0037] The central region 250 and the peripheral region 260 are defined as a predetermined range including the center point of the field of view display range 202, and the peripheral region 260 is defined as the area around the central region 250. The "predetermined range" may be determined in advance, or the width of the predetermined range may be changed depending on the walking state. For example, if the walking speed is relatively fast, the "predetermined range" may be relatively narrow, and if the walking speed is relatively slow, the "predetermined range" may be relatively wide.

[0038] In the embodiment of Fig. 3, the display items in the central area 250 are enlarged, so that the large display items 312 and 313 largely block the view in front of the user while walking, which can make it difficult for the user to see. In contrast, in the embodiment of Fig. 4, the size of the display items in the central area 250 is left unchanged to minimize the impact on the view in front of the user, and the display items in the peripheral area 260 are enlarged, which has the effect of improving visibility.

[0039] Third Embodiment A third embodiment will be described with reference to Fig. 5. Elements having similar features to those in Figs. 2, 3 and 4 are given the same numbers, and detailed descriptions thereof will be omitted.

[0040] 3 and 4 is the display size of the display items 212 and 213 in Fig. 2 in the central region 250 within the field of view display range 202. In Fig. 3, display items 312 and 313 are enlarged, while in Fig. 4, display items 212 and 213 are not enlarged. In Fig. 5, display items 512 and 513 are enlarged at a lower magnification than display item 311 in the peripheral region.

[0041] When the user is walking, the visibility of the display items is high in the peripheral region 260 and low in the central region 250, but dynamic visual acuity tends to be lower than static visual acuity. By utilizing this characteristic, when the user is walking, it is possible to ensure visibility by enlarging the display items in the central region 250 while minimizing the obstruction of the scenery in the central region 250 within the field of view display range 202.

[0042] While Figure 5 shows two-stage switching of the size of the displayed items between the central and peripheral regions, the display size may also be controlled smoothly or in multiple stages from the center to the periphery of the field of view display range. Multi-stage or smooth display size control can further reduce the scenery occlusion effect and ensure visibility.

[0043] <Fourth Embodiment> In contrast to the embodiment of FIG. 4 which describes walking forward, the display item display operation of the information processing device when the user is stationary or when the user starts walking forward and then walks diagonally left or right will be described with reference to FIGS. 6A and 6B .

[0044] 6A shows the field of view display range 602 and the display state of display items of the AR glasses 100 when walking diagonally to the left, and Fig. 6B shows the field of view display range 604 and the display state of display items of the AR glasses 100 when walking diagonally to the right, relative to the scenery 201 in front of the user when the user is stationary. In Fig. 6A and Fig. 6B, parts shown in Fig. 4 and assigned the same reference numerals have the same operations as those already explained in Fig. 4, and therefore detailed explanations thereof will be omitted.

[0045] 6A , when the user walks with his / her head turned diagonally left as indicated by arrow 601, the field of view display range of the AR glasses 100 moves to the left by an amount corresponding to the direction of walking diagonally left, as shown by field of view display range 602. Accordingly, the positions of a central region 650 and a peripheral region 660 within field of view display range 602 also change. For this reason, when the user is standing still or walking forward, the display item 213 in FIGS. 2 and 4 that was in the central region 250 of field of view display range 202 (for example, FIG. 5 ) moves to a peripheral region 660 within field of view display range 602 that is difficult for the user to reach.

[0046] Therefore, when walking diagonally left, if the display item 213 that has moved to the surrounding area 660 is displayed in the same relatively small display size, it will be difficult to see.

[0047] Therefore, in the fourth embodiment, the display item 213 that has come to be located in the peripheral area 660 when walking diagonally left is greatly enlarged and displayed as the display item 613. In other words, when walking diagonally left, the display item that has come to be moved to the peripheral area 660 is greatly enlarged and displayed, thereby improving the visibility of the display item that has moved to the peripheral area 660.

[0048] It should be noted that the display size of the display item 212 does not change because the display item 212 is included in the central region 650 within the field of view display range 602 .

[0049] 6B shows the case where the user walks diagonally to the right. When the user walks diagonally to the right as indicated by arrow 603, the field of view display range of the AR glasses 100 moves to the right by an amount corresponding to the direction of the user's walking diagonally to the right, as shown by field of view display range 604. Accordingly, the positions of a central region 651 and a peripheral region 661 within field of view display range 604 also change. Therefore, display item 212, which has been positioned in peripheral region 661 when the user walks diagonally to the right, is greatly enlarged and displayed as display item 612. This makes it possible to improve the visibility of the display item that has moved into peripheral region 661 when the user walks diagonally to the right.

[0050] On the other hand, since the display item 213 is included in the central region 651 within the field of view display range 604, the display size of the display item 213 does not change.

[0051] 6A and 6B, display items 211 in peripheral areas 660 and 661 of field of view display ranges 602 and 604 are greatly enlarged and displayed as display items 311 when walking diagonally to the left, in addition to when stationary or walking forward, thereby improving the visibility of these display items.

[0052] As explained above using Figures 6A and 6B, when walking forward or diagonally left or right, display items located in the peripheral area within the field of view display range are greatly enlarged and displayed, making it possible to avoid concentrating one's gaze on the periphery of the field of view and to easily view display items located in the periphery of the field of view display range as virtual objects in an easy-to-see manner.

[0053] Fifth Embodiment Next, the display operation within the field of view display range of an information processing device when the user moves their head by turning their head left and right, up and down, etc. while walking will be described with reference to Fig. 7A, Fig. 7B, and Fig. 7C. In Fig. 7A, Fig. 7B, and Fig. 7C, the parts with the same reference numerals as those shown in Fig. 4 and Fig. 6B have the same operations as those already described in Fig. 4 and Fig. 7B, and therefore detailed description thereof will be omitted.

[0054] 7A shows an example in which the user 10 is walking forward and turning his / her head to the left. The user has moved leftward from the field of view display range 202 in FIG. 4 to the field of view display range 701. As a result, the display item 713 in the peripheral area 760 of the field of view display range 701 is displayed at a larger size than the display item 213 in the central area 250 of the field of view display range 202 in FIG. 4.

[0055] Note that the display item 212 was in the central area 250 of the field of view display range 202 in FIG. 4, but remains in the central area 750 in the field of view display range 701 in FIG. 7A, so there is no change in the display size.

[0056] 7A shows an example in which the user 10 is standing still and turns his / her head to the left, but if the user changes the direction of his / her head while walking, a different display control may be performed. FIG. 7B shows such an example.

[0057] When the user 10 moves their head while walking, turning their head left and right, up and down, etc., the user 10 will first turn their head to the left, for example, out of curiosity, and at this time it can be said that the user is very conscious of looking at, reading, and visually recognizing display items that will be located in the peripheral area 760 on the right side, which is opposite the direction of movement of the central area 750 within the field of view display range 701. Therefore, it is thought that there is little need to intentionally enlarge and display display items that will move to the peripheral area 760 within the field of view display range 701 by turning their head.

[0058] Furthermore, if the display state of a display item, such as its size or position, changes in response to head movement, the user may become distracted and lose focus on the direction in which the user is turning their head.Furthermore, if the display state of a display item changes even though the scenery in the field of view does not change when the user moves their head, the screen within the field of view display range becomes noisy, adversely affecting the user's visibility.

[0059] Therefore, in the embodiment shown in Figure 7B, if the walking direction is maintained even when the acceleration sensor 104, which is a head movement detection sensor, detects movement of the user's head while walking, changes in the display state of the display items are suppressed by maintaining the central area 250 and the display state of the display items as they were before the head movement occurred (i.e., the same central area and display state as in Figure 4).

[0060] 4 in which the head is facing in the forward direction of walking, when the user 10 turns their head to the left in the forward direction of walking, the field of view display range moves to field of view display range 701. In Fig. 7A, the display item 713 is displayed large, but in Fig. 7B, the display item 213 is displayed in the same display size as in Fig. 4.

[0061] According to this embodiment, the display size is not changed when the walking direction is maintained, so a stable display can be provided without drawing the user's attention to unnecessary changes in the display size.

[0062] FIG. 7C shows an example in which the user 10 turns his head upward and left while walking diagonally to the right.

[0063] As shown in Fig. 6B , when the user 10 turns his head upward and left while walking diagonally to the right, the field of view display range moves from field of view display range 604 to the upper center of the forward scene 201, becoming field of view display range 702 shown in Fig. 7C . As in Fig. 7A , there are no display items in a central region 751 of field of view display range 702, and display items 311, 612, and 213 in a peripheral region 761 of field of view display range 702 may be enlarged and displayed.

[0064] However, in Fig. 7C, in order to suppress changes in the display state due to head movement, the display size of Fig. 6B when turning the head diagonally to the right while walking is maintained, and only the field of view display range is moved from field of view display range 604 in Fig. 6B to field of view display range 702 in Fig. 7C. Therefore, display items 311 and 612 in peripheral area 661 within field of view display range 604 when the head is turned in the walking direction are enlarged, and display item 213 in central area 651 of field of view display range 604 in Fig. 6B is not enlarged.

[0065] That is, when the walking direction is maintained and only the head is moved, the discrimination criterion between the central region and the peripheral region, which is the basis for determining the display size of the display item, is the actual field of view display range, which is the field of view display range 701 linked to the movement of the head, in Fig. 7A, but is the front field of view display range when the head is facing the walking direction in Fig. 7B and Fig. 7C, which are different in that it is the front field of view display range when the head is facing the walking direction. Incidentally, in all of Fig. 4, Fig. 5, Fig. 6A, and Fig. 6B, the head is facing the walking direction, so the actual field of view display range and the front field of view display range match.

[0066] The user may be able to switch between the actual field of view display range and the front field of view display range as the display size discrimination standard.

[0067] When the direction of the body and feet matches the walking direction, the frontal field of view display range is used as the reference, but when it is determined to be inconsistent, such as when walking in a "crab-like manner" (moving the body sideways while facing forward), the display range may be automatically switched to the actual field of view display range as the reference.

[0068] Furthermore, when the head is shifted in the walking direction or in front of the body by a predetermined angle, for example, approximately 10 degrees or more, the actual field of view display range standard may be prioritized after a predetermined time, such as approximately 0.5 seconds, has elapsed, respecting the user's intention to look in a direction other than the walking direction. Furthermore, the predetermined time may be controlled to decrease as the predetermined angle increases.

[0069] 8A, 8B, 8C, and 8D, the display item display operation of an information processing device in a case where the peripheral area within the field of view display range is divided into a plurality of peripheral areas, and display items in the central area of ​​the field of view display range are moved to the divided peripheral areas and enlarged and displayed while walking. In these figures, parts with the same reference numerals as those shown in Figures 3 and 4 have the same operations as those already explained, and therefore detailed explanations thereof will be omitted.

[0070] Figure 8A shows an example of dividing the peripheral areas, where a central area 850 is the area surrounded by a dashed frame 801 closest to the center, a first peripheral area 860 is the area outside the dashed frame 801 and inside the dashed frame 802, and a second peripheral area 870 is outside the dashed frame 802 and inside the field of view display range 202.

[0071] As shown in FIG. 8A, in the user stationary state, a display item 212 is displayed in a central region 850, a display item 813 is displayed in a first peripheral region 860, and a display item 211 is displayed in a second peripheral region 870.

[0072] When user 10 starts walking forward from a stationary state, it is desirable to erase or move the display items in central region 850 so that obstacles in the walking direction can be more easily seen. Thus, in Fig. 8B, display item 212 that was in central region 850 in Fig. 8A is moved to first peripheral region 860 and displayed as display item 812, thereby ensuring visibility ahead as the user walks.

[0073] Display item 813 in first peripheral area 860 is displayed without enlargement, as in Fig. 8A, and display item 211 that was in second peripheral area 870 in Fig. 8A is enlarged and displayed as display item 511. In this way, the visibility of the scenery in the walking direction is ensured, while the size of the surrounding display items is displayed larger to improve the visibility of the display items.

[0074] FIG. 8C shows an example in which display items located in the first peripheral area 860 in FIG. 8B are enlarged and displayed to ensure visibility ahead and improve visibility of the display items.

[0075] Display item 212 in central area 850 in Fig. 8A is moved to first peripheral area 860 and further enlarged to display item 822. Display item 813 displayed in first peripheral area 860 in Fig. 8A is enlarged to display item 823, and display item 211 in second peripheral area 870 in Fig. 8A is enlarged to display item 511. At this time, by making display items 822 and 823 in first peripheral area 860 larger than display item 511 in second peripheral area 870, it is possible to improve both the visibility of the scenery in the walking direction and the visibility of large surrounding display items.

[0076] 8D shows an example in which display item 212, which was in central region 850 in FIG. 8A , is moved to second peripheral region 870 and then enlarged to display item 832. In this example, display item 832 is displayed at a size similar to that of display item 511 displayed in second peripheral region 870. In this way, the size of the display item may be gradually changed depending on the destination peripheral region, thereby improving both the visibility of the scenery in the walking direction and the visibility of large surrounding display items.

[0077] Seventh Embodiment A display item display operation of an information processing device when controlling the enlarged display size of display items within the field of view display range in accordance with the walking speed of a user will be described with reference to Fig. 9A and Fig. 9B. Fig. 9A is an example of a diagram showing the relationship between walking speed and the display size of display items.

[0078] For a user 10 walking with the AR glasses 100 attached to their head, the faster the walking speed, the harder it becomes to see and read the display items displayed as virtual objects within the field of view display range, which tends to result in reduced visibility of the display items. Therefore, the faster the walking speed of the user 10, the more the display items within the field of view display range can be enlarged and displayed, thereby improving the visibility of the display items.

[0079] As indicated by the solid line 1002 in Fig. 9A , the enlarged display size of the display items may be gradually increased in accordance with the walking speed of the user 10. As another example, as indicated by the dashed line 1003 in Fig. 9 , the enlarged display size of the display items may be switched in multiple stages in accordance with the walking speed of the user 10. Switching the enlarged display size of the display items in multiple stages has the effect of preventing the display items from being stably viewed by the user 10 even if the walking speed changes slightly within a certain level. In multi-stage switching, the display is likely to become unstable when the display size is switched, so it is advisable to provide hysteresis as indicated by the dashed line 1003 to prevent critical display size switching.

[0080] FIG. 9B is an example of a diagram showing the relationship between walking speed and the destination of a displayed item in the central area.

[0081] For example, when the user is walking at a normal speed and not very fast, the display items in central region 850 are moved to first peripheral region 860 as shown in Fig. 8C , and when the user is walking at a fast speed and walking briskly, the display items in central region 850 are moved to second peripheral region 870 as shown in Fig. 8D . Furthermore, the faster the walking speed, the closer the display items in central region 850 are to the user's field of vision, and the more likely the display items are to obstruct the view of the scenery ahead.

[0082] 9B , by controlling the movement of display items in central region 850 to first peripheral region 860 and second peripheral region 870 located further outward as the walking speed increases, it is possible to reduce the visual obstruction of the scenery ahead caused by display items that occurs in conjunction with the walking speed of user 10. As with FIG. 9A , in FIG. 9B , it is preferable to provide hysteresis to prevent critical display items from moving.

[0083] Furthermore, the display positions of the display items may be switched in multiple stages according to the walking speed of the user 10, i.e., by further dividing the movement destination into smaller areas than the movement between the central area 850, the first peripheral area 860, and the second peripheral area 870. Switching the display positions of the display items in multiple stages has the effect of preventing the display positions of the display items from changing even if the walking speed changes slightly within a certain level, thereby enabling the user 10 to stably view the display items.

[0084] Furthermore, when user 10 moves quickly, such as when switching from walking to running or cycling, it may be desirable to enlarge the field of view area ahead in the direction of movement. For this reason, it is advisable to widen the central area 850 of the field of view display range according to the walking speed or moving speed. The size of this central area 850 may also be controlled in the same way as the walking speed vs. display size in FIG. 9A .

[0085] For example, when the user is walking at 4 km / h, the central region 850 may be a trapezoidal region about 3 m in front, with a width of about 10 degrees from the shoulder width of the user 10 to the left and right, and may be expanded to about 5 m in front when running at 5 km / h, and 10 m in front when riding a bicycle at 10 km / h. The forward distance may be increased depending on the traveling speed. The angle of opening to the left and right in front may also be increased depending on the running speed.

[0086] Eighth Embodiment An input operation of an information processing device when inputting and setting the display state, such as the display size and display position of a display item, while the user 10 is walking will be described with reference to Fig. 10. Fig. 10 shows an example in which an operation menu 1101 is displayed within the field of view display range while the user is stationary. In Fig. 10, parts with the same reference numerals as those shown in Fig. 2 perform the same operations as those already described in Fig. 2, and therefore detailed description thereof will be omitted.

[0087] Some users 10 have poor eyesight or a narrow field of view, and users with impaired eyesight may find it difficult to see the scenery ahead or display items while walking. Therefore, as shown in Fig. 10, the AR glasses 100 display an operation menu 1101 within the field of view display range 202 for inputting and setting display conditions such as the display size and display position of display items, and when the user 10 inputs and sets the desired display size and display position of display items from the operation menu 1101 while walking, the display size and display position of the display items are controlled according to the input setting information while walking.

[0088] Input settings such as the display size and display position of display items that are optimal for the user 10 can be made by operating the operation menu 1101 with hands or gaze. Input setting operations using gaze are performed by detecting the gaze direction of the user 10 with the left eye gaze sensor 101 and the right eye gaze sensor 102, and having the gaze direction point to the input setting button position on the operation menu 1101. Thus, the left eye gaze sensor 101 and the right eye gaze sensor 102 constitute an gaze input device that selects the operation menu 1101.

[0089] Furthermore, input setting operations can be performed with the hand by pointing the hand at the input setting button position on the operation menu 1101. For example, for a user 10 with poor eyesight, if the user 10 inputs and sets the display size in the operation menu 1101 so that the display items are enlarged to a larger size than normal, the visibility of the display items can be improved while walking.

[0090] In addition, the display position of a display item that moves from the central area to the peripheral area of ​​the field of view display range 202 may be set and input in the operation menu 1101 to be optimal for the eyesight of the user 10, thereby improving the visibility of the scenery ahead in the central area opened up by the movement of the display item while walking, and the display item itself.

[0091] Furthermore, while walking, the input setting operation by gaze may be restricted by erasing the display of the operation menu 1101 in which input setting is performed by gaze or by stopping the input setting operation by gaze on the operation menu 1101. This prevents the gaze of the user 10 from being directed toward the operation menu 1101, and allows the gaze to be focused on the scenery ahead within the field of view display range 202 or on display items that are virtual objects.

[0092] Furthermore, including operation menus other than input settings, the display of the operation menu 1101 may be suppressed when walking at a predetermined speed or above, such as 1 km / h, so as not to distract attention from where you are walking when using eye gaze input while walking.

[0093] <Processing Flow According to Each of the Above-described Embodiments> The display operation of the display items when walking starts and when walking stops will be described with reference to the flowcharts shown in FIGS. 11A and 11B.

[0094] 11A, the user 10 is wearing the AR glasses 100 and is about to start walking. Before the user starts walking, i.e., while the user is stationary (S1201), a display item is displayed within the field of view display range (S1202: Yes). In this state, the processor 1320 determines whether the user is walking or not based on sensor information from the walking detection sensor (S1203: walking determination step).

[0095] When the processor 1320 determines that the user 10 is walking (S1203: Yes), the AR glasses 100 control the display size of the display items, the display position of the display items, or a combination thereof, according to the positions of the display items displayed within the field of view display range (S1204). For example, the AR glasses 100 may enlarge and display the display items in the peripheral region of the field of view display range (corresponding to the enlarging step), or move the display items in the central region of the field of view display range to the peripheral region (corresponding to the display position moving step), and display them (corresponding to the display step).

[0096] After this, when the walking detection sensor detects that walking has stopped (S1205: Yes), as shown in FIG. 11B, the AR glasses 100 temporarily maintain the controlled display size and display position of the display item (S1206).

[0097] Here, when the user 10 performs an input operation indicating that the display state consisting of the maintained display size, display position, or a combination thereof of the display items is to be transitioned back to the display state of the display items when stationary (S1207: Yes), the display state is transitioned back to the display state of the display items when stationary (S1209).

[0098] Even if no input operation indicating a transition back is performed (S1207: No), once a predetermined control maintenance period has elapsed after the detection of cessation of walking (S1208: Yes), the AR glasses 100 transition back from the maintained display state of the display items to the display state of the display items before control, i.e., the display state of the display items when stationary (S1209).

[0099] In FIG. 11B, for ease of explanation, step S1208 is illustrated as being executed following step S1207, but steps S1207 and S1208 may be executed in the reverse order, or may be processed in parallel, and if a positive judgment is made in either step, the process may proceed to step S1209.

[0100] After transitioning back to the display state of the display items when the user is stationary, if the display items continue to be displayed when the user is walking (S1210: No), the process returns to the display item display confirmation in sequence S1202, and if the display items do not continue to be displayed (S1210: Yes), the sequence ends.

[0101] With the above operations, when walking stops, the display state of the display items while walking is maintained as is for the time being, and after a predetermined control maintenance period or by an input operation by the user, the display state of the display items while walking that was maintained is transitioned back to the original display state of the display items while stationary.

[0102] This reduces the possibility of a bad situation where, when walking stops, the display state of the display items, consisting of the display size, display position, or a combination thereof, changes suddenly at that moment, causing a user who is looking at or reading the display items to feel uncomfortable or abrupt.

[0103] In addition, after a user operation is performed or after a predetermined control maintenance period, the display state of the display item may be gradually and smoothly transitioned back to the original display state of the display item when it was stationary, which makes it possible to further reduce the sense of discomfort or abruptness felt by the user.

[0104] Furthermore, even when walking, when avoiding puddles or walking on bumpy roads, the display state of the display items displayed during the previous average walk may be maintained as is, in which case frequent changes in the display size and display position of the display items can be suppressed, and difficulty in viewing the display items can be reduced or eliminated.

[0105] An example of the configuration of an information processing device according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of the configuration of AR glasses, which is one form of HMD, as an example of the information processing device according to this embodiment. In Fig. 12, parts assigned the same reference numerals as those shown in Fig. 1 and the like have the same operations as those already explained in Fig. 1 and the like, and therefore detailed explanations thereof will be omitted.

[0106] In Figure 12, the AR glasses 100 are configured appropriately using a left eye gaze sensor 101, a right eye gaze sensor 102, a camera 103, an acceleration sensor 104, a geomagnetic sensor 1304, a gyro sensor 1305, a distance measurement sensor 1306, an operation input interface 1307, a display processing device 1308, a processor 1320, a storage 1330 that stores a program 1331 and information data 1332, a vibrator 1341, a microphone 1342, a speaker 1343, a communication device 1344, a timer 1345, and a RAM 1325, and each component is connected to each other via a bus 1350.

[0107] The camera 103 captures an image of the surroundings in front of the camera, and converts light incident from a lens into an electrical signal using an image sensor to obtain an image captured by the camera.

[0108] The camera 103 captures the scenery in front of the user 10 as well as the feet of the user 10. Therefore, the walking state of the user 10 can be analyzed from the captured state of the feet of the user 10, and is used as a walking detection sensor that detects the user's walking motion, walking direction, and walking speed. Note that capturing the feet with a camera can analyze not only the user's walking state, but also when the user is riding a bicycle or kick scooter while moving their feet, making it possible to control the display state, such as enlarging displayed items.

[0109] On the other hand, it is also possible to analyze the situation when the feet are not moving while riding on a train or bus, and even if the movement of the user 10 is detected by an acceleration sensor 104 or the like, it is determined that the user is stationary and the display state of the displayed items is not controlled.

[0110] If the camera captures not only your feet but also your hands, it can determine whether you are driving a car or not, and can prevent items from being displayed while you are driving.

[0111] In addition, in the video see-through type AR glasses, the camera 103 captures an image of an actual object in real space, i.e., the scenery in front of the user, and the image of the captured actual object is displayed on the display processing device 1308.

[0112] The left eye gaze sensor 101 and the right eye gaze sensor 102 are capable of detecting the movement and direction of the left eye and the right eye, respectively, and capturing the gaze of the user 10. Note that the process of detecting eye movement may utilize well-known techniques that are commonly used as eye tracking processing. For example, a method using corneal reflex is known in which an infrared LED (Light Emitting Diode) is irradiated onto the face, an image is captured with an infrared camera, and the position on the cornea of ​​the reflected light produced by the infrared LED irradiation (corneal reflex) is used as a reference point, and the movement of the eye and the gaze are detected based on the position of the pupil relative to the position of the corneal reflex.

[0113] The acceleration sensor 104 is a sensor that detects acceleration, which is a change in speed per unit time, and can capture movement, vibration, impact, etc. The AR glasses 100 are worn on the head of the user 10, and movement information such as the rotation, tilt, and vibration of the user 10's head can be analyzed and acquired from the acceleration data detected by the acceleration sensor 104 provided in the AR glasses 100.

[0114] The acceleration sensor 104 is used as a head movement detection sensor that detects the movement of the user's head, and can detect when the user 10 turns their head left, right, up, down, etc. Note that the relative relationship between the head and body (direction of the feet) cannot be directly detected by the acceleration sensor attached to the head alone, so an acceleration sensor may also be attached to the body, or the body and feet may be photographed and acquired using a camera built into the AR glasses 100.

[0115] In addition, when a user walks facing the same direction, the sensor information from the head movement detection sensor can be analyzed to detect the movement of the user's head, and the walking direction and walking speed of the user wearing the AR glasses 100 can also be detected, so it can also be used as a walking detection sensor.

[0116] The AR glasses 100 alone are limited to detecting average walking direction and walking speed, but this level is acceptable.

[0117] The gyro sensor 1305 is a sensor that detects angular velocity in the rotation direction and can capture the vertical, horizontal, and diagonal orientations. Accuracy can be improved by using the acceleration sensor 104 and gyro sensor 1305 of a smartphone or smartwatch placed in the user's pocket or the like in combination.

[0118] The geomagnetic sensor 1304 is a sensor that detects the Earth's magnetic force and detects the direction in which the AR glasses 100 are facing. A three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions can be used to detect the movement of the AR glasses 100 by capturing changes in geomagnetism in response to the movement of the AR glasses 100. This makes it possible to more accurately analyze and obtain the movement of the user's head and the walking state of the user wearing the AR glasses 100.

[0119] The ranging sensor 1306 is a sensor that can measure the distance and angle to an object and capture the shape of the object in three dimensions. Examples of sensors that can be used include LiDAR (Light Detection and Ranging), which irradiates an object with laser light such as infrared light, measures the reflected scattered light, and analyzes and detects the distance to a distant object and the state of the object; TOF (Time of Flight) sensors, which measure the distance by measuring the reflection time of pulsed light irradiated on a subject for each pixel; and millimeter-wave radar, which emits millimeter-wave radio waves, captures the reflected waves, and detects the distance to the object and the state of the object. The ranging sensor 1306 measures the distance and direction to a physical object in the scene ahead, and based on the measurement information, it becomes possible, for example, to virtually place a display item related to the physical object in a space near the physical object.

[0120] The processor 1320 is configured by a CPU or the like.

[0121] The storage 1330 is configured with a nonvolatile storage device or the like, and stores various programs 1331 and information data 1332 handled by the processor 1320 or the like. As the information data 1332, display item information 1335, walking information 1336, head movement information 1337, input setting information 1338, etc. are stored.

[0122] The processor 1320 loads an operating system (OS) 1333 and application programs 1334 for operation control, etc., stored in the storage 1330 into the RAM 1325 and executes them, thereby forming a walking state detection processing unit 1321, a head movement detection processing unit 1322, a display item display form control processing unit 1323, and an operation menu processing unit 1324, and realizing functions such as the OS, middleware, applications, and other functions.

[0123] In the case of optical see-through AR glasses, the display processing device 1308 is composed of a projection device that projects display items and notification information to the user, and a transparent half mirror that forms an image of the projected display items in front of the user's eyes.

[0124] This allows the user 10 to view both the actual object in the field of view in front of the user's eyes and the imaged display item together, as if they were floating.

[0125] In the case of video see-through AR glasses, the glasses are configured to include a display such as a liquid crystal panel that displays the actual object in front of the user photographed by the camera 103 together with display items, etc.

[0126] This allows the user 10 to visually recognize the actual objects in the field of view image in front of them superimposed on the display items, etc.

[0127] The operation input interface 1307 is an input means such as a line of sight, a hand, a pointer, a keyboard, key buttons, or touch keys, and is used by the user 10 to set and input the information that he or she wishes to input.

[0128] An input operation screen such as the operation menu 1101 may be displayed on the display screen of the display processing device 1308, and input operation information may be captured based on the position on the input operation screen to which the gaze is directed, or a hand or pointer may be displayed on the input operation screen and the hand or pointer may be operated using the operation input interface 1307 to capture input operation information.

[0129] Furthermore, when input is made using a keyboard, key buttons, touch keys, etc., the keyboard or key buttons may be provided in a position or form within the AR glasses 100 that makes it easy for the user 10 to perform input operations, or the keyboard or key buttons may be separated from the main body of the AR glasses 100 and connected via a wired or wireless connection.

[0130] Alternatively, the user 10 may vocalize a voice indicating an input operation, and the voice may be collected by the microphone 1342 to capture the input operation information.

[0131] The microphone 1342 collects external sounds and the user's own voice and converts them into digital audio data using an A / D converter (not shown). Instruction information based on the voice of the user 10 can be captured by the AR glasses 100, allowing the user 10 to easily perform actions in response to the instruction information.

[0132] The speaker 1343 outputs sound based on the audio data, and can notify the user 10 of notification information by sound. For example, the speaker 1343 outputs information by sound such as the direction in which the user 10 is running, the direction in which the user is facing, enlarged display of a displayed item, or a change in position.

[0133] The vibrator 1341 generates vibrations under the control of the processor 1320, and converts notification instruction information to the user 10 transmitted by the AR glasses 100 into vibrations. The vibrator 1341 transmits vibrations to the head of the user wearing the AR glasses 100, thereby making it possible to notify the user 10 of the notification instruction information and improving usability.

[0134] The communication device 1344 is a communication interface that performs wireless communication with other devices using short-range wireless communication such as Bluetooth (registered trademark) or ZigBee (registered trademark). The communication device 1344 includes a communication processing circuit, an antenna, etc. that correspond to various predetermined communication interfaces, and transmits and receives various information, control signals, etc. It may also include a telephone communication network.

[0135] The timer 1345 measures time, and measures the lapse of a predetermined control maintenance period after the detection of the cessation of walking.

[0136] The walking state detection processing unit 1321 analyzes the walking state of the user 10 from the image of the feet captured by the camera 103, and detects the walking direction and walking speed. It also analyzes the walking state of the user 10 from head movement information detected by the acceleration sensor 104, and detects the walking direction and walking speed. In addition to the acceleration sensor 104, a geomagnetic sensor 1304 and a gyro sensor 1305 may also be used.

[0137] The head movement detection processing unit 1322 detects the movement of the head of the user 10 using the acceleration sensor 104, and analyzes and acquires the state in which the user 10 turns his / her head left and right, up and down, etc. In addition to the acceleration sensor 104, a geomagnetic sensor 1304 and a gyro sensor 1305 may also be used.

[0138] The display item display form control processing unit 1323 controls the display size, the display position, or a combination thereof of the display items according to the positions, etc. of the display items displayed as virtual objects within the field of view display ranges 202, 602, 604, 701, and 702. For example, the display items in peripheral areas 660, 760, 761, 860, and 870 within the field of view display range are enlarged and displayed, or the display items in the central areas 650, 750, 751, and 850 of the field of view display range are moved to the peripheral areas 660, 760, 761, 860, and 870 and enlarged and displayed.

[0139] The operation menu processing unit 1324 displays an operation menu 1101 for setting and inputting the display state such as the display size and display position of the display items, and operation instructions for the AR glasses 100, and the information to be input is set and input by the user 10's gaze, hands, etc. towards the displayed operation menu 1101.

[0140] With the above configuration, in the AR glasses 100, when the walking state detection processing unit 1321 detects that the user 10 is walking, the display item display form control processing unit 1323 can enlarge and display the display items generated and displayed within the user's field of view display range, enlarge and display the display items in the peripheral areas 660, 760, 761, 860, 870 within the field of view display range 202, 602, 604, 701, 702, and also enlarge and display the display items in the peripheral areas 660, 760, 761, 860, 870 within the field of view display range 202, 602, 604, 701, 702 to a larger size than the enlarged display items in the central areas 650, 750, 751, 850.

[0141] In addition, in the AR glasses 100, when the walking state detection processing unit 1321 detects that the user 10 is walking, if the head movement detection processing unit 1322 detects movement of the user's head, the display item display form control processing unit 1323 can maintain the display size and display position of the display item as they were before the head movement occurred, and display the display item in the maintained display state.

[0142] In addition, in the AR glasses 100, when the walking state detection processing unit 1321 detects that the user 10 is walking, the display item display form control processing unit 1323 can move the display items in the central areas 650, 750, 751, and 850 of the field of view display range to peripheral areas outside the central areas 650, 750, and 850 of the field of view display range and enlarge the display.

[0143] When the field of view display range 202 is divided into a first peripheral region 860 that is adjacent to the outside of the central region 850 and forms a peripheral portion, and a second peripheral region 870 that is adjacent to the outside of the first peripheral region 860 and forms a peripheral portion, the display item display form control processing unit 1323 can move a display item in the central region 850 to the first peripheral region 860 and display it at a slightly enlarged display size or at the same display size, or move a display item in the central region 850 to the second peripheral region 870 and display it at a greatly enlarged display size. Furthermore, the display item in the central region 850 can also be moved to the outside of the field of view display range 202.

[0144] Furthermore, in the AR glasses 100, the walking state detection processing unit 1321 detects the walking speed of the user 10 while walking, and the display item display form control processing unit 1323 controls and displays the display size, display position, or a combination thereof of the display items according to the detected walking speed of the user 10. In this case, it is also possible to switch and control the display size, display position, or a combination thereof of the display items in multiple stages according to the walking speed of the user.

[0145] In the AR glasses 100, the operation menu processing unit 1324 sets and inputs the information that the user 10 wants to input by directing their gaze, hands, etc. toward the displayed operation menu 1101, and the display item display form control processing unit 1323 can control the display size and display position of the display items according to the input setting input information.

[0146] Furthermore, when the walking state detection processing unit 1321 detects that the user 10 is walking, the operation menu processing unit 1324 can be stopped to prohibit input operations to the operation menu by the user's line of sight.

[0147] In addition, in the AR glasses 100, when the walking state detection processing unit 1321 detects that the user 10 has stopped walking, the display item display form control processing unit 1323 maintains the display size and display position of the display items as they are, and after a predetermined control maintenance period or by an input operation by the user, the display size and display position of the display items maintained when walking can be transitioned to the original display size and display position of the display items when stationary.

[0148] Thus, an information processing device and an information processing method can be realized that, when walking, can easily view the displayed items within the field of view display range as virtual objects in an easy-to-see manner along with the scenery ahead, while avoiding focusing one's gaze on the periphery of the field of view, taking into consideration the safety of areas ahead and one's feet.

[0149] <Ninth Embodiment> A screen transmitted from a smartphone or a smartwatch may be displayed as a display item on the AR glasses 100. For example, when the user is stationary, as shown in Fig. 13A, a smartphone screen 1401 and a smartwatch screen 1402 are displayed as display items in the peripheral area 260 within the field of view display range 202. When the user is walking, as shown in Fig. 13B, the screens are enlarged to a larger display size and displayed as a smartphone screen 1411 and a smartwatch screen 1412 within the field of view display range 202. Note that in Figs. 13A and 13B, parts with the same reference numerals as those shown in Figs. 2 and 4 have the same operations as those already described, and therefore detailed description thereof will be omitted.

[0150] This makes it easier to see the smartphone screen or smartwatch screen that is displayed as a virtual object in the surrounding area that is difficult for the user to reach while walking.

[0151] The AR glasses 100 may enlarge the screen transmitted from the smartphone or smartwatch and display it on the AR glasses, or may detect that the user 10 has started walking, request an enlarged screen from the smartphone or smartwatch, receive the enlarged screen, and display it on the AR glasses 100.

[0152] In this case, the smartphone or smartwatch needs to have the function of receiving enlarged screen instructions and transmitting the enlarged screen, and it is advisable to authenticate in advance with the AR glasses 100 that it has this function.

[0153] In addition, in the case of optical see-through AR glasses, the text characters of the displayed items may be made bold in bright places to increase brightness, and may be made thin in dark places to reduce glare.

[0154] In the above explanation, AR glasses 100, which are a form of HMD, have been described as a specific example of an information processing device, but the present invention is applicable to all devices with similar functions, and it goes without saying that similar actions and effects can be obtained in information processing devices that display display items, which are virtual objects, in the user's field of view display range and view them together with the scenery ahead.

[0155] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0156] Furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0157] Furthermore, the above-mentioned components, functions, processing units, processing means, etc. may be implemented in part or in whole in hardware, for example, by designing them as integrated circuits, general-purpose processors, or application-specific processors. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. The above-mentioned components, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. Information such as programs, tables, and files that implement each function may be stored in memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communication network. Furthermore, the control lines and information lines shown are those considered necessary for explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be assumed that almost all components are interconnected.

[0158] The above-described embodiments include the following: (1) An information processing device worn on the user's head has a function of allowing the user to view a forward field of view and generating and displaying display items such as text characters and icons as virtual objects, the information processing device includes a walking detection sensor that detects the user's walking state, such as walking motion, walking direction, and walking speed, and when the walking detection sensor detects that the user is walking, the information processing device enlarges and displays the generated and displayed display items within the user's field of view display range.

[0159] (2) An information processing device according to (1), characterized in that when the walking detection sensor detects that the user is walking, the information processing device enlarges and displays display items in the peripheral area within the field of view display range.

[0160] (3) An information processing device according to (2), characterized in that when the walking detection sensor detects the user walking, the display items in the peripheral area within the field of view display range are enlarged and displayed larger than the display items in the central area of ​​the field of view display range that are enlarged and displayed.

[0161] (4) An information processing device according to (2), comprising a head movement detection sensor that detects head movement of a user wearing the information processing device, wherein when the user's walking is detected by the walking detection sensor and the head movement of the user is detected by the head movement detection sensor, the information processing device maintains the display state, such as the display size of the display items, that was present before the head movement was detected, and displays the display items in the maintained display state.

[0162] (5) An information processing device according to (2) is characterized in that the walking speed of the user is detected by the walking detection sensor, and the display size of the display items that are enlarged and displayed is controlled in accordance with the walking speed of the user detected by the walking detection sensor.

[0163] (6) The information processing device according to (5) is characterized in that the display size of the display item, which is enlarged and displayed in accordance with the walking speed of the user detected by the walking detection sensor, is switched in multiple stages.

[0164] (7) The information processing device described in (2) is characterized in that the information processing device has an operation menu through which the user sets and inputs the display state of the display items when walking, such as the display size and display position, and the information processing device controls the display size of the display items when walking in accordance with the setting input information set and input in the operation menu.

[0165] (8) The information processing device according to (1) is provided with an operation menu and an eye-gaze input device for selecting the operation menu, and is characterized in that when the walking detection sensor detects that the user is walking, the information processing device restricts input operations to the operation menu by the user's eye gaze.

[0166] (9) An information processing device according to (1), characterized in that even if the walking detection sensor detects that the user has stopped walking while walking, the display size of the display items when walking is maintained as it is, and after a predetermined control maintenance period from the user having stopped walking or by an input operation by the user, the display size of the display items when walking is transitioned from the maintained display size of the display items when walking to the original display size of the display items when stationary.

[0167] (10) An information processing device that is worn on the user's head and has the function of allowing the user to view the field of view ahead and generating and displaying display items such as text characters and icons as virtual objects, the information processing device is equipped with a walking detection sensor that detects the user's walking state, such as walking motion, walking direction, and walking speed, and is characterized in that when the walking detection sensor detects the user's walking, the information processing device moves display items in the central region of the field of view display range to outside the central region of the field of view display range.

[0168] (11) An information processing device according to (10), characterized in that when walking, display items in the central area of ​​the field of view display range are moved to the peripheral area of ​​the field of view display range and enlarged and displayed.

[0169] (12) An information processing device according to (11), wherein the peripheral area within the field of view display range is divided into a first peripheral area that is adjacent to the outside of the central area of ​​the field of view display range and forms the periphery, and a second peripheral area that is adjacent to the outside of the first peripheral area and forms the periphery, and wherein the information processing device is characterized in that when the walking detection sensor detects the user walking, the display items moved to the second peripheral area are enlarged at a larger magnification than the display items moved to the first peripheral area.

[0170] (13) The information processing device according to (10) is characterized in that the walking speed of the user is detected by the walking detection sensor, and the display position of the enlarged display item is controlled in accordance with the walking speed of the user detected by the walking detection sensor.

[0171] (14) The information processing device described in (13) is characterized in that the display position of the display item is switched in multiple stages depending on the walking speed of the user detected by the walking detection sensor.

[0172] (15) The information processing device described in (10) is provided with an operation menu for the user to input and set the display state of the display items when walking, such as the display size and display position, and the information processing device is characterized in that the display position of the display items when walking is controlled according to the setting input information input and set in the operation menu.

[0173] (16) The information processing device according to (10) is provided with an operation menu and an eye-gaze input device for selecting the operation menu, and when the walking detection sensor detects that the user is walking, the information processing device is characterized in that it restricts input operations to the operation menu by the user's eye gaze.

[0174] (17) An information processing device according to (10), characterized in that even if the walking detection sensor detects that the user has stopped walking while walking, the display positions of the display items during walking are maintained as they are, and after a predetermined control maintenance period from the user having stopped walking or by an input operation by the user, the display positions of the display items during walking that have been maintained are shifted to the original display positions of the display items during stationary state.

[0175] (18) An information processing method in an information processing device worn on a user's head, which has a function of visually recognizing the field of view ahead and generating and displaying display items such as text characters and icons as virtual objects, and which is equipped with a walking detection sensor that detects the user's walking state, such as walking motion, walking direction, and walking speed, the information processing method comprising: a step of detecting the user's walking by the walking detection sensor; and a step of enlarging and displaying the display items generated and displayed within the user's field of view display range or moving them outside the central area of ​​the field of view display range when the walking detection sensor detects the user's walking.

[0176] (19) An information processing method according to (18), comprising a step of enlarging and displaying display items in the peripheral area within the field of view display range when the walking detection sensor detects that the user is walking.

[0177] (20) An information processing device worn by a user, comprising: a display processing device that displays an augmented reality object within a field of view display range; a walking detection sensor that detects the user's walking state; and a processor, wherein the processor determines whether the user is walking based on sensor information from the walking detection sensor, and when it determines that the user is walking, executes control to enlarge the augmented reality object and display the enlarged augmented reality object within the field of view display range.

[0178] (21) An information processing device worn by a user, comprising: a display processing device that displays an augmented reality object within a field of view display range; a walking detection sensor that detects the user's walking state; and a processor, wherein the field of view display range includes a central region including the center point of the field of view display range and a peripheral region located on the outer edge of the central region, and the processor determines whether the user is walking based on sensor information from the walking detection sensor, and, if it determines that the user is walking, executes display control to move the augmented reality object displayed in the central region to the peripheral region.

[0179] (22) An information processing method executed by an information processing device worn by a user, comprising: a walking determination step for determining whether the user is walking based on sensor information from a walking detection sensor that detects the user's walking state; an enlargement step for enlarging an augmented reality object compared to when the user is stationary if it is determined that the user is walking; and a display step for displaying the enlarged augmented reality object on a display processing device.

[0180] 10: User 11: Foot 12: Foot 100: AR glasses 101: Left eye gaze sensor 102: Right eye gaze sensor 103: Camera 104: Acceleration sensor 111: Display item 112: Display item 113: Display item 201: Forward scenery 202: Field of view display range 211: Display item 212: Display item 213: Display item 250: Central area 260: Peripheral area 300: Arrow 311: Display item 312: Display item 313: Display item 511: Display item 512: Display item 601: Arrow 602: Field of view display range 603: Arrow 604: Field of view display range 612: Display item 613: Display item 650: Central area 651: Central area 660: Peripheral area 661: Peripheral area 701: View display range 702: View display range 713: Display item 750: Central area 751: Central area 760: Peripheral area 761: Peripheral area 801: Dashed frame 802: Dashed frame 812: Display item 813: Display item 822: Display item 823: Display item 832: Display item 850: Central area 860: First peripheral area 870: Second peripheral area 1002: Solid line 1003: Dashed line 1101: Operation menu 1304: Geomagnetic sensor 1305: Gyro sensor 1306: Distance measurement sensor 1307: Operation input interface 1308: Display processing device 1320: Processor 1321: Walking state detection processing unit 1322: Head movement detection processing unit 1323: Display item display form control processing unit 1324: Operation menu processing unit 1325: RAM 1330: Storage 1331: Program 1332: Information data 1334: Application program 1335: Display item information 1336: Walking information 1337: Head movement information 1338: Input setting information 1341: Vibrator 1342: Microphone 1343: Speaker 1344: Communication device 1345: Timer 1350: Bus 1401: Smartphone screen1402: Smart watch screen 1411: Smart phone screen 1412: Smart watch screen

Claims

1. An information processing device worn by a user, comprising: a display processing device that displays an augmented reality object within a field of view display range; a walking detection sensor that detects the user's walking state; and a processor, wherein the processor determines whether the user is walking based on sensor information from the walking detection sensor, and when it determines that the user is walking, enlarges the augmented reality object and executes control to display the enlarged augmented reality object within the field of view display range.

2. An information processing device as described in claim 1, wherein the field of view display range includes a central region including a center point of the field of view display range and a peripheral region located on the outer edge of the central region, and the processor enlarges the augmented reality object displayed in the peripheral region and does not enlarge the augmented reality object displayed in the central region.

3. An information processing device according to claim 2, characterized in that the processor enlarges the display size of the augmented reality object displayed in the peripheral area to a size larger than the display size of the augmented reality object displayed in the central area.

4. An information processing device according to claim 2, further comprising a head movement detection sensor that detects head movement of a user wearing the information processing device, and when the processor determines that the user is stationary based on sensor information from the walking detection sensor and detects head movement of the user based on sensor information from the head movement detection sensor, the processor moves the position of the field of view display range to follow the head movement, and the augmented reality object displayed in the peripheral area within the field of view display range after the movement is enlarged, while the augmented reality object displayed in the central area within the field of view display range after the movement is not enlarged.

5. An information processing device according to claim 4, wherein when the processor determines that the user is walking based on the sensor information of the walking detection sensor and detects head movement of the user based on the sensor information of the head movement detection sensor, the processor does not change the display size of the augmented reality object before and after the head movement is detected.

6. An information processing device according to claim 2, wherein the processor detects the walking speed of the user based on sensor information from the walking detection sensor, and controls the enlarged display size of the augmented reality object according to the walking speed.

7. An information processing device according to claim 1, wherein the processor controls the display of an operation menu for setting and inputting the display size of the augmented reality object within the field of view display range.

8. An information processing device worn by a user, comprising: a display processing device that displays an augmented reality object within a field of view display range; a walking detection sensor that detects the user's walking state; and a processor, wherein the field of view display range includes a central region that includes the center point of the field of view display range and a peripheral region located on the outer edge of the central region, and the processor determines whether the user is walking based on sensor information from the walking detection sensor, and, if it determines that the user is walking, executes display control to move the augmented reality object displayed in the central region to the peripheral region.

9. An information processing device according to claim 8, wherein the processor executes display control to enlarge the augmented reality object displayed in the central area and move it to the peripheral area.

10. An information processing device as described in claim 9, wherein the peripheral area within the field of view display range is divided into a first peripheral area adjacent to the outside of the central area and a second peripheral area adjacent to the outside of the first peripheral area, and the processor determines that the user is walking based on sensor information from the walking detection sensor, and makes the magnification rate of the augmented reality object moved to the second peripheral area larger than the magnification rate of the augmented reality object moved to the first peripheral area.

11. An information processing device according to claim 10, wherein the processor detects the walking speed of the user based on sensor information from the walking detection sensor, and controls the display position of the augmented reality object in accordance with the walking speed.

12. An information processing device according to claim 8, wherein the processor controls the display of an operation menu for setting and inputting the display size of the augmented reality object within the field of view display range.

13. An information processing device according to claim 1, characterized in that the walking detection sensor is at least one of a camera that photographs the user's feet, an acceleration sensor, a geomagnetic sensor, and a gyro sensor, and the head movement detection sensor is at least one of the acceleration sensor, the geomagnetic sensor, and the gyro sensor.

14. An information processing method executed by an information processing device worn by a user, comprising: a walking determination step of determining whether the user is walking based on sensor information from a walking detection sensor that detects the user's walking state; an enlargement step of enlarging an augmented reality object compared to when the user is stationary when it is determined that the user is walking; and a display step of displaying the enlarged augmented reality object on a display processing device.

15. An information processing method as described in claim 14, comprising a display position moving step of moving the display position of the augmented reality object within a field of view display range of the display processing device when it is determined that the user is walking, wherein the field of view display range includes a central region including a center point of the field of view display range and a peripheral region located on the outer edge of the central region, and wherein in the display position moving step, the augmented reality object displayed in the central region is moved to the peripheral region.