Information processing apparatus and information processing method

Abstract

An information processing apparatus worn by a user is provided with: a display processing device that displays an augmented reality object within a field-of-view display range; a walking detection sensor that detects a walking state of the user; and a processor that determines whether the user is walking based on sensor information of the walking detection sensor, enlarges the augmented reality object when it is determined that the user is walking, and performs control of displaying the enlarged augmented reality object within the field of view display range.

Description

Technical Field

[0001] The present invention relates to an information processing device and an information processing method for displaying virtual objects together with objects in real space. Background Art

[0002] In recent years, augmented reality (AR) technology, which overlaps virtual objects (virtual objects) in a virtual space composed of digital information produced by CG (Computer Graphics) and the like with the real space to visually enhance the real world, has been widely used, and information processing devices that can recognize objects in the real space and also easily visually recognize virtual objects are becoming popular. As an example, a head-mounted display (HMD) worn on the head and a glasses-type digital device, namely AR glasses, which is a form of HMD, can be cited. In HMD or AR glasses, a display unit set in front of the head displays a physical object or virtual object in the real space (the so-called video perspective type), or a virtual object is displayed through a display unit and the user visually recognizes the physical object in front of the user (the so-called optical perspective type), so that it is possible to experience as if a virtual object exists in the real world. Virtual objects are generated by digital information composed of various display items such as text, icons, and dynamic images, and are visualized by displaying them in front of the user.

[0003] Here, when a user wearing an HMD or AR glasses is walking outside, it is desired that display items as virtual objects can also be easily visually identified together with the field of vision in front. On the other hand, when display items as virtual objects affect the user's entire field of vision, the user feels annoyed and has a sense of discomfort when experiencing AR technology. In response to this issue, as a method for reducing the impact of display items on the user's field of vision, Patent Document 1 describes "an information processing device comprising a display control unit that controls the presence or absence of the display items in the display unit based on a display order of the display items determined based on environmental information or user information, triggered by changes in the user's behavior."

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-101743 Summary of the invention

[0007] Problems to be solved by the invention

[0008] Although the above-mentioned Patent Document 1 shows that the influence on the user's field of vision can be reduced by controlling the presence or absence of display items as virtual objects while walking, there is a problem that the display items as virtual objects may not be displayed while walking, making it difficult for the user to always obtain the indispensable AR technology.

[0009] The present invention takes the above-mentioned problems into consideration and provides an information processing device and an information processing method for displaying display items located within a visual field display range as virtual objects together with the scenery ahead in an easily visible display format while walking.

[0010] Means for solving problems

[0011] Among the inventions disclosed in this application, the outline of representative inventions will be briefly described as follows.

[0012] That is, the present invention is an information processing device installed on a user, characterized in that it comprises: a display processing device, which displays an augmented reality object within a field of view display range; a walking detection sensor, which detects the walking state of the user; and a processor, which determines whether the user is walking based on sensor information of the walking detection sensor, and when it is determined that the user is walking, enlarges the augmented reality object and executes control to display the enlarged augmented reality object within the field of view display range.

[0013] In addition, the present invention is an information processing device worn by a user, characterized in that it comprises: a display processing device, which displays an augmented reality object within a field of view display range; a walking detection sensor, which detects the walking state of the user; and a processor, wherein the field of view display range includes a central area and a peripheral area, the central area includes a center point of the field of view display range, and the peripheral area is located at the outer edge of the central area, and the processor determines whether the user is walking based on sensor information of the walking detection sensor, and when it is determined that the user is walking, executes display control to move the augmented reality object displayed in the central area to the peripheral area.

[0014] In addition, the present invention is an information processing method performed by an information processing device worn by a user, characterized in that the information processing method includes the following steps: a walking judgment step, judging whether the user is walking based on sensor information of a walking detection sensor that detects the walking state of the user; an enlargement step, when it is judged that the user is walking, enlarging the augmented reality object compared to when it is stationary; and a display step, causing a display processing device to display the enlarged augmented reality object.

[0015] Effects of the Invention

[0016] By using the technology of the present invention, an information processing device and an information processing method are provided that can realize displaying objects located within the visual field display range as virtual objects together with the scenery ahead in an easily observable display manner. In addition, through the description of the following embodiments, the above-mentioned problems, structures and effects become clear. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is an example of a diagram schematically illustrating the appearance of an example of an embodiment of the information processing device according to the present embodiment.

[0018] Figure 2 It is explained in Figure 1 An example of a diagram of a front scenery screen and a field of view display range when a user is stationary in the information processing device according to the embodiment described in .

[0019] Figure 3 It is explained in Figure 1 In the information processing device of the embodiment described in , an example of a diagram showing a situation where all displayed items are enlarged and displayed while the user is walking.

[0020] Figure 4 It is explained in Figure 1 FIG. 1 is a diagram showing an example of a case where displayed items in the surrounding area are greatly enlarged and displayed while the user is walking in the information processing device of the embodiment described in FIG.

[0021] Figure 5 It is explained in Figure 1 FIG. 1 is an example of a case where, in the information processing device of the embodiment described in , displayed items in the peripheral area are greatly enlarged and displayed, and displayed items in the central area are slightly enlarged when the user is walking.

[0022] Fig. 6A It is explained in Figure 1 FIG. 1 is a diagram showing an example of a situation in which, when walking diagonally to the left, display items that have moved to the peripheral portion are greatly enlarged and displayed in the information processing device of the embodiment described in FIG.

[0023] Figure 6B It is explained in Figure 1 FIG. 1 is a diagram showing an example of a situation in which, when walking diagonally to the right, display items that have moved to the peripheral portion are greatly enlarged and displayed in the information processing device of the embodiment described in FIG.

[0024] Fig. 7A It is explained in Figure 1 FIG. 1 is an example of a case where the user, in the information processing device according to the embodiment described in FIG. 1 , turns his head to the left while walking forward.

[0025] Figure 7B It is explained in Figure 1FIG. 1 is another example of a case where the user turns his head to the left while walking forward in the information processing device of the embodiment described in FIG.

[0026] Figure 7C It is explained in Figure 1 FIG. 1 is an example of a situation in which, in the information processing device of the embodiment described in FIG. 1 , the head is turned toward the upper left while walking diagonally to the right.

[0027] Fig. 8A It is explained in Figure 1 An example of a diagram showing the relationship between a front scenery screen, a field of view display range, a central area, a first peripheral area, and a second peripheral area when the user is stationary in the information processing device of the embodiment described in .

[0028] Figure 8B It is explained in Figure 1 FIG. 4 is an example of a case where, in the information processing device of the embodiment described in FIG. 4 , displayed items in the central area are moved to the first peripheral area when the user is walking.

[0029] Figure 8C It is explained in Figure 1 FIG. 4 is a diagram showing an example of a case where, in the information processing device of the embodiment described in FIG. 4 , displayed items in the central area are moved to the first peripheral area and enlarged while the user is walking.

[0030] Fig.8D It is explained in Figure 1 FIG. 4 is an example of a case where, in the information processing device of the embodiment described in FIG. 4 , displayed items in the central area are moved to the second peripheral area and enlarged while the user is walking.

[0031] Fig.9A It means in Figure 1 An example of a graph showing the relationship between walking speed and display size of displayed items in the information processing device according to the embodiment described in .

[0032] Fig. 9B It means in Figure 1 An example of a graph showing the relationship between walking speed and the destination of displayed articles in the central area in the information processing device of the embodiment described in .

[0033] Fig.10 It is explained in Figure 1 FIG. 1 is a diagram showing an example of a case where an operation menu is displayed in a stationary state in the information processing device according to the embodiment described in FIG.

[0034] Fig.11A This is an example of a flowchart (first half) for explaining the operation of the information processing device according to the present embodiment.

[0035] Fig. 11BThis is an example of a flowchart (latter half) for explaining the operation of the information processing device according to the present embodiment.

[0036] Fig.12 This is a block diagram showing a configuration example of AR glasses as an example of the information processing device according to the present embodiment.

[0037] Fig.13A It is explained in Figure 1 In the information processing device of the embodiment described in , when the user is stationary, the screen of a smartphone or a smart watch is displayed as an example of a diagram showing a situation in which an item is displayed.

[0038] Fig. 13B It is explained in Figure 1 In the information processing device of the embodiment described in , an example is given in which a screen of a smartphone or a smart watch is displayed as a diagram showing a situation where an item is displayed while the user is walking. DETAILED DESCRIPTION

[0039] The present invention can provide an information processing device and method that can help to improve the visual recognizability of augmented reality objects while walking and the visual recognizability of the scenery ahead, so it can be expected to improve the technology of labor-intensive industries, and thus it can be expected to contribute to 8.2 of the Sustainable Development Goals (SDGs: Sustainable Development Goals) advocated by the United Nations (improving economic productivity through diversification, technological advancement and innovation, focusing on industries that further increase the value of goods and services and labor-intensive industries).

[0040] In addition, by improving the visibility of displayed items, it is expected to support people with poor eyesight, and thus it is expected to contribute to Goal 3 of the Sustainable Development Goals (SDGs: Sustainable Development Goals) advocated by the United Nations, namely "promoting health and well-being for all."

[0041] <Implementation Method 1>

[0042] Hereinafter, an example of an embodiment of the present invention will be described using the accompanying drawings. As a specific example of the information processing device of the present embodiment, a glasses-type AR glasses is described as an example, and the glasses-type AR glasses are in the form of an HMD worn on the user's head for displaying and visually identifying information in real space and virtual space. Figure 1 FIG. 1 is an example of a diagram schematically illustrating the appearance of an example of an embodiment of an information processing device according to the present embodiment. Figure 2 , Figure 3 The following is an example of a diagram for explaining a display screen of a front scenery or a field of view display range of the information processing device according to the present embodiment.

[0043] exist Figure 1 In the figure, the user 10 wears AR glasses 100 on his head. The AR glasses 100 include a left-eye sight line sensor 101 for detecting the sight lines of the left and right eyes of the user 10, a right-eye sight line sensor 102, a camera 103 for photographing the outside world, and an acceleration sensor 104 for detecting the change in speed per unit time, i.e., acceleration.

[0044] For the AR glasses 100, in the optical perspective type of the main body, the user 10 directly uses the eye vision to recognize the physical objects in the field of vision around the front. On the other hand, in the video perspective type, the physical objects in the field of vision around the front are photographed by the camera 103, and the image of the photographed physical object is displayed on the display.

[0045] In addition, the AR glasses 100 generate various display items such as text, icons, and images as augmented reality objects in addition to physical objects, and display them within the visual field display range of the user 10. Figure 1 In the figure, display items 111, 112, and 113 represented by text characters, icons, and the like such as "Train delays in the Tokyo area. Rain tonight", "**Teacher", and "Turn right 8m to the station" are virtually configured in a real space composed of physical buildings, roads, and passers-by.

[0046] The visual field display range is a range in which the display items 111 , 112 , and 113 of real objects and virtual objects are displayed in an overlapping manner via the AR glasses 100 .

[0047] The user's line of sight can also be detected based on the movement of the user's eyeballs, and the user's field of view can be used as the field of view display range. In addition, the area behind the field of view when the user moves the eyeballs up and down and left and right can also be used as the field of view display range in addition to the field of view when the user observes the front distance.

[0048] The AR glasses 100 use the camera 103 provided in the AR glasses 100 to photograph the front of the user 10 and the left and right feet 11 and 12, and capture the user's walking state from under the feet of the photographed feet. In addition to the walking movements, walking direction, and walking speed, the AR glasses 100 also obtain the user's walking information such as the relative position relationship between the feet or body of the user 10 and the AR glasses 100.

[0049] In addition, the AR glasses 100 measure acceleration in three-dimensional directions through the acceleration sensor 104 provided in the AR glasses 100 , and analyze and detect motion information such as the position, direction, tilt, and vibration of the head of the user wearing the AR glasses 100 .

[0050] Moreover, the AR glasses 100 can capture the user's walking state by analyzing and detecting the movement information of the user's head, and obtain the user's walking information such as walking movements, walking direction, and walking speed.

[0051] Therefore, the AR glasses 100 are equipped with a walking detection sensor, which detects walking information such as walking movements, walking direction, and walking speed of the user wearing the AR glasses 100 through a camera 103 that captures the user's feet and an acceleration sensor 104 that measures the acceleration of the user's head in three dimensions.

[0052] The AR glasses 100 also include a head motion detection sensor that detects the movement of the head of the user wearing the AR glasses 100 using an acceleration sensor 104 that measures the acceleration of the user's head in three-dimensional directions.

[0053] In addition, in the AR glasses 100 , the line of sight of the user 10 can be detected by the left-eye line of sight sensor 101 and the right-eye line of sight sensor 102 .

[0054] Figure 2 The figure schematically shows that a user 10 wearing AR glasses 100 is in a Figure 1 In the state shown, the user's front scenery 201, the field of view display range 202 of the AR glasses 100, and the user 10 wearing the AR glasses 100 are determined by the camera 103 and the acceleration sensor 104 to be stationary rather than walking.

[0055] exist Figure 2 In the embodiment, since the user 10 stops walking, even if the displayed items 111, 112, 113 have a smaller display size, the user 10 can visually recognize the displayed items 211, 212, 213 within the visual field display range effortlessly without feeling any reading difficulty.

[0056] Furthermore, by moving the head up, down, left, and right to move the visual field display range 202 , the entire view of the scenery 201 ahead can be grasped.

[0057] Figure 3 Indicates that user 10 Figure 2 The display example shown is when the user starts walking from a stationary state.

[0058] When walking in the front direction indicated by arrow 300, the scenery in field of view display range 202 in the walking direction is mainly visually recognized, and the eyes are also allocated to display items 211, 212, and 213. The smaller the display size of the display items in field of view display range 202, the more difficult it is to view and read, and the visual recognizability of the display items in the field of view display range deteriorates.

[0059] Therefore, when the walking detection sensor that analyzes the outputs of the camera 103 and the acceleration sensor 104 determines that the user 10 is walking, Figure 3As shown, the displayed items 211, 212, 213 as virtual objects are greatly enlarged and displayed as displayed items 311, 312, 313. Thus, when the user walks, the user can easily view and read the greatly enlarged displayed displayed items, which can greatly improve the visual recognition of the displayed items as virtual objects.

[0060] In addition, the displayed item can be directly enlarged, or the characters or icons inside the item can be enlarged. When the area blocking the field of view becomes too large when the displayed item is enlarged, the number of characters can be limited to reduce the amount of displayed information, or the character spacing can be reduced.

[0061] <Implementation Method 2>

[0062] use Figure 4 A second embodiment is described. Figure 2 , Figure 3 The same reference numerals are used to designate the same features, and detailed description thereof will be omitted.

[0063] When the user is walking, the user avoids focusing on the peripheral part and focuses on the central area 250 of the walking field of view display range 202 in consideration of the safety of the front and the foot in the walking direction. Therefore, within the field of view display range 202, the user's visual recognition of the central area 250 is relatively good, but the displayed items in the peripheral area 260 are more difficult to visually recognize than the displayed items in the central area 250.

[0064] Therefore, when the user is walking, the central area 250 of the visual field display range 202 Figure 2 The displayed items 212 and 213 remain unchanged, and the surrounding area 260 within the visual field display range 202 Figure 2 Display items 211 and Figure 3 Similarly, the display is enlarged and displayed as displayed item 311. Thus, the visual recognizability of displayed items in peripheral area 260, which is difficult for the user's line of sight to reach compared to central area 250 of visual field display range 202, can be improved as the user walks, and displayed items in peripheral area 260 can be visually recognized as much as displayed items in central area 250.

[0065] In addition, the central area 250 and the peripheral area 260 are a predetermined range area including the center point of the visual field display range 202 as the central area 250, and an area existing around it as the peripheral area 260. The above-mentioned "predetermined range" can be predetermined, and the width of the predetermined range can be changed according to the walking state. For example, when the walking speed is relatively fast, the "predetermined range" can be relatively narrowed, and when the walking speed is relatively slow, the "predetermined range" can be relatively widened.

[0066] exist Figure 3 In the illustrated embodiment, since the displayed items in the central area 250 are enlarged, the large displayed items 312 and 313 largely block the front view when walking, which may make the user feel difficult to see. Figure 4 In the embodiment, the size of the display items in the central area 250 remains unchanged to minimize the impact on the front scenery, and the display items in the peripheral area 260 are enlarged to improve the visual recognition effect.

[0067] <Implementation Method 3>

[0068] use Figure 5 A third embodiment is described. Figure 2 , Figure 3 , Figure 4 The same reference numerals are used to designate the same features, and detailed description thereof will be omitted.

[0069] and Figure 3 , Figure 4 The difference of the embodiment is that the central area 250 in the field of view display range 202 Figure 2 The display size of displayed items 212 and 213. Figure 3 It is enlarged as shown in the display items 312 and 313. Figure 4 The items 212 and 213 are displayed without enlargement, but Figure 5 In the middle are display items 512 and 513 whose magnification is reduced compared to the display item 311 in the surrounding area.

[0070] When the user is walking, the visibility of displayed items is greater in the peripheral area 260 and smaller in the central area 250, but dynamic visual acuity tends to be lower than static visual acuity. By effectively utilizing this feature, when the user is walking, it is possible to suppress the obstruction of scenery in the central area 250 within the visual field display range 202, and to ensure visibility by enlarging the displayed items in the central area 250.

[0071] Figure 5 The size of the displayed object is shown to be switched in two levels, the central area and the peripheral area, but it is also possible to switch in multiple levels from the center to the periphery within the field of view display range, or to smoothly control the display size. The multi-level and smooth display size control can further enhance the effect of suppressing scenery occlusion and ensuring visual recognition.

[0072] <Implementation Method 4>

[0073] Compared to the description of walking forward Figure 4 The implementation method uses Fig. 6A and Figure 6BThe following describes the display item display operation of the information processing device when walking diagonally to the left or right after starting from a stationary state or walking forward.

[0074] With respect to the front scenery 201 when the user is stationary, Fig. 6A FIG. 6 shows the visual field display range 602 of the AR glasses 100 and the display status of displayed items when walking diagonally to the left. Figure 6B The figure shows the visual field display range 604 of the AR glasses 100 and the display status of displayed items when walking diagonally to the right. Fig. 6A and Figure 6B In, it is marked Figure 4 Parts with the same reference numerals as shown have the same Figure 4 The following are the same actions as those already described in , so detailed description of some of them will be omitted.

[0075] exist Fig. 6A In the example, when walking with the head facing diagonally to the left as shown by arrow 601, the visual field display range of the AR glasses 100 moves to the left by an amount corresponding to the direction of walking diagonally to the left, as shown in the visual field display range 602. Along with this, the positions of the central area 650 and the peripheral area 660 in the visual field display range 602 also change. Therefore, when the user is stationary or walking forward, the visual field display range 202 (for example, Figure 5 ) within the central area 250 Figure 2 or Figure 4 The displayed item 213 is displaced to a peripheral area 660 within the field of view display range 602 which is difficult for the user to reach.

[0076] Therefore, when walking diagonally to the left, if the displayed item 213 displaced to the peripheral area 660 is directly displayed with a relatively small display size, it will be difficult to visually recognize.

[0077] Therefore, in Embodiment 4, displayed item 213 located in peripheral area 660 when walking diagonally to the left is greatly enlarged to display item 613. That is, when walking diagonally to the left, displayed items moved to peripheral area 660 are greatly enlarged and displayed, thereby improving the visibility of displayed items moved to peripheral area 660.

[0078] Since the displayed item 212 is contained in the central area 650 within the field of view display range 602 , the displayed size of the displayed item 212 is not changed.

[0079] Figure 6BIn the case of walking diagonally to the right, when walking diagonally to the right as indicated by arrow 603, the visual field display range of the AR glasses 100 moves to the right by an amount corresponding to the direction of walking diagonally to the right, as shown in the visual field display range 604. Along with this, the positions of the central area 651 and the peripheral area 661 in the visual field display range 604 also change. Therefore, the display item 212 located in the peripheral area 661 when walking diagonally to the right is greatly enlarged and displayed as the display item 612. As a result, when walking diagonally to the right, the visual recognizability of the display item shifted to the peripheral area 661 can be improved.

[0080] On the other hand, since the displayed item 213 is included in the central area 651 within the field of view display range 604 , the display size of the displayed item 213 does not change.

[0081] In addition, Fig. 6A , Figure 6B In addition to when standing still or walking forward, the display item 211 located in the peripheral area 660, 661 of the field of view display range 602, 604 is greatly enlarged and displayed as the display item 311 when walking diagonally to the left, thereby improving the visual recognizability of the display item.

[0082] As mentioned above, if you use Fig. 6A and Figure 6B As described, when walking forward or diagonally to the left or right, by greatly enlarging the displayed objects in the peripheral area within the field of view display range, it is possible to avoid the line of sight being concentrated on the peripheral part of the field of view, and to easily visually identify the displayed objects located in the peripheral part of the field of view display range as virtual objects in an easily observable form.

[0083] <Implementation method 5>

[0084] Next, refer to Fig. 7A , Figure 7B and Figure 7C The following describes the display operation within the visual field display range of an information processing device when a user moves his head leftward, rightward, upward, or downward while walking. Fig. 7A , Figure 7B , Figure 7C In the Figure 4 , Figure 6B Parts with the same reference numerals as in Figure 4 , Figure 7B The following are the same actions as those already described in , so detailed description of some of them will be omitted.

[0085] Fig. 7A is an example of user 10 walking forward and turning his head to the left. Figure 4The visual field display range 202 is moved to the left side of the visual field display range 701. Therefore, the displayed items 713 in the peripheral area 760 of the visual field display range 701 are displayed at a higher resolution than those in the peripheral area 760 of the visual field display range 701. Figure 4 The display item 213 located in the central area 250 in the visual field display range 202 is displayed in a large display size.

[0086] In addition, it is shown that the item 212 is located at Figure 4 The central area 250 of the visual field display range 202, but Fig. 7A The image also stays in the central area 750 in the field of view display range 701, so the display size does not change.

[0087] Fig. 7A This is an example in which the user 10 stops and simply turns his head to the left. However, when the direction of the user's head changes while walking, other display controls may be performed. Figure 7B is the example.

[0088] When the user 10 moves his head by turning his head left, right, up, down, etc. while walking, the user 10 first turns his head, for example, to the left because of interest, and therefore it can be said that the user's awareness of viewing or reading displayed items in the peripheral area 760 on the right side opposite to the moving direction of the central area 750 in the visual field display range 701 is very low. Therefore, it is considered that it is not necessary to enlarge and display the displayed items in the peripheral area 760 that are displaced to the visual field display range 701 due to the user turning his head.

[0089] In addition, when the display state of displayed items, such as the display size and display position, changes according to the movement of the head, the user will be attracted by the change, and the consciousness will not be focused on the direction of the head rotation, resulting in a situation where the consciousness is scattered. In addition, when the head moves, if the scenery in the field of vision does not change, but the display state of the displayed items changes, the picture in the field of vision display range becomes noisy, which has an adverse effect on the user's visual recognition.

[0090] Therefore, in Figure 7B In the illustrated embodiment, when the user maintains the walking direction even when the acceleration sensor 104 as the head movement detection sensor detects the movement of the user's head, the display state of the central area 250 and the displayed items before the head movement occurs (i.e., the same as the state of the central area 250 and the displayed items before the head movement occurs) is maintained. Figure 4 same central area and display state) to suppress changes in the display state of displayed items.

[0091] Relative to the head facing the front direction of walking Figure 4 The visual field display range 202 of the embodiment of the present invention moves to the visual field display range 701 when the user 10 turns his head to the left relative to the front direction of walking. Fig. 7AIn the display, item 713 is displayed larger, but in Figure 7B China-Israel Figure 4 The same display size displays the display item 213 .

[0092] According to the present embodiment, since the display size is not changed while the walking direction is maintained, a stable display can be provided without drawing unnecessary attention of the user due to the change in display size.

[0093] Figure 7C An example is shown in which the user 10 turns his head to the upper left when walking diagonally to the right.

[0094] like Figure 6B As shown in FIG. 1 , when walking diagonally to the right, if the user 10 turns his head to the upper left, the visual field display range moves from the visual field display range 604 to the upper center of the front scenery 201 and becomes Figure 7C The field of view shown is 702. Fig. 7A Similarly, no items are displayed in the central area 751 of the visual field display range 702, and the displayed items 311, 612, and 213 located in the peripheral area 761 of the visual field display range 702 are displayed in an enlarged manner.

[0095] However, in Figure 7C In order to suppress the change of the display state caused by the head movement, the display state for indicating the situation of pointing the head diagonally to the right while walking is maintained. Figure 6B The display size only makes the field of view display range from Figure 6B The visual range 604 moves to Figure 7C Therefore, the displayed items 311 and 612 in the peripheral area 661 within the visual field display range 604 when the head is facing the walking direction are enlarged and displayed. Figure 6B The displayed item 213 in the central area 651 of the field of view display range 604 is not displayed in an enlarged manner.

[0096] That is, when only the head is moved while the walking direction is maintained, the central area and the peripheral area, which are the criteria for determining the display size of the displayed items, are determined. Fig. 7A The visual field display range 701 that is linked to the movement of the head is set as the actual visual field display range, but Figure 7B , Figure 7C The front view display range is different when the head is facing the walking direction. Figure 4 and Figure 5 , Fig. 6A , Figure 6B In both cases, the head is facing the walking direction, so the actual visual field display range is consistent with the front visual field display range.

[0097] The user may switch and set whether the display size determination criterion is the actual field of view display range or the front field of view display range.

[0098] Automatic switching can also be performed as follows: when the direction of the body and feet is consistent with the walking direction, the front field of view display range is used as the basis; when it is judged to be inconsistent, such as "moving sideways like a crab" (moving the body sideways while facing the front), the actual field of view display range is used as the basis.

[0099] In addition, when the head is offset from the walking direction or the front of the body by a predetermined angle of, for example, approximately 10 degrees or more, the user's intention to observe other than the walking direction may be respected, and after a predetermined time of approximately 0.5 seconds, the actual visual field display range reference may be prioritized. Furthermore, the predetermined time may be controlled to be shorter as the predetermined angle increases.

[0100] <Implementation Method 6>

[0101] use Fig. 8A , Figure 8B , Figure 8C , Fig.8D The following describes the display item display operation of the information processing device when the peripheral area within the visual field display range is divided into a plurality of peripheral areas, and when walking, the display items in the central area of ​​the visual field display range are moved to the divided peripheral areas and enlarged for display. Figure 3 and Figure 4 The parts shown with the same reference numerals have the same actions as those already described, and thus their detailed description is omitted.

[0102] Fig. 8A In the figure, as an example of the division of the peripheral areas, the central area 850 is the area surrounded by the dotted frame 801 which is closest to the center, the first peripheral area 860 is the area outside the dotted frame 801 and inside the dotted frame 802, and the second peripheral area 870 is the area outside the dotted frame 802 and inside the field of view display range 202.

[0103] like Fig. 8A As shown, when the user is in a stationary state, a display item 212 is displayed in the central area 850 , a display item 813 is displayed in the first peripheral area 860 , and a display item 211 is displayed in the second peripheral area 870 .

[0104] When the user 10 walks from a stationary state to the front direction, it is preferable to remove or move the displayed items in the central area 850 so that obstacles in the walking direction can be easily visually recognized. Figure 8B The Chinese Ambassador is located in Fig. 8AThe display item 212 in the central area 850 is moved to the first peripheral area 860 and displayed as the display item 812, thereby ensuring the field of vision ahead when walking.

[0105] The display items 813 of the first peripheral area 860 are Fig. 8A Likewise shown without magnification, at Fig. 8A The display item 211 in the second peripheral area 870 is enlarged and displayed as the display item 511. In this way, while ensuring the visibility of the scenery in the walking direction, the size of the display items in the periphery is displayed larger to improve the visibility of the display items.

[0106] Figure 8C is relative to Figure 8B This is an example of enlarging and displaying displayed items in the first peripheral area 860 to ensure the field of vision ahead while walking and to improve the visibility of displayed items.

[0107] Make it located Fig. 8A The display item 212 in the central area 850 moves toward the first peripheral area 860 and is enlarged to display the item 822 . Fig. 8A The display item 813 displayed in the first peripheral area 860 is enlarged to the display item 823, Fig. 8A The display item 211 in the second peripheral area 870 is enlarged to the display item 511. At this time, by making the display items 822 and 823 larger than the first peripheral area 860 and the display item 511 larger than the second peripheral area 870, the improvement of the visibility of the scenery in the walking direction and the improvement of the visibility of the surrounding large display items are taken into account.

[0108] Fig.8D is to make it located Fig. 8A An example in which the displayed item 212 in the central area 850 is moved to the second peripheral area 870 and further enlarged to be the displayed item 832. The displayed item 832 is an example in which the displayed item is displayed in the same size as the displayed item 511 displayed in the second peripheral area 870. In this way, the size of the displayed item can be gradually changed according to the peripheral area of ​​the movement destination, and the visual recognizability of the scenery in the walking direction and the visual recognizability of the larger displayed items in the periphery can be improved and taken into account.

[0109] <Implementation Method 7>

[0110] use Fig.9A , Fig. 9B A description will be given of a displayed item display operation of an information processing device when the enlarged display size of a displayed item within a visual field display range is controlled according to the walking speed of a user when walking. Fig.9A This is an example of a graph showing the relationship between walking speed and display size of displayed items.

[0111] For the user 10 who wears the AR glasses 100 on his head and walks, the faster the walking speed, the harder it is to see the displayed items as virtual objects in the visual field display range, which tends to reduce the visual recognition of the displayed items. Therefore, the faster the walking speed of the user 10 when walking, the more the displayed items in the visual field display range are magnified, thereby improving the visual recognition of the displayed items.

[0112] like Fig.9A As shown by the solid line 1002, the enlarged display size of the displayed item can be gradually increased according to the walking speed of the user 10 when walking. In addition, as another example, the enlarged display size of the displayed item can be switched in multiple stages according to the walking speed of the user 10 when walking, as shown by the dotted line 1003 in FIG. 9. When the enlarged display size of the displayed item is switched in multiple stages, even if the walking speed changes slightly within a certain level, the display size of the displayed item does not change, and the user 10 can visually recognize the displayed item stably. In the multi-stage switching, when the display size is switched, the display is likely to become unstable, so as shown by the dotted line 1003, it is sufficient to have a hysteresis to prevent unstable display size switching.

[0113] Fig. 9B This is an example of a graph showing the relationship between walking speed and the destination of displayed items in the central area.

[0114] For example, in the case of ordinary walking where the user's walking speed is not too fast, Figure 8C As shown, the displayed items in the central area 850 are moved to the first peripheral area 860. When the user walks at a fast walking speed, as shown in FIG. Fig.8D As shown, the displayed items in the central area 850 are moved to the second peripheral area 870. In addition, the faster the walking speed, the closer the displayed items in the central area 850 are to the user's field of vision, and there is a tendency that the displayed items are likely to obstruct the visual recognition of the scenery ahead.

[0115] Therefore, if Fig. 9B As shown, the faster the walking speed, the more the display items in the central area 850 are moved to the first peripheral area 860 and the second peripheral area 870 on the outside, thereby reducing the obstruction of the display items to the visual recognition of the scenery ahead caused by the walking speed of the user 10. Fig.9A Likewise, Fig. 9B Has hysteresis to prevent erratic movement of displayed items.

[0116] In addition, the display position of the displayed item may be switched in multiple stages according to the walking speed of the user 10, that is, the movement destination may be switched more finely than the movement among the central area 850, the first peripheral area 860, and the second peripheral area 870. When the display position of the displayed item is switched in multiple stages, even if the walking speed slightly changes within a certain stage, the display position of the displayed item does not change, and the user 10 can stably visually recognize the displayed item.

[0117] In addition, when the user 10 switches from walking to running or cycling, etc., it is sometimes desirable to increase the visual field area in the front of the moving direction. Therefore, the central area 850 of the visual field display range can be expanded according to the walking speed or the moving speed. The size of the central area 850 is also set to Fig.9A The walking speed has the same control on the display size.

[0118] For example, when the user is walking at a speed of 4 km / h, the central area 850 can be expanded from the shoulder width of the user 10 to the front and opened to the left and right by about 10 degrees to set as a trapezoidal area about 3 meters in front, when the user is traveling at a speed of 5 km / h, it can be expanded to about 5 meters in front, and when the bicycle is traveling at a speed of 10 km / h, it can be expanded to 10 meters in front, and the distance in front can be extended according to the moving speed. The opening angle in front can also be expanded according to the traveling speed.

[0119] <Implementation Method 8>

[0120] use Fig.10 The input operation of the information processing device when the user 10 inputs the display state such as the display size and display position of a displayed object while walking will be described. Fig.10 This is an example of a case where the operation menu 1101 is displayed within the visual field display range when the user is stationary. Fig.10 In the Figure 2 Parts with the same reference numerals as in Figure 2 The following are the same actions as those already described in , so detailed description of some of them will be omitted.

[0121] Some users 10 have poor eyesight and narrow vision. For users with poor eyesight, it is sometimes difficult to visually recognize the scenery and displayed objects in front of them when walking. Fig.10 As shown, an operation menu 1101 for inputting settings of display status such as display size and display position of a display item is displayed within the field of view display range 202. When the user 10 inputs settings of the desired display size and display position of a display item from the operation menu 1101 while walking, the display size and display position of the display item are controlled according to the setting input information while walking.

[0122] By operating the operation menu 1101 by hand or sight, input settings such as the display size or display position of the displayed item that is optimal for the user 10 can be performed. The input setting operation based on sight is performed by detecting the sight destination of the user 10 by the left eye sight sensor 101 and the right eye sight sensor 102, and the sight destination indicates the position of the input setting button on the operation menu 1101. Therefore, the left eye sight sensor 101 and the right eye sight sensor 102 constitute a sight input device for selecting the operation menu 1101.

[0123] Furthermore, the input setting operation can be performed by hand by pointing the input setting button position on the operation menu 1101. For example, for the user 10 with poor eyesight, when the display size is set by inputting the operation menu 1101 so that the displayed object is enlarged to a larger size than the standard size for display, the visual recognition of the displayed object can be improved while walking.

[0124] In addition, as the display position of the display item moves from the central area of ​​the field of view display range 202 to the peripheral area, the optimal display position that matches the vision of the user 10 can also be set and input through the operation menu 1101, which can improve the visual recognizability of the front scenery in the central area opened by the movement of the display item while walking and the display item itself.

[0125] Furthermore, while walking, the display of the operation menu 1101 for input setting by sight may be eliminated, or the input setting operation based on sight in the operation menu 1101 may be stopped, thereby limiting the input setting operation based on sight. Thus, the sight of the user 10 is no longer directed toward the operation menu 1101, and the sight can be focused on the scenery ahead and the displayed objects as virtual objects within the visual field display range 202.

[0126] Furthermore, it also includes an operation menu other than input settings. In order not to distract attention from the walking destination when performing visual input while walking, the display of the operation menu 1101 can be suppressed at a walking speed above a predetermined speed, such as 1 km per hour.

[0127] <Processing Flow of Each Embodiment>

[0128] use Fig.11A , Fig. 11B The flowchart shown in the figure explains the display operation of the display items at the start of walking and the stop of walking.

[0129] like Fig.11AAs shown, the user 10 wants to start walking after wearing the AR glasses 100. Before the user starts walking, that is, in a stationary state (S1201), an item is displayed in the field of view display range (S1202: Yes). In this state, the processor 1320 determines whether the user is walking based on the sensor information of the walking detection sensor (S1203: walking determination step).

[0130] If the processor 1320 determines that the user 10 is walking (S1203: Yes), the AR glasses 100 controls the display size, display position, or a combination thereof according to the position of the display item displayed in the visual field display range (S1204). For example, the display items in the peripheral area of ​​the visual field display range are enlarged and displayed (equivalent to the enlarging step), or the display items in the central area of ​​the visual field display range are moved to the peripheral area (equivalent to the display position moving step) and displayed (equivalent to the display step).

[0131] Afterwards, when the walking detection sensor detects that the walking stops (S1205: Yes), Fig. 11B As shown, in the AR glasses 100, the display size and display position of the controlled display item are temporarily maintained as is (S1206).

[0132] Here, when the user 10 performs an input operation to indicate that the display state consisting of the display size, display position or a combination thereof of the maintained display item is transferred to the display state of the display item when it is stationary (S1207: Yes), the display state of the display item when it is stationary is transferred to the display state (S1209).

[0133] Even if no input operation indicating transfer recovery is performed (S1207: No), when a predetermined control maintenance period has passed after walking cessation is detected (S1208: Yes), in the AR glasses 100, the display state of the displayed item is transferred from the maintained display state to the display state of the displayed item before control, that is, the display state of the displayed item when stationary (S1209).

[0134] exist Fig. 11B For ease of explanation, step S1208 is illustrated as being executed following step S1207, but steps S1207 and S1208 may be in the opposite order or may be processed in parallel. When a positive judgment is made in any step, step S1209 is entered.

[0135] After the transition is restored to the display state of the displayed items when the user is stationary, if the displayed items continue to be displayed when the user is walking (S1210: No), the process returns to the display confirmation of the displayed items in step S1202. When the display of the displayed items is not to continue (S1210: Yes), the process ends.

[0136] Through the above actions, when walking stops, the display state of the display items during walking is temporarily maintained as it is, and after a predetermined control maintenance period or through user input operation, the display state of the display items during walking is transferred from the maintained display state to the original display state of the display items during stationary state.

[0137] This can reduce the problem that when the user stops walking, the display size, display position or combination of the display items changes suddenly and instantly, causing the user who is viewing or reading the display items to feel discomfort or abruptness.

[0138] In addition, after a user operation is performed or after a predetermined control maintenance period, the display state of the display item can be gradually transferred at a gentle pace and restored to the original display state of the display item when it was stationary. In this case, the user's sense of disharmony and abruptness can be further reduced.

[0139] In addition, even when walking, avoiding puddles, or walking on uneven roads, the display state of the display items displayed during the previous average walking can be maintained as is. In this case, frequent changes in the display size or display position of the display items can be suppressed, thereby reducing or eliminating the situation where the display items are difficult to view.

[0140] use Fig.12 A configuration example of an information processing device according to this embodiment will be described. Fig.12 1 is a block diagram showing a configuration example of AR glasses as one form of HMD as an example of the information processing device of this embodiment. Fig.12 In the Figure 1 Parts with the same reference numerals as shown in FIG. Figure 1 The operations are the same as those already described in etc., so detailed description of some of them is omitted.

[0141] exist Fig.12 In the AR glasses 100, a left-eye sight sensor 101, a right-eye sight sensor 102, a camera 103, an acceleration sensor 104, a geomagnetic sensor 1304, a gyroscope sensor 1305, a distance sensor 1306, an operation input interface 1307, a display processing device 1308, a processor 1320, a storage device 1330 for storing a program 1331 and information data 1332, a vibrator 1341, a microphone 1342, a speaker 1343, a communication device 1344, a timer 1345 and a RAM 1325 are appropriately used, and each component is connected to each other via a bus 1350.

[0142] The camera 103 captures a field of view in front of the camera and around the camera, and obtains a camera captured image by converting light incident from a lens into an electrical signal via an imaging element.

[0143] The camera 103 captures the scenery under the feet of the user 10 and in front of the user 10. Therefore, the walking state of the user 10 can be analyzed based on the captured state of the feet of the user 10, and is used as a walking detection sensor to detect the walking motion, walking direction, and walking speed of the user. In addition, by capturing the feet of the user with the camera, not only the walking state of the user can be analyzed, but also the situation of riding a bicycle or a scooter while moving the feet can be analyzed, and display state control such as enlarging a displayed object can be performed.

[0144] On the other hand, the situation where the user 10 is riding a train or a bus and the feet are not moving can be analyzed. Even if the acceleration sensor 104 detects the movement of the user 10, it is determined that the user is stationary and the display state control of the displayed items is not performed.

[0145] If not only the area under your feet but also your hands are photographed, it is possible to determine whether you are driving a car, and it is also possible to hide objects from display while driving.

[0146] In addition, in the video see-through AR glasses, a physical object in the real space such as the scenery in front of the eyes is photographed by the camera 103, and the image of the photographed physical object is displayed on the display processing device 1308.

[0147] The left eye sight sensor 101 and the right eye sight sensor 102 detect the movement and direction of the left eye and the right eye, respectively, and can capture the sight destination of the user 10. In addition, regarding the process of detecting the movement of the eyeball, it is sufficient to use the well-known technology commonly used as the eye tracking process. For example, in the method using corneal reflection, the following technology is known: an infrared LED (Light Emitting Diode) is irradiated to the face and photographed with an infrared camera, and the position of the reflected light formed by the infrared LED irradiation on the cornea (corneal reflection) is used as a reference point, and the movement of the eyeball and the sight are detected based on the position of the pupil relative to the position of the corneal reflection.

[0148] The acceleration sensor 104 is a sensor that detects the change in speed per unit time, i.e., acceleration, and can capture motion, vibration, impact, etc. The AR glasses 100 are worn on the head of the user 10, and can analyze and obtain motion information such as rotation, tilt, and vibration of the head of the user 10 based on the acceleration data detected by the acceleration sensor 104 provided in the AR glasses 100.

[0149] The acceleration sensor 104 is used as a head motion detection sensor for detecting the movement of the user's head, and can detect the state of the user 10 turning his head left, right, up, down, etc. In addition, the relative relationship between the head and the body (the direction of the feet) cannot be directly detected only by the acceleration sensor worn on the head, so the acceleration sensor can also be worn on the body, or the body and feet can be photographed by a camera built into the AR glasses 100 to obtain the above relative relationship.

[0150] In addition, when the user walks in the same direction, the sensor information of the head movement detection sensor can be analyzed to detect the movement of the user's head, and the walking direction and walking speed of the user wearing the AR glasses 100 can also be detected, so it can also be used as a walking detection sensor.

[0151] The AR glasses 100 alone are limited to the detection of the average walking direction and walking speed, but may also be at this level.

[0152] In addition, the gyro sensor 1305 is a sensor for detecting the angular velocity in the rotation direction, and can capture the state of the vertical, horizontal, and tilted posture. The gyro sensor 1305 can be used together with the acceleration sensor 104 of a smartphone or smart watch placed in the user's pocket to improve accuracy.

[0153] The geomagnetic sensor 1304 is a sensor that detects the magnetic force of the earth, and detects the direction in which the AR glasses 100 are facing. A three-axis type that detects geomagnetism in the up-down direction in addition to the front-back direction and the left-right direction is used to capture the geomagnetic changes relative to the movement of the AR glasses 100, thereby also detecting the movement of the AR glasses 100. In this way, the movement of the head of the user wearing the AR glasses 100 and the walking state of the user can be analyzed more accurately.

[0154] The distance measuring sensor 1306 is a sensor that can measure the distance and angle to an object and capture the shape of an object such as an object in three dimensions. LiDAR (Light Detection and Ranging), TOF (Time Of Flight) sensor, millimeter wave radar, etc. are used. Among them, LiDAR irradiates a laser such as infrared light to an object, measures the rebounded scattered light to analyze and detect the distance to an object at a long distance and the state of the object, TOF sensor measures the reflection time of the pulse light irradiated to the subject for each pixel to measure the distance, and millimeter wave radar emits millimeter wave radio waves and captures its reflected waves to detect the distance to the object that reflects the radio waves and the state of the object. The distance and direction to the physical object in the front scenery are measured by the distance measuring sensor 1306, and based on the measurement information, for example, a display item related to the physical object can be virtually set in the space near the physical object.

[0155] The processor 1320 is composed of a CPU and the like.

[0156] The storage device 1330 is composed of a nonvolatile storage device and stores various programs 1331 and information data 1332 processed by the processor 1320. The information data 1332 stores display item information 1335, walking information 1336, head movement information 1337, input setting information 1338, and the like.

[0157] The processor 1320 loads the operating system (OS) 1333 and the application program 1334 for motion control, etc. stored in the storage device 1330 into the RAM 1325 and executes them, thereby forming a walking state detection processing unit 1321, a head movement detection processing unit 1322, a display item display mode control processing unit 1323, and an operation menu processing unit 1324, and realizing the functions of OS, middleware, application, and other functions.

[0158] In the case of optical see-through AR glasses, the display processing device 1308 includes a projection device for projecting display items and notification information to the user, and a transparent half mirror for imaging and displaying the projected display items in front of the eyes.

[0159] As a result, the user 10 can visually recognize both the imaged display object in a floating form and the real object in the field of vision in front of the user.

[0160] In the case of video see-through AR glasses, a display such as a liquid crystal panel is included that displays a real object in front of the eyes captured by the camera 103 together with display items and the like.

[0161] This allows the user 10 to visually recognize a real object and a displayed item in the visual field image in front of the user in an overlapping manner.

[0162] The operation input interface 1307 is used to set and input information that the user 10 wants to input, for example, using an input unit such as a line of sight, a hand, a pointer, a keyboard, a key button, or a touch key.

[0163] An input operation screen such as an operation menu 1101 can be displayed on the display screen of the display processing device 1308, and input operation information can be acquired according to the position of the line of sight on the input operation screen. Alternatively, a hand or a pointer can be displayed on the input operation screen, and the input operation information can be acquired by operating the hand or pointer through the operation input interface 1307.

[0164] In addition, when input is performed through a keyboard, key buttons, touch keys, etc., it is sufficient to set the position and method in the AR glasses 100 so that the user 10 can easily perform input operations, or it can be separated from the main body of the AR glasses 100 and connected by wire or wirelessly.

[0165] Alternatively, the user 10 may make a sound indicating an input operation, and the microphone 1342 may collect the sound to acquire input operation information.

[0166] The microphone 1342 collects external sounds or the user's own voice, and converts it into voice data composed of digital data through an A / D converter (not shown). The instruction information obtained based on the user's 10 voice can be taken into the AR glasses 100, and the action corresponding to the instruction information can be easily executed.

[0167] The speaker 1343 outputs sound based on the sound data, and can notify the user 10 of notification information by sound. For example, information such as the direction in which the user 10 is traveling, the direction in which the head is facing, the enlarged display of displayed items, and the position movement can be output by sound.

[0168] The vibrator 1341 generates vibrations under the control of the processor 1320, and converts the notification indication information sent by the AR glasses 100 to the user 10 into vibrations. By transmitting vibrations to the head of the user wearing the AR glasses 100 through the vibrator 1341, the notification indication information can be notified to the user 10, and the convenience of use can be improved.

[0169] The communication device 1344 is a communication interface for wireless communication with other devices through short-range wireless communication such as Bluetooth (registered trademark) and ZigBee (registered trademark). The communication device 1344 includes a communication processing circuit, an antenna, etc. corresponding to various predetermined communication interfaces, and performs transmission and reception of various information, control signals, etc. In addition, it may also include a telephone communication network.

[0170] The timer 1345 measures time, and measures the time from when a predetermined control maintenance period has elapsed after the walking stop is detected.

[0171] The walking state detection processing unit 1321 analyzes the walking state of the user 10 based on the captured image of the foot captured by the camera 103, and detects the walking direction and walking speed. In addition, the walking state of the user 10 is analyzed based on the head movement information detected by the acceleration sensor 104, and the walking direction and walking speed are detected. In addition to the acceleration sensor 104, the geomagnetic sensor 1304 and the gyro sensor 1305 can also be used.

[0172] The head movement detection processing unit 1322 detects the movement of the head of the user 10 through the acceleration sensor 104, and analyzes and obtains the state of the user 10 turning the head left, right, up, down, etc. In addition to the acceleration sensor 104, a geomagnetic sensor 1304 and a gyro sensor 1305 may be used.

[0173] The display item display mode control processing unit 1323 controls the display size of the display item, the display position of the display item, or a combination thereof, based on the position of the display item displayed as a virtual object in the visual field display range 202, 602, 604, 701, 702. For example, the display items in the peripheral area 660, 760, 761, 860, 870 in the visual field display range are enlarged and displayed, or the display items in the central area 650, 750, 751, 850 of the visual field display range are moved to the peripheral area 660, 760, 761, 860, 870 and enlarged and displayed.

[0174] In the operation menu processing unit 1324, an operation menu 1101 for setting the display status of the input display item, such as the display size and display position, and the action instructions of the AR glasses 100 is displayed. The user 10 sets the information he wants to input by looking at the displayed operation menu 1101 or by hand.

[0175] Through the above structure, in the AR glasses 100, when the walking state detection processing unit 1321 detects the walking of the user 10, the display item generated and displayed within the user's field of view display range can be enlarged and displayed through the display item display mode control processing unit 1323, or the display items in the peripheral area 660, 760, 761, 860, 870 within the field of view display range 202, 602, 604, 701, 702 can be enlarged and displayed, or the display items in the peripheral area 660, 760, 761, 860, 870 within the field of view display range 202, 602, 604, 701, 702 can be enlarged and displayed more than the display items enlarged in the central area 650, 750, 751, 850.

[0176] In addition, in the AR glasses 100, when the walking state detection processing unit 1321 detects the walking of the user 10, and when the head movement detection processing unit 1322 detects the movement of the user's head, the display item display mode control processing unit 1323 can maintain the display size and display position of the display item before the head movement occurs, and display the display item in the maintained display state.

[0177] In addition, in the AR glasses 100, when the walking state detection processing unit 1321 detects the walking of the user 10, the display item display mode control processing unit 1323 can move the display items located in the central area 650, 750, 751, 850 of the field of view display range to the peripheral area outside the central area 650, 750, 850 of the field of view display range and enlarge the display.

[0178] When the display area 202 is divided into the first peripheral area 860 adjacent to the outside of the central area 850 and constituting the peripheral part, and the second peripheral area 870 adjacent to the outside of the first peripheral area 860 and constituting the peripheral part, the display item display mode control processing unit 1323 can move the display items located in the central area 850 to the first peripheral area 860 and display them in a slightly enlarged display size or the same display size, or move the display items located in the central area 850 to the second peripheral area 870 and display them in a greatly enlarged display size. In addition, the display items located in the central area 850 can also be moved to the outside of the visual field display range 202.

[0179] In addition, in the AR glasses 100, the walking speed of the user 10 when walking can be detected by the walking state detection processing unit 1321, and the display size, display position, or a combination thereof of the displayed items can be controlled and displayed according to the detected walking speed of the user 10 by the display item display mode control processing unit 1323. In this case, the display size, display position, or a combination thereof of the displayed items can be switched in multiple stages according to the walking speed of the user.

[0180] In the AR glasses 100, the information desired to be input is set by operating the menu processing unit 1324 by pointing the user 10's eyes, hands, etc. toward the displayed operation menu 1101, and the display size and display position of the displayed object can be controlled according to the input setting information by the display object display mode control processing unit 1323.

[0181] Furthermore, when the walking state detection processing unit 1321 detects that the user 10 is walking, the operation menu processing unit 1324 can be stopped to prohibit the user's line of sight from inputting operations to the operation menu.

[0182] In addition, in the AR glasses 100, when the walking state detection processing unit 1321 detects that the user 10 has stopped walking, the display size and display position of the displayed item are maintained as they were through the display item display mode control processing unit 1323, and after a predetermined control maintenance period or through user input operation, the display size and display position of the displayed item during walking can be transferred to the original display size and display position of the displayed item during stationary state.

[0183] Thus, the following information processing device and information processing method can be realized: when walking, the scenery ahead and displayed objects as virtual objects within the field of view can be easily seen in an easy-to-observe form while avoiding focusing the line of sight on the peripheral part of the field of vision by considering the safety of the front and the feet in the walking direction.

[0184] <Implementation Method 9>

[0185] As a display item, a picture sent from a smartphone or a smart watch can be displayed on the AR glasses 100. For example, when the user is stationary, such as Fig.13A As shown, in the case where the surrounding area 260 within the visual field display range 202 displays the smartphone screen 1401 and the smart watch screen 1402 as display items, when the user is walking, as shown in FIG. Fig. 13B As shown, it is enlarged to a large display size and displayed as a smartphone screen 1411 and a smart watch screen 1412 in the visual field display range 202. Fig.13A , Fig. 13B In the Figure 2 , Figure 4 The parts with the same reference numerals shown have the same actions as those already described, and thus detailed description of some of them will be omitted.

[0186] As a result, while walking, it is possible to easily visually recognize the smartphone screen or smartwatch screen displayed as a virtual object in a peripheral area that is difficult for the user's line of sight to reach.

[0187] The AR glasses 100 can enlarge the images transmitted from the smartphone or smart watch and display them on the AR glasses. They can also detect when the user 10 starts walking, request the smartphone or smart watch to enlarge the image, receive the enlarged image and display it on the AR glasses 100.

[0188] In this case, the smartphone or smart watch needs to have the function of receiving an instruction to enlarge the screen and sending the enlarged screen, and can be pre-authenticated to have the same function as the AR glasses 100.

[0189] In addition, in the case of optical see-through AR glasses, the text of the displayed items can be made bold in bright places to gain brightness, and the text of the displayed items can be made thin in dark places to reduce glare.

[0190] In the above description, the AR glasses 100 as a form of HMD are described as a specific example of an information processing device, but the same function and effect can of course be obtained in an information processing device that takes all devices with the same function as the object, displays display items as virtual objects within the user's field of view, and visually recognizes them together with the scenery in front.

[0191] In addition, the present invention is not limited to the above-mentioned embodiments, and includes various modified examples. For example, the above-mentioned embodiments are embodiments described in detail in order to easily explain the present invention, and are not limited to having all the structures described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of another embodiment, and in addition, the structure of another embodiment can be added to the structure of a certain embodiment. In addition, with respect to a part of the structure of each embodiment, other structures can be added, deleted, or replaced.

[0192] In addition, the numerical values, messages, etc. that appear in the text and drawings are merely examples, and even if different numerical values ​​or messages are used, the effects of the present invention will not be impaired.

[0193] In addition, part or all of the above-mentioned structures, functions, processing parts, processing units, etc. can be implemented by hardware, for example, by designing integrated circuits, general-purpose processors, special-purpose processors, etc. The processor includes transistors and other circuits and is regarded as a circuit (circuitry) or a processing circuit (processing circuitry). In addition, the above-mentioned structures, functions, etc. can also be implemented by software by interpreting and executing programs for implementing each function by the processor. Information such as programs, tables, files, etc. that implement each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, a DVD, or a device stored on a communication network. In addition, control lines and information lines represent lines that are considered necessary for the description, and may not represent all control lines and information lines on the product. In fact, it can be considered that almost all structures are connected to each other.

[0194] The above-mentioned embodiment includes the following aspects.

[0195] (1) An information processing device that is worn on the head of a user and has the function of enabling the user to visually recognize the field of vision ahead and to generate and display display items such as text characters and icons as virtual objects. The information processing device has a walking detection sensor that detects the user's walking state such as walking movements, walking direction, and walking speed. In the information processing device, when the walking detection sensor detects the user's walking, the display items generated and displayed within the user's field of vision display range are enlarged and displayed.

[0196] (2) The information processing device according to (1), wherein when the walking detection sensor detects the user's walking, display items in the peripheral area within the field of view display range are displayed in an enlarged manner.

[0197] (3) An information processing device according to (2), wherein when the walking detection sensor detects the user's walking, displayed items in the peripheral area of ​​the field of view display range are enlarged and displayed larger than displayed items in the central area of ​​the field of view display range.

[0198] (4) The information processing device according to (2) includes a head movement detection sensor, which detects the movement of the head of a user wearing the information processing device. In the information processing device, when the head movement detection sensor detects the movement of the user's head while the walking detection sensor detects the user's walking, the display state such as the display size of the display item before the head movement was detected is maintained as it is, and the display item is displayed in the maintained display state.

[0199] (5) The information processing device according to (2), wherein the walking speed of the user is detected by the walking detection sensor, and the display size of the display item displayed in an enlarged manner is controlled according to the walking speed of the user detected by the walking detection sensor.

[0200] (6) The information processing device according to (5), wherein the display size of the display item to be displayed in an enlarged manner is switched and controlled in multiple stages according to the walking speed of the user detected by the walking detection sensor.

[0201] (7) The information processing device according to (2) has an operation menu for the user to set and input display status such as display size and display position of display items when walking. In the information processing device, the display size of display items when walking is controlled based on the setting input information set and input in the operation menu.

[0202] (8) An information processing device according to (1), comprising an operation menu and a line of sight input device for selecting the operation menu, wherein when the walking detection sensor detects the user's walking, input operations to the operation menu through the user's line of sight are restricted.

[0203] (9) According to the information processing device described in (1), in the information processing device, even if the walking detection sensor detects that the user stops walking while walking, the display size of the display items during walking is maintained as it is, and after a predetermined control maintenance period has passed since the walking stops or through the user's input operation, the display size of the display items during walking is transferred from the maintained display size of the display items during walking to the original display size of the display items during stationary state.

[0204] (10) An information processing device that is worn on the head of a user and has the function of enabling the user to visually recognize the field of view in front of the user and to generate display items such as text characters and icons as virtual objects. The information processing device has a walking detection sensor that detects the user's walking state such as walking movements, walking direction, and walking speed. In the information processing device, when the walking detection sensor detects the user's walking, display items located in the central area of ​​the field of view display range are moved to the outside compared to the central area of ​​the field of view display range.

[0205] (11) The information processing device according to (10), wherein during walking, the displayed items located in the central area of ​​the visual field display range are moved to the peripheral area of ​​the visual field display range and displayed in an enlarged manner.

[0206] (12) According to the information processing device described in (11), the peripheral area within the field of view display range is divided into a first peripheral area adjacent to the outer side of the central area of ​​the field of view display range and constituting a peripheral portion, and a second peripheral area adjacent to the outer side of the first peripheral area and constituting a peripheral portion. In the information processing device, the magnification of the display items that are moved to the second peripheral area by the walking detection sensor detecting the user's walking is made greater than the magnification of the display items that are moved to the first peripheral area.

[0207] (13) The information processing device according to (10), wherein the walking detection sensor detects the walking speed of the user, and the display position of the display item to be displayed in an enlarged manner is controlled according to the walking speed of the user detected by the walking detection sensor.

[0208] (14) The information processing device according to (13), wherein the display position of the displayed item is switched and controlled in multiple stages according to the walking speed of the user detected by the walking detection sensor.

[0209] (15) The information processing device according to (10) has an operation menu for the user to set and input display states such as display size and display position of display items when walking. In the information processing device, the display position of display items when walking is controlled based on setting input information set and input in the operation menu.

[0210] (16) An information processing device according to (10), comprising an operation menu and a line of sight input device for selecting the operation menu, wherein when the walking detection sensor detects the user's walking, input operations on the operation menu through the user's line of sight are restricted.

[0211] (17) According to the information processing device described in (10), in the information processing device, even if the walking detection sensor detects that the user stops walking while walking, the display position of the display items during walking is maintained as it is, and after a predetermined control maintenance period has passed since the user stopped walking or through the user's input operation, the display position of the display items during walking is shifted from the maintained display position to the original display position of the display items during stationary state.

[0212] (18) An information processing method in an information processing device, wherein the information processing device is worn on the head of a user, has the function of visually recognizing the field of vision in front of it and generating and displaying text characters, icons and other display items as virtual objects, and has a walking detection sensor for detecting the user's walking state such as walking movements, walking direction, and walking speed, the information processing method comprising: a step of detecting the user's walking by the walking detection sensor; and when the user's walking is detected by the walking detection sensor, a step of enlarging the display items generated and displayed within the user's field of vision display range, or moving them to the outside compared to the central area within the field of vision display range.

[0213] (19) The information processing method according to (18), further comprising: when the walking detection sensor detects the user's walking, a step of enlarging and displaying displayed items in a peripheral area within a field of view display range.

[0214] (20) An information processing device worn by a user, comprising: a display processing device that displays an augmented reality object within a field of view display range; a walking detection sensor that detects the walking state of the user; and a processor that determines whether the user is walking based on sensor information from the walking detection sensor, and when it is determined that the user is walking, enlarges the augmented reality object and executes control to display the enlarged augmented reality object within the field of view display range.

[0215] (21) An information processing device worn by a user, comprising: a display processing device that displays an augmented reality object within a field of view display range; a walking detection sensor that detects the user's walking state; and a processor, wherein the field of view display range includes a central area and a peripheral area, the central area includes a center point of the field of view display range, and the peripheral area is located at the outer edge of the central area, the processor determines whether the user is walking based on sensor information from the walking detection sensor, and when it is determined that the user is walking, executes display control to move the augmented reality object displayed in the central area to the peripheral area.

[0216] (22) An information processing method executed by an information processing device worn by a user, the information processing method comprising the following steps: a walking judgment step, judging whether the user is walking based on sensor information of a walking detection sensor that detects the walking state of the user; an enlargement step, when it is judged that the user is walking, enlarging an augmented reality object compared to when the user is stationary; and a display step, causing a display processing device to display the enlarged augmented reality object.

[0217] Description of Reference Numerals

[0218] 10: User

[0219] 11: Feet

[0220] 12: Feet

[0221] 100: AR glasses

[0222] 101: Left eye sight sensor

[0223] 102: Right eye sight sensor

[0224] 103: Camera

[0225] 104: Accelerometer

[0226] 111: Display items

[0227] 112: Display items

[0228] 113: Display items

[0229] 201: The scenery ahead

[0230] 202: Field of view display range

[0231] 211: Display Items

[0232] 212: Display Items

[0233] 213: Display Items

[0234] 250: Central area

[0235] 260: Surrounding area

[0236] 300: Arrow

[0237] 311: Display items

[0238] 312: Display items

[0239] 313: Display items

[0240] 511: Display items

[0241] 512: Display items

[0242] 601: Arrow

[0243] 602: Field of view display range

[0244] 603: Arrow

[0245] 604: Field of view display range

[0246] 612: Display Item

[0247] 613: Display Items

[0248] 650: Central area

[0249] 651: Central Area

[0250] 660: Surrounding area

[0251] 661: Surrounding Area

[0252] 701: Field of view display range

[0253] 702: Field of view display range

[0254] 713: Display Items

[0255] 750: Central area

[0256] 751: Central Area

[0257] 760: Surrounding area

[0258] 761: Surrounding Area

[0259] 801: Dashed box

[0260] 802: Dashed box

[0261] 812: Display items

[0262] 813: Display Items

[0263] 822: Display items

[0264] 823: Display Items

[0265] 832: Display items

[0266] 850: Central area

[0267] 860: First peripheral area

[0268] 870: Second peripheral area

[0269] 1002: Solid line

[0270] 1003: Dashed line

[0271] 1101: Operation Menu

[0272] 1304: Geomagnetic sensor

[0273] 1305: Gyroscope sensor

[0274] 1306: Distance sensor

[0275] 1307: Operation input interface

[0276] 1308: Display processing device

[0277] 1320: Processor

[0278] 1321: Walking state detection processing unit

[0279] 1322: Head motion detection processing unit

[0280] 1323: Display item display mode control processing unit

[0281] 1324: Operation menu processing unit

[0282] 1325: RAM

[0283] 1330: Storage device

[0284] 1331: Procedure

[0285] 1332: Information data

[0286] 1334: Application

[0287] 1335: Display item information

[0288] 1336: Walking information

[0289] 1337: Head movement information

[0290] 1338: Enter setting information

[0291] 1341: Vibrator

[0292] 1342: Microphone

[0293] 1343: Speaker

[0294] 1344: Communication equipment

[0295] 1345: Timer

[0296] 1350: Bus

[0297] 1401: Smartphone screen

[0298] 1402: Smart watch screen

[0299] 1411: Smartphone screen

[0300] 1412: Smart watch screen.

Claims

1. An information processing device worn by a user, characterized in that: have: A display processing device that displays an augmented reality object within a field of view display; A walking detection sensor that detects a user's walking state; as well as processor, The processor determines whether the user is walking based on the sensor information of the walking detection sensor, and when it is determined that the user is walking, enlarges the augmented reality object and performs control to display the enlarged augmented reality object within the field of view display range.

2. The information processing device according to claim 1, characterized in that The visual field display range includes a central area and a peripheral area, wherein the central area includes the center point of the visual field display range, and the peripheral area is located at the outer edge of the central area. The processor enlarges the augmented reality object displayed in the peripheral area and does not enlarge the augmented reality object displayed in the central area.

3. The information processing device according to claim 2, characterized in that: The processor enlarges a display size of the augmented reality object displayed in the peripheral area to a size larger than a display size of the augmented reality object displayed in the central area.

4. The information processing device according to claim 2, characterized in that The information processing device further includes a head movement detection sensor that detects movement of a head of a user wearing the information processing device. When the processor determines that the user is stationary based on the sensor information of the walking detection sensor, and when the processor detects the movement of the user's head based on the sensor information of the head movement detection sensor, the processor moves the position of the field of view display range to follow the movement of the head, and the augmented reality object displayed in the peripheral area within the field of view display range after the movement is enlarged, while the augmented reality object displayed in the central area within the field of view display range after the movement is not enlarged.

5. The information processing device according to claim 4, characterized in that When it is determined that the user is walking based on the sensor information of the walking detection sensor, and when the movement of the user's head is detected based on the sensor information of the head movement detection sensor, the processor does not change the display size of the augmented reality object before and after the movement of the head is detected.

6. The information processing device according to claim 2, characterized in that: The processor detects the walking speed of the user according to the sensor information of the walking detection sensor, and controls the display size of the augmented reality object to be enlarged according to the walking speed.

7. The information processing device according to claim 1, characterized in that: The processor controls the display of an operation menu within the field of view display range, wherein the operation menu is used to set a display size of the augmented reality object.

8. An information processing device worn by a user, characterized in that: have: A display processing device that displays an augmented reality object within a field of view display; A walking detection sensor that detects a user's walking state; as well as processor, The visual field display range includes a central area and a peripheral area, wherein the central area includes the center point of the visual field display range, and the peripheral area is located at the outer edge of the central area. The processor determines whether the user is walking based on sensor information of the walking detection sensor, and when it is determined that the user is walking, performs display control to move the augmented reality object displayed in the central area to the peripheral area.

9. The information processing device according to claim 8, characterized in that: The processor performs display control to enlarge and move the augmented reality object displayed in the central area to the peripheral area.

10. The information processing device according to claim 9, characterized in that: The peripheral area within the visual field display range is divided into a first peripheral area adjacent to the outer side of the central area and a second peripheral area adjacent to the outer side of the first peripheral area. The processor determines that the user is walking based on sensor information of the walking detection sensor, and makes the magnification of the augmented reality object moved to the second surrounding area greater than the magnification of the augmented reality object moved to the first surrounding area.

11. The information processing device according to claim 10, characterized in that: The processor detects the walking speed of the user according to the sensor information of the walking detection sensor, and controls the display position of the augmented reality object according to the walking speed.

12. The information processing device according to claim 8, characterized in that: The processor controls the display of an operation menu within the field of view display range, where the operation menu is used to set the display size of the augmented reality object.

13. The information processing device according to claim 1, characterized in that: The walking detection sensor is at least one of a camera that captures the user's feet, an acceleration sensor, a geomagnetic sensor, and a gyro sensor. The head motion detection sensor is at least one of the acceleration sensor, the geomagnetic sensor, and the gyro sensor.

14. An information processing method performed by an information processing device worn by a user, characterized in that: The information processing method comprises the following steps: a walking determination step of determining whether the user is walking based on sensor information of a walking detection sensor that detects a walking state of the user; A magnification step, when it is determined that the user is walking, magnifying the augmented reality object compared to when the user is stationary; as well as The display step enables the display processing device to display the enlarged augmented reality object.

15. The information processing method according to claim 14, characterized in that: The information processing method comprises a display position moving step, in which, when it is determined that the user is walking, the display position of the augmented reality object is moved within the visual field display range of the display processing device, The visual field display range includes a central area and a peripheral area, wherein the central area includes the center point of the visual field display range, and the peripheral area is located at the outer edge of the central area. In the display position moving step, the augmented reality object displayed in the central area is moved to the peripheral area.

Images