Processor, image processing device, eyeglass-type information display device, image processing method, and image processing program

Abstract

To provide a processor, an image processing apparatus, a spectacle-type information display device, an image processing method, and an image processing program that can perform shake correction in which a display position of a display image is made appropriate.SOLUTION: A processor derives, based on a change in the relative position between a transmission-type display device and the eyes of a user, a first position of a display image displayed in a display area of the display device and a second position according to the change in the relative position, when the second position is not beyond a correction area where a display position can be corrected, changes the first position to the second position, and when the second position is beyond the correction area, changes the first position to a third position and displays the display image in the correction area.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present disclosure relates to a processor, an image processing apparatus, a glasses-type information display apparatus, an image processing method, and an image processing program.

Background Art

[0002] Conventionally, as a display device for displaying a stereoscopic video, a transmissive display device such as an augmented reality (AR) device that displays an image in a superimposed state in the real space is known.

[0003] When the relative position between the transmissive display device and the user's eyes changes, such as when the user is running, the displayed image on the display device may blur and be difficult to view. Therefore, in order to suppress blurring during viewing, a technique called blur correction is known, which changes the display position of the display image so as to cancel out the change in the relative position between the transmissive display device and the user's eyes (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the conventional technology, when the position of the display image is changed for blur correction, there are cases where the display image cannot be displayed properly, such as when the display image cannot be displayed at all.

[0006] In view of the above circumstances, the present disclosure has been made, and an object thereof is to provide a processor, an image processing apparatus, a glasses-type information display apparatus, an image processing method, and an image processing program that can perform blur correction while appropriately setting the display position of the display image. [Means for solving the problem]

[0007] To achieve the above objective, the processor of the first aspect of this disclosure is a transparent display device Detected by a vibration detection sensor that detects vibrations, Display device and Yu Changes in relative position to the eye Information representing the direction and amount of movement. Based on, display device Located within the correction region, which is the largest area in which the displayed image can be shown. The first position of the displayed image in the display area, and the change in its relative position, It is the position that cancels out the change in relative position. Derive the second position, the second position but If the correction range is not exceeded, the first position is changed to the second position; if the second position exceeds the correction range, the first position is changed. It is a position within the correction area. Change to the third position and display the image within the correction area.

[0008] A processor according to a second aspect of the present disclosure, in the processor according to the first aspect, changes the position of the display area with respect to the correction area in accordance with the change in relative position.

[0009] A processor according to a third aspect of this disclosure, in the case of a processor according to a second aspect, independently corrects the display positions of multiple display images when multiple display images are displayed in the display area.

[0010] A processor according to a fourth aspect of the present disclosure, in the processor according to a third aspect, performs control to change the display position of the second display image according to the first display image after changing the display position to the third position, when the first display image is superimposed on the second display image by changing the display position to the third position.

[0011] A processor according to a fifth aspect of the present disclosure, in a processor according to a third aspect, selectively performs one of the following controls when the first display image is superimposed on the second display image by changing the display position to a third position: a first control that changes the display position of the second display image according to the first display image after changing the display position to the third position; a second control that displays the first display image and the second display image superimposed on each other; or a third control that sets the display position of the first display image to the second position.

[0012] A processor according to a sixth aspect of this disclosure, in a processor according to a third aspect, has a displayable range in which a display image can be displayed predetermined according to the degree of change in relative position, and the display position of the display image is changed within the displayable range.

[0013] A processor according to a seventh aspect of this disclosure, in the processor according to a sixth aspect, selects a displayable range according to the user's movement state and changes the display position of the display image within the selected displayable range.

[0015] History of this disclosure 8 In the first embodiment of the processor, the display image is divided into a plurality of sub-regions, and the third position is a position in which at least a predetermined sub-region among the plurality of sub-regions is within the correction region.

[0016] History of this disclosure 9 In the first embodiment of the processor, a priority is assigned to the displayed image according to the area within the correction region, and if the second position exceeds the correction region, it is changed to the third position based on the priority.

[0017] History of this disclosure 10 In the first embodiment of the processor, the case in which the second position exceeds a correction area in which the display position can be corrected means that at least a part of the display image is outside the correction area.

[0018] History of this disclosure 11 In the first embodiment of the processor, the case in which the second position exceeds the correction area in which the display position can be corrected is when 90% or more of the area of ​​the display image is outside the correction area.

[0019] To achieve the above objectives, the disclosure of the following 12 The image processing apparatus in this embodiment is a transparent display device. Detected by a vibration detection sensor that detects vibrations, Display device and Yu Changes in relative position to the eye Information representing the direction and amount of movement. Based on, display device Located within the correction region, which is the largest area in which the displayed image can be shown.Derive a second position according to the first position of the display image displayed in the display area and the change in the relative position, and when the second position It is the position that cancels out the change in relative position. does not exceed the correction area, change the first position to the second position, and when the second position exceeds the correction area, change the first position to but a third position, and a processor for displaying the display image within the correction area. It is a position within the correction area.

[0020] 13 aspects To achieve the above object, the first The glasses-type information display device includes a transmissive display device and the processor of the present disclosure.

[0021] 14 To achieve the above object, the image processing method according to the fifteenth aspect of the present disclosure is a transmissive display device Detected by a vibration detection sensor that detects vibrations, display device and Yu - Based on the change in the relative position between the user's eyes and Information representing the direction and amount of movement. derive a second position according to the first position of the display image displayed in the display area of the display device and the change in the relative position, and when the second position Located within the correction region, which is the largest area in which the displayed image can be shown. does not exceed the correction area, change the first position to the second position, and when the second position exceeds the correction area, change the first position to It is the position that cancels out the change in relative position. a third position, and perform a process of displaying the display image within the correction area. but It is a position within the correction area.

[0022] To achieve the above object, the image processing program according to the fifteenth aspect of the present disclosure is a transmissive display device Detected by a vibration detection sensor that detects vibrations, display device and Yu - Based on the change in the relative position between the user's eyes and Information representing the direction and amount of movement. derive a second position according to the first position of the display image displayed in the display area of the display device and the change in the relative position, and when the second position <QQ000110>does not exceed the correction area, change the first position to the second position, and when the second position exceeds the correction area, change the first position to It is the position that cancels out the change in relative position. a third position, and cause the processor to but perform a process of displaying the display image within the correction area. It is a position within the correction area. execution [[ID=~]]

[0023]

Advantages of the Invention

[0023] According to this disclosure, it is possible to perform blur correction while appropriately positioning the displayed image. [Brief explanation of the drawing]

[0024] [Figure 1] This is a configuration diagram showing an example of the configuration of the eyeglass-type information display device according to the embodiment. [Figure 2] This is a perspective view showing an example of AR glasses according to an embodiment. [Figure 3] A block diagram showing an example of the hardware configuration of a smartphone according to an embodiment. [Figure 4] This block diagram shows an example of the processor configuration of the embodiment. [Figure 5A] This is a diagram illustrating the image stabilization provided by the processor in this embodiment. [Figure 5B] This is a diagram illustrating the image stabilization provided by the processor in this embodiment. [Figure 6A] This diagram illustrates an example of a case where the display position due to basic image stabilization exceeds the correction range. [Figure 6B] This diagram illustrates another example where the display position exceeds the correction range due to basic image stabilization. [Figure 7] This flowchart shows an example of image processing performed by the processor of the first embodiment. [Figure 8] This diagram illustrates an example of positioning the displayed image within the correction area. [Figure 9A] This diagram illustrates an example of the displayable range during walking. [Figure 9B] This diagram illustrates an example of the displayable range during driving. [Figure 9C] This diagram illustrates an example of the displayable range while riding in a vehicle. [Figure 10] This flowchart shows an example of image processing performed by the processor of the second embodiment. [Modes for carrying out the invention]

[0025] Hereinafter, with reference to the drawings, examples of embodiments for carrying out the technology of this disclosure will be described in detail.

[0026] [First Embodiment] Referring to Figure 1, the configuration of the glasses-type information display device 1 of this embodiment will be described. As shown in Figure 1, the glasses-type information display device 1 of this embodiment includes AR (Augmented Reality) glasses 10 and a smartphone 12.

[0027] The AR glasses 10 are a device that allows a user to view a projected image superimposed on a real-world image, projected from an OLED (Organic Light Emitting Diode) 26. Figure 2 shows a perspective view of an example of the AR glasses 10 of this embodiment. As shown in Figures 1 and 2, the AR glasses 10 comprises a pair of transparent sections 20L for the left eye and 20R for the right eye, an OLED 26, and a vibration detection sensor 29. The AR glasses 10 of this embodiment is an example of a transparent display device of this disclosure.

[0028] The OLED26 projects an image representing information (projected image) onto the right-eye transparent section 20R in order to superimpose information onto the field of view of the real image that is viewed by the user through the right-eye transparent section 20R.

[0029] The right-eye transmissive section 20R includes a right-eye lens 22R and a light guide plate 24. Light corresponding to the projected image projected from the OLED 26 is incident on one end of the light guide plate 24. The light propagating through the light guide plate 24 changes direction at an output section (not shown) and is emitted towards the user's eye. The light corresponding to the projected image emitted from the light guide plate 24 passes through the right-eye lens 22R and is guided to the user's right eye, where it is viewed as a projected image. The user also views the real space through the right-eye lens 22R as a real image with their right eye.

[0030] Therefore, while a projection image is being projected from the OLED 26, the image perceived by the user's right eye is a superposition of the real image representing the real space seen through the right-eye lens 22R and the projected image corresponding to the projection image projected onto the light guide plate 24. Furthermore, when no projection image is being projected from the OLED 26, the image perceived by the user is the real image representing the real space seen through the right-eye lens 22R and the light guide plate 24.

[0031] On the other hand, the left eye's transparent section 20L includes the left eye's lens 22L. The user views the real world through the left eye's lens 22L with their left eye.

[0032] The vibration detection sensor 29 is a sensor that detects changes in the relative position between the AR glasses 10 and the user's eye as vibrations of the AR glasses 10. In other words, the vibration detection sensor 29 is a sensor that detects so-called "shakes". Specifically, the vibration detection sensor 29 is a sensor that detects the direction of movement of the AR glasses 10, which represents whether the AR glasses 10 has moved in the in-plane direction of the right-eye lens 22R, or in other words, in the left, right, up, or down direction relative to the user, and the amount of movement in that direction. The direction of movement and amount of movement detected by the vibration detection sensor 29 are output to the smartphone 12 as information representing vibrations.

[0033] On the other hand, the smartphone 12 is equipped with a processor 41. The processor 41 in this embodiment controls the OLED 26 to project an image onto the light guide plate 24. The processor 41 in this embodiment also controls the change in the relative position between the AR glasses 10 and the user's eye, i.e., the correction of blur. The smartphone 12 in this embodiment is an example of an image processing device of the present disclosure.

[0034] Figure 3 shows a block diagram illustrating an example of the hardware configuration of a smartphone 12. As shown in Figure 3, the smartphone 12 comprises a CPU (Central Processing Unit) 40, memory 42, an I / F (Interface) unit 43, a storage unit 44, a display 46, and an input device 48. The CPU 40, memory 42, I / F unit 43, storage unit 44, display 46, and input device 48 are connected to each other via a bus 49, such as a system bus or control bus, enabling the exchange of various types of information.

[0035] The CPU 40 reads various programs, including the image processing program 45 stored in the memory unit 44, into the memory 42 and executes processing according to the read program. This allows the CPU 40 to control blur correction and the display of the projected image by the OLED 26. As an example, the processor 41 in this embodiment consists of a combination of the CPU 40 and the image processing program 45. The memory 42 is a work memory for the CPU 40 to execute processing.

[0036] The image processing program 45 executed by the CPU 40 is stored in the memory unit 44. The memory unit 44 also stores image data (not shown) of the projected image projected from the OLED 26, as well as various other information. Specific examples of the memory unit 44 include HDDs (Hard Disk Drives) and SSDs (Solid State Drives).

[0037] The I / F unit 43 communicates various information with the OLED 26 via wireless or wired communication. The display 46 and input device 48 function as a user interface. The display 46 provides the user with various information regarding the projection of the projected image. The display 46 is not particularly limited and includes liquid crystal monitors and LED (Light Emitting Diode) monitors, etc. The input device 48 is operated by the user to input various instructions regarding the projection of the projected image. The input device 48 is not particularly limited and includes, for example, a keyboard, a stylus, and a mouse. In the case of the smartphone 12, a touch panel display is used in which the display 46 and the input device 48 are integrated.

[0038] Figure 4 shows a functional block diagram illustrating an example of the configuration of the processor 41 in this embodiment. As shown in Figure 4, the processor 41 includes an acquisition unit 50, a blur correction unit 52, and a display control unit 54. As an example, in the smartphone 12 of this embodiment, the CPU 40 executes an image processing program 45 stored in the storage unit 44, thereby allowing the CPU 40 to function as the acquisition unit 50, the blur correction unit 52, and the display control unit 54.

[0039] The acquisition unit 50 acquires information representing vibration from the vibration detection sensor 29 of the AR glasses 10. The acquisition unit 50 outputs the acquired information representing vibration to the correction unit 52.

[0040] The image blur correction unit 52 of this embodiment has a function to correct the display position of the displayed image in order to cancel out blur based on information representing vibration. As shown in Figure 4, the image blur correction unit 52 includes a display area position changing unit 53A and a display image display position changing unit 53B. The image blur correction and the change in the display position of the displayed image by the image blur correction unit 52 will be described with reference to Figures 5A and 5B.

[0041] As shown in Figure 5A, the initial state in which the user's eye U and the AR glasses 10 are facing each other is considered to be the initial state in which no blurring occurs. The AR glasses 10 are provided with a display area 70 for displaying a display image 80 that is projected from the OLED 26. In Figure 5A, display images 80A to 80D are shown as examples of display images 80. In this embodiment, when display images 80A to 80D are referred to collectively as display image 80 without distinguishing between them individually.

[0042] Furthermore, the AR glasses 10 are provided with a correction area 72 for correcting the display position of the display image 80 in the event of blurring. The position of the correction area 72 is fixed with respect to the right eye lens 22R. In this embodiment, for example, the correction area 72 is defined as the largest area in which the display image 80 can be displayed in the AR glasses 10. As shown in Figure 5A, the area of ​​the correction area 72 is larger than the area of ​​the display area 70. In this embodiment, for example, the display area 70 is provided inside the correction area 72, and in the initial state, the center positions of the display area 70 and the correction area 72 are the same.

[0043] Figure 5B shows the case where, from the state shown in Figure 5A, at least one of the following changes in relative position occurs: downward movement of the user's eye U and upward movement of the AR glasses 10. In this case, the display area position changing unit 53A performs blur correction by changing the position of the display area 70 relative to the right eye lens 22R downward in order to counteract the change in relative position. As shown in Figure 5B, by changing the position of the display area 70, the relative position between the user's eye U and the display area 70 does not change from the initial state. Since the display position of the display image 80 is also changed in accordance with the change in the position of the display area 70, the relative position between the user's eye U and the display position of the display image 80 also does not change from the initial state. Therefore, the user can perceive the display image 80 as not moving.

[0044] However, as shown in Figure 5B, if the amount of blur is large, the display area 70 whose position has been changed by the display area position changing unit 53A may extend beyond the correction area 72. The display image 80 whose display position is within the display area 70 that is outside the correction area 72 will actually not be displayed on the right eye lens 22R and will not be visible to the user's eye U. In the example shown in Figure 5B, the display position of the display image 80A, indicated by the dotted line, is outside the correction area 72. Therefore, the display image 80A will actually not be displayed on the right eye lens 22R and will not be visible to the user's eye U.

[0045] Therefore, the display image position changing unit 53B of this embodiment changes the display position of the display image 80, which has a display position outside the correction area 72, to be inside the correction area 72. In the example shown in Figure 5B, the display position of the display image 80A, shown by the dotted line, is moved in the direction of arrow Z, so that the entire display image 80A is inside the correction area 72, thereby changing its display position to be inside the correction area 72. In the example shown in Figure 5B, the display image 80A after the display position has been changed to be inside the correction area 72 is shown by the solid line. In this way, by changing the display position of the display image 80A to be inside the correction area 72, the user can see the display image 80A.

[0046] In this manner, the display area position changing unit 53A performs basic blur correction by changing the position of the display area 70 relative to the right eye lens 22R based on the change in the relative position between the AR glasses 10 (right eye lens 22R) and the user's eye U. Specifically, the display area position changing unit 53A in this embodiment derives the display position of the display image 80 (hereinafter referred to as the basic correction display position) in accordance with the change in the relative position and the display position of the display image 80 displayed in the display area 70 in the initial state (hereinafter referred to as the initial display position). If the display position of the display image 80 derived by the display area position changing unit 53A exceeds the correction area 72, the display image position changing unit 53B changes the display position of the display image 80 to a position within the correction area 72. The initial display position in this embodiment is an example of the first position in this disclosure. Furthermore, the basic correction display position of the display image 80 when basic blur correction is performed in this embodiment is an example of the second position in this disclosure. Furthermore, the display position of the display image 80 derived by the display image position changing unit 53B of this embodiment is an example of the third position of this disclosure.

[0047] The specific conditions under which the display position of the display image 80 exceeds the correction area 72 are not particularly limited and can be predetermined. For example, as shown in Figure 6A, it may be the case when at least a part of the display image 80 is outside the correction area 72, in other words, when even a small part of the display image 80 extends outside the correction area 72. Alternatively, as shown in Figure 6B, it may be the case when 90% or more of the area of ​​the display image 80 is outside the correction area 72.

[0048] The blur correction unit 52 outputs the display positions of the display images 80A to 80D, derived by the display area position changing unit 53A and the display image display position changing unit 53B, to the display control unit 54.

[0049] The display control unit 54 has the function of displaying display images 80A to 80D at the display position input from the correction unit 52. Specifically, the display control unit 54 acquires image data for each of the display images 80A to 80D projected by the OLED 26. In this embodiment, for example, since the image data for each of the display images 80A to 80D is stored in the storage unit 44, the display control unit 56 acquires the image data for each of the display images 80A to 80D from the storage unit 44. In addition to this embodiment, the image data for each of the display images 80A to 80D may be acquired from an external device of the smartphone 12 via the I / F unit 43. In the following, the image data for the display images 80A to 80D will simply be referred to as the display image, for example, when acquiring display image 80A.

[0050] Next, the operation of the processor 41 in this embodiment will be explained. Figure 7 shows a flowchart illustrating an example of the image processing flow by the processor 41 of the smartphone 12 in this embodiment. As an example, in this embodiment, when the projection of the projected image by the OLED 26 is started and the display of the display image 80 is started, the processor 41 executes the image processing shown as an example in Figure 7.

[0051] In step S100 of Figure 7, the acquisition unit 50 starts acquiring information representing vibration, as described above. Specifically, it starts acquiring information representing vibration output from the vibration detection sensor 29.

[0052] In the next step S102, the display area position changing unit 53A determines whether the amount of movement of the AR glasses 10 is greater than or equal to the blur threshold, based on the vibration information acquired by the acquisition unit 50. As an example, in this embodiment, if the amount of movement is extremely small, blur correction is performed, that is, the position of the display area 70 is not changed, and the position of the display image 80 is also not changed. Therefore, in this embodiment, if the amount of movement is greater than or equal to the blur threshold, the amount of movement is considered large, and blur correction is performed. Note that the method for determining the blur threshold, which is the criterion for whether or not to perform blur correction, is not particularly limited, and for example, an experimentally obtained value may be used, or it may be adjustable by individual users.

[0053] If the amount of movement is less than the deviation threshold, the judgment in step S102 is negative. On the other hand, if the amount of movement is greater than or equal to the deviation threshold, the judgment in step S102 is positive, and the process proceeds to step S104.

[0054] In step S104, the display area position changing unit 53A derives the basic correction display position of the display image 80 when basic shake correction is performed, based on the change in the relative position between the AR glasses 10 and the user's eye U, and the initial display position, as described above. Specifically, the display area position changing unit 53A derives the display position of the display image 80 according to the direction and amount of movement of the AR glasses 10 included in the vibration information acquired in step S100. For example, if the amount of movement is 5 mm and the direction of movement is downward, the display position of the display image 80 is derived as the basic correction display position, which is a display position moved 5 mm upward from the initial display position. When multiple display images 80 are displayed in the display area 70, the basic correction display position is corrected for each of the multiple display images 80.

[0055] In the next step S106, the display image display position changing unit 53B determines whether the basic correction display position derived in step S104 is within the correction area 72. If the basic correction display position is within the correction area 72, the determination in step S106 becomes a positive determination, and the process proceeds to step S112. On the other hand, if the basic correction display position is outside the correction area 72, the determination in step S106 becomes a negative determination, and the process proceeds to step S108.

[0056] In step S108, the display image position changing unit 53B changes the display position, which was outside the correction area 72, to a display position within the correction area 72, as described above. The amount of change in the display position and the direction of change are not particularly limited for the display image position changing unit 53B to change the display position from outside the correction area 72 to a display position within the correction area 72. For example, the display image position changing unit 53B may change the display position of the display image 80 to within the correction area 72 in a manner that minimizes the amount of change.

[0057] The display image position changing unit 53B changes the display position from outside the correction area 72 to a display position within the correction area 72, but the extent to which the display image 80 is within the correction area 72 is not limited. For example, as shown in the display image 80A of Figure 5B above, the display position may be changed so that the entire display image 80 is within the correction area 72. Alternatively, as shown in Figure 8, for example, partial areas 811 to 814 are defined for the display image 80, and the display position may be changed so that at least one of the predetermined partial areas 81 (partial area 811 in Figure 8) is within the correction area 72. The method of determining which partial area 81 is at least within the correction area 72 may be determined by the content of the display image 80 and the display position of the display image 80, and there are no particular limitations on how this is determined.

[0058] Furthermore, the display image position changing unit 53B may, for example, assign a priority to the display image 80 according to the area to be within the correction area 72, and based on the priority assigned to the display image 80, set the display position of the display image 80 to a display position within the correction area 72. For example, if the assigned priority is the highest, the display image position changing unit 53B may set the display position so that the entire area of ​​the display image 80 is within the correction area 72, as shown in Figure 5B above, and as the priority decreases, the display position may allow a portion of the area of ​​the display image 80 to be within the correction area 72.

[0059] When multiple display images 80 are displayed in the display area 70, the determinations in steps S106 and S108 are performed independently for each display image 80. That is, the display image display position changing unit 53B independently corrects the display position of each of the multiple display images 80.

[0060] In the next step S110, the display control unit 54 controls the display of the display image 80. Specifically, for display images 80 whose display position, derived in step S104, is within the correction region 72, the display control unit 54 displays the display image 80 at the display position derived in step S104. In addition, for display images 80 whose display position was changed to be within the correction region 72 in step S108, the display control unit 54 displays the display image 80 at the changed display position.

[0061] In the next step S112, the display area position changing unit 53A, similar to step S102 above, determines whether the amount of movement of the AR glasses 10 is greater than or equal to the vibration threshold based on the vibration information acquired by the acquisition unit 50. That is, it determines whether or not the vibration is continuing. If the amount of movement is greater than or equal to the vibration threshold, i.e., if the vibration is continuing, the determination in step S112 becomes a positive determination, and the process proceeds to step S116.

[0062] On the other hand, if the amount of movement is greater than or equal to the vibration threshold, that is, if the vibration stops, the determination in step S112 becomes a negative determination, and the process proceeds to step S114. In step S114, the vibration correction unit 52 changes the display position of the currently displayed image 80 to the initial display position.

[0063] In the next step, S116, the acquisition unit 50 determines whether or not to terminate the image processing shown in Figure 7, i.e., blur correction. As an example, in this embodiment, while the display image 80 is being displayed, the determination in step S116 is negative, and the process returns to step S102, repeating the processing in steps S102 to S114. On the other hand, if the display of the display image 80 is to be terminated, the determination in step S116 becomes positive, and the process proceeds to step S118.

[0064] In step S118, the acquisition unit 50 finishes acquiring the vibration information that was started in step S100. When the processing in step S118 is completed, the image processing shown in Figure 7 is completed.

[0065] Thus, according to the processor 41 of this embodiment, the display position of the displayed image 80 can be appropriately adjusted in the blur correction performed in response to changes in the relative position between the AR glasses 10 and the user's eye U.

[0066] [Second Embodiment] In this embodiment, a configuration is described in which the initial display position of the display image 80 is changed to a position within a predetermined displayable range in accordance with the change in the relative position between the AR glasses 10 and the user's eye U. The displayable range is the area within which, for the display image 80, the basic corrected display position in accordance with the change in the relative position between the AR glasses 10 and the user's eye U is predicted not to exceed the correction area 72.

[0067] For example, the amount of change in the relative position between the AR glasses 10 and the user's eye U differs depending on the user's movement. Therefore, in this embodiment, a displayable range is selected according to the user's movement, and the initial display position of the display image 80 is set within the selected evaluable range.

[0068] For example, when a user is walking, the relative position between the AR glasses 10 and the user's eye U tends to change in the vertical direction of the user. Therefore, as shown in Figure 9A, when a user is walking, a range that is narrower vertically than the display area 70 is provided within the display area 70 as the displayable range 71.

[0069] Furthermore, for example, when a user is running, the relative position between the AR glasses 10 and the user's eyes U tends to change more easily in the vertical direction, and this change is greater than when the user is walking. Therefore, as shown in Figure 9B, when a user is running, a narrower vertical range is provided within the display area 70 as the displayable range 71 compared to when the user is running.

[0070] Furthermore, for example, when a user is riding in a vehicle such as a car or train, the relative position between the AR glasses 10 and the user's eyes U may change in the vertical and horizontal directions. Therefore, as shown in Figure 9C, when a user is riding in a vehicle, a range that is narrower vertically and horizontally than the display area 70 is provided within the display area 70 as the displayable range 71.

[0071] Furthermore, if the user is stationary, such as standing still or sitting, there is no change in the relative position between the AR glasses 10 and the user's eye U, and the display position of the displayed image 80 can remain at its initial display position. Therefore, no specific display range 71 is defined.

[0072] Unlike the first embodiment, in this embodiment, the display area position changing unit 53A of the blur correction unit 52 predicts the user's motion state based on the vibration information acquired by the acquisition unit 50, and changes the initial display position of the display image 80 within the displayable range 71 according to the prediction result.

[0073] Figure 10 shows a flowchart illustrating an example of the image processing flow by the processor 41 of the smartphone 12 in this embodiment.

[0074] In step S200 of Figure 10, the acquisition unit 50 starts acquiring information representing vibration, as described above. Specifically, it starts acquiring information representing vibration output from the vibration detection sensor 29.

[0075] In the next step S202, the display area position changing unit 53A predicts the user's movement state as described above. The method by which the display area position changing unit 53A predicts the user's movement state is not particularly limited. For example, a machine learning model that has learned the correspondence between vibration information and the user's movement state may be used to predict the user's movement state according to the vibration information acquired by the acquisition unit 50. Alternatively, the user's movement state may be predicted using information other than, or in addition to, vibration information. For example, the user's movement speed may be detected, and based on the detection result, the user's movement state may be predicted as either stopped, walking, running, or riding in a vehicle.

[0076] In the next step, S204, it is determined whether the predicted result is a walking state or not. If it is a walking state, the determination in step S204 becomes positive, and the process moves to step S206. In step S206, the display area position changing unit 53A selects the maximum displayable range 71, specifically the displayable range 71 shown in Figure 9A, as the displayable range 71. On the other hand, if it is not a walking state, the determination in step S204 becomes negative, and the process moves to step S208.

[0077] In step S208, it is determined whether the prediction result indicates a driving state. If it indicates a driving state, the determination in step S208 becomes positive, and the process proceeds to step S210. In step S210, the display area position changing unit 53A selects an intermediate displayable range 71, specifically the displayable range 71 shown in Figure 9B, as the displayable range 71. On the other hand, if it does not indicate a driving state, the determination in step S208 becomes negative, and the process proceeds to step S212.

[0078] In step S212, it is determined whether the prediction result indicates that the user is in a state of being in a vehicle. If the user is in a state of being in a vehicle, the determination in step S212 becomes positive, and the process proceeds to step S214. In step S214, the display area position changing unit 53A selects the smallest displayable range 71, specifically the displayable range 71 shown in Figure 9C, as the displayable range 71. On the other hand, if the user is not in a state of being in a vehicle, i.e., if the user is stopped, the determination in step S212 becomes negative, and the process returns to step S2102.

[0079] In step S216, the display area position changing unit 53A changes the display position of the display image 80 to the displayable range 71. Specifically, it changes the display position of the display image 80 to within the displayable range 71 selected in any of steps S206, S210, or S214, and sets the changed display position as the initial display position.

[0080] In the next step S218, the display control unit 54 displays the display image 80 at the initial display position changed in step S216.

[0081] In the next step S220, the display area position changing unit 53A, similar to step S112 of the image processing in the first embodiment (see Figure 7), determines whether the amount of movement of the AR glasses 10 is greater than or equal to the vibration threshold based on the vibration information acquired by the acquisition unit 50. That is, it determines whether or not the vibration is continuing. If the amount of movement is greater than or equal to the vibration threshold, i.e., if the vibration is continuing, the determination in step S220 becomes a positive determination, and the process proceeds to step S222.

[0082] The processing in steps S222 to S228 is the same as the processing in steps S104 to S110 of the image processing in the first embodiment (see Figure 7), so the explanation is omitted. In this embodiment, after the completion of step S228, the process returns to step S220.

[0083] On the other hand, if the amount of movement is greater than or equal to the blur threshold, that is, if the vibration stops, the judgment in step S220 becomes a negative judgment, and the process proceeds to step S230. In step S230, the blur correction unit 52 changes the display position of the currently displayed image 80 to the initial display position. The initial display position here is the initial display position after the change in step S216.

[0084] In the next step, S232, the acquisition unit 50 determines whether or not to terminate the image processing shown in Figure 10, namely, blur correction. As described above, while the display image 80 is being displayed, the determination in step S232 becomes a negative determination, and the process returns to step S220, repeating the processing in steps S220 to S230. On the other hand, if the display of the display image 80 is to be terminated, the determination in step S232 becomes a positive determination, and the process proceeds to step S234.

[0085] In step S234, the acquisition unit 50 finishes acquiring the vibration information that was started in step S200. When the processing in step S118 is completed, the image processing shown in Figure 10 is completed.

[0086] Thus, according to the processor 41 of this embodiment, the initial display position is kept within the displayable range 71 in response to the relative change between the AR glasses 10 and the user's eye U, thereby preventing the basic corrected display position from exceeding the correction area 72. Consequently, the amount of correction required for the display position of the displayed image 80 can be reduced.

[0087] In each of the above embodiments, if the display image 80 overlaps with other display images 80 due to the display image position changing unit 53B changing the display position of the display image 80 within the correction area 72, the display positions of the other display images 80 may be changed according to the display image 80 that has been changed to fit within the correction area 72.

[0088] In addition to the control to change the display position of other display images 80 as described above, control may also be performed to display the display images 80 superimposed on each other. Alternatively, control may be performed to keep the display position within the correction area 72 unchanged and maintain the basic correction display position. Furthermore, these three controls may be performed selectively by the display image display position changing unit 53B based on user instructions or predetermined settings.

[0089] The shape of the glasses-type information display device is not limited to the shape, use, or mounting location of ordinary eyeglasses. Furthermore, the glasses-type information display device may be monocular or bicular, and while the above description explains a configuration where the projected image is viewed with one eye, it may also be a configuration where the projected image is viewed with both eyes. It may also have a shape where the left and right sides are connected, like goggles. Moreover, it is not limited to devices worn on the human head, such as so-called head-mounted displays (for example, if a robot mimics human functions but has the external form of a dog, and the functions of human eyes are realized by a camera on the robot's knee, then the control device of this disclosure would be mounted on the knee). Such a control device is also included in the technology of this disclosure.

[0090] Furthermore, some or all of the functions of the processor 41 in the above configuration may be provided by the AR glasses 10, or by a device other than the glasses-type information display device 1.

[0091] Furthermore, in the above configuration, the hardware structure of the processing unit that performs various processes, such as the acquisition unit 50, the blur correction unit 52 (display area position changing unit 53A and display image display position changing unit 53B), and the display control unit 54, can be the various processors shown below. As mentioned above, these various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to perform specific processes.

[0092] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0093] Examples of configuring multiple processing units with a single processor include, firstly, as exemplified by client and server computers as in the above embodiment, where a single processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by a System on a Chip (SoC). Thus, various processing units are configured as hardware structures using one or more of the above-mentioned various processors.

[0094] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits, which are combinations of circuit elements such as semiconductor devices.

[0095] Furthermore, although the above embodiments describe a configuration in which the image processing program 45 is pre-stored (installed) in the storage unit 44, the invention is not limited to this configuration. The image processing program 45 may be provided in the form of a recording medium such as a CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the image processing program 45 may be provided in the form of a download from an external device via a network.

[0096] The following additional information is disclosed regarding the above-described embodiments.

[0097] (Note 1) Based on the change in relative position between the transparent display device and the user's eyes, A first position of the display image displayed in the display area of ​​the display device and a second position corresponding to the change in the relative position are derived. If the second position does not exceed the correction range in which the display position can be corrected, the first position is changed to the second position. If the second position exceeds the correction area, the first position is changed to the third position, and the display image is displayed within the correction area. Processor.

[0098] (Note 2) The position of the display area relative to the correction area is changed in accordance with the change in the relative position. The processor described in Appendix 1.

[0099] (Note 3) When multiple display images are displayed in the aforementioned display area, The display positions of the multiple display images are corrected independently. The processor described in Appendix 1 or Appendix 2.

[0100] (Note 4) When the first displayed image is superimposed on the second displayed image by changing the display position to the third position, The third position is controlled to change the display position of the second display image according to the first display image after the display position has been changed. The processor described in any one of the appendices 1 through 3.

[0101] (Note 5) When the first displayed image is superimposed on the second displayed image by changing the display position to the third position, Selectively perform one of the following controls: a first control that changes the display position of the second display image according to the first display image after changing its display position to the third position; a second control that displays the first display image and the second display image superimposed on each other; or a third control that sets the display position of the first display image to the second position. The processor described in any one of the appendices 1 through 3.

[0102] (Note 6) The displayable range in which the aforementioned display image can be displayed is predetermined according to the degree of change in the relative position. The display position of the display image is changed within the displayable range. The processor described in any one of the appendices 1 through 5.

[0103] (Note 7) Select the displayable range according to the user's exercise state, The display position of the display image is changed to within the selected displayable range. The processor described in Appendix 6.

[0104] (Note 8) If the amount of change in the relative position exceeds a predetermined threshold, instead of changing the first position to the third position, the display image is hidden. The processor described in any one of the appendices 1 through 7.

[0105] (Note 9) The aforementioned display image is divided into multiple sub-regions, The third position is a position in which at least one predetermined sub-region among the plurality of sub-regions falls within the correction region. The processor described in any one of the appendices 1 through 8.

[0106] (Note 10) Priority is assigned to the displayed image according to the area within the correction region. If the second position exceeds the correction region, The position is changed to the third position based on the aforementioned priority. The processor described in any one of the appendices 1 through 8.

[0107] (Note 11) The case in which the second position exceeds the correction range in which the display position can be corrected is: This is the case where at least a portion of the displayed image falls outside the correction area. A processor listed in any one of the appendices 1 through 10.

[0108] (Note 12) The case in which the second position exceeds the correction range in which the display position can be corrected is: This is the case where 90% or more of the area of ​​the displayed image is outside the correction area. A processor listed in any one of the appendices 1 through 10.

[0109] (Note 13) Based on the change in relative position between the transparent display device and the user's eyes, A first position of the display image displayed in the display area of ​​the display device and a second position corresponding to the change in the relative position are derived. If the second position does not exceed the correction range in which the display position can be corrected, the first position is changed to the second position. If the second position exceeds the correction area, the first position is changed to the third position, and the display image is displayed within the correction area. Processor, Equipped with an image processing device.

[0110] (Note 14) A transparent display device, A processor described in any one of the appendices 1 to 12, A spectacle-type information display device equipped with [unspecified feature].

[0111] (Note 15) Based on the change in relative position between the transparent display device and the user's eyes, A first position of the display image displayed in the display area of ​​the display device and a second position corresponding to the change in the relative position are derived. If the second position does not exceed the correction range in which the display position can be corrected, the first position is changed to the second position. If the second position exceeds the correction area, the first position is changed to the third position, and the display image is displayed within the correction area. An image processing method in which the processor performs the processing.

[0112] (Note 16) Based on the change in relative position between the transparent display device and the user's eyes, A first position of the display image displayed in the display area of ​​the display device and a second position corresponding to the change in the relative position are derived. If the second position does not exceed the correction range in which the display position can be corrected, the first position is changed to the second position. If the second position exceeds the correction area, the first position is changed to the third position, and the display image is displayed within the correction area. An image processing program that enables the processor to execute the processing. [Explanation of Symbols]

[0113] 1 Glass-type information display device 10 AR Glasses 12 Smartphones 20L transparent section for left eye, 20R transparent section for right eye 22L lens for the left eye, 22R lens for the right eye 24 Light guide plate 26 OLED 29. Vibration detection sensor 40 CPU 41 processors 42 memory 43 I / F section 44 Memory section 45 Control Program 46 displays 48 Input devices 49 bus 50 Acquisition Department 52 Image stabilization unit, 53A Display area position change unit, 53B Display image display position change unit 54 Display Control Unit 70 display area 72 Correction area 80A~80D Display Images 811~814 partial area U User's Eye Z arrow

Claims

1. Based on information representing the direction and amount of movement that indicates a change in the relative position between the display device and the user's eye, detected by a vibration detection sensor that detects vibrations of the transmissive display device, A first position of the display image displayed in a display area provided within the correction area, which is the largest area in the display device where the display image can be displayed, and a second position that cancels out the change in the relative position, corresponding to the change in the relative position, are derived. If the second position does not exceed the correction region, the first position is changed to the second position. If the second position exceeds the correction area, the first position is changed to a third position which is within the correction area, and the display image is displayed within the correction area. Processor.

2. The position of the display area relative to the correction area is changed in accordance with the change in the relative position. The processor according to claim 1.

3. When multiple display images are displayed in the aforementioned display area, The display positions of the multiple display images are corrected independently. The processor according to claim 2.

4. When the first displayed image is superimposed on the second displayed image by changing the display position to the third position, The third position is controlled to change the display position of the second display image according to the first display image after the display position has been changed. The processor according to claim 3.

5. When the first displayed image is superimposed on the second displayed image by changing the display position to the third position, Selectively perform one of the following controls: a first control that changes the display position of the second display image according to the first display image after changing its display position to the third position; a second control that displays the first display image and the second display image superimposed on each other; or a third control that sets the display position of the first display image to the second position. The processor according to claim 3.

6. The displayable range in which the aforementioned display image can be displayed is predetermined according to the degree of change in the relative position. The display position of the display image is changed within the displayable range. The processor according to claim 3.

7. Select the displayable range according to the user's exercise state, The display position of the display image is changed to within the selected displayable range. The processor according to claim 6.

8. The aforementioned display image is divided into multiple sub-regions, The third position is a position in which at least one of the plurality of sub-regions, predetermined sub-regions, falls within the correction region. The processor according to claim 1.

9. Priority is assigned to the displayed image according to the area within the correction region. If the second position exceeds the correction region, The position is changed to the third position based on the aforementioned priority. The processor according to claim 1.

10. The case in which the second position exceeds the correction range in which the display position can be corrected is: This is the case where at least a portion of the displayed image falls outside the correction area. The processor according to claim 1.

11. The case in which the second position exceeds the correction range in which the display position can be corrected is: This is the case where 90% or more of the area of ​​the displayed image is outside the correction area. The processor according to claim 1.

12. Based on information representing the direction and amount of movement that indicates a change in the relative position between the display device and the user's eye, detected by a vibration detection sensor that detects vibrations of the transmissive display device, A first position of the display image displayed in a display area provided within the correction area, which is the largest area in the display device where the display image can be displayed, and a second position that cancels out the change in the relative position, corresponding to the change in the relative position, are derived. If the second position does not exceed the correction region, the first position is changed to the second position. If the second position exceeds the correction area, the first position is changed to a third position which is within the correction area, and the display image is displayed within the correction area. Processor, Equipped with an image processing device.

13. A transparent display device, A processor according to any one of claims 1 to 11, A spectacle-type information display device equipped with [unspecified feature].

14. Based on information representing the direction and amount of movement that indicates a change in the relative position between the display device and the user's eye, detected by a vibration detection sensor that detects vibrations of the transmissive display device, A first position of the display image displayed in a display area provided within the correction area, which is the largest area in the display device where the display image can be displayed, and a second position that cancels out the change in the relative position, corresponding to the change in the relative position, are derived. If the second position does not exceed the correction region, the first position is changed to the second position. If the second position exceeds the correction area, the first position is changed to a third position which is within the correction area, and the display image is displayed within the correction area. An image processing method in which the processor performs the processing.

15. Based on information representing the direction and amount of movement that indicates a change in the relative position between the display device and the user's eye, detected by a vibration detection sensor that detects vibrations of the transmissive display device, A first position of the display image displayed in a display area provided within the correction area, which is the largest area in the display device where the display image can be displayed, and a second position that cancels out the change in the relative position, corresponding to the change in the relative position, are derived. If the second position does not exceed the correction region, the first position is changed to the second position. If the second position exceeds the correction area, the first position is changed to a third position which is within the correction area, and the display image is displayed within the correction area. An image processing program that directs the processor to perform the processing.

Images