Image correction method, image correction device, storage medium and display device

Abstract

The present invention discloses an image correction method, an image correction device, a storage medium and a display device. The image correction method comprises: determining the coordinates of each new pixel point of a monocular image to be displayed; determining the original pixel point coordinates corresponding to each new pixel point coordinate in the original monocular image according to a preset coordinate transformation relationship; extracting the pixel value of the original pixel point coordinate as the pixel value of the corresponding new pixel point coordinate to form a monocular display image, and the monocular display image is used for binocular combination imaging. By correcting the monocular virtual image, the spatial position of the combination point in the binocular combination imaging process is adjusted to obtain a flat virtual image, avoid bending of the image, prevent obvious differences in image depth in different regions, and avoid dislocation and ghosting of the left and right eye images, thereby improving the image display quality and improving the user experience.

Description

Technical Field

[0001] The present invention belongs to the technical field of augmented reality near-eye display, and in particular, relates to an image correction method, an image correction device, a computer-readable storage medium and a display device. Background Art

[0002] Augmented reality (AR) near-eye display technology is a display technology that can superimpose a virtual image to be displayed on a real scene. The fusion effect of virtual images and real scenes in the existing technology is poor, and it is difficult to ensure that the observer can clearly see the virtual image and real scene displayed by AR at the same time. There is a contradictory problem that if you can see the virtual image displayed by AR clearly, you cannot see the real scene clearly, and if you can see the real scene clearly, you cannot see the virtual image displayed by AR clearly, which can easily cause visual fatigue and other discomfort. Figure 1 As shown in the figure, O1 is the virtual image to be displayed, O2 is the real scene object, E1 and E2 are the left and right eyes of the observer respectively, s1 and s2 are the visual axis directions of the left and right eyes when the observer looks at the object, and R1 is the real indoor scene range that the observer is concerned about (usually within 10m). Figure 1 In a, the observer is looking at object O2 in the real scene. At this time, the observer cannot see the virtual image O1 clearly at the same time, because the binocular visual axis angle when looking at O2 does not match the binocular visual axis angle when looking at O1, and vice versa. The binocular visual axis angle when looking at O1 is as follows: Figure 1 As shown in b. As a result, the visual fusion effect of the virtual image and the real scene is poor. Therefore, when the observer's gaze point switches between the virtual image displayed by AR and the real scene, the convergence of the human eye needs to be repeatedly adjusted significantly, which can easily lead to visual fatigue, dizziness, nausea and other discomfort.

[0003] In order to solve this technical problem, the usual method is to adjust the main optical axis angle of the left and right glasses, such as Figure 2 As shown, Figure 2 In the figure, w1 and w2 are the left and right lenses of the AR near-eye display device, c1 and c2 are the main optical axes of the left and right lenses, and a1 is the angle between c1 and c2. The advantage of this setting is that it shortens the convergence distance when the human eye observes the virtual image to be displayed, so that the virtual image originally imaged at infinity is imaged within a distance range of about 25cm to 10m. This distance range basically covers the distance of most objects that people pay attention to when working and living indoors. When the human eye observes virtual images and real scenes, the difference in imaging distance between the two is small, so the convergence change is also small, such as Figure 1 As shown in c, the human eye muscles can switch between virtual images and real scenes with only slight adjustments, making the user's visual experience more comfortable.

[0004] However, due to the angle between the main optical axes of the left and right eye lenses, the virtual image displayed by the left and right eyes will cause the depth of the image perceived by the human eye to be bent after the binocular image is combined, and the image edges will be blurred or even have double images. Figure 3 As shown, Figure 3 It shows the virtual image seen by a single eye when there is an angle between the main optical axes of the left and right eye images of the AR near-eye display device. L is the full-frame virtual image seen by the left eye, and R is the full-frame virtual image seen by the right eye. The two full-frame virtual image planes L and R intersect in space. This will result in that, except for the image points in the horizontal and vertical directions in the center of the virtual image, the two lines of sight formed by any image point in the rest of the virtual image area and the left and right eyes will not intersect in space in theory. This will result in the same image point seen by the left and right eyes not being able to overlap when the left and right eye images are combined, that is, ghosting occurs. And the closer to the edge of the virtual image, the greater the distance between the left and right eye lines of sight in space, and the more severe the ghosting. In addition, there is also the problem of depth curvature of the virtual image after binocular combination. Figure 4 The transformation relationship between the virtual image seen by one eye and the virtual image seen after binocular image fusion is a top view. For the convenience of analysis, the problem of non-intersection of the aforementioned sight lines is ignored here. Among them, S1 is the full-frame virtual image perceived by the human eye after binocular image fusion. Consider Figure 4 There is an image point L11 on the virtual image L1 to be displayed for the left eye. If the same point on the corresponding virtual image R1 to be displayed for the right eye is R11, then after the binocular combination of the human eye (calculating the intersection of the image point and the line connecting the corresponding eyes), the image point and its same point are superimposed to form an image point S11 on the curved surface S1, that is, when the human eye views it with both eyes, it is felt that the image point is located at point S11 on the curved surface S1. Similarly, the image point L12 on the virtual image L1 to be displayed for the left eye has a same point on the virtual image R1 to be displayed for the right eye, which is R12. After the binocular combination of the human eye, the image point and its same point are superimposed to form an image point S12 on the curved surface S1. When the images to be displayed for the left and right eyes are exactly the same, it can be calculated according to the above method that the full-frame virtual image perceived by the human eye after the binocular combination is a curved surface S1.

[0005] Therefore, the bending of binocular images and the blurred edges and ghosting of virtual images in AR glasses are technical problems that need to be urgently solved in this field. Summary of the invention

[0006] The technical problem solved by the present invention is: how to avoid the bending of the virtual image obtained by binocular combination in AR glasses, blurred edges of the virtual image and double images.

[0007] The present invention discloses an image correction method, which comprises:

[0008] Determine the coordinates of each new pixel point of the monocular image to be displayed;

[0009] Determining the original pixel coordinates corresponding to the new pixel coordinates in the original monocular image according to a preset coordinate transformation relationship;

[0010] The pixel value of the original pixel point coordinate is extracted as the pixel value of the corresponding new pixel point coordinate to form a monocular display image, and the monocular display image is used for binocular combined imaging.

[0011] Preferably, the method for determining the coordinates of each new pixel point of the monocular image to be displayed is:

[0012] Obtaining the coordinates of each combined image point of the virtual image to be displayed and the coordinates of each pixel point with the same name of the monocular reference image;

[0013] The coordinates of the new pixel point are obtained according to the coordinates of the combined image point and the coordinates of the pixel point with the same name.

[0014] Preferably, the expression of the preset coordinate transformation relationship is:

[0015]

[0016]

[0017]

[0018] Among them, (u′, v′) represents the coordinates of the original pixel point, (u, v) represents the coordinates of the new pixel point, θ represents the angle between the main optical axes of the left and right eye images, h represents the binocular image distance, d represents the average pupil distance of the human eye, (uc, vc) represents the coordinates of the central pixel point of the monocular image to be displayed, and M represents the main optical axis rotation matrix of the monocular image to be displayed.

[0019] Preferably, the image correction method further comprises: obtaining a main optical axis rotation matrix M by calculating the rotation angle of the main optical axis of the monocular image to be displayed relative to each coordinate axis in the world coordinate system.

[0020] The present application also discloses an image display method, which comprises:

[0021] Track and obtain the user's gaze direction in real time;

[0022] When the gaze direction reaches a predetermined condition, the above-mentioned image correction method is executed.

[0023] The present application also discloses an image correction device, which includes:

[0024] A new coordinate determination module, the new coordinate determination module is used to determine the coordinates of each new pixel point of the monocular image to be displayed;

[0025] A coordinate transformation module, the coordinate transformation module is used to determine the original pixel coordinates corresponding to the coordinates of each of the new pixel points in the original monocular image according to a preset coordinate transformation relationship;

[0026] A pixel value extraction module is used to extract the pixel value of the original pixel point coordinates as the pixel value of the corresponding new pixel point coordinates to form a monocular display image, and the monocular display image is used for binocular combined imaging.

[0027] Preferably, the image correction device further comprises:

[0028] A pixel storage module, wherein the pixel storage module is used to store the original monocular image.

[0029] The present application also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores an image correction program, and when the image correction program is executed by a processor, the above-mentioned image correction method is implemented.

[0030] The present application also discloses a display device, which includes a computer-readable storage medium, a processor, and an image correction program stored in the computer-readable storage medium. When the image correction program is executed by the processor, the above-mentioned image correction method is implemented.

[0031] The present invention discloses an image correction method and an image correction device, which have the following technical effects:

[0032] By correcting the monocular virtual image, the spatial position of the image point during binocular image formation is adjusted to obtain a flat virtual image, avoid image bending and prevent obvious differences in image depth in different areas, as well as avoid misalignment and ghosting of left and right eye images, thereby improving image display quality and enhancing user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the fusion of virtual images and real scenes in the prior art;

[0034] Figure 2 It is a schematic diagram of the case where the principal optical axes of the left and right eyeglass lenses have an included angle in the prior art;

[0035] Figure 3 It is a schematic diagram of left and right eye displayed images when the main optical axes of the left and right eyeglasses have an included angle in the prior art;

[0036] Figure 4 A schematic diagram of the transformation relationship between a virtual image seen by a single eye and a virtual image seen after binocular images are combined in the prior art;

[0037] Figure 5is a flow chart of an image correction method according to Embodiment 1 of the present invention;

[0038] Figure 6 Schematic diagram of the transformation relationship between a virtual image seen by a single eye and a virtual image seen after binocular imaging according to the first embodiment of the present invention;

[0039] Figure 7 This is a principle block diagram of an image correction device according to a third embodiment of the present invention;

[0040] Figure 8 It is a schematic diagram of a display device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] Before describing the various embodiments of the present application in detail, the technical concept of the present application is briefly described first: as described in the background technology section, when the main optical axes of the left and right eye images have an angle, the virtual image obtained after the left and right eye images are binocularly combined is a curved image, and the edges of the virtual image are blurred and double images. To this end, the image correction method proposed in the present invention adaptively adjusts the coordinates of each pixel point of the original monocular image and retains each pixel value of the original monocular image, so that when the corrected monocular display image is used for binocular combination, the combined image can be on the same plane, that is, the virtual image obtained by combining the images is a flat image, thereby avoiding the bending of the image and the difference in image depth, and avoiding the misalignment and double images of the left and right eye images.

[0043] In the first embodiment, the AR device is the executor of the image correction method, and the AR device includes but is not limited to AR glasses and AR head-mounted display devices.

[0044] Specifically, if Figure 5 As shown, the image correction method of the first embodiment includes the following steps:

[0045] Step S10, determining the coordinates of each new pixel point of the monocular image to be displayed;

[0046] Step S20, determining the original pixel coordinates corresponding to the new pixel coordinates in the original monocular image according to a preset coordinate transformation relationship;

[0047] Step S30: extracting the pixel value of the original pixel point coordinate as the pixel value of the corresponding new pixel point coordinate to form a monocular display image, wherein the monocular display image is used for binocular combined imaging.

[0048] Specifically, the first embodiment of this invention defines that, in the world coordinate system, the pixel coordinates of a point in the physical space in the left eye image and the pixel coordinates in the right eye image are two pixel coordinates with the same name, and a virtual image is obtained after the left eye image and the right eye image are binocularly combined. Figure 6 As shown, for a certain point in space, its corresponding pixel point in the left eye image L2 is L21, its corresponding pixel point in the right eye image R2 is R21, and the image point in the virtual image S2 obtained by the combined image is S21. The image point S21 is the intersection of the line connecting the pixel point L21 and the left eye and the line connecting the pixel point R21 and the right eye. Since the relative position between the left eye and the right eye remains unchanged in the world coordinate system, it can be considered that the coordinates of the image point S21 depend on the coordinates of the pixel point L21 and the pixel point R21.

[0049] Furthermore, the basic idea of ​​step S10 is to assume that one of the monocular images remains unchanged and the binocular combined virtual image to be displayed is set on a certain plane, and the position of the other monocular image after correction is solved based on the two. Specifically, first obtain the coordinates of each combined image point of the binocular combined virtual image to be displayed and the coordinates of each pixel point with the same name of the monocular reference image, and then obtain the coordinates of the new pixel points of the other monocular image according to the coordinates of the binocular combined image points and the coordinates of the pixel points with the same name on the monocular reference image. Exemplarily, when it is determined that the combined image point of a certain point in space in the binocular combined virtual image to be displayed is S21 and the pixel point in the left eye reference image is L21, according to the principle of light intersection, it can be reversed to obtain that the pixel point in the right eye image to be displayed is R21, and by analogy, the coordinates of each pixel point in the right eye image to be displayed, that is, the coordinates of the new pixel point, can be obtained.

[0050] Furthermore, in order to complete the image transformation, in addition to determining the transformed pixel coordinates, it is also necessary to determine the pixel values ​​corresponding to each pixel coordinate. In order to determine the pixel values ​​corresponding to each pixel coordinate, it is first necessary to determine the original pixel coordinates corresponding to each of the new pixel coordinates after the transformation in the original monocular image, and then use the pixel values ​​of the original pixel coordinates as the pixel values ​​of the new pixel coordinates.

[0051] Specifically, in step S20, the original pixel coordinates corresponding to the coordinates of each new pixel can be determined according to a preset coordinate transformation relationship, wherein the expression of the preset coordinate transformation relationship is:

[0052]

[0053]

[0054]

[0055] like Figure 6 As shown in the figure, (u′, v′) represents the original pixel coordinates, (u, v) represents the new pixel coordinates, θ represents the main optical axis angle of the left and right eye images, h represents the binocular image distance, d represents the average pupil distance of the human eye, (uc, vc) represents the central pixel coordinates of the monocular image to be displayed, and M represents the main optical axis rotation matrix of the monocular image to be displayed.

[0056] The main optical axis rotation matrix M is calculated based on the rotation angle of the main optical axis of the monocular image to be displayed relative to each coordinate axis in the world coordinate system. Specifically, assuming that in the world coordinate system, the rotation angles of the main optical axis relative to the three coordinate axes XYZ are α, β, and γ respectively, the expression of the main optical axis rotation matrix M is:

[0057] M=R z (γ)*R y (β)*R x (α)

[0058]

[0059]

[0060]

[0061] According to the above calculation process, the original pixel coordinates (u, v) corresponding to the new pixel coordinates in the original monocular image can be calculated. ′ ,v′), for example, the coordinates of the pixel point R21 in the right eye image to be displayed are calculated to obtain the coordinates of the corresponding original pixel point R11 in the right eye original image.

[0062] Furthermore, the original pixel coordinates (u ′ ,v′), then extract the coordinates of each original pixel point (u ′ ,v′) as the pixel value of the corresponding new pixel point coordinate (u,v), so that the coordinates of each new pixel point of the monocular image to be displayed and its corresponding pixel value are determined, and the required monocular display image can be formed. When the monocular display image and the monocular reference image are binocularly combined, a flat virtual image can be obtained. For example, the corrected right eye image R2 and the left eye image L2 as a reference are binocularly combined to obtain a flat virtual image S2.

[0063] It should be noted that the above-mentioned first embodiment uses the left eye image L2 as a reference image to adjust the right eye image R2. In other embodiments, the right eye image R2 may be used as a reference image to adjust the left eye image L2. For example, the original pixel point L12 of the left eye image L1 is adjusted to the new pixel point L22. The new pixel point L22 is binocularly combined with the pixel point R22 of the same name in the right eye image R2 to obtain the combined image point S22. Furthermore, the left eye image L2 and the right eye image R2 may be adjusted at the same time. For example, the left eye image L2 is used as a reference to adjust part of the right eye image R2, and then the unadjusted part of the right eye image R2 is used as a reference to adjust part of the left eye image L2 to obtain a flat virtual image S2.

[0064] The image correction method provided in the first embodiment of the present invention adjusts the spatial position of the image point during binocular image formation by correcting the monocular virtual image, thereby obtaining a flat virtual image, avoiding image bending and preventing obvious differences in image depth in different areas, and avoiding misalignment and ghosting of left and right eye images, thereby improving image display quality and enhancing user experience.

[0065] Furthermore, the second embodiment also discloses an image display method, which includes real-time tracking and obtaining the user's gaze direction, and when the gaze direction reaches a predetermined condition, executing the image correction method in the first embodiment. Exemplarily, in combination with eye tracking technology, when the AR display device displays the image, the monocular virtual image is not adjusted by default, that is, the binocular virtual image is not corrected. Once the eye tracking detects that the viewer's gaze is fixed on the edge of the virtual image, the correction of the binocular virtual image is immediately started to reduce the requirements for computing power and power consumption.

[0066] like Figure 7 As shown, the second embodiment discloses an image correction device, which includes a new coordinate determination module 100, a coordinate transformation module 200 and a pixel value extraction module 300. The new coordinate determination module 100 is used to determine the coordinates of each new pixel point of the monocular image to be displayed; the coordinate transformation module 200 is used to determine the original pixel point coordinates corresponding to each new pixel point coordinate in the original monocular image according to a preset coordinate transformation relationship; the pixel value extraction module 300 is used to extract the pixel value of the original pixel point coordinate as the pixel value of the corresponding new pixel point coordinate to form a monocular display image, and the monocular display image is used for binocular combined imaging.

[0067] Furthermore, the image correction device further includes a pixel storage module 400, and the pixel storage module 400 is used to store the original monocular image.

[0068] Among them, the image correction device can be an AR device such as AR glasses, AR head display, etc. The more detailed working process of each module in the image correction device can be referred to the description of Example 1, which will not be repeated here.

[0069] The third embodiment also discloses a computer-readable storage medium, which stores an image correction program. When the image correction program is executed by a processor, the above-mentioned image correction method is implemented.

[0070] This fourth embodiment also discloses a display device, at the hardware level, such as Figure 8 As shown, the display device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding program from the computer-readable storage medium and then runs it, forming a request processing device at the logical level. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices. An image correction program is stored on the computer-readable storage medium 11, and the image correction program implements the above-mentioned image correction method when executed by the processor.

[0071] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be display device readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0072] The specific implementation methods of the present invention are described in detail above. Although some embodiments have been shown and described, those skilled in the art should understand that these embodiments can be modified and improved without departing from the principles and spirit of the present invention, the scope of which is defined by the claims and their equivalents. These modifications and improvements should also be within the scope of protection of the present invention.

Claims

1. An image correction method, characterized in that: The image correction method comprises: Determining the coordinates of each new pixel point of the monocular image to be displayed, including: obtaining the coordinates of each combined image point of the virtual image to be displayed and the coordinates of each pixel point with the same name of the monocular reference image; obtaining the coordinates of the new pixel point according to the combined image point coordinates and the coordinates of the pixel point with the same name, wherein the coordinates of each combined image point of the virtual image to be displayed are in the same plane; Determining the original pixel coordinates corresponding to the new pixel coordinates in the original monocular image according to a preset coordinate transformation relationship; Extracting the pixel value of the original pixel point coordinate as the pixel value of the corresponding new pixel point coordinate to form a monocular display image, wherein the monocular display image and the monocular reference image are used for binocular combined imaging; The expression of the preset coordinate transformation relationship is: in, Represents the original pixel coordinates, Represents the new pixel coordinates, Represents the angle between the main optical axes of the left and right eye images, represents the binocular image distance, The average pupil distance of the human eye. Indicates the coordinates of the central pixel of the monocular image to be displayed. Represents the main optical axis rotation matrix of the monocular image to be displayed; The image correction method further comprises: obtaining a main optical axis rotation matrix M by calculating the rotation angle of the main optical axis of the monocular image to be displayed relative to each coordinate axis in the world coordinate system; Assume that in the world coordinate system, the rotation angles of the principal optical axis relative to the three coordinate axes XYZ are , the expression of the main optical axis rotation matrix M is: 。 2. An image display method, characterized in that: The image display method comprises: Track and obtain the user's gaze direction in real time; When the gaze direction reaches a predetermined condition, the image correction method of claim 1 is executed.

3. An image correction device, characterized in that: The image correction device comprises: A new coordinate determination module, the new coordinate determination module is used to determine the coordinates of each new pixel point of the monocular image to be displayed, including: obtaining the coordinates of each combined image point of the virtual image to be displayed and the coordinates of each pixel point with the same name of the monocular reference image; obtaining the coordinates of the new pixel point according to the combined image point coordinates and the coordinates of the pixel point with the same name, wherein the coordinates of each combined image point of the virtual image to be displayed are in the same plane; A coordinate transformation module, the coordinate transformation module is used to determine the original pixel coordinates corresponding to the coordinates of each of the new pixel points in the original monocular image according to a preset coordinate transformation relationship; A pixel value extraction module, the pixel value extraction module is used to extract the pixel value of the original pixel point coordinate as the pixel value of the corresponding new pixel point coordinate to form a monocular display image, and the monocular display image is used for binocular combined imaging; The expression of the preset coordinate transformation relationship is: in, Represents the original pixel coordinates, Represents the new pixel coordinates, Represents the angle between the main optical axes of the left and right eye images, represents the binocular image distance, The average pupil distance of the human eye. Indicates the coordinates of the central pixel of the monocular image to be displayed. Represents the main optical axis rotation matrix of the monocular image to be displayed; The image correction device is also used to: calculate the main optical axis rotation matrix M according to the rotation angle of the main optical axis of the monocular image to be displayed relative to each coordinate axis in the world coordinate system; Assume that in the world coordinate system, the rotation angles of the principal optical axis relative to the three coordinate axes XYZ are , the expression of the main optical axis rotation matrix M is: 。 4. The image correction device according to claim 3, characterized in that: The image correction device also includes: A pixel storage module, wherein the pixel storage module is used to store the original monocular image.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an image correction program, and when the image correction program is executed by a processor, the image correction method of claim 1 is implemented.

6. A display device, characterized in that: The display device includes a computer-readable storage medium, a processor, and an image correction program stored in the computer-readable storage medium, and the image correction program implements the image correction method of claim 1 when executed by the processor.

Images