Method and device for obtaining pre-distortion mapping relationship of near-eye display device

Abstract

The application provides an acquisition method and device for a pre-distortion mapping relationship of a near-eye display device, which comprises the following steps: acquiring a calibration board picture and a virtual image picture after center alignment under the camera visual angle by using a camera; identifying a plurality of sampling points of the calibration board picture according to the features of the picture card of the calibration board to obtain a sampling point array image; for a plurality of virtual image points corresponding to the plurality of sampling point positions identified in the sampling point array image, determining the offset amount of the image source pixel coordinates of the virtual image point by moving the display image of the near-eye display device to make the virtual image point coincide with the corresponding sampling point according to the offset condition between each virtual image point and the corresponding sampling point, determining the original image source pixel coordinates of the virtual image point and the corresponding pre-distortion image source pixel coordinates; and generating a pre-distortion mapping relationship data matrix of the near-eye display device according to the original image source pixel coordinates of the plurality of virtual image points and the corresponding pre-distortion image source pixel coordinates.

Description

Technical Field

[0001] This invention relates to the field of optical display technology, and in particular to a method and apparatus for obtaining the pre-distortion mapping relationship of a near-eye display device. Background Technology

[0002] Near-eye displays (NIVs) are fundamental display devices used to implement Virtual Reality (VR) and Augmented Reality (AR) technologies. They display virtual information in front of the user by magnifying the virtual image. Users wearing NIVs can interact with the virtual images displayed through gestures, voice, and other methods. Distortion in NIVs can cause the virtual image to become distorted and inaccurate, affecting the user's interactive experience.

[0003] Currently, image electronic correction methods can be used to correct distortion in near-eye display devices. For example, by acquiring a virtual image of the calibration image displayed on the near-eye display device, an ideal image is generated, and the pre-distortion mapping relationship of the near-eye display device is obtained. Then, this pre-distortion mapping relationship is used to pre-distort the object-side image, resulting in a pre-distorted image. This pre-distorted image is then mapped and displayed as a virtual image according to the display mapping relationship. Thus, by pre-distorting the object-side image, the distortion of the virtual image is corrected. Therefore, obtaining an accurate pre-distortion mapping relationship is crucial for the successful correction of virtual image distortion.

[0004] However, in the existing technology, when obtaining the pre-distortion mapping relationship, it is necessary to use an ultra-low distortion wide-angle industrial lens, and then combine it with a complicated camera calibration process to correct the residual distortion of the lens itself. Even so, the obtained pre-distortion mapping relationship still cannot completely eliminate the influence of the lens distortion on the electronic correction of the image.

[0005] Therefore, there is an urgent need to provide a method for obtaining a more accurate pre-distortion mapping relationship without relying on camera lens distortion and camera calibration accuracy. Summary of the Invention

[0006] This invention provides a method and apparatus for obtaining the pre-distortion mapping relationship of a near-eye display device, which can obtain a more accurate pre-distortion mapping relationship without relying on camera lens distortion and camera calibration accuracy.

[0007] In a first aspect, embodiments of the present invention provide a method for obtaining the pre-distortion mapping relationship of a near-eye display device, comprising:

[0008] The camera acquires a calibration board image and a virtual image after center alignment from the camera's perspective. The calibration board image is the image acquired by the camera based on the chart on the physical calibration board. The near-eye display device to be distorted outputs a display image identical to the chart. The virtual image is the image acquired by the camera based on the virtual image formed by the optical system of the near-eye display device after the display image passes through it. This virtual image is distorted relative to the calibration board image.

[0009] Based on the characteristics of the image card, multiple sampling points on the calibration board screen are identified to obtain a sampling point array image;

[0010] For the multiple sampling points identified in the sampling point array image, multiple virtual image points in the virtual image are determined that correspond one-to-one with the multiple sampling points. For each virtual image point, based on the offset between each virtual image point and the corresponding sampling point, the virtual image point is moved to coincide with the corresponding sampling point to determine the offset of the image source pixel coordinates of the virtual image point, and then the original image source pixel coordinates of the virtual image point and its corresponding pre-distorted image source pixel coordinates are determined.

[0011] Based on the original image source pixel coordinates of multiple virtual image points and their corresponding pre-distorted image source pixel coordinates, a pre-distortion mapping relationship data matrix for the near-eye display device is generated.

[0012] Secondly, embodiments of the present invention also provide a device for obtaining the pre-distortion mapping relationship of a near-eye display device, comprising:

[0013] An image acquisition unit is used to acquire, using a camera, a calibration board image and a virtual image after center alignment from the camera's viewpoint; the calibration board image is the image acquired by the camera for the chart on the physical calibration board; the near-eye display device to be distorted outputs a display image identical to the chart, and the virtual image is the image acquired by the camera for the virtual image formed by the optical system of the near-eye display device after the display image passes through it, and the virtual image is distorted relative to the calibration board image;

[0014] The identification unit is used to identify multiple sampling points on the calibration board image according to the characteristics of the image card, so as to obtain a sampling point array image;

[0015] The determining unit is used to determine multiple virtual image points in the virtual image that correspond one-to-one with the multiple sampling points identified in the sampling point array image, and for each virtual image point, based on the offset between each virtual image point and the corresponding sampling point, to determine the offset of the image source pixel coordinates of the virtual image point by moving the display image of the near-eye display device to make the virtual image point coincide with the corresponding sampling point, thereby determining the original image source pixel coordinates of the virtual image point and its corresponding pre-distorted image source pixel coordinates;

[0016] The generation unit is used to generate a pre-distortion mapping relationship data matrix of the near-eye display device based on the original image source pixel coordinates of multiple virtual image points and their corresponding pre-distortion image source pixel coordinates.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method described in any embodiment of this specification.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in any embodiment of this specification.

[0019] This invention provides a method and apparatus for obtaining the pre-distortion mapping relationship of a near-eye display device. A distortion-free physical calibration board is used as the display standard for the virtual image output by the near-eye display device. The same camera is used to acquire the calibration board image and the virtual image image after center alignment from the camera's perspective. Although the camera itself has slight distortion, since the calibration board image and the virtual image image are center-aligned and acquired by the same camera, the degree of deformation at corresponding positions is the same, which can offset the influence of the camera's own distortion on the display result. Thus, using the sampling points of the calibration board image and the virtual image points at corresponding positions in the virtual image image, for each virtual image point, based on the offset between the virtual image point and the corresponding sampling point, the display image of the near-eye display device is moved to make the virtual image point and the sampling point coincide, thereby determining the original image source pixel coordinates of each virtual image point and its corresponding pre-distortion image source pixel coordinates, and thus generating the pre-distortion mapping relationship data matrix of the near-eye display device. As can be seen, this solution can obtain a more accurate pre-distortion mapping relationship without relying on camera lens distortion and camera calibration accuracy. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a method for obtaining pre-distortion mapping relationships according to an embodiment of the present invention;

[0022] Figure 2 It is a schematic diagram of the principle of the camera capturing virtual images and calibration board images;

[0023] Figure 3 This is a schematic diagram of the virtual image captured by the camera with the center aligned and the image on the calibration board;

[0024] Figure 4 This is a schematic diagram showing the superimposed display of the sampled point array image and the virtual image;

[0025] Figure 5 This is a hardware architecture diagram of an electronic device provided in an embodiment of the present invention;

[0026] Figure 6 This is a structural diagram of a pre-distortion mapping relationship acquisition device provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0028] Please refer to Figure 1 This invention provides a method for obtaining pre-distortion mapping relationships, the method comprising:

[0029] Step 100: Use a camera to acquire the calibration board image and the virtual image image after center alignment from the camera's perspective; the calibration board image is the image acquired by the camera for the chart on the physical calibration board; the near-eye display device to be distorted outputs a display image identical to the chart; the virtual image image is the image acquired by the camera through the optical system of the near-eye display device for the display image, and the virtual image formed by the optical system of the near-eye display device is distorted relative to the display image, and the virtual image image is distorted relative to the calibration board image;

[0030] Step 102: Based on the characteristics of the image card, identify multiple sampling points on the calibration board screen to obtain a sampling point array image;

[0031] Step 104: For the multiple sampling points identified in the sampling point array image, determine multiple virtual image points in the virtual image that correspond one-to-one with the multiple sampling points. For each virtual image point, based on the offset between each virtual image point and the corresponding sampling point, move the display image to make the virtual image point coincide with the corresponding sampling point to determine the offset of the image source pixel coordinates of the virtual image point, and then determine the original image source pixel coordinates of the virtual image point and its corresponding pre-distorted image source pixel coordinates.

[0032] Step 106: Based on the original image source pixel coordinates of multiple virtual image points and their corresponding pre-distortion image source pixel coordinates, generate the pre-distortion mapping relationship of the near-eye display device. The schematic mapping relationship can be represented in the form of a data matrix, etc.

[0033] In this embodiment of the invention, a distortion-free physical calibration board image is used as the display standard for the virtual image output by the near-eye display device. The same camera is used to acquire the calibration board image and the virtual image image after center alignment from the camera's viewpoint. Although the camera itself has slight distortion, since the calibration board image and the virtual image are center-aligned and acquired by the same camera, the degree of distortion at corresponding positions is the same, which can offset the influence of the camera's own distortion on the display result. Thus, using the sampling points of the calibration board image and the corresponding virtual image points in the virtual image image, for each virtual image point, based on the offset between the virtual image point and the corresponding sampling point, the display image of the near-eye display device is moved to make the virtual image point and the sampling point coincide, thereby determining the original image source pixel coordinates of each virtual image point and its corresponding pre-distorted image source pixel coordinates, and thus generating the pre-distortion mapping relationship of the near-eye display device. It can be seen that this solution can obtain a more accurate pre-distortion mapping relationship without relying on camera lens distortion and camera calibration accuracy.

[0034] The following description Figure 1The execution method of each step is shown.

[0035] First, in step 100, a camera is used to acquire a calibration board image and a virtual image after center alignment is achieved from the camera's perspective. The calibration board image is the image acquired by the camera from the chart on the physical calibration board. The near-eye display device to be distorted outputs a display image identical to the chart. The virtual image is the image acquired by the camera from the virtual image formed by the optical system of the near-eye display device after the display image passes through it. This virtual image is distorted relative to the calibration board image.

[0036] In embodiments of the present invention, such as Figure 2 As shown, when using a distortion-free calibration board chart as the display standard for the virtual image output by a near-eye display device, in order to reduce the impact of camera distortion on the accuracy of the pre-distortion mapping relationship, calibration board 2 can be fixed in front of the camera lens, with calibration board 2 parallel to the object plane of the camera lens, and the distance between calibration board 2 and the camera lens is D. V and D H That is, the lens object distance. Additionally, the near-eye display device 1 is also positioned in front of the camera lens, between the camera 3 and the calibration plate 2, and is closer to the camera 3. The distance between the near-eye display device and the camera lens is much smaller than the lens object distance. When the near-eye display device is an AR device with a field of view larger than the virtual display screen's field of view in the perspective direction, the camera can capture the calibration plate image 20 and the virtual image image 10 through the AR device. Figure 3 The display shows the calibration board image 20 and the virtual image 10, both centered and aligned with the O image, captured by the camera, and the principle of superimposed display of the two images. The calibration board image 20 is the image acquired by the camera from the calibration board's map. The near-eye display device 1 outputs a display image identical to the map. The virtual image 10 is the image acquired by the camera from this display image after passing through the optical system of the near-eye display device; the virtual image is distorted relative to the display image on the near-eye display device. To enable the camera to capture both the calibration board image and the virtual image clearly with a single focus, the distance between the calibration board and the camera can be set to the virtual image distance D of the near-eye display device. This avoids camera aberrations introduced when the camera focuses on the calibration board image and the virtual image separately. In other words, the calibration board image is acquired by the camera at a distance D from the camera.

[0037] Considering that in practice, the object distance and virtual image distance may not correspond precisely, a camera lens with a large depth of field can be used to capture the calibration board image and the virtual image simultaneously with a single focus, so as to ensure that the sampling point coordinates of the calibration board image and the virtual image captured by the camera lens can be correctly identified, thereby improving the accuracy of the pre-distortion mapping relationship.

[0038] When the near-eye display device is an AR device with a perspective field of view smaller than that of the virtual display screen, or when the near-eye display device is a VR device, the near-eye display device can be removed from between the camera and the calibration board to capture the image of the calibration board, and then the near-eye display device can be placed to capture the virtual image. The shooting parameters of the camera should be consistent for the two captures.

[0039] In this embodiment of the invention, the calibration board can use a checkerboard pattern, a cross pattern, a wireframe pattern, or a dot pattern. These patterns can all be used as calibration boards to perform image correction on near-eye display devices.

[0040] Because the field of view of the camera lens is different from that of the near-eye display device, in order to ensure that the camera can capture the image of the calibration board within the field of view of the near-eye display device at the lens object distance, the size of the calibration board needs to meet certain requirements and match the field of view of the near-eye display device.

[0041] Specifically, the calibration board image is acquired by the camera at a distance D from the camera, where D represents the virtual image distance of the near-eye display device; and the size of the calibration board chart is L*H, where L is the width of the chart and H is the height of the chart, wherein L, H, and D satisfy the following formula:

[0042]

[0043]

[0044] D = D V =D H

[0045] Wherein, the distance between the physical calibration plate and the exit pupil of the camera lens is D. V D H D V D is the object distance corresponding to the half-vertical field of view angle. H The object distance corresponding to the half-horizontal field of view; FOV V The vertical field of view (FOV) of the near-eye display device is... H The horizontal field of view of the near-eye display device is denoted as .

[0046] The field of view of a near-eye display device can be directly determined based on the calibration parameters of the near-eye display device, including the half-horizontal field of view of the near-eye display device. and semi-vertical field of view

[0047] like Figure 2As shown, taking a checkerboard pattern card used for the calibration board as an example, let the number of rows and columns on the checkerboard pattern card used for the calibration board be A and B respectively, the width of a single checkerboard grid be X centimeters, and the height be Y centimeters. A and B must be integers not less than 2, and A is not equal to B. Since the size of a single checkerboard grid is fixed, the calibration board must meet certain size requirements, which are actually requirements for the number of rows and columns on the checkerboard pattern card used for the calibration board. Specifically, the size requirement for the calibration board is BX*AY, where A, B, X, and Y satisfy the following formula:

[0048]

[0049]

[0050] D = D V =D H .

[0051] In this embodiment of the invention, the required number of rows and columns of the chessboard can be calculated using the above formula, and a calibration board that meets the requirements can be obtained based on this number of rows and columns. It should be noted that when using other charts for the calibration board, the dimensions of the calibration board still need to meet L*H, that is, match the field of view of the near-eye display device.

[0052] To reduce the discrepancy between the virtual image captured by the camera lens and the virtual image output by the near-eye display device, the center of the camera lens's aperture and the center of the near-eye display device's exit pupil can be aligned. Then, the camera lens captures the image of the calibration plate, and the near-eye display device captures the virtual image. A high-precision optical platform and fixture can be used to fix the camera lens and the near-eye display device separately.

[0053] In this embodiment of the invention, due to the slight distortion inherent in the camera lens itself, to reduce the impact of the camera lens's distortion on the pre-distortion mapping relationship, the calibration board image and the virtual image can be center-aligned under the same camera lens. This ensures that the acquired calibration board image and virtual image experience the same degree of distortion at their respective positions. Acquiring the calibration board image and virtual image using the same camera lens can offset the influence of the camera lens's distortion on the results, thereby obtaining a more accurate pre-distortion mapping relationship. For AR devices with a perspective field of view, the calibration board image and the virtual image can be simultaneously captured through the AR device's perspective field of view, and the centers of the two images can be aligned by adjusting the position of the physical calibration board or near-eye display device. For VR devices without a perspective field of view, either the calibration board image or the virtual image can be acquired first and aligned with a fixed position of the camera (e.g., the center of the field of view). Then, by adjusting the position of the physical calibration board or near-eye display device, the center position of the other image can be aligned with that fixed position, achieving the same alignment effect.

[0054] To achieve center alignment between the calibration board image and the virtual image, a near-eye display device (specifically, a microdisplay controlling the near-eye display device) can be controlled by a computer program to output a display image identical to the calibration board's pattern card. For example, when the calibration board uses a checkerboard pattern card, the display image output by the near-eye display device will be a checkerboard pattern, with the same number of rows and columns as the checkerboard pattern card on the calibration board. Furthermore, the size of the virtual image formed by the optical system of the near-eye display device will also be the same as the size of the calibration board's pattern card. The machine vision function of a camera is used to identify the center of the calibration board image and the center of the virtual image, and the pixel coordinates of these centers are recorded. By fine-tuning the position of the calibration board, the centers of the calibration board image and the virtual image are made to coincide, thus achieving center alignment between them.

[0055] It is understandable that during the process of aligning the centers of the calibration board image and the virtual image, the center of the camera lens aperture and the center of the exit pupil of the near-eye display device coincide. Thus, after adjusting the position of the calibration board so that the center of the calibration board image and the center of the virtual image coincide, there is no need to further adjust the alignment of the center of the camera lens aperture and the center of the exit pupil of the near-eye display device. Otherwise, if it is necessary to adjust the alignment of the center of the camera lens aperture and the center of the exit pupil of the near-eye display device, it will cause a deviation between the centers of the calibration board image and the virtual image.

[0056] In one embodiment of the present invention, if the camera simultaneously acquires both the calibration board image and the virtual image, the two images overlap, which can affect the accurate identification of subsequent corner point positions. Therefore, to improve the accuracy of subsequent sampling point identification, the calibration board image and the virtual image are acquired separately, and are not interfered with by each other when acquired separately.

[0057] Specifically, after fixing the camera lens, near-eye display device, and calibration board in the required positions, the calibration board image and the virtual image can be obtained through at least one of the following methods:

[0058] A1. Determine whether the field of view of the camera lens after viewing the near-eye display device is not less than the virtual image field of view of the near-eye display device to be distorted. If yes, proceed to step A2; otherwise, proceed to step A3.

[0059] A2. Turn off the display image output by the near-eye display device, and the camera lens captures the calibration plate image through the near-eye display device; remove the calibration plate, and turn on the near-eye display device to output the display image, and the camera lens captures the virtual image image through the near-eye display device.

[0060] A3. Remove the near-eye display device, and the camera lens captures the image from the calibration plate; place the near-eye display device back in the required position, and remove the calibration plate. The camera lens then captures a virtual image of the displayed image through the near-eye display device.

[0061] It should be noted that the order in which the calibration board image and the virtual image are acquired in steps A2 and A3 can also be reversed.

[0062] It should also be noted that, in addition to the above methods for obtaining the calibration board image and the virtual image, other methods can also be used. For example, regardless of the relationship between the field of view of the camera lens after passing through the near-eye display device and the field of view of the virtual image to be distorted on the near-eye display device, the near-eye display device can be removed before obtaining the calibration board image, thereby avoiding the influence of the near-eye display device on the calibration board image.

[0063] Then, the following explanations are given regarding step 102, "Based on the characteristics of the image card, identify multiple sampling points on the calibration board screen to obtain a sampling point array image" and step 104, "For the multiple sampling points identified in the sampling point array image, determine multiple virtual image points in the virtual image screen that correspond one-to-one with the multiple sampling points, and for each virtual image point, based on the offset between each virtual image point and the corresponding sampling point, determine the offset of the image source pixel coordinates of the virtual image point by moving the display image to make the virtual image point coincide with the corresponding sampling point, and then determine the original image source pixel coordinates of the virtual image point and its corresponding pre-distorted image source pixel coordinates."

[0064] To facilitate obtaining the offset between virtual image points in the virtual image and sampling points in the calibration board image, when different charts are used for the calibration board, the sampling points for sampling the calibration board image can be determined based on the characteristics of the charts. Specifically:

[0065] If the card is a chessboard pattern card, then the corner points of the chessboard are used as sampling points;

[0066] If the chart is a cross-shaped chart, then the intersection point of the cross will be used as the sampling point;

[0067] If the drawing is a wireframe drawing, then the corner points of the wireframe are used as sampling points;

[0068] If the chart is a dotted chart, then the chart points are used as sampling points.

[0069] When determining multiple sampling points in the calibration board image according to the above method, all points that meet the requirements can be used as sampling points, or only a portion of the points that meet the requirements can be used as sampling points. It can be understood that the more sampling points there are, the more accurate the final pre-distortion mapping relationship will be. Therefore, considering the number of sampling points within the field of view, at least an 8×8 array of sampling points should be guaranteed, and preferably all points that meet the requirements should be used as sampling points.

[0070] Since the checkerboard pattern in the checkerboard chart is black and white alternating, the black and white alternating feature of the checkerboard corners can make the identification of the sampling points more accurate when the corners of the checkerboard are used as sampling points. Therefore, the checkerboard chart can be preferred as a calibration board.

[0071] In this embodiment of the invention, when marking sampling points, the marking symbols can include at least a cross symbol, a dot symbol, and a star symbol. The following explanation uses a checkerboard pattern card as the calibration board and a cross symbol as the marking symbol. Please refer to... Figure 4 This is a schematic diagram showing the sampling point array image and the virtual image image after center alignment and superposition in the camera's CMOS (Complementary Metal Oxide Semiconductor) coordinate system. The sampling point array image consists of multiple cross marks. Since the virtual image image is distorted relative to the calibration board image, almost all virtual image points in the virtual image image do not coincide with the corresponding sampling points in the sampling point array image. Figure 4 In the sampling point array image, the sampling point is a corner point A*. Taking the corner point A* located at the first position on the left of the third row of the sampling point array image as an example, the virtual image point corresponding to the position of corner point A* in the virtual image is the corner point A at the top left corner of the first checkerboard square on the left of the third row of the virtual image. It can be understood that multiple virtual image points corresponding to the positions of multiple sampling points can be determined in the virtual image.

[0072] In one embodiment of the present invention, when determining the offset of the original image source pixel coordinates of the virtual image point by moving the display image source to make the virtual image point coincide with the corresponding sampling point based on the offset between each virtual image point and the corresponding sampling point, and then determining the pre-distorted image source pixel coordinates corresponding to the original image source pixel coordinates of the virtual image point, the specific steps may include:

[0073] S1. For the plurality of sampling points, select one sampling point that has not been selected from the plurality of sampling points, and execute step S2 for the sampling point;

[0074] When selecting an unselected sampling point from multiple sampling points, the sampling points can be selected in the order of top to bottom and left to right, or in the order of bottom to top and left to right, or in any other order. This embodiment does not limit the selection, as long as it ensures that all sampling points in the sampling point array image are selected and selected only once.

[0075] S2. Obtain the original image source pixel coordinates of the virtual image point corresponding to the sampling point, determine the deviation direction of the sampling point relative to the corresponding virtual image point, and adjust the position of the display image of the near-eye display device to shift the virtual image from the initial position along the deviation direction. When the shift distance between the moved virtual image point and the sampling point is 0, stop the adjustment and determine the total pixel shift of the display image during the image shift process. The sum of the original image source pixel coordinates of the virtual image point and the total pixel shift is determined as the pre-distorted image source pixel coordinates corresponding to the virtual image point.

[0076] S3. Adjust the position of the displayed image of the near-eye display device to restore the virtual image to its initial position where its center is aligned with the center of the calibration board image, and return to step S1 until there are no unselected sampling points among the plurality of sampling points.

[0077] For a sampling point and its corresponding virtual image point, the sampling point serves as the display standard for that virtual image point. In other words, to achieve a distortion-free virtual image, the virtual image point should be displayed at the position of the sampling point. Based on this, it is necessary to determine the deviation direction of the sampling point relative to the virtual image point. This can be achieved by adjusting the near-eye display device, specifically by adjusting the microdisplay of the near-eye display device to shift the displayed image from its initial position along the deviation direction. Since a precise mapping relationship cannot be directly found between the pixel coordinate system of the microdisplay screen and the camera CMOS pixel coordinate system, a computer traversal algorithm is needed to continuously attempt to move the position of the displayed image on the microdisplay, making corner point A and corner point A* coincide. This yields the original coordinates and the moved coordinates of corner point A in the coordinate system of the microdisplay screen, thus establishing a mapping relationship between the original image source pixel coordinates of corner point A and the corresponding pre-distorted image source pixel coordinates.

[0078] In this embodiment of the invention, various methods can be used to achieve the overall movement of the display image. One implementation method is to refill the display image with pixels. Specifically, when it is necessary to move the entire display image by w pixels along the positive X-axis of the microdisplay's screen coordinate system, simply subtract w pixels from the edge of the display image along the positive X-axis and add w pixels in the negative X-axis direction. This method can achieve an image offset of w pixels relative to the microdisplay's screen coordinate system along the positive X-axis.

[0079] In another implementation, due to corner point A and corner point A * In the CMOS coordinate system, there are corresponding image source pixel coordinates, so the coordinates from corner point A to corner point A can be determined. * The deviation distance (in pixels of the image source) is decomposed along the X and Y axes in the CMOS pixel coordinate system. For example, corner point A and corner point A * The image source pixel coordinates in the CMOS pixel coordinate system are respectively A(x p ,y p ), A * (x q ,y q Therefore, by controlling the overall offset of the displayed image through the microdisplay, the offset direction is decomposed along the X and Y axes of the microdisplay's screen coordinate system, from reference corner point A to corner point A. * The offset distance (in pixels) between the X and Y axes in the CMOS pixel coordinate system is used to offset the displayed image of the microdisplay according to the offset direction, with each offset amount being (x... m ,y m ),m∈N + (N + (Represents a positive integer), m represents the m-th movement of the displayed image during the process of adjusting the microdisplay to make the two corner points coincide. Each time the displayed image moves, it is necessary to determine the position of corner point A and corner point A'. * The offset distance in the CMOS pixel coordinate system is calculated until the offset distance is determined to be 0, thus determining corner point A and corner point A'. * coincide.

[0080] It should be noted that in the CMOS pixel coordinate system, the distance from corner point A to corner point A' is... * There is no precise correspondence between the offset distance of the two corner points and the offset distance of the virtual image on the microdisplay. Even if the deviation distance of the two corner points in the CMOS pixel coordinate system is known, it cannot be directly converted into the accurate source pixel value that needs to be offset for the image displayed on the microdisplay. Therefore, it is necessary to continuously adjust the method to determine the deviation of the two corner points on the X and Y axes in the CMOS pixel coordinate system, and continuously offset the display image in the microdisplay to try to align corner point A and corner point A. * Alignment is not a one-step process. Let's assume the deviation between the two corner points along the X-axis is Δx = x. q -x p The deviation in the Y-axis direction is Δy = y q -y p When Δx>0 or Δy>0, it means that corner point A needs to be offset along the positive X-axis or Y-axis of the CMOS pixel coordinate system to shorten the distance between corner point A and corner point A. * The deviation distance is the same for both sides.

[0081] Furthermore, after aligning the two corner points, the total pixel offset (ΔX) of the displayed image during the image offset process can be calculated. ij , △Y ij ):

[0082] △X ij =∑x1+x2+x3…x m ,m∈N +

[0083] △Y ij =∑y1+y2+y3…y m ,m∈N +

[0084] The subscripts i and j represent the row and column numbers of corner point A within the checkerboard pattern, respectively. The original image source pixel coordinates of corner point A when the image is initially displayed are (X... ij ,Y ij The original image source pixel coordinates and pixel offsets are summed.

[0085] X′ ij =X ij +△X ij

[0086] Y′ ij =Y ij +△Y ij

[0087] A(X ij ',Y ij The coordinates of the pre-distorted image source pixel corresponding to corner point A are '). Save and record the coordinates of the pre-distorted image source pixel, where i and j represent the row number and column number of the chessboard corner point corresponding to the corner point, respectively.

[0088] After determining the pre-distortion image source pixel coordinates of corner point A, it is necessary to determine the pre-distortion image source pixel coordinates of other corner points in the same way. Therefore, after determining the pre-distortion image source pixel coordinates of corner point A, it is necessary to restore the display image to its initial position where its center is aligned with the center of the calibration board screen, so as to determine the pre-distortion image source pixel coordinates for the next sampling point by image offsetting the display image, until the pre-distortion image source pixel coordinates of all sampling points have been determined.

[0089] Finally, for step 106, the pre-distortion mapping relationship data matrix of the near-eye display device is generated based on the original image source pixel coordinates of multiple virtual image points and their corresponding pre-distortion image source pixel coordinates.

[0090] The pre-distortion mapping relationship includes the original image source pixel coordinates and their corresponding pre-distortion image source pixel coordinates.

[0091] To achieve distortion correction for near-eye display devices, it is necessary to obtain the pre-distortion mapping relationship of any image source pixel coordinates within the entire display image area. Specifically, based on the pre-distortion mapping relationship of multiple virtual image points, at least through bilinear interpolation, the pre-distortion mapping relationship of any image source pixel coordinates within the entire display image area can be obtained, thereby obtaining a pre-distortion mapping relationship data matrix that includes all original image source pixel coordinates and their corresponding pre-distorted image source pixel coordinates.

[0092] After obtaining the pre-distortion mapping relationship of the coordinates of any source pixel within the entire display image area, a pre-distorted image can be generated by pre-distorting the data source acquired by the near-eye display device using any conventional electronic distortion correction method, and then displayed, thereby achieving accurate distortion correction for the near-eye display device. The method of generating the pre-distorted image based on the pre-distortion mapping relationship is a method that can be implemented by those skilled in the art, such as the vertex displacement method implemented based on computer graphics.

[0093] The operational accuracy of this invention embodiment is mainly related to the camera resolution. As long as the camera resolution is not lower than the angular resolution of the near-eye display device, the accuracy of the pre-distortion mapping relationship can be guaranteed at the pixel level, thereby enabling pixel-level distortion correction.

[0094] like Figure 5 , Figure 6 As shown, this embodiment of the invention provides a device for obtaining the pre-distortion mapping relationship of a near-eye display device. The device embodiment can be implemented by software, hardware, or a combination of both. From a hardware perspective, as... Figure 5 The diagram shown is a hardware architecture diagram of an electronic device containing a pre-distortion mapping relationship acquisition device for a near-eye display device, as provided in an embodiment of the present invention. Except for... Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing packets. Taking software implementation as an example, such as... Figure 6 As shown, a device in a logical sense is formed by the CPU of the electronic device in which it resides reading the corresponding computer program from the non-volatile memory into memory and running it. This embodiment provides a device for obtaining the pre-distortion mapping relationship of a near-eye display device, comprising:

[0095] Image acquisition unit 401 is used to acquire, using a camera, a calibration board image and a virtual image after center alignment under the camera's viewpoint; the calibration board image is the image acquired by the camera for the calibration board's chart; the near-eye display device to be distorted outputs a display image identical to the chart, and the virtual image is the image acquired by the camera for the virtual image formed by the optical system of the near-eye display device after the display image passes through it; the virtual image formed by the optical system of the near-eye display device is distorted relative to the original display image, and the virtual image is distorted relative to the calibration board image;

[0096] The identification unit 402 is used to identify multiple sampling points of the calibration board image according to the characteristics of the image card, so as to obtain a sampling point array image;

[0097] The determining unit 403 is used to determine multiple virtual image points in the virtual image that correspond one-to-one with the multiple sampling points in the sampling point array image, and for each virtual image point, according to the offset between each virtual image point and the corresponding sampling point, to determine the offset of the image source pixel coordinates of the virtual image point by moving the display image of the near-eye display device to make the virtual image point coincide with the corresponding sampling point, thereby determining the original image source pixel coordinates of the virtual image point and its corresponding pre-distorted image source pixel coordinates;

[0098] The generation unit 404 is used to generate a pre-distortion mapping relationship data matrix of the near-eye display device based on the original image source pixel coordinates of multiple virtual image points and their corresponding pre-distortion image source pixel coordinates.

[0099] In one embodiment of the present invention, the card is a checkerboard card, a cross card, a wireframe card, or a dotted card.

[0100] In one embodiment of the present invention, if the card is a chessboard card, then the corner points of the chessboard are used as sampling points;

[0101] If the chart is a cross-shaped chart, then the intersection point of the cross will be used as the sampling point;

[0102] If the drawing is a wireframe drawing, then the corner points of the wireframe are used as sampling points;

[0103] If the chart is a dotted chart, then the chart points are used as sampling points.

[0104] In one embodiment of the present invention,

[0105] The calibration board image is acquired by the camera at a distance D from the camera, where D represents the virtual image distance of the near-eye display device; and the size of the chart is L*H, where L is the width of the chart and H is the height of the chart, wherein L, H, and D satisfy the following formula:

[0106]

[0107]

[0108] D = D V =D H

[0109] Wherein, the distance between the physical calibration plate and the exit pupil of the camera lens is D. V D H D V D is the object distance corresponding to the half-vertical field of view angle. H The object distance corresponding to the half-horizontal field of view; FOV V The vertical field of view (FOV) of the near-eye display device is... H The horizontal field of view of the near-eye display device is denoted as .

[0110] In one embodiment of the present invention, when acquiring the calibration board image and the virtual image, the center of the camera's aperture coincides with the center of the exit pupil of the near-eye display device.

[0111] In one embodiment of the present invention, when the determining unit determines the pre-distortion coordinates corresponding to each virtual image point based on the offset distance between each virtual image point and the corresponding sampling point, it specifically implements the following steps:

[0112] S1. For the plurality of sampling points, select one sampling point that has not been selected from the plurality of sampling points, and execute step S2 for the sampling point;

[0113] S2. Obtain the original image source pixel coordinates of the virtual image point corresponding to the sampling point, determine the deviation direction of the sampling point relative to the corresponding virtual image point, and adjust the position of the display image of the near-eye display device to shift the virtual image from the initial position along the deviation direction. When the shift distance between the moved virtual image point and the sampling point is 0, stop the adjustment and determine the total pixel shift of the display image during the image shift process. The sum of the original image source pixel coordinates of the virtual image point and the total pixel shift is determined as the pre-distorted image source pixel coordinates corresponding to the virtual image point.

[0114] S3. Adjust the position of the displayed image of the near-eye display device to restore the virtual image to its initial position where its center is aligned with the center of the calibration board image, and return to step S1 until there are no unselected sampling points among the plurality of sampling points.

[0115] In one embodiment of the present invention, the generation unit is specifically used to: based on the pre-distortion mapping relationship of multiple virtual image points, obtain the pre-distortion mapping relationship of any image source pixel coordinate within the entire display image area using a bilinear interpolation method, thereby obtaining a pre-distortion mapping relationship data matrix; the pre-distortion mapping relationship data matrix includes all original image source pixel coordinates and their corresponding pre-distorted image source pixel coordinates. It can be understood that the generation unit can also obtain the pre-distortion mapping relationship of any image source pixel coordinate within the entire display image area based on the pre-distortion mapping relationship of multiple virtual image points using other known data processing methods.

[0116] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on a pre-distortion mapping relationship acquisition device for a near-eye display device. In other embodiments of the present invention, a pre-distortion mapping relationship acquisition device for a near-eye display device may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0117] The information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.

[0118] This invention also provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a method for obtaining the pre-distortion mapping relationship of a near-eye display device according to any embodiment of this invention.

[0119] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program causes the processor to perform a method for obtaining the pre-distortion mapping relationship of a near-eye display device according to any embodiment of this invention.

[0120] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0121] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0122] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0123] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0124] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0125] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0126] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for obtaining the pre-distortion mapping relationship of a near-eye display device, characterized in that, include: The camera acquires a calibration board image and a virtual image after center alignment from the camera's perspective. The calibration board image is the image acquired by the camera based on the chart on the physical calibration board. The near-eye display device to be distorted outputs a display image identical to the chart. The virtual image is the image acquired by the camera based on the virtual image formed by the optical system of the near-eye display device after the display image passes through it. This virtual image is distorted relative to the calibration board image. Based on the characteristics of the image card, multiple sampling points on the calibration board screen are identified to obtain a sampling point array image; For the multiple sampling points identified in the sampling point array image, multiple virtual image points in the virtual image are determined that correspond one-to-one with the multiple sampling points. For each virtual image point, based on the offset between each virtual image point and the corresponding sampling point, the virtual image point is moved to coincide with the corresponding sampling point to determine the offset of the image source pixel coordinates of the virtual image point, and then the original image source pixel coordinates of the virtual image point and its corresponding pre-distorted image source pixel coordinates are determined. Based on the original image source pixel coordinates of multiple virtual image points and their corresponding pre-distortion image source pixel coordinates, a pre-distortion mapping relationship data matrix for the near-eye display device is generated.

2. The method according to claim 1, characterized in that, The cards are chessboard cards, cross cards, wireframe cards, or dotted cards.

3. The method according to claim 2, characterized in that, If the card is a chessboard pattern card, then the corner points of the chessboard are used as sampling points; If the chart is a cross-shaped chart, then the intersection point of the cross will be used as the sampling point; If the drawing is a wireframe drawing, then the corner points of the wireframe are used as sampling points; If the chart is a dotted chart, then the chart points are used as sampling points.

4. The method according to claim 2, characterized in that, The calibration board image is acquired by the camera at a distance D from the camera, where D represents the virtual image distance of the near-eye display device; and the size of the chart is L*H, where L is the width of the chart and H is the height of the chart, wherein L, H, and D satisfy the following formula: D=D V =D H Wherein, the distance between the physical calibration plate and the exit pupil of the camera lens is D. V D H D V D is the object distance corresponding to the half-vertical field of view angle. H The object distance corresponding to the half-horizontal field of view; FOV V The vertical field of view (FOV) of the near-eye display device is... H The horizontal field of view of the near-eye display device is denoted as .

5. The method according to claim 1, characterized in that, When acquiring the calibration plate image and the virtual image image, the center of the camera's aperture coincides with the center of the exit pupil of the near-eye display device.

6. The method according to claim 1, characterized in that, For each virtual image point, based on the offset between each virtual image point and its corresponding sampling point, the offset of the original image source pixel coordinates of the virtual image point is determined by moving the display image source to make the virtual image point coincide with the corresponding sampling point. This, in turn, determines the pre-distorted image source pixel coordinates corresponding to the original image source pixel coordinates of the virtual image point, including: S1. For the plurality of sampling points, select one sampling point that has not been selected from the plurality of sampling points, and execute step S2 for the sampling point; S2. Obtain the original image source pixel coordinates of the virtual image point corresponding to the sampling point, determine the deviation direction of the sampling point relative to the corresponding virtual image point, and adjust the position of the display image of the near-eye display device to shift the virtual image from the initial position along the deviation direction. When the shift distance between the moved virtual image point and the sampling point is 0, stop the adjustment and determine the total pixel shift of the display image during the image shift process. The sum of the original image source pixel coordinates of the virtual image point and the total pixel shift is determined as the pre-distorted image source pixel coordinates corresponding to the virtual image point. S3. Adjust the position of the displayed image of the near-eye display device to restore the virtual image to its initial position where its center is aligned with the center of the calibration board image, and return to step S1 until there are no unselected sampling points among the plurality of sampling points.

7. The method according to any one of claims 1-6, characterized in that, The step of generating the pre-distortion mapping relationship data matrix of the near-eye display device based on the original image source pixel coordinates of multiple virtual image points and their corresponding pre-distortion image source pixel coordinates includes: obtaining the pre-distortion mapping relationship of any image source pixel coordinate within the entire display image area through bilinear interpolation based on the pre-distortion mapping relationship of multiple virtual image points, thereby obtaining the pre-distortion mapping relationship data matrix; the pre-distortion mapping relationship data matrix includes all original image source pixel coordinates and their corresponding pre-distortion image source pixel coordinates.

8. A device for obtaining the pre-distortion mapping relationship of a near-eye display device, characterized in that, include: An image acquisition unit is used to acquire, using a camera, a calibration board image and a virtual image after center alignment from the camera's viewpoint; the calibration board image is the image acquired by the camera for the chart on the physical calibration board; the near-eye display device to be distorted outputs a display image identical to the chart, and the virtual image is the image acquired by the camera for the virtual image formed by the optical system of the near-eye display device after the display image passes through it, and the virtual image is distorted relative to the calibration board image; The identification unit is used to identify multiple sampling points of the calibration board image according to the characteristics of the image card, so as to obtain a sampling point array image; The determining unit is used to determine multiple virtual image points in the virtual image that correspond one-to-one with the multiple sampling points identified in the sampling point array image, and for each virtual image point, based on the offset between each virtual image point and the corresponding sampling point, to determine the offset of the image source pixel coordinates of the virtual image point by moving the display image of the near-eye display device to make the virtual image point coincide with the corresponding sampling point, thereby determining the original image source pixel coordinates of the virtual image point and its corresponding pre-distorted image source pixel coordinates; The generation unit is used to generate a pre-distortion mapping relationship data matrix of the near-eye display device based on the original image source pixel coordinates of multiple virtual image points and their corresponding pre-distortion image source pixel coordinates.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-7.

Images