Method and device for distortion calibration and anti-distortion image generation of VR display module

Abstract

The invention discloses a distortion calibration and anti-distortion image generation method and device for a VR display module. The method comprises the following steps: projecting a calibration graphic card onto a VR screen of the VR display module; according to the calibration graph card, internal parameters and distortion parameters of the combined camera are obtained through a calibration method; constructing a distortion model according to the internal reference and the distortion parameter; constructing a target anti-distortion mapping matrix according to the distortion model; and performing anti-distortion processing on an initial image according to the target anti-distortion mapping matrix to generate a target image. The method can improve the efficiency and precision of image generation, and can be widely applied to the technical field of virtual reality.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a method and device for distortion calibration and anti-distortion image generation of a VR display module. Background Art

[0002] In VR experience, users usually wear a head-mounted display (HMD), which projects images in front of the user's eyes to create a realistic virtual environment. However, due to factors such as the optical system of the VR headset, the shape of the lens, and the position of the user's eyes, image distortion and distortion may occur. These distortions and distortions will affect the user's sense of reality and comfort of the virtual environment and reduce the overall visual quality. At present, the traditional VR anti-distortion method used in VR systems relies on mathematical modeling of the VR lens model to simulate distortion, thereby adjusting and correcting the image so that it presents a shape and proportion that is more consistent with the real world in the user's field of view. However, the traditional anti-distortion method has the problems of high computational complexity, time delay, and impact on system performance. On the other hand, due to manufacturing and assembly errors, theoretical modeling analysis cannot fully simulate and solve the real distortion, and the distortion simulation results need to be manually adjusted in the later stage, resulting in low efficiency and insufficient accuracy in generating images. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a method and device for distortion calibration and anti-distortion image generation of a VR display module, so as to improve the efficiency and accuracy of image generation.

[0004] To achieve the above-mentioned purpose, an embodiment of the present invention provides a method for distortion calibration and anti-distortion image generation of a VR display module, the method comprising the following steps:

[0005] Project the calibration chart onto the VR screen of the VR display module;

[0006] According to the calibration chart, the intrinsic parameters and distortion parameters of the combined camera are obtained through a calibration method;

[0007] Constructing a distortion model according to the internal parameters and the distortion parameters;

[0008] According to the distortion model, constructing a target anti-distortion mapping matrix;

[0009] According to the target anti-distortion mapping matrix, the initial image is subjected to anti-distortion processing to generate a target image.

[0010] In some embodiments, before the step of obtaining the intrinsic parameters and distortion parameters of the combined camera by a calibration method according to the calibration chart, the following steps are also included:

[0011] Performing optical axis alignment operation on the VR display module and the industrial camera;

[0012] Performing distortion correction on the industrial lens of the industrial camera;

[0013] Wherein, the VR display module is provided with the VR screen and VR lens;

[0014] The combined camera includes the VR lens and the industrial camera;

[0015] The industrial camera includes the industrial lens and a camera body.

[0016] In some embodiments, obtaining the intrinsic parameters and distortion parameters of the combined camera by a calibration method according to the calibration chart includes the following steps:

[0017] Obtaining the physical coordinates of each corner point in the calibration chart on the VR screen;

[0018] Obtaining the pixel coordinates of each corner point;

[0019] Associating the pixel coordinates with the physical coordinates;

[0020] According to the pixel coordinates and the physical coordinates, the intrinsic parameters and the distortion parameters of the combined camera are calibrated by a calibration method to obtain the intrinsic parameters and the distortion parameters.

[0021] In some embodiments, constructing the distortion model according to the internal parameter and the distortion parameter comprises the following steps:

[0022] Performing distortion correction on the pixel coordinates of each corner point according to the intrinsic parameter and the distortion parameter;

[0023] Obtaining an average size of the two-dimensional code of the calibration chart in an image of an industrial camera;

[0024] According to the average size, obtaining the magnification of the pinhole imaging;

[0025] Obtaining the object distance through the magnification of the pinhole imaging;

[0026] A distortion model is constructed according to the internal parameter, the distortion parameter and the object distance.

[0027] In some embodiments, constructing a target anti-distortion mapping matrix according to the distortion model comprises the following steps:

[0028] Constructing a first anti-distortion mapping matrix according to the distortion model, wherein the first anti-distortion mapping matrix is ​​used to store a first mapping relationship between a horizontal coordinate of the initial image and a horizontal coordinate of the target image;

[0029] Constructing a second anti-distortion mapping matrix according to the distortion model, wherein the second anti-distortion mapping matrix is ​​used to store a second mapping relationship between the ordinate of the initial image and the ordinate of the target image;

[0030] The target anti-distortion mapping matrix includes the first anti-distortion mapping matrix and the second anti-distortion mapping matrix.

[0031] In some embodiments, the step of performing anti-distortion processing on the initial image according to the target anti-distortion mapping matrix to generate the target image comprises the following steps:

[0032] Obtaining, by means of the target coordinates of the target image, initial coordinates corresponding to the target coordinates in the initial image according to the first mapping relationship and the second mapping relationship of the target anti-distortion mapping matrix;

[0033] Assigning the grayscale value in the initial coordinates to the target coordinates to generate the target image;

[0034] Wherein, the initial coordinates include the horizontal coordinate of the initial image and the vertical coordinate of the initial image;

[0035] The target coordinates include a horizontal coordinate of the target image and a vertical coordinate of the target image.

[0036] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a distortion calibration and anti-distortion image generation device for a VR display module, the device comprising:

[0037] The first module is used to project the calibration chart onto the VR screen of the VR display module;

[0038] The second module is used to obtain the internal parameters and distortion parameters of the combined camera through a calibration method according to the calibration chart;

[0039] A third module is used to construct a distortion model according to the internal parameters and the distortion parameters;

[0040] The fourth module is used to construct a target anti-distortion mapping matrix according to the distortion model;

[0041] The fifth module is used to perform anti-distortion processing on the initial image according to the target anti-distortion mapping matrix to generate a target image.

[0042] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the distortion calibration and anti-distortion image generation method of a VR display module as described above.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the distortion calibration and anti-distortion image generation method of a VR display module as described above.

[0044] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the aforementioned method for distortion calibration and anti-distortion image generation of a VR display module.

[0045] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a method and device for distortion calibration and anti-distortion image generation of a VR display module, the scheme projects a calibration chart onto the VR screen of the VR display module; according to the calibration chart, the internal parameters and distortion parameters of the combined camera are obtained through the calibration method; according to the internal parameters and the distortion parameters, a distortion model is constructed, through which the same measurement pattern can simultaneously adapt to the situation that the VR screen and the VR lens optical axis are vertical or non-vertical, further reducing the complexity of the operation and improving the efficiency of generating images; according to the distortion model, a target anti-distortion mapping matrix is ​​constructed; according to the target anti-distortion mapping matrix, the initial image is subjected to anti-distortion processing to generate a target image. The embodiments of the present invention can improve the efficiency and accuracy of generating images. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 It is a flow chart of a method for distortion calibration and anti-distortion image generation of a VR display module provided by an embodiment of the present invention;

[0048] Figure 2is a schematic diagram of an AprilTag card provided by an embodiment of the present invention;

[0049] Figure 3a-3c is a schematic diagram of the original AprilTag card and its derivative cards provided by an embodiment of the present invention;

[0050] Figure 4 It is a schematic diagram of optical axis alignment of a VR display module and an industrial camera provided by an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of a cross BMW logo card provided by an embodiment of the present invention;

[0052] Figure 6 is a schematic diagram of a pinhole imaging model and an object-side normalized plane provided by an embodiment of the present invention;

[0053] Figure 7 It is a schematic diagram of the principle of obtaining a distorted virtual image by a user directly observing through a VR lens provided by an embodiment of the present invention;

[0054] Figure 8 It is a schematic diagram of the principle of using an industrial camera instead of human eyes to capture a VR virtual image provided by an embodiment of the present invention;

[0055] Fig. 9 It is a schematic diagram of the principle that a user sees a normal non-distorted grid virtual image after using the distortion model provided by an embodiment of the present invention;

[0056] Fig.10 is a schematic diagram of an image coordinate system and a pixel coordinate system provided by an embodiment of the present invention;

[0057] Fig.11 It is a schematic diagram of the principle of assigning the grayscale values ​​of corresponding points of the initial image to the anti-distorted image provided by an embodiment of the present invention;

[0058] Fig.12 is a schematic diagram of incomplete corner points after imaging provided by an embodiment of the present invention;

[0059] Fig.13 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention.

[0060] Reference numerals:

[0061] VR display module 100, VR screen 101, VR lens 102;

[0062] Industrial cameras 200;

[0063] Fixture 300, 6-axis adjustment platform 400, optical axis 500;

[0064] Distorted grid virtual image 601, non-distorted grid virtual image 602, distorted grid image 603, non-distorted grid image 604, anti-distorted grid image 605;

[0065] Human eye: 700. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention, and they are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0067] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the flow chart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned drawings may be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiment of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0068] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0070] Before describing the embodiments of the present invention in detail, some nouns and terms involved in the embodiments of the present invention are first described. The nouns and terms involved in the embodiments of the present invention are subject to the following explanations.

[0071] Virtual Reality (VR) technology is a computer-generated simulated environment that allows users to immerse themselves in and interact with the virtual world. In a VR experience, users usually wear a head-mounted display (HMD), which projects images in front of the user's eyes to create a realistic virtual environment.

[0072] At present, traditional anti-distortion methods have the problems of high computational complexity, time delay, and impact on system performance. On the other hand, due to manufacturing and assembly errors, theoretical modeling and analysis cannot fully simulate and solve the real distortion, and the distortion simulation results need to be manually adjusted in the later stage, resulting in low efficiency and insufficient accuracy in generating images in VR systems.

[0073] In view of this, if Figure 1 As shown, an embodiment of the present invention provides a method for distortion calibration and anti-distortion image generation of a VR display module, the method including but not limited to steps S100 to S500:

[0074] Step S100, projecting the calibration chart onto a VR screen of a VR display module;

[0075] Step S200, obtaining the intrinsic parameters and distortion parameters of the combined camera through a calibration method according to the calibration chart;

[0076] Step S300, constructing a distortion model according to the internal parameters and the distortion parameters;

[0077] Step S400, constructing a target anti-distortion mapping matrix according to the distortion model;

[0078] Step S500: performing anti-distortion processing on the initial image according to the target anti-distortion mapping matrix to generate a target image.

[0079] In step S100 of some embodiments, a calibration card having a ratio of the length and width of the VR screen of the VR display module is projected onto the VR screen of the VR display module, and the calibration card covers the entire VR screen, such as Figure 2 As shown in the figure, the AprilTag chart can be used as a VR lens distortion calibration chart. In addition, charts with coded information such as CCT (Circle Coded Target) and CCTag (ConcentircCircle Tag) can also be used.

[0080] In some embodiments, Figure 3a-3c As shown in , the feature corner point distribution can be increased by rotating and translating the AprilTag card, such as Figure 3aThe original AprilTag chart shown in the figure is rotated and translated without scaling to generate multiple derivative charts, among which the following can be generated: Figure 3b The derivative chart generated by rotating the original AprilTag chart as shown in the figure can also be generated as follows Figure 3c As shown, the derivative chart card generated by translating the original AprilTag chart card is projected onto the VR screen respectively, and then the image is captured by the camera to collect multiple images and then calibrate the distortion parameters. Optionally, the distortion parameters can be calibrated with a white chart card with a black background, or the distortion parameters of the red, green and blue channels can be calibrated with a red chart card with a black background, a green chart card with a black background, and a blue chart card with a black background.

[0081] Through two-dimensional transformation (rotation, translation), the distribution of feature corner points can be more sufficient, making the fitting effect of the distortion model better.

[0082] Before step S200 in some embodiments, the method for distortion calibration and anti-distortion image generation of the VR display module may further include steps S610 to S620:

[0083] Step S610, performing an optical axis alignment operation on the VR display module and the industrial camera;

[0084] Step S620, performing distortion correction on the industrial lens of the industrial camera.

[0085] In step S610 of some embodiments, before calibrating the intrinsic parameters and distortion parameters of the combined camera, it is necessary to align the optical axis 500 of the VR display module and the industrial camera. Figure 4 As shown, the VR display module 100 is fixed by a fixture 300, and the industrial camera 200 is installed on a 6-axis adjustment platform 400. Figure 5 The cross BMW mark card shown in the figure adjusts the posture of the 6-axis adjustment platform so that the central cross mark of the industrial camera itself is aligned with the cross BMW mark in the VR display module, and the grayscale of the cross BMW mark is symmetrical and uniform, so that the optical axis alignment can be completed. Optionally, the industrial lens of the industrial camera can also be an anthropomorphic eye lens, a wide-angle lens, or an ultra-wide-angle lens.

[0086] In step S620 of some embodiments, before calibrating the internal parameters and distortion parameters of the combined camera, the distortion of the industrial lens of the industrial camera is first calibrated and corrected, and its influence on the imaging process of the combined camera is removed, so that the distortion calibrated by the subsequent combined camera belongs only to the VR lens itself.

[0087] In some embodiments, the VR screen is regarded as a physical calibration chart, the VR display module is provided with a VR screen and a VR lens, the VR lens and the industrial camera are regarded as a combined camera, and the industrial camera includes an industrial lens and a camera body.

[0088] In some embodiments, step S200 may include but is not limited to steps S210 to S240:

[0089] Step S210, obtaining the physical coordinates of each corner point in the calibration chart on the VR screen;

[0090] Step S220, obtaining the pixel coordinates of each corner point;

[0091] Step S230, associating the pixel coordinates with the physical coordinates;

[0092] Step S240: Calibrate the intrinsic parameters and the distortion parameters of the combined camera according to the pixel coordinates and the physical coordinates through a calibration method to obtain the intrinsic parameters and the distortion parameters.

[0093] In some embodiments, when designing a calibration chart, Figure 2 As shown, the ratio of the pixel sizes of the AprilTag QR code and the small black square on the AprilTag card relative to the entire AprilTag card is known, that is, the pixel ratio of the AprilTag QR code and the small black square can be obtained.

[0094] In steps S210 to S240 of some embodiments, the actual physical size of the AprilTag QR code and the black small square can be calculated according to the pixel ratio of the AprilTag QR code and the black small square and the actual length and width of the VR screen. Select a corner point on the calibration card as the origin of the calibration plate coordinate system, and then calculate the physical coordinates (in millimeters) of each corner point on the calibration card on the VR screen. Use an industrial camera to capture the virtual image of the AprilTag card of the VR display module, and calculate the pixel coordinates of each AprilTag corner point. Associate the physical coordinates and pixel coordinates of the corner points, and calibrate the internal parameters and distortion parameters of the combined camera through the calibration method to obtain the internal parameters and distortion parameters (k1, k2, k3, p1, p2). Among them, k1, k2, k3 are radial distortion parameters, and p1 and p2 are tangential distortion parameters.

[0095] In some embodiments, step S300 may include but is not limited to steps S310 to S350:

[0096] Step S310, performing distortion correction on the pixel coordinates of each corner point according to the intrinsic parameter and the distortion parameter;

[0097] Step S320, obtaining the average size of the two-dimensional code of the calibration chart in the image of the industrial camera;

[0098] Step S330, obtaining the magnification of the pinhole imaging according to the average size;

[0099] Step S340, obtaining the object distance through the magnification of the pinhole imaging;

[0100] Step S350: construct a distortion model according to the internal parameters, the distortion parameters and the object distance.

[0101] In steps S310 to S350 of some embodiments, the extracted corner pixel coordinates are calibrated using the combined camera internal parameters and distortion parameters. After the virtual image of the AprilTag card of the VR display module is captured by an industrial camera, the average size s of the AprilTag QR code on the AprilTag card in the image of the industrial camera can be calculated. For example, if the actual physical size of the AprilTag QR code on the VR screen is S, the magnification can be obtained as:

[0102] β=s / S

[0103] The expression of image distance is:

[0104] v=f′ v *Δx′

[0105] According to the magnification and distance of the pinhole imaging, the object distance can be obtained as:

[0106] u=v / β

[0107] In the formula, β represents the magnification of pinhole imaging; s represents the average size of the AprilTag QR code on the image of the industrial camera; S represents the actual physical size of the AprilTag QR code on the VR screen; v represents the image distance; f′ x represents the main focal length in the horizontal direction (x-axis direction) of the image end; Δx′ represents the pixel spacing in the horizontal direction (x-axis direction) of the image end; and u represents the object distance. Figure 6 As shown, the distance from the VR screen to the optical center of the combined camera is the object distance u, and the distance from the optical center of the combined camera to the photosensitive chip of the industrial camera is the image distance v. Optionally, the distance between the normalized plane on the object end and the optical center of the combined camera is set to 1.

[0108] Generally, the image acquired by the industrial camera is distorted by using a distortion model to correct the original image to remove the distortion effect, but the human eye and brain cannot achieve distortion correction, such as Figure 7As shown, a distortion-free grid image 604 is projected onto the VR screen 101. Due to the distortion introduced by the VR lens 102, a distorted grid virtual image 601 is generated. The user's human eye 700 observes through the VR lens 102 and sees a deformed grid, which may be a grid with pincushion distortion or a grid with barrel distortion. Therefore, the anti-distortion method of the VR display module is to perform distortion correction processing on the image projected on the VR screen through a distortion model, which is called anti-distortion. The anti-distortion of the projected image and the distortion of the VR lens cancel each other out, so that the picture seen by the human eye is not distorted. As shown in FIG. Figure 8 As shown, an industrial camera 200 can be used to capture VR virtual images instead of human eyes, and the distortion parameters of the combined camera can be calculated through the distortion grid map 603 to obtain a distortion model. Fig. 9 As shown, the distortion model is used to perform anti-distortion processing on the original non-distortion grid image 604 to be projected, and an anti-distortion grid image 605 is obtained. After the VR screen 101 projects the anti-distortion grid image 605, the anti-distortion effect and the distortion of the VR lens itself offset each other, and the user can see a normal non-distortion grid virtual image 602.

[0109] Exemplarily, according to the intrinsic parameter, the distortion parameter and the object distance, the distortion model is constructed as follows:

[0110] x′=x*(1+k1+r 2 +k2*r 4 +k3*r 6 )+2*p1*x*y+p2*(r 2 +2*x 2 )

[0111] y'=y*(1+k1*r 2 +k2*r 4 +k3*r 6 )+2*p1*x*y+p2*(r 2 +2*y 2 )

[0112] In the formula,

[0113] Δx=W / w

[0114] Δy=H / h

[0115] f x =u / Δx

[0116] f y =u / Δy

[0117] u0=(w-1) / 2

[0118] v0=(h-1) / 2

[0119] x=(i-u0) / f x

[0120] y=(j-v0) / f y

[0121]

[0122] in addition,

[0123] i′=f x *x′+u0

[0124] j′=f y *y′+v0

[0125] Wherein, W represents the physical width of the VR screen; H represents the physical height of the VR screen; w represents the number of pixels of the width of the VR screen; h represents the number of pixels of the height of the VR screen; Δx represents the pixel spacing in the horizontal direction (x-axis direction) of the object end; Δy represents the pixel spacing in the vertical direction (y-axis direction); i represents the distance from the VR screen to the optical center of the combined camera, that is, the object distance; f x Represents the main focal length in the horizontal direction (x-axis direction) of the object end; f y Represents the main focal length in the vertical direction (y-axis direction); Fig.10 As shown, u0 represents the horizontal (x-axis) coordinate of the principal point in the (screen) pixel coordinate system; v0 represents the vertical (y-axis) coordinate of the principal point in the (screen) pixel coordinate system; x represents the horizontal (x-axis) coordinate of the image coordinate system on the normalized plane of the object side; y represents the vertical (y-axis) coordinate of the image coordinate system on the normalized plane of the object side; i represents the width coordinate of the (screen) pixel coordinate system after dedistortion; j represents the height coordinate of the (screen) pixel coordinate system after dedistortion; r represents the distance from a certain point in the image coordinate system on the normalized plane of the object side to the origin of the coordinates; x′ represents the horizontal (x-axis) coordinate of the image coordinate system on the normalized plane of the object side after dedistortion; y′ represents the vertical (y-axis) coordinate of the image coordinate system on the normalized plane of the object side after dedistortion; i′ represents the coordinate corresponding to i before dedistortion; j′ represents the coordinate corresponding to j before dedistortion.

[0126] In some embodiments, step S400 may include but is not limited to steps S410 to S420:

[0127] Step S410: constructing a first anti-distortion mapping matrix according to the distortion model, where the first anti-distortion mapping matrix is ​​used to store a first mapping relationship between the horizontal coordinate of the initial image and the horizontal coordinate of the target image;

[0128] Step S420: constructing a second anti-distortion mapping matrix according to the distortion model, where the second anti-distortion mapping matrix is ​​used to store a second mapping relationship between the ordinate of the initial image and the ordinate of the target image;

[0129] The target anti-distortion mapping matrix includes the first anti-distortion mapping matrix and the second anti-distortion mapping matrix.

[0130] In steps S410 to S420 of some embodiments, a first anti-distortion mapping matrix MapI and a second anti-distortion mapping matrix MapJ can be created in advance according to the distortion model, and the width and height of the first anti-distortion mapping matrix and the second anti-distortion mapping matrix are consistent with the pixel size of the VR screen, wherein the first anti-distortion mapping matrix stores the mapping relationship of the horizontal coordinate during anti-distortion, that is, the first mapping relationship between the horizontal coordinate of the initial image and the horizontal coordinate of the target image; the second anti-distortion mapping matrix stores the mapping relationship of the vertical coordinate during anti-distortion, that is, the second mapping relationship between the vertical coordinate of the initial image and the vertical coordinate of the target image.

[0131] In some embodiments, step S500 may include but is not limited to steps S510 to S520:

[0132] Step S510, obtaining the initial coordinates corresponding to the target coordinates in the initial image according to the first mapping relationship and the second mapping relationship of the target anti-distortion mapping matrix through the target coordinates of the target image;

[0133] Step S520, assigning the grayscale value in the initial coordinates to the target coordinates to generate the target image;

[0134] Wherein, the initial coordinates include the horizontal coordinate of the initial image and the vertical coordinate of the initial image;

[0135] The target coordinates include a horizontal coordinate of the target image and a vertical coordinate of the target image.

[0136] In steps S510 to S520 of some embodiments, Fig.11As shown, according to the first mapping relationship and the second mapping relationship of the target anti-distortion mapping matrix, the mapping relationship between the initial coordinates of the initial image and the target coordinates of the target image (i.e., the anti-distortion image) can be obtained. The corresponding initial coordinates in the initial image can be found from the mapping relationship by the target coordinates of the target image. The row and column coordinate points (i, j) of the anti-distortion image are traversed, and the row and column coordinate points (i, j) of each anti-distortion image are found one by one according to the corresponding mapping relationship. The corresponding coordinate point (i′, j′) in the initial image is assigned to the grayscale value (gray scale) of the corresponding coordinate point (i′, j′) in the initial image. If the mapped coordinate point is a floating point coordinate, the grayscale value can be obtained by the interpolation method. Through the distortion model and the target anti-distortion mapping matrix, the original distortion-free image to be projected is subjected to anti-distortion processing to obtain an anti-distortion image. After the anti-distortion image is projected on the VR screen, the user can see the normal distortion-free image through the VR display module.

[0137] An embodiment of the present invention also provides a distortion calibration and anti-distortion image generation device for a VR display module, which can implement the above-mentioned distortion calibration and anti-distortion image generation method for a VR display module. The device includes:

[0138] The first module is used to project the calibration chart onto the VR screen of the VR display module;

[0139] The second module is used to obtain the internal parameters and distortion parameters of the combined camera through a calibration method according to the calibration chart;

[0140] A third module is used to construct a distortion model according to the internal parameters and the distortion parameters;

[0141] The fourth module is used to construct a target anti-distortion mapping matrix according to the distortion model;

[0142] The fifth module is used to perform anti-distortion processing on the initial image according to the target anti-distortion mapping matrix to generate a target image.

[0143] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0144] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned method for distortion calibration and anti-distortion image generation of a VR display module is implemented. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0145] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0146] refer to Fig.13 , Fig.13 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0147] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0148] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes a distortion calibration and anti-distortion image generation method for a VR display module in an embodiment of the present invention;

[0149] Input / output interface 803, used to implement information input and output;

[0150] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0151] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0152] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0153] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned distortion calibration and anti-distortion image generation method of a VR display module.

[0154] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0155] The embodiment of the present invention further provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned method for distortion calibration and anti-distortion image generation of a VR display module.

[0156] In summary, the method and device for distortion calibration and anti-distortion image generation of a VR display module according to an embodiment of the present invention have the following advantages:

[0157] 1. The embodiment of the present invention uses a customized AprilTag chart as a VR lens distortion calibration chart, which can provide sufficiently rich feature corner points to make the distortion calibration more practical, and each AprilTag QR code of the AprilTag chart will have its own unique coding information, and each AprilTag QR code is bound to four ordered corner points (lower left corner, lower right corner, upper right corner, upper left corner). Therefore, the four corner points can be indexed by using the encoding of the AprilTag QR code, and the physical coordinates can be bound to each corner point through the above indexing method. After the industrial camera is imaged, a special AprilTag recognition algorithm is used to identify the valid AprilTag QR code, and the pixel coordinates of the corner points and the physical coordinates can be associated one by one. Since the corner points already have their own coordinate information, even if the corner points after imaging are incomplete due to various reasons (such as Fig.12 As shown in the figure, due to the limitation of the field of view aperture or local dark spots, the corner points after imaging are incomplete), and some corner points cannot be extracted. The pixel coordinates of the valid corner points can also be associated with the physical coordinates, which does not affect the original coordinate system and the reading of the corner point information, making the calibration have a certain degree of anti-interference. At the same time, the chart has a strong derivative ability. Through two-dimensional transformation (rotation, translation), multiple derivative charts can be generated to provide enough feature corner points, so that the fitting effect of the distortion model is better. In addition, when it is necessary to enrich the distribution of feature corner points by rotating or translating the chart, some corner points will run out of the original image range. At this time, even if these corner points are lost, the algorithm can detect and identify the remaining valid corner points.

[0158] 2. The embodiment of the present invention does not require pre-configuration of the VR screen size and the distance between the VR lens and the VR screen. It only needs to configure the length and width of the VR screen or the length and width ratio, thereby reducing the number of parameters that may deviate from the design value and the actual value.

[0159] 3. The embodiment of the present invention adopts a distortion model including radial distortion and tangential distortion, so that the same measurement pattern can simultaneously adapt to the situation where the optical axes of the VR screen and the VR lens are vertical or non-vertical, further reducing the complexity of the operation.

[0160] 4. The embodiment of the present invention regards the VR lens and the industrial camera as a combined camera, realizes the calibration of distortion parameters through the calculation method of the pinhole imaging model + polynomial distortion model, and also realizes the anti-distortion processing of the original image, which can effectively correct the image distortion and distortion appearing in the head display, and reduce the calculation complexity and delay while ensuring the image quality, so as to realize more accurate and faster image anti-distortion processing, so as to provide a better VR experience.

[0161] 5. Using an industrial camera to calibrate the distortion of the VR lens will inevitably introduce the distortion of the industrial lens of the industrial camera. In order to obtain the distortion of the VR lens itself, the distortion of the industrial lens of the industrial camera itself can be corrected by the image after imaging, and then the combined camera can be calibrated with the corrected image. The calibrated distortion parameters describe the distortion of the VR lens itself.

[0162] 6. The commonly used imaging distortion correction method uses the distortion model of the normalized plane at the image end and the camera internal parameters to correct image distortion. The embodiment of the present invention applies this idea to the anti-distortion processing of the image at the object end (i.e., the screen), that is, the image distortion correction method at the image end is applied to the anti-distortion of the image at the object end. The distortion model has the same geometric meaning at the object end and the image end of the pinhole imaging model, making the anti-distortion of the image at the object end feasible.

[0163] 7. Since the combined camera is composed of a VR lens and an industrial camera, the position of the optical center is unpredictable, and the object end image anti-distortion requires the object distance of the combined lens (VR lens and industrial lens), that is, the distance from the VR screen to the optical center of the combined camera. The embodiment of the present invention estimates the object distance of the combined lens using the geometric relationship of the pinhole imaging model.

[0164] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0165] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0166] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0167] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0168] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0169] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0170] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0171] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0172] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for distortion calibration and anti-distortion image generation of a VR display module, characterized in that: The following steps are involved: Project the calibration chart onto the VR screen of the VR display module; According to the calibration chart, the intrinsic parameters and distortion parameters of the combined camera are obtained through a calibration method; Constructing a distortion model according to the internal parameters and the distortion parameters; According to the distortion model, constructing a target anti-distortion mapping matrix; According to the target anti-distortion mapping matrix, the initial image is subjected to anti-distortion processing to generate a target image.

2. The method for distortion calibration and anti-distortion image generation of a VR display module according to claim 1, characterized in that: Before the step of obtaining the intrinsic parameters and distortion parameters of the combined camera by a calibration method according to the calibration chart, the method further includes the following steps: Performing optical axis alignment operation on the VR display module and the industrial camera; Performing distortion correction on the industrial lens of the industrial camera; Wherein, the VR display module is provided with the VR screen and VR lens; The combined camera includes the VR lens and the industrial camera; The industrial camera includes the industrial lens and a camera body.

3. The method for distortion calibration and anti-distortion image generation of a VR display module according to claim 1, characterized in that: The method of obtaining the intrinsic parameters and distortion parameters of the combined camera according to the calibration chart by a calibration method comprises the following steps: Obtaining the physical coordinates of each corner point in the calibration chart on the VR screen; Obtaining the pixel coordinates of each corner point; Associating the pixel coordinates with the physical coordinates; According to the pixel coordinates and the physical coordinates, the intrinsic parameters and the distortion parameters of the combined camera are calibrated by a calibration method to obtain the intrinsic parameters and the distortion parameters.

4. The method for distortion calibration and anti-distortion image generation of a VR display module according to claim 1, characterized in that: The step of constructing a distortion model according to the internal parameters and the distortion parameters comprises the following steps: Performing distortion correction on the pixel coordinates of each corner point according to the intrinsic parameter and the distortion parameter; Obtaining an average size of the two-dimensional code of the calibration chart in an image of an industrial camera; According to the average size, obtaining the magnification of the pinhole imaging; Obtaining the object distance through the magnification of the pinhole imaging; A distortion model is constructed according to the internal parameter, the distortion parameter and the object distance.

5. The method for distortion calibration and anti-distortion image generation of a VR display module according to claim 1, characterized in that: The step of constructing a target anti-distortion mapping matrix according to the distortion model comprises the following steps: Constructing a first anti-distortion mapping matrix according to the distortion model, wherein the first anti-distortion mapping matrix is ​​used to store a first mapping relationship between a horizontal coordinate of the initial image and a horizontal coordinate of the target image; Constructing a second anti-distortion mapping matrix according to the distortion model, where the second anti-distortion mapping matrix is ​​used to store a second mapping relationship between the ordinate of the initial image and the ordinate of the target image; The target anti-distortion mapping matrix includes the first anti-distortion mapping matrix and the second anti-distortion mapping matrix.

6. The method for distortion calibration and anti-distortion image generation of a VR display module according to claim 1, characterized in that: The step of performing anti-distortion processing on the initial image according to the target anti-distortion mapping matrix to generate a target image comprises the following steps: Obtaining, by means of the target coordinates of the target image, initial coordinates corresponding to the target coordinates in the initial image according to the first mapping relationship and the second mapping relationship of the target anti-distortion mapping matrix; Assigning the grayscale value in the initial coordinates to the target coordinates to generate the target image; Wherein, the initial coordinates include the horizontal coordinate of the initial image and the vertical coordinate of the initial image; The target coordinates include a horizontal coordinate of the target image and a vertical coordinate of the target image.

7. A distortion calibration and anti-distortion image generation device for a VR display module, characterized in that: include: The first module is used to project the calibration chart onto the VR screen of the VR display module; The second module is used to obtain the internal parameters and distortion parameters of the combined camera through a calibration method according to the calibration chart; A third module is used to construct a distortion model according to the internal parameters and the distortion parameters; The fourth module is used to construct a target anti-distortion mapping matrix according to the distortion model; The fifth module is used to perform anti-distortion processing on the initial image according to the target anti-distortion mapping matrix to generate a target image.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images