Calibration parameter acquisition method and display method of augmented reality equipment and storage medium

Abstract

The invention provides a calibration parameter acquisition method and display method of augmented reality equipment and a storage medium, calibration parameters comprise a first conversion relation between an imaging plane coordinate system of a virtual camera and a display screen coordinate system of the augmented reality equipment, and the acquisition method comprises the following steps: acquiring external parameters of an industrial camera for simulating human eyes, taking the external parameter of the industrial camera as the external parameter of the virtual camera; obtaining the internal reference of the virtual camera based on the internal reference of the industrial camera, the resolution of the display screen and the geometrical relationship between the imaging surface of the industrial camera and the imaging surface of the virtual camera; acquiring a second conversion relation between the imaging plane coordinate system of the industrial camera and the display screen coordinate system of the augmented reality equipment; and obtaining a first conversion relation based on the second conversion relation, the internal reference of the industrial camera and the internal reference of the virtual camera. In the method, the external parameter of the industrial camera is directly used as the external parameter of the virtual camera, so that the calculation process of the calibration parameter can be simplified.

Description

Technical Field

[0001] This application relates to the field of near-eye display technology, and in particular to a method for obtaining calibration parameters, a display method, and a storage medium for an augmented reality device. Background Technology

[0002] Augmented Reality (AR) glasses are head-mounted display devices that enhance a user's perception of the real world by overlaying computer-generated images and information onto their field of vision. AR glasses utilize augmented reality technology to combine virtual information with the real world, providing users with a richer and more interactive experience.

[0003] To ensure that the virtual objects observed by the user accurately align with the real world, the parameters of the target virtual camera used by the augmented reality device to render the displayed images need to be calibrated first. This aligns the virtual images rendered by the target virtual camera with the real world; in other words, it involves virtual-real calibration, or object spatial see-through (OST) calibration. The virtual-real calibration process involves modeling the display screen of the augmented reality device as the imaging plane of the target virtual camera used for rendering the displayed images, and calculating the intrinsic and extrinsic parameters of the target virtual camera.

[0004] Currently, the parameter calibration process for a target virtual camera typically involves first defining an initial virtual camera whose optical axis passes through the center of the virtual imaging plane. The intrinsic parameters of the target virtual camera are obtained through the pose relationship between the initial virtual camera and an industrial camera used to simulate the human eye, as well as the pose of the target virtual camera. The extrinsic parameters of the target virtual camera are then calculated using the extrinsic parameters of the industrial camera and the glasses themselves. This calibration method is complex, requiring the construction of an initial virtual camera before calculating the calibration parameters. Furthermore, the calculated calibration parameters depend on the relative rotation and displacement between the industrial camera and the initial virtual camera. However, during the mass production of AR glasses, defects may exist in the display screen during assembly, affecting the accuracy of the calculation of the relative rotation and displacement between the industrial camera and the initial virtual camera. This leads to inconsistencies between the calculated calibration parameters and the actual values. If the calibration parameters calculated using the above method are subsequently used for imaging, virtual objects of the same preset size will appear to be of different sizes and distances, resulting in poor imaging quality and severely impacting the consistency of AR glasses mass production. Summary of the Invention

[0005] This application provides a method for obtaining calibration parameters of an augmented reality device, a display method, and a storage medium. In this method, by directly using the extrinsic parameters of an industrial camera as the extrinsic parameters of a virtual camera, the calculation process in the virtual-real calibration process can be simplified, and errors caused by the assembly of the display screen on the augmented reality device can be avoided, thereby improving the subsequent imaging effect and improving the mass production consistency of the augmented reality device.

[0006] In a first aspect, this application provides a method for obtaining calibration parameters of an augmented reality device. The calibration parameters include a first transformation relationship between the imaging plane coordinate system of a virtual camera and the display screen coordinate system of the augmented reality device. The virtual camera is used to render the image on the display screen of the augmented reality device. The method includes: obtaining the extrinsic parameters of an industrial camera used to simulate the human eye, and using the extrinsic parameters of the industrial camera as the extrinsic parameters of the virtual camera; obtaining the intrinsic parameters of the virtual camera based on the intrinsic parameters of the industrial camera, the resolution of the display screen, and the geometric relationship between the imaging plane of the industrial camera and the imaging plane of the virtual camera; obtaining a second transformation relationship between the imaging plane coordinate system of the industrial camera and the display screen coordinate system of the augmented reality device; and obtaining the first transformation relationship based on the second transformation relationship, the intrinsic parameters of the industrial camera, and the intrinsic parameters of the virtual camera.

[0007] In this acquisition method, there is no need to construct an initial virtual camera. The extrinsic parameters of the industrial camera are directly used as the extrinsic parameters of the virtual camera, that is, the poses of the industrial camera and the virtual camera are kept consistent. The intrinsic parameters of the virtual camera are described using a pinhole camera model. Finally, the coordinate transformation relationship between the image of the virtual camera and the display screen of the augmented reality device is calculated, thus completing the virtual-real calibration. This simplifies the calibration parameter calculation process and avoids calibration errors caused by the assembly of the display screen on the augmented reality device. The calculated calibration parameters do not contain errors caused by the assembly of the display screen, improving the consistency between the calculated calibration parameters and the actual values, and improving the mass production consistency of the augmented reality device. Subsequent imaging using these calibration parameters can improve the imaging effect.

[0008] In some embodiments, the geometric relationship between the imaging surface of the industrial camera and the imaging surface of the virtual camera includes the height and width of the imaging surface of the virtual camera on the imaging surface of the industrial camera, the starting coordinates of the origin of the imaging plane coordinate system of the virtual camera in the imaging plane coordinate system of the industrial camera, the intrinsic parameters of the industrial camera include the focal length and the optical center coordinates of the industrial camera, the resolution includes the horizontal resolution and the vertical resolution, and the process of obtaining the intrinsic parameters of the virtual camera based on the intrinsic parameters of the industrial camera, the resolution of the display screen, and the geometric relationship between the imaging surface of the industrial camera and the imaging surface of the virtual camera includes: obtaining the focal length of the virtual camera based on the focal length, the width, the height, the horizontal resolution, and the vertical resolution; obtaining the optical center coordinates of the virtual camera based on the optical center coordinates, the width, the height, the horizontal resolution, the vertical resolution, and the starting coordinates of the industrial camera, and using the focal length and the optical center coordinates of the virtual camera as the intrinsic parameters of the virtual camera.

[0009] In this embodiment, by analyzing the relationship between the intrinsic parameters of the industrial camera and the intrinsic parameters of the virtual camera based on the pinhole model, the intrinsic parameters of the virtual camera can be easily calculated, thereby simplifying the computational complexity of the virtual camera's intrinsic parameters and reducing the time required to calculate them.

[0010] In some embodiments, obtaining the second transformation relationship between the imaging plane coordinate system of the industrial camera and the display screen coordinate system of the augmented reality device includes: obtaining a first image captured by the industrial camera of a first calibration image displayed on the display screen by the augmented reality device, wherein the first calibration image includes at least one feature pattern; determining first coordinate information of the feature pattern in the first image under the imaging plane coordinate system of the industrial camera, and second coordinate information of the feature pattern in the first calibration image under the display screen coordinate system of the augmented reality device; and obtaining the second transformation relationship based on the first coordinate information and the second coordinate information.

[0011] In this embodiment, a second transformation relationship is obtained by relating the first coordinate information of the feature pattern in the imaging plane coordinate system of the industrial camera to the second coordinate information in the display screen coordinate system of the augmented reality device. Furthermore, using a homography transformation matrix to describe the mapping relationship between the imaging plane of the industrial camera and the plane of the display screen improves the accuracy of the mapping relationship description.

[0012] In some embodiments, obtaining the first transformation relationship based on the second transformation relationship, the intrinsic parameters of the industrial camera, and the intrinsic parameters of the virtual camera includes: transforming the second transformation relationship based on the intrinsic parameters of the industrial camera to obtain a third transformation relationship between the display screen coordinate system and the camera coordinate system of the industrial camera; and transforming the third transformation relationship based on the intrinsic parameters of the virtual camera to obtain the first transformation relationship.

[0013] In this embodiment, the first transformation relationship can be calculated through the above coordinate transformation, simplifying the calculation process of the first transformation relationship.

[0014] In a second aspect, embodiments of this application provide a display method for an augmented reality device, characterized in that it includes: obtaining calibration parameters of the augmented reality device based on the acquisition method described in any one of the first aspects; obtaining an initial rendered image of the augmented reality device; mapping the initial rendered image according to the calibration parameters to obtain a target image; and presenting the target image on the display screen of the augmented reality device.

[0015] In this embodiment, the calibration parameters of the augmented reality device can be obtained by using the acquisition method provided in this application, which simplifies the calculation.

[0016] In some embodiments, mapping the initial rendered image according to the calibration parameters to obtain a target image includes: obtaining the distortion mapping relationship of the augmented reality device, wherein the distortion mapping relationship is used to correct the distortion of the initial rendered image; mapping the initial rendered image based on the distortion mapping relationship to obtain a distorted-mapped image; and performing homography transformation on the distorted-mapped image based on a first transformation relationship in the calibration parameters to obtain the target image.

[0017] In this embodiment, by using the distortion mapping relationship to map the initial rendered image, the distortion caused by the assembly of the display screen can be corrected, thereby improving the image imaging effect and user experience.

[0018] In some embodiments, obtaining the distortion mapping relationship of the augmented reality device includes: obtaining a second image obtained by an industrial camera used to simulate the human eye capturing a second calibration image displayed by the augmented reality device on the display screen, wherein the second calibration image is used to determine the display screen of the augmented reality device in the second image; and determining the mapping relationship between the actual coordinates of each pixel of the display screen in the second image in the display screen coordinate system of the augmented reality device and the ideal coordinates of each pixel of the display screen as the distortion mapping relationship of the augmented reality device.

[0019] In this embodiment, the mapping relationship is calculated in the above manner, and the initial rendered image can be mapped using this mapping relationship to correct the distortion caused by screen assembly and improve the subsequent imaging effect.

[0020] In some embodiments, determining the mapping relationship between the actual coordinates of each pixel of the display screen in the second image under the display screen coordinate system of the augmented reality device and the ideal coordinates of each pixel of the display screen as the distortion mapping relationship of the augmented reality device includes: determining a first vertical direction difference between the actual vertical direction coordinates and the corresponding ideal vertical direction coordinates of each point on the upper edge of the display screen under the imaging plane coordinate system of the industrial camera; determining a first relationship between the first vertical direction difference and the actual horizontal direction coordinates of each point on the upper edge of the display screen under the imaging plane coordinate system of the industrial camera; and determining the actual horizontal direction coordinates of each point on the lower edge of the display screen under the imaging plane coordinate system of the industrial camera. The system calculates a second vertical direction difference between the actual vertical coordinates and the corresponding ideal vertical coordinates; it then determines a second relationship between the second vertical direction difference and the actual horizontal coordinates of each point on the lower edge of the display screen in the imaging plane coordinate system of the industrial camera; it performs coordinate transformation on the first and second relationships to obtain a third relationship between the first vertical direction difference and the actual horizontal coordinates of each point on the upper edge of the display screen in the display screen coordinate system of the augmented reality device, and a fourth relationship between the second vertical direction difference and the actual horizontal coordinates of each point on the lower edge of the display screen; based on the vertical resolution of the display screen, the third relationship, and the fourth relationship, it performs linear interpolation to obtain the mapping relationship.

[0021] In this embodiment, a simple distortion correction method is provided. By calibrating the distortion of the display screen in the vertical direction caused by assembly, the image display effect after distortion correction can be improved.

[0022] Thirdly, embodiments of this application provide an augmented reality device, characterized in that it includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in either the first or second aspect.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method described in either the first or second aspect.

[0024] Fifthly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in the first or second aspect above. Attached Figure Description

[0025] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise stated, the figures in the drawings do not constitute a limitation on scale.

[0026] Figure 1 This is a schematic diagram of the hardware structure of a method for obtaining calibration parameters of an augmented reality device provided in an embodiment of this application;

[0027] Figure 2 This is a schematic diagram of the hardware structure of another method for obtaining calibration parameters of an augmented reality device provided in an embodiment of this application;

[0028] Figure 3 This is a flowchart of a method for obtaining calibration parameters of an augmented reality device provided in an embodiment of this application;

[0029] Figure 4 This is a schematic diagram of the geometric relationship between the imaging surface of an industrial camera and the imaging surface of a virtual camera in a method for obtaining calibration parameters of an augmented reality device provided in an embodiment of this application.

[0030] Figure 5 This is a schematic diagram of a calibration image provided in an embodiment of this application;

[0031] Figure 6 This is a flowchart of a display method provided in an embodiment of this application;

[0032] Figure 7 This is a schematic diagram of the display link of an augmented reality device provided in an embodiment of this application;

[0033] Figure 8 This is a schematic diagram of an ideal display screen provided in an embodiment of this application;

[0034] Figure 9 This is a schematic diagram of an actual display screen provided in an embodiment of this application;

[0035] Figure 10 This is a schematic diagram of the upper edge of a display screen provided in an embodiment of this application;

[0036] Figure 11This is a structural block diagram of the augmented reality device provided in the embodiments of this application. Detailed Implementation

[0037] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0038] To facilitate understanding of this application, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0039] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram, in some cases, they can be divided differently from those in the device. In addition, the terms "first" and "second" used herein do not limit the data or execution order, but only distinguish between identical or similar items with essentially the same function and effect.

[0040] In augmented reality devices, such as AR glasses, to ensure that the virtual objects observed by the user are accurately integrated with the real world, it is usually necessary to calibrate the parameters of the target virtual camera used by the AR device to render the displayed images; this is known as virtual-real calibration. The virtual-real calibration process typically involves modeling the AR glasses' display screen S1 as the imaging plane of the virtual camera and calculating the virtual camera's intrinsic and extrinsic parameters. This process usually involves characterizing the virtual camera as a standard pinhole camera model, projecting a checkerboard pattern or other calibration image onto the AR glasses' display screen S1, and then calculating the homography transformation relationship between the virtual camera and the industrial camera with respect to the display screen S1 based on the image displayed on S1. Based on this homography transformation, the intrinsic parameters of the virtual camera itself and the extrinsic parameters with respect to the industrial camera are calculated. The industrial camera is a pre-calibrated binocular camera that can be used to represent the position of the human eye; this industrial camera is also called the "human eye camera."

[0041] In the existing methods for calculating the calibration parameters of the target virtual camera, please refer to... Figure 1 and Figure 2First, define an initial virtual camera 11, or on-axis virtual camera. The principal optical axis of this initial virtual camera 11 is perpendicular to the virtual imaging plane, and the intersection of the principal optical axis of the initial virtual camera 11 and the virtual imaging plane is the exact center of the virtual imaging plane. Then, the intrinsic parameters of the initial virtual camera 11 are: The parameters for the focal length are: in, These are the horizontal and vertical coordinates of the optical center of the initial virtual camera 11 in the pixel coordinate system of the initial virtual camera 11, respectively. These are the focal lengths of the initial virtual camera 11 in the horizontal and vertical directions, respectively. The initial value can be set according to the viewing angle of the display screen S1 itself, so that... The initial values ​​are set. The coordinates of the projection points of the same 3D point p on the display screen S1 onto the initial virtual camera 11 and the industrial camera 12 satisfy the following relationship:

[0042]

[0043]

[0044] in, Here is the intrinsic parameter matrix of the initial virtual camera 11; the scalar λ is the scale factor, x v =(x v ,y v ,1.0) T The x-coordinate is the homogeneous pixel coordinate of the 3D point p in the imaging plane coordinate system of the initial virtual camera 11. e It is the homogeneous pixel coordinate of a 3D point p in the imaging plane coordinate system of the industrial camera 12, and the scalar d is... e n is the distance between the industrial camera 12 and the display screen S1. e R is the normal vector of the display screen S1 plane in the camera coordinate system of industrial camera 12. ve and t ve These are the 3x3 rotation matrix and 3x1 translation vector from industrial camera 12 to initial virtual camera 11, respectively.

[0045] The goal of virtual-real calibration is to calculate the intrinsic parameters of a target virtual camera 13 and the extrinsic parameters of the target virtual camera 13 with respect to the industrial camera 12, which can be used by the rendering engine to draw virtual images. The target virtual camera 13 is also an off-axis virtual camera. For example... Figure 2 As shown, the position of the target virtual camera 13 is the same as that of the industrial camera 12, and the rotational attitude of the target virtual camera 13 is consistent with that of the initial virtual camera 11. Formula (1) can be transformed to obtain the relationship shown in Formula (3) below:

[0046]

[0047] Where, n v It is the normal vector of the display screen S1 plane in the camera coordinate system of the initial virtual camera 11. After transformation, the relationship shown in formula (4) can be obtained:

[0048]

[0049] Therefore, the intrinsic parameter matrix of the target virtual camera 13 can be obtained.

[0050]

[0051] Since the rendering engine needs to draw the content corresponding to the left and right display screens S1 based on the coordinate system of the AR glasses body, that is, the extrinsic parameters of the target virtual camera 13 need to be rendered with the coordinate system of the AR glasses body or other sensors on the AR glasses body such as cameras and inertial measurement units as the reference system. Therefore, it is necessary to first calibrate the industrial camera 12 and the AR glasses body to obtain the extrinsic parameters of the industrial camera 12 and the AR glasses body, and then, through coordinate system transformation, obtain the extrinsic parameters between the left and right target virtual cameras 13 and the AR glasses body or other sensors on the AR glasses body, thereby completing the calculation process of the calibration parameters of the target virtual cameras.

[0052] However, the display screen S1 on the AR glasses is prone to distortion due to defects in the assembly process, resulting in a large error between the calculated relative rotation and displacement between the industrial camera 12 and the initial virtual camera 11 and the preset value, i.e., the rotation matrix R in formula (1). ve and displacement vector t ve The existence of a large error leads to inconsistencies between the calculated calibration parameters and the actual values. If the calibration parameters calculated under the above calibration method are used for imaging in the future, the actual observed virtual objects of the same preset size will be of different sizes and distances, resulting in poor imaging effects, which will seriously affect the mass production consistency of AR glasses.

[0053] To address the aforementioned technical problems, this application provides a method for obtaining calibration parameters for augmented reality (AR) devices, a display method, and a storage medium. In this method, there is no need to construct an initial virtual camera. Instead, the extrinsic parameters of an industrial camera are directly used as the extrinsic parameters of the virtual camera, ensuring that the poses of the industrial camera and the virtual camera remain consistent. The intrinsic parameters of the virtual camera are described using a pinhole camera model. Finally, the coordinate transformation relationship between the virtual camera's image and the AR device's display screen is calculated, thus completing the virtual-real calibration. This simplifies the calibration parameter calculation process and avoids calibration errors caused by the assembly of the display screen on the AR device. The calculated calibration parameters are free from errors caused by the display screen assembly, improving the consistency between the calculated calibration parameters and the actual values, thus improving the mass production consistency of the AR device. Subsequent imaging using these calibration parameters can improve the imaging effect.

[0054] In a first aspect, this application provides a method for obtaining calibration parameters of an augmented reality device. The calibration parameters include a first transformation relationship between the imaging plane coordinate system of a virtual camera and the display screen coordinate system of the augmented reality device. The virtual camera is used to render the image on the display screen of the augmented reality device.

[0055] Augmented reality devices include AR devices, Virtual Reality (VR) devices, Mixed Reality (MR) devices, and other head-mounted augmented reality display devices. AR devices can be AR glasses.

[0056] A virtual camera is a virtual camera used to render images on the display screen of an augmented reality device. It's important to note that this virtual camera is not actually a real camera; rather, it's a virtual camera constructed to simulate the image rendering process on the augmented reality device's display screen. The imaging plane coordinate system of a virtual camera is a two-dimensional Cartesian coordinate system established with the origin at a point on the imaging plane of the virtual camera (such as the upper left corner of the imaging plane).

[0057] Please see Figure 3 The method for obtaining this information includes the following steps:

[0058] Step S10: Obtain the extrinsic parameters of the industrial camera used to simulate the human eye, and use the extrinsic parameters of the industrial camera as the extrinsic parameters of the virtual camera.

[0059] The extrinsic parameters of an industrial camera refer to its attitude parameters in a reference coordinate system, such as the coordinate system of the augmented reality device (AR) or the coordinate system of the sensor on the AR device. These parameters can include rotation matrices and displacement vectors that characterize the camera's pose in that reference coordinate system. For example, in AR glasses, the extrinsic parameters of an industrial camera can be either the extrinsic parameters of the industrial camera relative to the AR glasses' coordinate system or the extrinsic parameters of the industrial camera relative to the coordinate system of a sensor (such as an inertial measurement unit) on the AR glasses.

[0060] The extrinsic parameters of a virtual camera refer to the attitude parameters of the virtual camera in a reference coordinate system, such as the coordinate system of the augmented reality device itself or the coordinate system of the sensor on the augmented reality device. These parameters may include rotation matrices and displacement vectors that characterize the pose of the virtual camera in that reference coordinate system. For example, in AR glasses, the extrinsic parameters of the virtual camera may be the extrinsic parameters of the virtual camera relative to the coordinate system of the AR glasses itself, or the extrinsic parameters of the virtual camera relative to the coordinate system of the sensor (such as the inertial measurement unit) on the AR glasses itself.

[0061] The extrinsic parameters of the industrial camera are used as the extrinsic parameters of the virtual camera. That is, the pose of the virtual camera in the reference coordinate system is kept consistent with the pose of the industrial camera in the reference coordinate system, so that the position vector and rotation matrix of the virtual camera in the reference coordinate system are consistent with the position vector and rotation matrix of the industrial camera in the reference coordinate system, respectively.

[0062] Step S20: Based on the intrinsic parameters of the industrial camera, the resolution of the display screen, and the geometric relationship between the imaging surface of the industrial camera and the imaging surface of the virtual camera, obtain the intrinsic parameters of the virtual camera.

[0063] Intrinsic parameters (IPMs) of an industrial camera refer to its internal parameters, used to describe its internal optical characteristics. These IPMs include the optical center coordinates and focal length. The focal length characterizes the camera's ability to focus light onto the sensor, while the optical center coordinates are the coordinates of the camera's optical center in the imaging plane coordinate system. They characterize the offset of the imaging plane relative to the optical center. Specifically, the IPMs can be obtained by calibrating the industrial camera using conventional calibration methods.

[0064] The intrinsic parameters of a virtual camera refer to the internal parameters of the virtual camera, used to describe the internal optical characteristics of the virtual camera. The intrinsic parameters of a virtual camera include the optical center coordinates and focal length. The focal length is used to characterize the ability of the virtual camera lens to focus light onto the sensor, and the optical center coordinates are the coordinates of the virtual camera's optical center in the virtual camera's imaging plane coordinate system. They can be used to characterize the offset of the virtual camera's imaging plane relative to the virtual camera's optical center.

[0065] After obtaining the intrinsic parameters of the industrial camera and the resolution of the display screen, the geometric relationship between the virtual camera and the industrial camera can be obtained based on the pinhole camera model. The intrinsic parameters of the virtual camera can then be calculated using this geometric relationship, the intrinsic parameters of the industrial camera, and the resolution of the display screen.

[0066] Step S30: Obtain the second transformation relationship between the imaging plane coordinate system of the industrial camera and the display screen coordinate system of the augmented reality device.

[0067] The imaging plane coordinate system of an industrial camera is a two-dimensional Cartesian coordinate system, typically, such as... Figure 4 As shown, the origin of the imaging plane coordinate system of the industrial camera is located at the upper left corner of the imaging plane S2 of the industrial camera. The two axes of the imaging plane coordinate system are the x1 axis and the y1 axis, where the positive direction of the x1 axis points to the right side of the imaging plane S2 of the industrial camera, and the positive direction of the y1 axis points to the bottom of the imaging plane S2 of the industrial camera. Typically, the imaging plane S2 of the industrial camera is a plane.

[0068] The display screen coordinate system of the augmented reality device is a two-dimensional rectangular coordinate system. Typically, the origin of the display screen coordinate system is located at the upper left corner of the display screen S1. The two axes of the display screen coordinate system are the x2 axis and the y2 axis, where the positive direction of the x2 axis points to the right side of the plane of the display screen S1, and the positive direction of the y2 axis points to the bottom of the plane of the display screen S1.

[0069] Specifically, an image can be displayed on the display screen S1, and the coordinate information of the image in the imaging plane coordinate system of the industrial camera can be obtained by taking a picture of the image using an industrial camera. The coordinate information of the image in the display screen coordinate system of the augmented reality device is then calculated to obtain the second transformation relationship.

[0070] Step S40: Based on the second transformation relationship, the intrinsic parameters of the industrial camera and the virtual camera, obtain the first transformation relationship.

[0071] After obtaining the second transformation relationship, the intrinsic parameters of the industrial camera, and the intrinsic parameters of the virtual camera, the corresponding coordinate transformation can be performed to obtain the first transformation relationship.

[0072] By calculating the extrinsic and intrinsic parameters of the virtual camera, the geometric relationship between the virtual camera and the real-world camera can be determined, thereby achieving precise alignment between the virtual and real worlds. This ensures that the position and posture of virtual objects in the user's field of vision are consistent with the geometric relationship of the real world, creating a more realistic and immersive experience. Furthermore, this calibration parameter acquisition method eliminates the need to construct an initial virtual camera. By directly using the extrinsic parameters of the industrial camera as the extrinsic parameters of the virtual camera, the pose of the industrial camera and the virtual camera are kept consistent. The intrinsic parameters of the virtual camera are described using a pinhole camera model. Finally, the coordinate transformation relationship between the virtual camera's image and the display screen of the augmented reality device is calculated, thus completing the virtual-real calibration. This simplifies the calibration parameter calculation process and avoids calibration errors caused by the assembly of the display screen on the augmented reality device. The calculated calibration parameters are free from errors caused by the display screen assembly, improving the consistency between the calculated calibration parameters and the actual values, and improving the mass production consistency of the augmented reality device. Subsequent imaging using these calibration parameters can improve the imaging effect.

[0073] In any optional embodiment, see [reference] Figure 4 The geometric relationship between the imaging plane of the industrial camera and the imaging plane of the virtual camera includes the height of the virtual camera's imaging plane S3 on the industrial camera's imaging plane S2, the width of the virtual camera's imaging plane S3 on the industrial camera's imaging plane S2, and the starting coordinates of the origin of the virtual camera's imaging plane coordinate system in the industrial camera's imaging plane coordinate system. The intrinsic parameters of the industrial camera include the focal length and optical center coordinates of the industrial camera. The resolution of the display screen includes horizontal resolution and vertical resolution. Step S20 includes the following steps: Step S21: Obtain the focal length of the virtual camera based on the industrial camera's focal length, width, height, horizontal resolution, and vertical resolution. Step S22: Obtain the optical center coordinates of the virtual camera based on the industrial camera's optical center coordinates, width, height, horizontal resolution, vertical resolution, and starting coordinates, using the virtual camera's focal length and optical center coordinates as the virtual camera's intrinsic parameters.

[0074] Typically, the imaging surface S3 of a virtual camera is a plane. After obtaining the intrinsic parameters of the industrial camera, the geometric relationship between the intrinsic parameters of the industrial camera and the intrinsic parameters of the virtual camera can be analyzed based on the camera pinhole model and the geometric relationship between the industrial camera and the virtual camera. In this way, the intrinsic parameters of the virtual camera can be calculated based on the aforementioned dimensions, starting coordinates, resolution, and the intrinsic parameters of the industrial camera.

[0075] Specifically, based on the camera pinhole model, it is possible to draw such as Figure 4 The geometric relationship between the display screen S1, the imaging surface S2 of the industrial camera, and the imaging surface S3 of the virtual camera is shown. Based on this geometric relationship, the intrinsic parameters of the virtual camera can be calculated using the following formulas:

[0076]

[0077]

[0078]

[0079] in, v f x The focal length of the virtual camera in the horizontal direction. v f y The focal length of the virtual camera in the vertical direction. e f x This refers to the focal length of the industrial camera in the horizontal direction. e f y This refers to the focal length of the industrial camera in the vertical direction. To display the horizontal resolution of the screen, To display the screen's vertical resolution, The height of the virtual camera's imaging surface on the industrial camera's imaging surface. This represents the width of the virtual camera's imaging surface on the industrial camera's imaging surface. e x0 is the starting coordinate of the origin of the virtual camera's imaging plane coordinate system in the horizontal direction within the industrial camera's imaging plane coordinate system. e y0 is the starting coordinate of the origin of the virtual camera's imaging plane coordinate system in the vertical direction within the industrial camera's imaging plane coordinate system. v c x Let be the coordinates of the optical center of the virtual camera in the horizontal direction within the imaging plane coordinate system of the virtual camera. v c y Let be the coordinates of the optical center of the virtual camera in the vertical direction within the imaging plane coordinate system of the virtual camera. e c x Let be the coordinates of the optical center of the industrial camera in the horizontal direction within the imaging plane coordinate system of the industrial camera. e c y Let be the coordinates of the optical center of the industrial camera in the vertical direction within the imaging plane coordinate system of the industrial camera.

[0080] In this embodiment, by analyzing the relationship between the intrinsic parameters of the industrial camera and the intrinsic parameters of the virtual camera based on the pinhole model, the intrinsic parameters of the virtual camera can be easily calculated, thereby simplifying the computational complexity of the virtual camera's intrinsic parameters and reducing the time required to calculate them.

[0081] In any optional embodiment, step S30 may include the following steps:

[0082] Step S31: Acquire a first image obtained by an industrial camera from a first calibration image displayed on a display screen of an augmented reality device, wherein the first calibration image includes at least one feature pattern.

[0083] The first calibration image includes at least one feature pattern. The shape of the feature pattern can be a circle, rectangle, or other regular or irregular shape, and its color can be a suitable color such as red or black. Specifically, for example... Figure 5 As shown, the first calibration image may include multiple dots arranged in an array.

[0084] Specifically, the augmented reality device is controlled to display a first calibration image on a screen, and then an industrial camera is controlled to take a picture, thereby obtaining the first image.

[0085] Step S32: Determine the first coordinate information of the feature pattern in the first image in the imaging plane coordinate system of the industrial camera, and the second coordinate information of the feature pattern in the first calibration image in the display screen coordinate system of the augmented reality device.

[0086] The first coordinate information includes the first coordinates of each feature pattern in the first calibration image in the imaging plane coordinate system of the industrial camera, and the second coordinate information includes the second coordinates of each feature pattern in the first calibration image in the display screen coordinate system. Figure 5 In the illustrated embodiment, the first coordinate information includes the first coordinates of the center of each circular spot in the imaging coordinate system of the industrial camera. e x i , e y i The second coordinate information includes the second coordinates of the center of each circular spot in the display screen coordinate system. s x i , s y i ), where i represents the number of the circular spot. The first image can be processed using existing image processing methods to obtain the first coordinate information and the second coordinate information.

[0087] Step S33: Based on the first coordinate information and the second coordinate information, obtain the second transformation relationship.

[0088] The second transformation relationship is the mapping relationship between the imaging plane of the industrial camera and the plane of the display screen. This mapping relationship can be understood as being characterized by a homography transformation matrix; that is, there exists a homography transformation matrix. This makes the following formula (5) true:

[0089]

[0090] Where λ1 is the scale factor. This represents the first transformation relationship. After obtaining the first and second coordinates, the solution can be obtained using the least squares method.

[0091] In this embodiment, a second transformation relationship is obtained by relating the first coordinate information of the feature pattern in the imaging plane coordinate system of the industrial camera to the second coordinate information in the display screen coordinate system of the augmented reality device. Furthermore, using a homography transformation matrix to describe the mapping relationship between the imaging plane of the industrial camera and the plane of the display screen improves the accuracy of the mapping relationship description, and using the least squares method to solve the second transformation relationship minimizes the overall reprojection error.

[0092] In any optional embodiment, step S40 includes the following steps: Step S41: Transform the second transformation relationship based on the intrinsic parameters of the industrial camera to obtain a third transformation relationship between the display screen coordinate system and the camera coordinate system of the industrial camera; Step S42: Transform the third transformation relationship based on the intrinsic parameters of the virtual camera to obtain a first transformation relationship.

[0093] Specifically, the first transformation relationship can be calculated according to the following formula (6):

[0094]

[0095] in, This is the second transformation relation. For the first transformation relation, K e K is an internal reference for industrial cameras. v This refers to the internal parameters of the virtual camera.

[0096] In this embodiment, the first transformation relationship can be calculated through the above coordinate transformation, which simplifies the calculation process of the first transformation relationship.

[0097] Secondly, this application provides a display method for an augmented reality device; please refer to [link to relevant documentation]. Figure 6 The display method includes:

[0098] Step S100: Obtain calibration parameters of the augmented reality device based on the acquisition method of any embodiment of the first aspect.

[0099] In this embodiment, the acquisition method has the same process and function as the acquisition method in any embodiment of the first aspect, and will not be described again here. The acquisition method provided by this application can obtain calibration parameters, such as a first transformation relationship, which can be used to represent the mapping relationship between the pixel coordinates of the display screen of the augmented reality device and the pixel coordinates of the imaging plane of the virtual camera.

[0100] Step S200: Obtain the initial rendered image of the augmented reality device.

[0101] Specifically, for augmented reality devices, their display chain is as follows: Figure 7 As shown, the system includes a rendering module 100 and a mapping and display module 200, which operate independently. The rendering engine 110 in the rendering module 100 generates textures at a specific frequency (typically 60Hz) based on the pose of the augmented reality device and the intrinsic and extrinsic parameters of the virtual camera. These textures are then stored in the cache area of ​​the memory 120. The mapping and display module 200 can subsequently display the corresponding image based on these textures. The initial rendered image of the augmented reality device refers to the texture generated by the rendering engine 110 in the rendering module 100. In this embodiment, a texture is defined as a two-dimensional image containing color and other visual attributes at each pixel coordinate.

[0102] Step S300: Map the initial rendered image according to the calibration parameters to obtain the target image.

[0103] Specifically, the mapper 210 in the mapping and display module 200 reads the corresponding initial rendered image in the memory 120 according to the timestamp, and maps the initial rendered image based on the calibration parameters to obtain the mapped image, which is the target image.

[0104] Step S400: Present the target image on the display screen of the augmented reality device.

[0105] Finally, the mapping and display module 200 of the augmented reality device will control the display screen 220 to display the target image.

[0106] In this embodiment, the calibration parameters of the augmented reality device are obtained by using the acquisition method provided in this application, which simplifies the calculation. Moreover, in this augmented reality device, the rendering module 100 and the mapping and display module 200 operate independently, which can improve the flexibility and efficiency of the system, allowing rendering and display to be performed in parallel, thereby reducing latency and improving the user experience.

[0107] Augmented reality (AR) devices are prone to distortion due to assembly defects. When distortion occurs, the virtual image displayed on the screen appears warped, causing users to perceive a "pendulum effect" or "distortion effect," negatively impacting the user experience. For example, when displaying a rectangle, ideally, the AR device's display screen should... Figure 8 As shown in the middle screen S11, in actual applications, when the display screen on the augmented reality device displays a rectangle, distortion will occur, and the actual display effect will be as follows. Figure 9 As shown in screen S12. Therefore, it is necessary to perform distortion mapping on the rendered image and correct its distortion.

[0108] To perform distortion correction on the rendered image, in any optional embodiment, step S300 includes step S310: obtaining the distortion mapping relationship of the augmented reality device, wherein the distortion mapping relationship is used to correct the distortion of the initial rendered image. Step S320: mapping the initial rendered image based on the distortion mapping relationship to obtain a distorted-mapped image. Step S330: performing homography transformation on the distorted-mapped image based on the first transformation relationship in the calibration parameters to obtain the target image.

[0109] The distortion mapping relationship can be pre-stored in the memory. In this way, the distortion mapping relationship can be obtained through the memory. After obtaining the initial rendered image, the mapper first uses the distortion mapping relationship to map the initial rendered image to obtain the distorted image, thereby correcting the distortion caused by the assembly of the display screen. Then, the first transformation relationship is used to perform homography transformation on the distorted image to obtain the target image.

[0110] In this embodiment, by mapping the initial rendered image using a distortion mapping relationship, distortions caused by assembly on the display screen can be corrected, improving image imaging quality and user experience. Furthermore, by performing homography transformation mapping on the rendered image using a second transformation relationship, it can be ensured that the image on the display screen adapts better, improving display quality and accuracy.

[0111] In some embodiments, obtaining the distortion mapping relationship of the augmented reality device includes step S311: obtaining a second image obtained by an industrial camera used to simulate the human eye capturing a second calibration image displayed on the display screen of the augmented reality device, the second calibration image being used to determine the edges of the display screen of the augmented reality device in the second image.

[0112] Specifically, the second calibration image can be a rectangle filled with white, and the four sides of the rectangle are the edges of the display screen. In this step, by making all pixels on the display screen display white and acquiring a second image taken by an industrial camera, the edges of the display screen can be detected based on the second calibration image in the second image.

[0113] Step S312: Determine the mapping relationship between the actual coordinates of each pixel point on the edge in the display screen coordinate system of the augmented reality device and the ideal coordinates of each pixel point on the edge as the distortion mapping relationship of the augmented reality device.

[0114] Next, after edge detection of the second image is completed, the relationship between the actual coordinates and ideal coordinates of each pixel on the edge of the display screen in the imaging plane coordinate system of the industrial camera can be obtained. After coordinate transformation, the mapping relationship between the actual coordinates and ideal coordinates of each pixel on the edge of the display screen in the display screen coordinate system of the augmented reality device can be obtained.

[0115] In this embodiment, the mapping relationship is calculated in the above manner, and the initial rendered image can be mapped using this mapping relationship to correct the distortion caused by screen assembly and improve the subsequent imaging effect.

[0116] By comparison Figure 8 and Figure 9 It was discovered that the display screen distortion of augmented reality devices exhibits a unique characteristic: horizontal straight lines on the ideal screen are distorted into downward-curving horizontal arcs, while vertical straight lines on the ideal screen remain straight after the distortion mapping. Figure 10 As shown, the ideal top edge of a display screen should be a straight line AB, but the actual edge of the display screen is a curve AB. This means that the distortion mainly occurs in the vertical direction of the screen coordinate system, while the distortion in the horizontal direction is relatively slight and can be ignored. Therefore, based on this premise, the vertical distortion of the screen can be calibrated and removed.

[0117] Specifically, in any optional embodiment, step S312 includes: Step S3121: Determining a first vertical direction difference between the actual vertical coordinates and the corresponding ideal vertical coordinates of each point on the upper edge of the display screen in the imaging plane coordinate system of the industrial camera. Step S3122: Determining a first relationship between the first vertical direction difference and the actual horizontal coordinates of each point on the upper edge of the display screen in the imaging plane coordinate system of the industrial camera. Step S3123: Determining a second vertical direction difference between the actual vertical coordinates and the corresponding ideal vertical coordinates of each point on the lower edge of the display screen in the imaging plane coordinate system of the industrial camera. Step S3124: Determining a second relationship between the second vertical direction difference and the actual horizontal coordinates of each point on the lower edge of the display screen in the imaging plane coordinate system of the industrial camera. Step S3125: Perform coordinate transformation on the first and second relationships to obtain a third relationship between the first vertical difference and the actual horizontal coordinates of each point on the upper edge of the display screen in the coordinate system of the augmented reality device, and a fourth relationship between the second vertical difference and the actual horizontal coordinates of each point on the lower edge of the display screen. Step S3126: Based on the vertical resolution of the display screen, the third relationship, and the fourth relationship, perform linear interpolation to obtain a mapping relationship.

[0118] Specifically, such as Figure 10As shown, after image processing of the second image, the actual edge (curve AB) and ideal upper edge (straight line AB) of the display screen can be obtained. By sampling the actual edge and ideal upper edge (straight line AB) of the display screen, the coordinate data (x, y, x) of multiple pixels on the upper edge of the display screen in the imaging plane coordinate system of the industrial camera can be obtained. e ,Δy e ), where x e Let Δy be the actual horizontal coordinate of the pixel on the upper edge in the imaging plane coordinate system of the industrial camera. e It is the coordinate difference between the actual pixel point (located on curve AB) and the ideal pixel point (located on straight line AB) in the vertical direction of the imaging plane coordinate system of the industrial camera, which is also the first vertical direction difference.

[0119] Next, after fitting the coordinate data, the first vertical difference Δy of each point on the upper edge of the display screen in the industrial camera image coordinate system can be obtained. e Regarding the actual horizontal coordinate x e The first relationship between them As shown in formula (7):

[0120]

[0121] Similarly, a second relationship can be obtained between the second vertical difference of the lower edge of the display screen and the corresponding actual horizontal coordinate in the industrial camera image coordinate system.

[0122] Next, by using the first relationship Second Relationship The third relationship between the first vertical difference of each point on the top edge of the display screen and the actual horizontal coordinates after coordinate transformation to the display screen coordinate system of the augmented reality device. The fourth relationship between the second vertical difference between points at the bottom edge of the display screen in the display screen coordinate system of the augmented reality device and the actual horizontal coordinate.

[0123] Finally, the mapping relationship Δy can be obtained by linear interpolation using the following formula (8). s :

[0124]

[0125] Where H is the vertical resolution of the display screen, x s Let y be the horizontal coordinate of a pixel in the initial rendered image. sLet x be the vertical coordinate of a pixel in the initial rendered image. The coordinates of the pixel in the initial rendered image can then be calculated using formula (8). s ,y s The coordinates (x) of the mapping point s ,y s +Δy s ).

[0126] In this embodiment, a simple distortion correction method is provided. By calibrating the distortion of the display screen in the vertical direction caused by assembly, the image display effect after distortion correction can be improved.

[0127] Thirdly, this application also provides an augmented reality device 30, please refer to [link to relevant documentation]. Figure 11 It illustrates the hardware structure of an augmented reality device capable of executing the calibration parameter acquisition and display methods provided in this application.

[0128] The augmented reality device 30 includes: at least one processor 31; and a memory 32 communicatively connected to the at least one processor 31. Figure 11 Taking a processor 31 as an example, the memory 32 stores instructions executable by the at least one processor 31. These instructions, when executed by the at least one processor 31, enable the at least one processor 31 to perform the aforementioned calibration parameter acquisition and display methods. The processor 31 and the memory 32 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0129] The memory 32, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the calibration parameter acquisition method and display method in the embodiments of this application. The processor 31 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 32, thereby implementing the calibration parameter acquisition method and display method described in the above method embodiments.

[0130] The memory 32 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the augmented reality device 30. Furthermore, the memory 32 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 32 may optionally include memory remotely located relative to the processor 31, and this remote memory may be connected to the augmented reality device 30 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0131] The one or more modules are stored in the memory 32, and when executed by the one or more processors 31, they execute the calibration parameter acquisition method and display method of the augmented reality device 30 in any of the above method embodiments.

[0132] The augmented reality device 30 described above can execute the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0133] This application also provides a computer-readable storage medium storing computer-executable instructions for causing an electronic device to execute the calibration parameter acquisition method or the display method of the augmented reality device provided in this application.

[0134] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0135] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0136] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0137] As an example, executable instructions can be deployed to execute on a single computing device (including devices such as smart terminals and servers), or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0138] This application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to execute the calibration parameter acquisition method for augmented reality devices or the display method for augmented reality devices provided in this application.

[0139] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for at least one computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for acquiring a calibration parameter of an augmented reality device, characterized in that, The calibration parameters include a first conversion relationship between an imaging plane coordinate system of a virtual camera and a display screen coordinate system of the augmented reality device, the virtual camera being used to render a picture of the display screen of the augmented reality device, and the obtaining method includes: obtaining an external parameter of an industrial camera used to simulate a human eye, and taking the external parameter of the industrial camera as an external parameter of a virtual camera; obtaining an internal parameter of the virtual camera based on an internal parameter of the industrial camera, a resolution of the display screen, and a geometric relationship between an imaging plane of the industrial camera and an imaging plane of the virtual camera; obtaining a second conversion relationship between the imaging plane coordinate system of the industrial camera and the display screen coordinate system of the augmented reality device; obtaining the first conversion relationship based on the second conversion relationship, the internal parameter of the industrial camera, and the internal parameter of the virtual camera.

2. The acquisition method of claim 1, wherein, The geometric relationship between the imaging plane of the industrial camera and the imaging plane of the virtual camera includes a height and a width of the imaging plane of the virtual camera on the imaging plane of the industrial camera, and a starting coordinate of an origin of an imaging plane coordinate system of the virtual camera in the imaging plane coordinate system of the industrial camera, the internal parameter of the industrial camera includes a focal length and a light center coordinate of the industrial camera, the resolution includes a horizontal resolution and a vertical resolution, and the obtaining of the internal parameter of the virtual camera based on the internal parameter of the industrial camera, the resolution of the display screen, and the geometric relationship between the imaging plane of the industrial camera and the imaging plane of the virtual camera includes: obtaining the focal length of the virtual camera based on the focal length of the industrial camera, the width, the height, the horizontal resolution, and the vertical resolution; obtaining the light center coordinate of the virtual camera based on the light center coordinate of the industrial camera, the width, the height, the horizontal resolution, the vertical resolution, and the starting coordinate, and taking the focal length of the virtual camera and the light center coordinate of the virtual camera as the internal parameter of the virtual camera.

3. The acquisition method of claim 1, wherein, The obtaining of the second conversion relationship between the imaging plane coordinate system of the industrial camera and the display screen coordinate system of the augmented reality device includes: obtaining a first image captured by the industrial camera for a first calibration image displayed in the display screen by the augmented reality device, wherein the first calibration image includes at least one feature pattern; determining first coordinate information of the feature pattern in the first image in the imaging plane coordinate system of the industrial camera, and second coordinate information of the feature pattern in the first calibration image in the display screen coordinate system of the augmented reality device; obtaining the second conversion relationship based on the first coordinate information and the second coordinate information.

4. The acquisition method according to any one of claims 1 to 3, characterized in that, The obtaining of the first conversion relationship based on the second conversion relationship, the internal parameter of the industrial camera, and the internal parameter of the virtual camera includes: transforming the second conversion relationship based on the internal parameter of the industrial camera to obtain a third conversion relationship between the display screen coordinate system and a camera coordinate system of the industrial camera; Transform the third conversion relationship based on an intrinsic parameter of the virtual camera to obtain the first conversion relationship.

5. A display method of an augmented reality device, characterized by, The method comprises: obtaining the calibration parameters of the augmented reality device based on the method of any one of claims 1-4; obtaining an initial rendering image of the augmented reality device; mapping the initial rendering image according to the calibration parameters to obtain a target image; presenting the target image on a display screen of the augmented reality device.

6. The display method according to claim 5, wherein The method of mapping the initial rendering image according to the calibration parameters to obtain a target image comprises: obtaining a distortion mapping relationship of the augmented reality device, wherein the distortion mapping relationship is used for distortion correction of the initial rendering image; mapping the initial rendering image based on the distortion mapping relationship to obtain a distortion-mapped image; performing a homography transformation on the distortion-mapped image based on a first conversion relationship in the calibration parameters to obtain the target image.

7. The display method according to claim 6, wherein The method of obtaining the distortion mapping relationship of the augmented reality device comprises: obtaining a second image of a second calibration image displayed by the augmented reality device on the display screen, the second calibration image being used to determine the display screen of the augmented reality device in the second image; determining a mapping relationship between actual coordinates of each pixel point of the display screen in the second image in a display screen coordinate system of the augmented reality device and ideal coordinates of each pixel point of the display screen as the distortion mapping relationship of the augmented reality device.

8. The display method according to claim 7, wherein The method of determining a mapping relationship between actual coordinates of each pixel point of the display screen in the second image in a display screen coordinate system of the augmented reality device and ideal coordinates of each pixel point of the display screen as the distortion mapping relationship of the augmented reality device comprises: determining a first vertical direction difference between actual vertical direction coordinates of each point on the upper edge of the display screen in an imaging plane coordinate system of the industrial camera and corresponding ideal vertical direction coordinates; determining a first relationship between the first vertical direction difference and actual horizontal direction coordinates of each point on the upper edge of the display screen in the imaging plane coordinate system of the industrial camera; determining a second vertical direction difference between actual vertical direction coordinates of each point on the lower edge of the display screen in the imaging plane coordinate system of the industrial camera and corresponding ideal vertical direction coordinates; determining a second relationship between the second vertical direction difference and actual horizontal direction coordinates of each point on the lower edge of the display screen in the imaging plane coordinate system of the industrial camera; performing coordinate conversion on the first relationship and the second relationship to obtain a third relationship between the first vertical direction difference and actual horizontal direction coordinates of each point on the upper edge of the display screen in a display screen coordinate system of the augmented reality device, and a fourth relationship between the second vertical direction difference and actual horizontal direction coordinates of each point on the lower edge of the display screen; performing linear interpolation based on the vertical resolution of the display screen, the third relationship and the fourth relationship to obtain the mapping relationship.

9. An augmented reality device, characterized by The method comprises: at least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method of any one of claims 1 to 8.

Images