Method, device, equipment, system and storage medium for calibrating augmented reality equipment

Abstract

The embodiments of the present application disclose a method, apparatus, device, system and storage medium for calibrating an augmented reality device, which includes: obtaining a first transformation parameter between a first three-dimensional coordinate system where an observation sensor is located and a second three-dimensional coordinate system where a target camera is located, the target camera being located within the corresponding eye movement range of the augmented reality device; generating a first calibration map having the same resolution parameters as the imaging screen; obtaining the first pixel coordinates of each calibration point in the first calibration map; sending the first calibration map to the corresponding imaging screen for display; obtaining a second calibration map obtained after the target camera shoots the imaging screen and determining the second pixel coordinates of the calibration point in the second calibration map; obtaining the calibration result of the augmented reality device based on the first pixel coordinates, the second pixel coordinates and the first transformation parameter. The above method can solve the technical problems of low production efficiency and large manual workload caused by manual participation in the calibration process in the related art.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of augmented reality technology, and in particular to a method, apparatus, device, system, and storage medium for calibrating an augmented reality device. Background Art

[0002] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. With the development of AR technology, various augmented reality devices (AR devices for short) have appeared in people's daily lives, such as AR glasses, AR helmets, etc. The goal of AR technology is to overlay the virtual world on the real world on the screen of the AR device and interact with it. For example, when a user uses an AR device, he or she can see the real world through the screen of the AR device. Figure 1 The content shown, at this time, the user can view the real world and virtual information at the same time ( Figure 1 The words "real world" and "virtual information" shown are used to assist in understanding the screen content, and the user will not see the aforementioned words when watching.)

[0003] The core of AR technology is to use two-dimensional or three-dimensional objects in the real world as markers to align virtual information with real-world information. This means that the position, size, and movement path of virtual objects perfectly match the real environment, achieving a state of virtual and real coexistence. Generally speaking, by calibrating the AR device, it is possible to accurately overlay virtual information with real-world objects.

[0004] In the related art, AR devices are usually calibrated manually. Specifically, the user wears the AR device and moves, and during the movement, the correspondence between a specific point in the real space and a pixel point on the screen is obtained. Alternatively, a mouse is connected to the AR device, and after the user wears the AR device, the user selects a pixel point on the screen of the AR device with the mouse, and then obtains the correspondence between the pixel point and the specific point in the real space corresponding to the pixel point. Afterwards, based on the correspondence, the virtual information and the objects in the real world can be correctly superimposed together, and the correspondence can be considered as the calibration result of the AR device. However, the above calibration process requires manual participation, so it is impossible to achieve automatic calibration of the AR device, which reduces the production efficiency of the AR device and increases the manual workload. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, device, system, and storage medium for calibrating an augmented reality device to solve the technical problems in the related art of low production efficiency and high manual workload caused by manual participation in the calibration process.

[0006] In a first aspect, an embodiment of the present application provides a method for calibrating an augmented reality device, wherein the augmented reality device includes: an imaging screen and an observation sensor;

[0007] The method comprises:

[0008] Obtaining a first transformation parameter between a first three-dimensional coordinate system where the observation sensor is located and a second three-dimensional coordinate system where the target camera is located, where the target camera is a left camera or a right camera, and the left camera and the right camera are respectively located within a left eye activity range and a right eye activity range of the augmented reality device;

[0009] generating a first calibration map, wherein the first calibration map and the imaging screen have the same resolution parameters;

[0010] Obtaining first pixel coordinates of each calibration point in the first calibration image;

[0011] Sending the first calibration map to an imaging screen corresponding to the target camera, so that the imaging screen displays the first calibration map;

[0012] Obtaining a second calibration image obtained after the target camera captures the imaging screen, and determining a second pixel coordinate of the calibration point in the second calibration image;

[0013] Based on the first pixel coordinates, the second pixel coordinates and the first transformation parameters, a calibration result of the augmented reality device is obtained, and the calibration result is the projection matrix required when the three-dimensional coordinate points in the first three-dimensional coordinate system are mapped to the two-dimensional coordinate system where the imaging screen is located.

[0014] In a second aspect, an embodiment of the present application further provides an apparatus for calibrating an augmented reality device, wherein the augmented reality device comprises: an imaging screen and an observation sensor;

[0015] The device comprises:

[0016] a first acquisition unit, configured to acquire a first transformation parameter between a first three-dimensional coordinate system where the observation sensor is located and a second three-dimensional coordinate system where the target camera is located, wherein the target camera is a left camera or a right camera, and the left camera and the right camera are respectively located within an activity range of a left eye and an activity range of a right eye of the augmented reality device;

[0017] a generating unit, configured to generate a first calibration map, wherein the first calibration map has the same resolution parameters as the imaging screen;

[0018] a second acquiring unit, configured to acquire first pixel coordinates of each calibration point in the first calibration image;

[0019] a sending unit, configured to send the first calibration map to an imaging screen corresponding to the target camera, so that the imaging screen displays the first calibration map;

[0020] a third acquiring unit, configured to acquire a second calibration image obtained after the target camera shoots the imaging screen, and determine a second pixel coordinate of the calibration point in the second calibration image;

[0021] A result determination unit is used to obtain a calibration result of the augmented reality device based on the first pixel coordinates, the second pixel coordinates and the first transformation parameters, wherein the calibration result is the projection matrix required when the three-dimensional coordinate points in the first three-dimensional coordinate system are mapped to the two-dimensional coordinate system where the imaging screen is located.

[0022] In a third aspect, an embodiment of the present application further provides a device for calibrating an augmented reality device, wherein the augmented reality device includes: an imaging screen and an observation sensor;

[0023] The device comprises:

[0024] one or more processors;

[0025] a memory for storing one or more programs;

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for calibrating an augmented reality device as described in the first aspect.

[0027] In a fourth aspect, an embodiment of the present application further provides a system for calibrating an augmented reality device, comprising: an augmented reality device, a target camera, and the device for calibrating the augmented reality device according to the third aspect; the augmented reality device comprises: an imaging screen and an observation sensor; the target camera is a left camera or a right camera, and the left camera and the right camera are respectively located within the left eye activity range and the right eye activity range of the augmented reality device;

[0028] The device obtains a first transformation parameter between a first three-dimensional coordinate system where the observation sensor is located and a second three-dimensional coordinate system where the target camera is located;

[0029] The device generates a first calibration map, wherein the first calibration map has the same resolution parameters as the imaging screen;

[0030] The device obtains first pixel coordinates of each calibration point in the first calibration image;

[0031] The device sends the first calibration image to an imaging screen corresponding to the target camera;

[0032] The imaging screen displays the first calibration image;

[0033] The target camera shoots the imaging screen to obtain a second calibration image;

[0034] The device obtains the second calibration image and determines a second pixel coordinate of the calibration point in the second calibration image;

[0035] The device obtains a calibration result of the augmented reality device based on the first pixel coordinates, the second pixel coordinates and the first transformation parameters. The calibration result is the projection matrix required when the three-dimensional coordinate points in the first three-dimensional coordinate system are mapped to the two-dimensional coordinate system where the imaging screen is located.

[0036] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for calibrating an augmented reality device as described in the first aspect.

[0037] In one embodiment of the present application, a target camera is placed in the eye movement range of the user when using an AR device to simulate the human eye viewing the imaging screen, and a first calibration map with the same resolution parameters as the imaging screen is generated and displayed on the imaging screen. Then, a second calibration map obtained by shooting the first calibration map with the target camera is obtained. Then, based on the first pixel coordinates of the calibration point in the first calibration map, the second pixel coordinates in the second calibration map, and the first transformation parameters between the observation sensor and the target camera, a technical means is used to obtain the projection matrix required for mapping the three-dimensional coordinate point in the first three-dimensional coordinate system where the observation sensor is located to the two-dimensional coordinate system where the imaging screen is located. This solves the technical problems of low production efficiency and high manual workload caused by manual participation in the calibration process. By simulating the human eye with the target camera, automatic calibration of the AR device is achieved, which reduces manual workload and improves the production efficiency of the AR device. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of imaging content of an imaging screen of an AR device in the prior art;

[0039] Figure 2 A schematic diagram of AR glasses provided in one embodiment of the present application;

[0040] Figure 3 A schematic diagram of imaging content of an uncalibrated imaging screen provided by one embodiment of the present application;

[0041] Figure 4 A schematic diagram of imaging content of an imaging screen after calibration provided in one embodiment of the present application;

[0042] Figure 5 A schematic diagram of a fixing frame provided in accordance with an embodiment of the present application;

[0043] Figure 6 A flowchart of a method for calibrating an AR device provided in one embodiment of the present application;

[0044] Figure 7 A schematic diagram of a ChArUco calibration diagram provided in one embodiment of the present application;

[0045] Figure 8 A schematic diagram of a second calibration diagram provided in one embodiment of the present application;

[0046] Figure 9 A schematic diagram of a coaxial projection provided in one embodiment of the present application;

[0047] Figure 10 A schematic diagram of a virtual space provided for one embodiment of the present application;

[0048] Figure 11 A schematic diagram of a coordinate system provided for one embodiment of the present application;

[0049] Figure 12 A schematic diagram of another coordinate system provided for one embodiment of the present application;

[0050] Figure 13 A top view of a coordinate system provided in one embodiment of the present application;

[0051] Figure 14 A schematic diagram of imaging content of another calibrated imaging screen provided in one embodiment of the present application;

[0052] Figure 15 A schematic diagram of imaging content of another calibrated imaging screen provided in one embodiment of the present application;

[0053] Figure 16 A schematic diagram of the structure of an apparatus for calibrating an AR device provided in one embodiment of the present application;

[0054] Figure 17 A schematic diagram of the device structure for calibrating an AR device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0055] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended to explain the present application, not to limit the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present application, not all structures.

[0056] An embodiment of the present application provides a method for calibrating an augmented reality device (referred to as an AR device for short). The AR device may be AR glasses or an AR helmet, etc. As a personal mobile computing platform, the AR device has an imaging screen, and the imaging screen adopts an optically transmissive helmet display. At this time, the imaging screen can be considered as a translucent optical screen. On the one hand, the imaging screen is similar to an ordinary eye lens, which can pass through the external ambient light to allow the user to see the real world in front of them (also recorded as the real world). On the other hand, the imaging screen can display the virtual image generated by the AR device. The human eye can see the real world superimposed with the virtual image through the imaging screen. Optionally, two imaging screens of the same size and resolution can be integrated on the AR device, one for the user's left eye and the other for the right eye to watch, or one imaging screen can be integrated on the AR device, and the imaging screen is divided into two areas of the same size, one for the user's left eye and the other for the right eye to watch.

[0057] The AR device also has an observation sensor, which is used to perform functions such as posture estimation, environmental perception, and object tracking of the AR device. That is, by observing the real world through the observation sensor, the AR device can clearly generate what kind of virtual image and the display position of the virtual image on the imaging screen. The observation sensor can be a camera, a depth lens (also called a TOF camera), a lidar, etc. The observation sensor is generally located near the imaging screen, and the number of observation sensors can be set according to actual needs. In one embodiment, the AR device is described as augmented reality glasses (i.e., AR glasses) as an example, reference Figure 2 , AR glasses are AR devices in the form of glasses, Figure 2 The AR glasses shown include two imaging screens 14 and are provided with three observation sensors 15, which are respectively located between the two imaging screens, on the left side of the left imaging screen, and on the right side of the right imaging screen (when a user uses the AR glasses, the user's left side (i.e., the position of the left eye) corresponds to the left imaging screen, and the user's right side (i.e., the position of the right eye) corresponds to the right imaging screen). It should be noted that Figure 2 The shape, structure, number of imaging screens, number of observation sensors and position of the AR glasses are only used for illustrative purposes and do not limit the shape, structure, number of imaging screens, number of observation sensors and position of the AR device in this disclosure.

[0058] It is understandable that AR devices may also include processors and other devices for data storage, operation, and imaging.

[0059] When using an AR device, there is a virtual space (used to generate virtual images) and a real space (i.e., the real world) in the AR device. Correctly superimposing the virtual space and the real space is the main purpose of calibrating the AR device. In one embodiment, calibrating the AR device specifically refers to obtaining the geometric relationship between the three-dimensional coordinate system used by the observation sensor to observe the real world and the two-dimensional coordinate system where the imaging screen is located, so that based on this geometric relationship, the virtual image generated according to the observation result of the observation sensor can be accurately superimposed with the real world viewed through the imaging screen. For example, Figure 3 It is the content viewed by one eye of the user through the imaging screen when it is not calibrated. Figure 4 It is the content viewed by one eye of the user through the imaging screen after calibration. Figure 3 and Figure 4 In the image processing, the object in the real world viewed through the imaging screen is a chessboard. After the observation sensor observes the chessboard, it generates a virtual image composed of points and rectangles. Figure 3 In the game, the points of the virtual image and the cuboid cannot coincide with the corners of the checkerboard. Figure 4 In the example, the points of the virtual image and the cuboid coincide with the corner points of the checkerboard. Figure 3 and Figure 4 It can be seen that the user experience of using a calibrated AR device is significantly better than that of using an uncalibrated AR device.

[0060] In one embodiment, when calibrating an AR device, a camera is used to simulate human eye imaging. In this case, the camera simulating the left eye is referred to as the left camera. The left camera is located within the left eye's range of motion to simulate the left eye viewing the imaging screen. The camera simulating the right eye is referred to as the right camera. The right camera is located within the right eye's range of motion to simulate the right eye viewing the imaging screen. The images captured by the left and right cameras can simulate the images on the left and right retina, respectively.

[0061] In one embodiment, reference Figure 5 , the AR device 11( Figure 5 Taking AR glasses as an example), the left camera 12 and the right camera are both placed on the fixing frame 13 for fixing ( Figure 5 (This is a side view after being fixed and the left camera can be seen). The structure and material of the fixing frame and the method of fixing the camera and AR device are not currently limited. Figure 5 The AR device 11 and the left camera 12 are placed on the fixing frame 13 for illustrative purposes only, and the devices and their relative positional relationship are not limited thereto.

[0062] In one embodiment, the AR device, the left camera, and the right camera are all connected to a control device (optionally a wired connection). The control device can be implemented by software and / or hardware. The control device can be composed of two or more physical entities, or it can be composed of one physical entity. The embodiment does not limit this. In one embodiment, the control device can be a computer device such as a laptop computer or a desktop computer for performing data analysis and calculations. It is understandable that the AR device can also be used as a control device. In this case, the left camera and the right camera are both connected to the AR device. Exemplarily, the control device is a device for calibrating the AR device (that is, the control device is a device for calibrating the augmented reality device), which is used to execute the method for calibrating the AR device provided in the embodiment.

[0063] In one embodiment, for ease of understanding, capital letters are used to represent points in three-dimensional space, and lowercase letters are used to represent points in two-dimensional space, and the coordinate system to which the corresponding point belongs is shown in the upper left corner of the letter. For example, World X represents point X in the World coordinate system, which is a three-dimensional world coordinate system.

[0064] Generally speaking, a three-dimensional coordinate point in a three-dimensional coordinate system can be mapped to another three-dimensional coordinate system through transformation parameters, where the transformation parameters represent the spatial transformation relationship between the two three-dimensional coordinate systems. The transformation parameters are composed of a rotation matrix and a displacement vector (also called a translation vector). In this case, the transformation parameters can be recorded as T = [R|t], where T represents the transformation parameters, R represents the rotation matrix, which is a 3×3 matrix, and t represents the displacement vector, which is a 3×1 vector. For example, the transformation relationship between three-dimensional coordinate system a and three-dimensional coordinate system b can be expressed as:

[0065]

[0066] in, a X represents a point in the three-dimensional coordinate system a, b X represents a The corresponding point of X in the three-dimensional coordinate system b, Represents the transformation parameters used to transform from three-dimensional coordinate system a to three-dimensional coordinate system b, represents the rotation matrix required to transform from three-dimensional coordinate system a to three-dimensional coordinate system b, Represents the displacement vector required to transform from three-dimensional coordinate system a to three-dimensional coordinate system b.

[0067] Figure 6 This is a flowchart of a method for calibrating an AR device provided in one embodiment of the present application, refer to Figure 6 , the method comprising:

[0068] Step 110: Obtain a first transformation parameter between a first three-dimensional coordinate system where the observation sensor is located and a second three-dimensional coordinate system where the target camera is located, where the target camera is a left camera or a right camera, and the left camera and the right camera are respectively located within the left eye activity range and the right eye activity range of the augmented reality device.

[0069] In one embodiment, when calibrating the AR device, the calibration of the simulated left eye is completed based on the left camera and an observation sensor in the AR device, and the calibration of the simulated right eye is completed based on the right camera and an observation sensor in the AR device. When the AR device has multiple observation sensors, the calibration of the simulated left eye and the calibration of the simulated right eye can use the same observation sensor or different observation sensors. Currently, the same observation sensor is used as an example for explanation. It can be understood that the calibration of the simulated left eye and the calibration of the simulated right eye can be performed simultaneously or separately. When performed separately, the order of the two calibrations is not limited. Currently, the calibration of the simulated left eye and the calibration of the simulated right eye have the same calibration method. Therefore, in the embodiment, a monocular is used as an example to describe the method of calibrating the AR device, and the camera used in the current calibration is recorded as the target camera, that is, the target camera can be the left camera or the right camera.

[0070] In one embodiment, the observation sensor is a camera. In this case, the observation sensor is a sensor that uses a camera, and the observation sensor and the target camera form a binocular vision system. The binocular vision system is composed of two cameras (currently the observation sensor and the target camera) located at different positions to simulate two eyes.

[0071] Currently, the three-dimensional coordinate system where the observation sensor is located is recorded as the first three-dimensional coordinate system, and the first three-dimensional coordinate system is expressed as the Tracker coordinate system (can also be recorded as {Tracker}). The Tracker coordinate system is the three-dimensional coordinate system used when the observation sensor observes the real world (that is, shoots the real world), and its origin is determined based on the position of the observation sensor. The three-dimensional coordinate system where the target camera is located is recorded as the second three-dimensional coordinate system, and the second three-dimensional coordinate system is recorded as the Eye coordinate system (can also be recorded as {Eye}). The Eye coordinate system is the three-dimensional coordinate system used when the target camera shoots, and its origin is determined based on the position of the target camera. After the observation sensor and the target camera form a binocular vision system, the transformation parameters from the Tracker coordinate system to the Eye coordinate system can be obtained according to the calibration method of the binocular vision system. Currently, the transformation parameters are recorded as the first transformation parameters. At this time, the transformation process from the Tracker coordinate system to the Eye coordinate system can be expressed as:

[0072]

[0073] in,Tracker X represents the point in the Tracker coordinate system, Eye X represents Tracker The corresponding point when X is mapped to the Eye coordinate system, represents the first transformation parameter, represents the rotation matrix of the first transformation parameters, A displacement vector representing the first transformation parameter.

[0074] In addition, during calibration, the internal parameter matrix of the target camera can also be obtained. Currently, the internal parameter matrix of the target camera is recorded as the first internal parameter matrix, and the first internal parameter matrix is ​​recorded as Eye K, Eye K is a 3×3 matrix, which contains the physical parameters of the target camera. It can be understood that based on the principle of pinhole imaging, according to Eye K can project each point in the second three-dimensional coordinate system into the two-dimensional coordinate system of the image captured by the target camera. The two-dimensional coordinate system used in the image captured by the target camera is denoted as the Image_Eye coordinate system (also denoted as {Image_Eye}). Optionally, the origin of the Image_Eye coordinate system is the upper left corner pixel of the image. In this case, the projection process can be expressed as:

[0075] λ Image_Eye x= Eye K Eye X (3)

[0076] in, Eye X represents the point in the Eye coordinate system, Image_Eye x represents Eye The corresponding point when X is projected into the Image_Eye coordinate system, λ represents the scale factor, which can be any scalar.

[0077] It should be noted that there are many ways to calibrate the binocular vision system, which are not described or limited here. It is understandable that when the observation sensor uses other types of sensors, other calibration methods can be used to obtain the first transformation parameters and the first intrinsic parameter matrix.

[0078] Step 120: Generate a first calibration image, where the first calibration image has the same resolution parameters as the imaging screen.

[0079] The calibration map is an image used for calibration. The calibration map contains multiple calibration points, and the calibration points can be considered as relatively obvious feature points in the calibration map. In one embodiment, the calibration map is composed of multiple square grids of the same size. In this case, the calibration map can be a checkerboard calibration map, an ArUco calibration map, or a ChArUco calibration map. Among them, the checkerboard calibration map is composed of black and white square grids, and a square grid can be considered as a checkerboard grid. The ArUco calibration map is also a square grid map, and an ArUco code is displayed in each square grid. The ChArUco calibration map is a combination of a checkerboard calibration map and an ArUco calibration map. It is also composed of black and white square grids, and the ArUco code is displayed inside the white square grid. For example, Figure 7 The ChArUco calibration diagram is shown, which consists of black and white square grids, and the ArUco code is displayed in the white square grid. Figure 7 The calibration points in the figure are corner points, which are the intersection points of the sides of the square grid (intersection points of non-edges), and can also be understood as the intersection points between any two white square grids. In one embodiment, the corner points of the non-edge square grids in the calibration diagram (i.e., the vertices of the squares) are used as calibration points.

[0080] Currently, a calibration map is generated by the control device, and the calibration map is recorded as the first calibration map and represented as I Screen The first calibration image and the imaging screen have the same resolution parameters. Optionally, when the target camera is the left camera, the imaging screen currently used during the calibration process refers to the imaging screen viewed by the left eye. When the target camera is the right camera, the imaging screen currently used during the calibration process refers to the imaging screen viewed by the right eye. For example, the resolution parameter provided by the manufacturer of the imaging screen is w screen ×h screen Among them, w screen Indicates the horizontal width of the imaging screen (i.e., the resolution of the width), h screen represents the height of the imaging screen in the vertical direction (i.e., the resolution of the height), w screen and h screen The units are all pixels. After the control device obtains the resolution parameters, it generates a w screen ×h screen The first calibration map, and the width of each square grid in the first calibration map is w grid pixels.

[0081] Step 130: Obtain the first pixel coordinates of each calibration point in the first calibration image.

[0082] After the first calibration image is generated, a two-dimensional coordinate system can be obtained based on the first calibration image. The origin of the two-dimensional coordinate system is the pixel point in the upper left corner of the first calibration image. The X-axis and Y-axis of the two-dimensional coordinate system are respectively parallel to the two edges of the first calibration image. At this time, each pixel point in the first calibration image has a two-dimensional coordinate (also recorded as pixel coordinates) in the two-dimensional coordinate system. For example, based on this two-dimensional coordinate system, the pixel coordinates of each calibration point in the first calibration image can be obtained and recorded as the first pixel coordinates.

[0083] In one embodiment, pixels representing calibration points in the first calibration image are detected, and the pixel coordinates of the pixels are obtained as first pixel coordinates. The method used to detect the calibration points is not currently limited. For example, a corner detection algorithm can be used to detect each corner point in the first calibration image and then obtain the first pixel coordinates of each corner point. The corner detection algorithm is an implemented detection algorithm and is not described here.

[0084] Step 140: Send the first calibration image to an imaging screen corresponding to the target camera, so that the imaging screen displays the first calibration image.

[0085] Exemplarily, the control device sends the first calibration map to the AR device, and then the AR device sends the first calibration map to the imaging screen for display. It can be understood that when the target camera is the left camera, the imaging screen is a screen for viewing by the left eye, and when the target camera is the right camera, the imaging screen is a screen for viewing by the right eye. Since the first calibration map and the imaging screen have the same resolution, when the imaging screen displays the first calibration map, the pixel coordinates of each pixel in the first calibration map are equal to the two-dimensional coordinates of the pixel in the imaging screen. In one embodiment, a two-dimensional coordinate system can be obtained based on the imaging screen. Currently, the two-dimensional coordinate system is recorded as the Image_Screen coordinate system (also recorded as {Image_Screen}), and the origin of the Image_Screen coordinate system is the upper left pixel of the imaging screen (or the upper left pixel among the pixels of the first calibration map displayed on the imaging screen). Each pixel in the imaging screen has a two-dimensional coordinate (also recorded as pixel coordinates) in the Image_Screen coordinate system. At this time, the pixel coordinates of each calibration point in the first calibration image in the Image_Screen coordinate system are its own first pixel coordinates. In one embodiment, the first pixel coordinates are expressed as Image_Screen x i,j , i = 1, ..., M, M represents the total number of rows of corner points, j = 1, ..., N, N represents the total number of columns of corner points, Image_Screen x i,j It can be the first pixel coordinate of the corner point in the i-th row and j-th column.

[0086] Step 150: Obtain a second calibration image obtained after the target camera shoots the imaging screen, and determine the second pixel coordinates of the calibration point in the second calibration image.

[0087] For example, after the imaging screen displays the first calibration image, the control device instructs the target camera to shoot the imaging screen, or the user controls the target camera to shoot the imaging screen. Afterwards, the control device obtains the image captured by the target camera. In one embodiment, the image is recorded as the second calibration image and represented as I Eye The second calibration image includes the first calibration image displayed on the imaging screen. Optionally, the imaging screen is covered when shooting the second calibration image to avoid the influence of the real world on the imaging effect of the first calibration image when shooting the second calibration image. The covering method is not currently limited. For example, a black cloth is used to cover the side of the imaging screen away from the target camera. Figure 8 , which is the second calibration image captured by the target camera after being covered with a black cloth. The background of the second calibration image is black and includes the first calibration image displayed on the imaging screen.

[0088] Exemplarily, the two-dimensional coordinate system where the second calibration image is located is the Image_Eye coordinate system, and each pixel point in the second calibration image has a two-dimensional coordinate (which can also be recorded as a pixel coordinate) in the Image_Eye coordinate system. Exemplarily, the pixel coordinates of each calibration point in the second calibration image can be determined according to the Image_Eye coordinate system and recorded as the second pixel coordinates. Optionally, the second pixel coordinates are determined in the same way as the first pixel coordinates. For example, the pixel points representing the corner points in the second calibration image are detected by a corner detection algorithm, and then the pixel coordinates of the pixel points are obtained as the second pixel coordinates of the calibration points. At this time, each corner point has a first pixel coordinate and a second pixel coordinate. In one embodiment, the second pixel coordinate is expressed as Image_ Eye x i,j , i = 1, ..., M, M represents the total number of rows of corner points, j = 1, ..., N, N represents the total number of columns of corner points, Image_Eye x i,j It can be the second pixel coordinate of the corner point in the i-th row and j-th column.

[0089] Step 160: Obtain a calibration result of the augmented reality device based on the first pixel coordinates, the second pixel coordinates, and the first transformation parameter. The calibration result is a projection matrix required when mapping the three-dimensional coordinate points in the first three-dimensional coordinate system to the two-dimensional coordinate system where the imaging screen is located.

[0090] For example, the calibration result of the AR device is the projection matrix required when the three-dimensional coordinate points in the first three-dimensional coordinate system are mapped to the two-dimensional coordinate system where the imaging screen is located, that is, the transformation relationship required when the Tracker coordinate system is mapped to the Image_Screen coordinate system. It can be understood that when the target camera is the left camera, the calibration result is the projection matrix corresponding to the imaging screen used by the left eye, and when the target camera is the right camera, the calibration result is the projection matrix corresponding to the imaging screen used by the right eye. In one embodiment, the projection matrix is ​​recorded as At this time, the process of mapping the three-dimensional coordinate point in the first three-dimensional coordinate system to the two-dimensional coordinate system where the imaging screen is located can be expressed as:

[0091]

[0092] in, Tracker X represents the point in the Tracker coordinate system, Image_Screen x represents Tracker X is the corresponding point when it is mapped to the Image_Screen coordinate system, and λ represents the scale factor, which can be any scalar. It can be understood that when using AR devices, the observation sensor observes the real world and obtains the real object. Tracker X, then the AR device gets the corresponding Image_Screen x, and on the imaging screen Image_Screen A virtual image is displayed at x. The target camera takes a picture of the imaging screen and obtains an image. Tracker X and Image_Screen x is located at the same position to achieve accurate superposition of virtual images with the real world.

[0093] In one embodiment, the projection matrix is ​​determined by the first pixel coordinate, the second pixel coordinate and the first transformation parameter. Exemplarily, a three-dimensional coordinate system can also be obtained based on the imaging screen. Currently, the three-dimensional coordinate system is recorded as the third three-dimensional coordinate system and expressed as the Screen coordinate system (also recorded as {Screen}). Each pixel point in the imaging screen has a corresponding three-dimensional coordinate in the Screen coordinate system. The origin of the Screen coordinate system is the upper left corner pixel point of the first calibration image in the imaging screen. The X-axis and Y-axis of the Screen coordinate system are respectively parallel to the two sides of the first calibration image and are both on the plane of the imaging screen (physical plane). Afterwards, the Z-axis of the Screen coordinate system is perpendicular to the plane where the imaging screen is located and can be considered as the normal vector of the plane where the imaging screen is located. At this time, the three-dimensional coordinates of the calibration point in the Screen coordinate system can be determined according to the first pixel coordinate of the calibration point and recorded as the first three-dimensional coordinate of the calibration point. Then, based on the first three-dimensional coordinates and the second pixel coordinates, the transformation parameters between the screen coordinate system and the eye coordinate system can be determined. Then, the transformation parameters between the screen coordinate system and the eye coordinate system and the transformation parameters between the eye coordinate system and the tracker coordinate system can be determined. Afterwards, Sent to the AR device for application by the AR device. It is understandable that a projection matrix obtained based on the left camera and the observation sensor, and a projection matrix obtained based on the right camera and the observation sensor, both need to be sent to the AR device for use by the AR device. When the AR device includes multiple observation sensors, there are also transformation parameters between the three-dimensional coordinate systems where the observation sensors are located. At this time, combined with the projection matrix obtained by the AR device and the transformation parameters between the observation sensors, it is possible to achieve accurate superposition of the virtual world and the real world on the corresponding imaging screen.

[0094] In one embodiment, a virtual camera is introduced when determining the projection matrix. The virtual camera is a virtual camera, and the imaging plane of the virtual camera is the plane where the imaging screen is located. Among them, the virtual camera exists behind the imaging screen (that is, the side where the target camera is located). A three-dimensional coordinate system can be obtained based on the virtual camera. Currently, the three-dimensional coordinate system of the virtual camera is recorded as the fourth three-dimensional coordinate system and expressed as the Render coordinate system (also recorded as {Render}). At this time, the Render coordinate system can be understood as the three-dimensional coordinate system used in the virtual space of the AR device. The virtual camera can shoot the virtual space to form an image and display the imaging result on the imaging screen, that is, the imaging screen is equivalent to the imaging plane of the virtual camera. At this time, the first calibration map displayed on the imaging screen can be simulated as the imaging content of the virtual camera in the virtual space.

[0095] The virtual camera also has a corresponding internal parameter matrix. Currently, the internal parameter matrix of the virtual camera is recorded as the second internal parameter matrix and expressed as Render K, Render K is a 3×3 matrix. In one embodiment, reference Figure 9 , the virtual camera and the imaging screen are on-axis projection. Render K describes an ideal 3D to 2D pinhole imaging model consisting of the render coordinate system and the plane of the imaging screen. The Z axis of the render coordinate system is perpendicular to the plane of the imaging screen and intersects with the imaging screen at the center of the imaging screen (i.e., the Z axis of the fourth 3D coordinate system is perpendicular to the imaging screen and intersects with the center point of the imaging screen). The resolution parameter of the imaging screen is w screen ×h screen When the Z axis of the Render coordinate system is Intersect, where [] T represents the transposed matrix. At this time, Render K can be expressed as:

[0096]

[0097] Among them, w screen and h screen The meaning of hfov is the horizontal field of view angle parameter of the virtual camera (or imaging screen), and vfov is the vertical field of view angle parameter of the virtual camera (or imaging screen). The units of hfov and vfov are degrees.

[0098] Based on the pinhole imaging principle, Render K can map each point in the Render coordinate system to the Image_Screen coordinate system where the imaging screen is located. At this time, the projection process can be expressed as:

[0099] λ Image_Screen x= Render KR ender X (6)

[0100] in, Render X represents the point in the Render coordinate system. Image_Screen x represents Render The corresponding point when X is mapped to the Image_Screen coordinate system, λ represents the scale factor, which can be any scalar. It is understood that the scale factor currently used can be the same as or different from the scale factor used in formula (3). Figure 10, which is a schematic diagram of a virtual camera in Unity (a game engine) shooting a virtual space. After shooting, the virtual image displayed in the imaging plane (i.e., the imaging screen) can be obtained by formula (6). The virtual image is displayed on Figure 10 The window in the lower left corner. Understandably, Figure 10 The "Z axis", "{Render}", "Imaging plane" and "I Screen ” is for explanation purposes only and does not actually appear in the virtual space.

[0101] After the introduction of the virtual camera, the coordinate system involved in calibrating the AR device is as follows Figure 11 In order to facilitate the understanding of each coordinate system, Figure 11 The world coordinate system (denoted as World) is shown in the figure, and the physical object ( Figure 11 The actual object has a checkerboard pattern), refer to Figure 11 , the observation sensor has the Tracker coordinate system, the target camera has the Eye coordinate system and the Image_Eye coordinate system, the imaging screen has the Image_Screen coordinate system and the Screen coordinate system, and the virtual camera has the Render coordinate system. Figure 11 It can be seen that the transformation parameters from the Screen coordinate system to the Render coordinate system can be determined based on the first pixel coordinate and the first three-dimensional coordinate. Then, the projection matrix can be obtained by combining the transformation parameters from the Screen coordinate system to the Eye coordinate system, the transformation parameters from the Tracker coordinate system to the Eye coordinate system, the projection relationship between the Render coordinate system and the Image_Screen coordinate system, and the projection relationship between the Eye coordinate system and the Image_Eye coordinate system. In one embodiment, after the virtual camera is introduced, step 160 includes steps 161 to 166:

[0102] Step 161: Obtain the first three-dimensional coordinates of the calibration point in the third three-dimensional coordinate system where the imaging screen is located based on the resolution parameters, field of view parameters, and focal length of the virtual camera. The virtual camera is a virtual camera, and the imaging plane of the virtual camera is the plane where the imaging screen is located.

[0103] The imaging screen's field of view parameters, including horizontal and vertical field of view parameters, are provided by the imaging screen manufacturer. Since the imaging screen serves as the virtual camera's imaging plane, these parameters also serve as the virtual camera's field of view parameters. The virtual camera's focal length is the distance from the origin of the render coordinate system to the imaging screen. Its specific value can be set based on actual conditions; for example, the focal length is set to 28mm.

[0104] The conversion relationship between meters and pixels, the physical unit of measurement on the imaging screen, can be determined based on the imaging screen's resolution and field of view parameters, as well as the focal length of the virtual camera. This conversion relationship can then be used to determine the 3D coordinates of the calibration point in the screen coordinate system. Currently, the 3D coordinates of the calibration point in the screen coordinate system are labeled as the first 3D coordinates.

[0105] In one embodiment, step 161 includes steps 1611 and 1612:

[0106] Step 1611: Obtain a conversion coefficient between the size of the imaging screen and pixels according to the resolution parameter and field angle parameter of the imaging screen and the focal length of the virtual camera.

[0107] Exemplarily, the unit of the size of the imaging screen is meter.

[0108] In one embodiment, the conversion coefficient is expressed as:

[0109]

[0110] Among them, ρ represents the conversion coefficient, f represents the focal length of the virtual camera, vfov represents the vertical field of view angle parameter in the field of view angle parameter, h Screen Indicates the resolution of the height in the resolution parameter.

[0111] Due to f, vfov, h Screen They are all known quantities, so after substituting them into formula (7), the conversion coefficient ρ can be obtained.

[0112] Step 1612: Determine the first three-dimensional coordinates of the calibration point in the third three-dimensional coordinate system where the imaging screen is located according to the conversion coefficient.

[0113] For example, based on the pixel position of the calibration point on the imaging screen and the conversion coefficient, the physical position of the calibration point on the imaging screen in meters can be obtained. The three-dimensional coordinates of this physical position in the third three-dimensional coordinate system are the first three-dimensional coordinates. It will be understood that the first calibration image is composed of a plurality of square grids and has the same resolution parameters as the imaging screen. Therefore, the pixel position of the calibration point on the imaging screen can be determined based on the width of the square grid and the number of rows and columns in which the calibration point is located.

[0114] In one embodiment, the first calibration image is composed of a plurality of square grid patterns of the same size, and the calibration points are the corner points of each square grid in the first calibration image. The expression of the first three-dimensional coordinate is:

[0115] S creen X i，j =[ρw grid j,ρw gridi, 0] T (8)

[0116] in, Screen X i，j represents the first three-dimensional coordinates of the corner point in the i-th row and j-th column, i = 1, ..., M, M represents the total number of rows of corner points, j = 1, ..., N, N represents the total number of columns of corner points, ρ represents the conversion coefficient, w grid Represents the pixel width of the square grid. grid j is the X-axis coordinate of the first three-dimensional coordinate, ρw grid i is the Y-axis coordinate of the first three-dimensional coordinate, and 0 is the Z-axis coordinate of the first three-dimensional coordinate, where the X-axis coordinate is related to the column number where the calibration point is located, and the Y-axis coordinate is related to the row number where the calibration point is located. The calibration point is located on the imaging screen, so the Z-axis coordinate is 0.

[0117] Step 162: Obtain second transformation parameters between the third three-dimensional coordinate system and the fourth three-dimensional coordinate system where the virtual camera is located according to the second intrinsic parameter matrix of the virtual camera, the first three-dimensional coordinates, and the first pixel coordinates.

[0118] For example, according to the second intrinsic parameter matrix and the first pixel coordinate of the virtual camera, and combined with formula (6), the three-dimensional coordinates of the calibration point in the Render coordinate system can be obtained, that is, Render The three-dimensional coordinates of X. Then, according to Render The three-dimensional coordinates of X and the three-dimensional coordinates of the calibration point in the third three-dimensional coordinates (i.e. Screen The transformation parameters from the Screen coordinate system to the Render coordinate system can be obtained by using the three-dimensional coordinates of X. Currently, the transformation parameters are recorded as the second transformation parameters. The second transformation parameters can be expressed as: represents the rotation matrix in the second transformation parameter, Represents the displacement vector in the second transformation parameters.

[0119] In one embodiment, a PNP algorithm is used to determine the second transformation parameters. In this case, step 162 specifically includes: using the second intrinsic parameter matrix of the virtual camera, the first three-dimensional coordinates, and the first pixel coordinates as input, and using the PNP (pespective-n-point) algorithm to determine the second transformation parameters between the third three-dimensional coordinate system and the fourth three-dimensional coordinate system where the virtual camera is located.

[0120] The PNP algorithm uses multiple pairs of 3D and 2D matching points (currently using the first 3D coordinates and first pixel coordinates of the calibration points) to determine the camera extrinsic parameters (currently the second transformation parameters) by minimizing the reprojection error, given known or unknown camera intrinsic parameters (currently using the third intrinsic parameter matrix). The PNP algorithm is already implemented and will not be described here.

[0121] Step 163 : Obtain third transformation parameters between the third three-dimensional coordinate system and the second three-dimensional coordinate system according to the first intrinsic parameter matrix, the first three-dimensional coordinates, and the second pixel coordinates of the target camera.

[0122] For example, according to the first intrinsic parameter matrix and the second pixel coordinates of the target camera, and in combination with formula (3), the three-dimensional coordinates of the calibration point in the second three-dimensional coordinate system can be obtained, that is, Eye The three-dimensional coordinates of X. Then, according to Eye The three-dimensional coordinates of X and the three-dimensional coordinates of the third three-dimensional coordinates of the calibration point (i.e. Screen The transformation parameters from the Screen coordinate system to the Eye coordinate system can be obtained by using the three-dimensional coordinates of X. Currently, the transformation parameters are recorded as the third transformation parameters. The third transformation parameters can be expressed as: represents the rotation matrix in the third transformation parameter, Represents the displacement vector in the third transformation parameter.

[0123] In one embodiment, the third transformation parameters are determined using a PNP algorithm. In this case, step 163 specifically involves using the first intrinsic parameter matrix, the first 3D coordinates, and the second pixel coordinates of the target camera as inputs, and determining the third transformation parameters between the third 3D coordinate system and the second 3D coordinate system using the PNP algorithm. This process is similar to the process of determining the second transformation parameters using the PNP algorithm and is not further described here.

[0124] It can be understood that the order of determining the second transformation parameters and the third transformation parameters is not currently limited. Steps 162 and 163 are only for illustrating how to determine the second transformation parameters and the third transformation parameters, and do not limit the order of determining the second transformation parameters and the third transformation parameters.

[0125] After the second transformation parameter and the third transformation parameter are determined, step 164 is executed.

[0126] Step 164 : Obtain fourth transformation parameters between the second three-dimensional coordinate system and the fourth three-dimensional coordinate system according to the second transformation parameters and the third transformation parameters.

[0127] For example, according to and The transformation parameters from the Eye coordinate system to the Render coordinate system can be obtained. Currently, the transformation parameters are recorded as the fourth transformation parameters. In one embodiment, the fourth transformation parameters can be expressed as: represents the rotation matrix in the fourth transformation parameter, Represents the displacement vector in the fourth transformation parameter.

[0128] Step 165 : Obtain a transformation matrix according to the focal length, the distance between the imaging screen and the origin of the second three-dimensional coordinate system, and the fourth transformation parameter.

[0129] For example, the distance from the imaging screen to the origin of the Eye coordinate system is denoted as d Eye , d Eye Can be achieved through And the normal vector of the Eye coordinate system is obtained, that is in, Eye n represents the normal vector of the plane where the imaging screen is located in the Eye coordinate system, Scree n represents the normal vector of the plane where the imaging screen is located in the Screen coordinate system. Screen n is parallel to the Z axis of the Screen coordinate system, that is, Screen n = [0, 0, 1] T At this time, d Eye The expression is:

[0130]

[0131] Among them, d Eye Indicates the distance between the imaging screen and the origin of the second three-dimensional coordinate system, Eye represents the second three-dimensional coordinate system, and Screen represents the third three-dimensional coordinate system where the imaging screen is located. Screen n represents the normal vector of the plane where the imaging screen is located in the third three-dimensional coordinate system, Screen n = [0, 0, 1] T , represents the rotation matrix in the third transformation parameter, represents the displacement vector in the third transformation parameter, represents the third transformation parameter. Screen n and Substituting into formula (9) we can get d Eye .

[0132] Exemplarily, the conversion matrix is ​​an intermediate quantity required to calculate the calibration result. In one embodiment, the expression of the conversion matrix is:

[0133]

[0134] in, represents the transformation matrix, f represents the focal length of the virtual camera, d Eye represents the distance between the imaging screen and the origin of the second three-dimensional coordinate system, represents the displacement vector (a 1×3 vector) in the fourth transformation parameter, It can represent the displacement in three directions respectively. represents the fourth transformation parameter. f and d Eye All of them have been calculated, so after substituting them into the above formula (10), the transformation matrix can be obtained.

[0135] After the conversion matrix is ​​obtained, step 166 is executed.

[0136] Step 166: Obtain a calibration result of the augmented reality device according to the conversion matrix, the second intrinsic parameter matrix of the virtual camera, the fourth transformation parameter, and the first transformation parameter.

[0137] For example, the expression of the calibration result is:

[0138]

[0139] in, Represents the projection matrix, Image_Screen represents the two-dimensional coordinate system where the imaging screen is located, and Tracker represents the first three-dimensional coordinate system where the observation sensor is located. Render K represents the second intrinsic parameter matrix of the virtual camera, Render represents the fourth three-dimensional coordinate system where the virtual camera is located, represents the transformation matrix, Represents the rotation matrix in the fourth transformation parameter, Eye represents the second three-dimensional coordinate system where the target camera is located, represents the first transformation parameter. Render K. and They have all been calculated, so after substituting them into the above formula (11), the calibration results can be obtained.

[0140] It is understandable that in order to better understand the method of calibrating AR devices, the derivation process of the projection matrix and the transformation matrix is ​​described below:

[0141] For example, the virtual space of the AR device is connected to the real space through the virtual camera. At this time, the positional relationship between the fourth three-dimensional coordinate system where the virtual camera is located and the imaging screen and the second three-dimensional coordinate system is as follows: Figure 12 Reference Figure 12 , find a point in the Eye coordinate system Eye X, Eye X is located on the side of the imaging screen facing away from the target camera. Eye The intersection of the straight line formed by X and the origin of the Eye coordinate system on the imaging screen is recorded as Eye X Screen ,at this time, Eye X and EyeX Screen Has the following relationship:

[0142] Eye X Screen =β Eye X (12)

[0143] Wherein, β is a scalar, and its value can be set according to actual conditions.

[0144] Depend on Figure 12 It can be seen that a three-dimensional coordinate point X on the imaging screen Screen The relationship of coplanar imaging (Planar Homography) is formed with the Render coordinate system and the Eye coordinate system, that is, the virtual camera and the target camera are currently used to image the point X on the plane where the screen is located. Screen When imaging, click X Screen In the Eye coordinate system, it is expressed as Eye X Screen , click X Screen In the Render coordinate system, it is represented as Render X Screen , Eye X Screen and Render X Screen The relationship of the following formula (13) is satisfied:

[0145]

[0146] The meaning of each letter in the above formula can be referred to the above description. Render X Screen Also located in the imaging screen.

[0147] For example, Render X Screen The two-dimensional coordinate point after mapping to the Image_Screen coordinate system is expressed as Image_Screen x, according to formula (6), we can get the following expression:

[0148] λ Image_Screen x= Render K Render X Screen (14)

[0149] After that, after substituting formula (13) into formula (14), formula (14) can be transformed as follows:

[0150]

[0151] Among them, I 3×3 represents the 3×3 identity matrix, Rendern represents the normal vector of the plane where the imaging screen is located in the Render coordinate system. Since the virtual camera and the imaging screen are coaxial projections, the Z axis of the Render coordinate system is perpendicular to the plane where the imaging screen is located, so the Z axis of the Render coordinate system can be used as Render n, and Render n T =[0,0,1]. The meaning of each letter in the above formula (15) can refer to the above description.

[0152] Understandably, because Render n T =[0,0,1], and and Represent the displacement in three directions respectively. Therefore, the part in the brackets in formula (15) can be transformed as follows:

[0153]

[0154]

[0155] Figure 13 This is a top view of the system consisting of the Render coordinate system, Eye coordinate system, and imaging screen, refer to Figure 13 , the distance from the origin of the Render coordinate system to the imaging screen is used as the focal length f of the virtual camera. And, That is, the distance from the origin of the Render coordinate system to the imaging screen is the distance d from the origin of the Eye coordinate system to the imaging screen. eye And the displacement of the Eye coordinate system and the Render coordinate system on the Z axis At this time, combined with Figure 13 According to the relationship shown in the figure, formula (16) can be transformed into:

[0156]

[0157] For example, the transformation matrix is ​​defined as and At this time, formula (17) can be expressed as:

[0158]

[0159] After that, after substituting formula (18) and formula (12) into formula (15), formula (15) can be expressed as:

[0160]

[0161] In formula (19), After being combined with β as a scalar into λ, formula (19) can be expressed as:

[0162]

[0163] After that, after substituting formula (2) into formula (20), formula (20) can be transformed into:

[0164]

[0165] From formula (21), we can get Tracker X maps to Image_Screen xThe required projection matrix and That is, we get formula (11).

[0166] From the above derivation process, we can know that during the calibration process, we only need to obtain Render K. You can get

[0167] The projection matrix obtained from the above derivation process It is applied in AR devices. In addition, a checkerboard-patterned object is placed in the real space of the AR device. When the camera is used to simulate the human eye viewing the imaging screen, the image obtained after the camera captures the imaging screen is as follows: Figure 14 As shown. Figure 14 As you can see, using the projection matrix After that, the virtual image (grid image) displayed on the imaging screen is accurately superimposed with the checkerboard pattern of the object. In order to verify the accuracy of the projection matrix, the Figure 15 Camera and shooting Figure 14 The camera model used is different, based on Figure 14 The projection matrix used, using Figure 15 The corresponding camera shoots the object with the checkerboard pattern and obtains Figure 15 The images shown, reference Figure 15 , the virtual image (grid image) displayed on the screen is accurately superimposed on the checkerboard pattern of the object. That is, when the target camera is used instead of the human eye to calibrate the AR device, the influence of the target camera model on the calibration result can be basically ignored. It should be noted that for ease of understanding, Figure 15 The virtual image is marked with a dotted box.

[0168] In the above, by placing the target camera in the eye movement range of the user when using the AR device, simulating the human eye viewing the imaging screen, and the control device generates a first calibration map with the same resolution parameters as the imaging screen and displays the first calibration map on the imaging screen, then the target camera shoots the first calibration map to obtain a second calibration map, and then, according to the first pixel coordinates of the calibration point in the first calibration map, the second pixel coordinates in the second calibration map, and the first transformation parameters between the observation sensor and the target camera, the calibration result is obtained. This technical means solves the technical problems of low production efficiency and large manual workload caused by manual participation in the calibration process. By simulating the human eye through the target camera, automatic calibration of the AR device is achieved, reducing manual workload and improving the production efficiency of the AR device. In addition, during the calibration process, by setting a virtual camera, the virtual space of the AR device and the real space can be connected, thereby ensuring the accuracy of the calibration result. In addition, the calculation process of the calibration result is simple, easy to implement and apply.

[0169] Figure 16 A schematic diagram of a device structure for calibrating an AR device provided in one embodiment of the present application. The AR device includes: an imaging screen and an observation sensor. Figure 16 The apparatus for calibrating an AR device includes a first acquiring unit 201 , a generating unit 202 , a second acquiring unit 203 , a sending unit 204 , a third acquiring unit 205 and a result determining unit 206 .

[0170] Among them, the first acquisition unit 201 is used to obtain a first transformation parameter between a first three-dimensional coordinate system where the observation sensor is located and a second three-dimensional coordinate system where the target camera is located, where the target camera is a left camera or a right camera, and the left camera and the right camera are respectively located within the left eye activity range and the right eye activity range of the augmented reality device; the generation unit 202 is used to generate a first calibration map, and the first calibration map has the same resolution parameter as the imaging screen; the second acquisition unit 203 is used to obtain the first pixel coordinates of each calibration point in the first calibration map; the sending unit 204 is used to send the first calibration map to the imaging screen corresponding to the target camera, so that the imaging screen displays the first calibration map; the third acquisition unit 205 is used to obtain a second calibration map obtained after the target camera shoots the imaging screen, and determine the second pixel coordinates of the calibration point in the second calibration map; the result determination unit 206 is used to obtain a calibration result of the augmented reality device based on the first pixel coordinates, the second pixel coordinates and the first transformation parameter, where the calibration result is the projection matrix required when the three-dimensional coordinate point in the first three-dimensional coordinate system is mapped to the two-dimensional coordinate system where the imaging screen is located.

[0171] In one embodiment of the present application, the result determination unit 206 includes: a first three-dimensional coordinate determination subunit, which is used to obtain the first three-dimensional coordinates of the calibration point in the third three-dimensional coordinate system where the imaging screen is located according to the resolution parameters, the field of view parameters and the focal length of the virtual camera of the imaging screen, wherein the virtual camera is a virtual camera, and the imaging plane of the virtual camera is the plane where the imaging screen is located; a second transformation parameter determination subunit, which is used to obtain the second transformation parameter between the third three-dimensional coordinate system and the fourth three-dimensional coordinate system where the virtual camera is located according to the second intrinsic parameter matrix of the virtual camera, the first three-dimensional coordinates and the first pixel coordinates; a third transformation parameter determination subunit, which is used to obtain the second transformation parameter between the third three-dimensional coordinate system and the fourth three-dimensional coordinate system where the virtual camera is located according to the second intrinsic parameter matrix of the virtual camera, the first three-dimensional coordinates and the first pixel coordinates; According to the first intrinsic parameter matrix, the first three-dimensional coordinate and the second pixel coordinate of the target camera, the third transformation parameter between the third three-dimensional coordinate system and the second three-dimensional coordinate system is obtained; the fourth transformation parameter determination subunit is used to obtain the fourth transformation parameter between the second three-dimensional coordinate system and the fourth three-dimensional coordinate system according to the second transformation parameter and the third transformation parameter; the conversion matrix determination subunit is used to obtain the conversion matrix according to the focal length, the distance between the imaging screen and the origin of the second three-dimensional coordinate system and the fourth transformation parameter; the calibration result determination subunit is used to obtain the calibration result of the augmented reality device according to the conversion matrix, the second intrinsic parameter matrix of the virtual camera, the fourth transformation parameter and the first transformation parameter.

[0172] In one embodiment of the present application, the expression of the calibration result is:

[0173]

[0174] in, Represents the projection matrix, Image_Screen represents the two-dimensional coordinate system where the imaging screen is located, and Tracker represents the first three-dimensional coordinate system where the observation sensor is located. Render K represents the second intrinsic parameter matrix of the virtual camera, Render represents the fourth three-dimensional coordinate system where the virtual camera is located, represents the transformation matrix, Represents the rotation matrix in the fourth transformation parameter, Eye represents the second three-dimensional coordinate system where the target camera is located, represents the first transformation parameter;

[0175] The expression of the transformation matrix is:

[0176]

[0177] in, represents the transformation matrix, f represents the focal length of the virtual camera, d Eye represents the distance between the imaging screen and the origin of the second three-dimensional coordinate system, represents the displacement vector in the fourth transformation parameter, represents the fourth transformation parameter.

[0178] In one embodiment of the present application, the distance between the imaging screen and the origin of the second three-dimensional coordinate system is expressed as:

[0179]

[0180] Among them, d Eye Indicates the distance between the imaging screen and the origin of the second three-dimensional coordinate system, Eye represents the second three-dimensional coordinate system, and Screen represents the third three-dimensional coordinate system where the imaging screen is located. Screen n represents the normal vector of the plane where the imaging screen is located in the third three-dimensional coordinate system, Screen n = [0, 0, 1] T , represents the rotation matrix in the third transformation parameter, represents the displacement vector in the third transformation parameter, Represents the third transformation parameter.

[0181] In one embodiment of the present application, the first three-dimensional coordinate determination subunit includes: a conversion coefficient determination grandson unit, which is used to obtain the conversion coefficient of the size and pixel of the imaging screen based on the resolution parameters, field of view angle parameters and focal length of the virtual camera of the imaging screen; and a first three-dimensional coordinate determination grandson unit, which is used to determine the first three-dimensional coordinate of the calibration point in the third three-dimensional coordinate system where the imaging screen is located based on the conversion coefficient.

[0182] In one embodiment of the present application, the conversion coefficient is expressed as:

[0183]

[0184] Among them, ρ represents the conversion coefficient, f represents the focal length of the virtual camera, vfov is the vertical field of view angle parameter in the field of view angle parameter, h Screen The resolution parameter indicates the height resolution.

[0185] In one embodiment of the present application, the first calibration map is composed of a plurality of square grid patterns, and the calibration points are the corner points of each square grid in the first calibration map.

[0186] The expression for the first three-dimensional coordinate is:

[0187] Screen X i，j =[ρw grid j,ρw grid i, 0] T

[0188] in, Screen X i，jrepresents the first three-dimensional coordinates of the corner point in the i-th row and j-th column, i = 1, ..., M, M represents the total number of rows of corner points, j = 1, ..., N, N represents the total number of columns of corner points, ρ represents the conversion coefficient, w grid Indicates the pixel width of the square grid.

[0189] In one embodiment of the present application, the Z axis of the fourth three-dimensional coordinate system is perpendicular to the plane where the imaging screen is located and intersects with the center point of the imaging screen.

[0190] In one embodiment of the present application, the second transformation parameter determination subunit is specifically used to: use the second intrinsic parameter matrix, the first three-dimensional coordinate and the first pixel coordinate of the virtual camera as input, and use the PNP algorithm to determine the second transformation parameter between the third three-dimensional coordinate system and the fourth three-dimensional coordinate system where the virtual camera is located; the third transformation parameter determination subunit is specifically used to: use the first intrinsic parameter matrix, the first three-dimensional coordinate and the second pixel coordinate of the target camera as input, and use the PNP algorithm to determine the third transformation parameter between the third three-dimensional coordinate system and the second three-dimensional coordinate system.

[0191] In one embodiment of the present application, the observation sensor is a sensor using a camera, and the observation sensor and the target camera form a binocular vision system.

[0192] In one embodiment of the present application, the AR device is AR glasses.

[0193] The apparatus for calibrating an AR device provided above can be used to execute the method for calibrating an AR device provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0194] It is worth noting that in the embodiment of the apparatus for calibrating AR equipment described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0195] Figure 17 This is a schematic diagram of a device structure for calibrating an AR device provided in one embodiment of the present application. Figure 17 As shown, the device for calibrating an AR device includes a processor 30, a memory 31, an input device 32, and an output device 33; the number of processors 30 in the device for calibrating an AR device can be one or more. Figure 17 In the device for calibrating the AR device, the processor 30, the memory 31, the input device 32, and the output device 33 can be connected via a bus or other means. Figure 17 The bus connection is taken as an example.

[0196] The memory 31, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and modules, such as program instructions / modules corresponding to the method for calibrating an AR device in one embodiment of the present application (for example, the first acquisition unit 201, generation unit 202, second acquisition unit 203, sending unit 204, third acquisition unit 205, and result determination unit 206 in the apparatus for calibrating an AR device). The processor 30 executes the various functional applications and data processing of the device for calibrating the AR device by running the software programs, instructions, and modules stored in the memory 31, thereby implementing the above-mentioned method for calibrating the AR device.

[0197] The memory 31 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the device for calibrating the AR device. Furthermore, the memory 31 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 31 may further include a memory remotely located relative to the processor 30, and these remote memories may be connected to the device for calibrating the AR device via a network. Examples of the aforementioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0198] The input device 32 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device for calibrating the AR device. The output device 33 may include a display device such as a display screen. In addition, the device for calibrating the AR device may also include a communication device to communicate with the AR device and the target camera (such as sending a first calibration map and obtaining a second calibration map).

[0199] The above-mentioned device for calibrating an AR device as a control device may include an apparatus for calibrating an AR device, and may execute the relevant descriptions in the method for calibrating an AR device, and have corresponding functions and beneficial effects.

[0200] The present application also provides a system for calibrating an AR device, comprising: an AR device, a target camera, and the aforementioned device for calibrating the AR device; the AR device includes an imaging screen and an observation sensor; the target camera is a left camera or a right camera, and the left camera and the right camera are located within the left eye activity range and the right eye activity range of the AR device, respectively. The device for calibrating the AR device may be a control device.

[0201] The method for calibrating the AR device system is as follows:

[0202] The device (i.e., the control device) obtains a first transformation parameter between a first three-dimensional coordinate system where the observation sensor is located and a second three-dimensional coordinate system where the target camera is located; the device (i.e., the control device) generates a first calibration map, and the first calibration map has the same resolution parameter as the imaging screen; the device (i.e., the control device) obtains the first pixel coordinates of each calibration point in the first calibration map; the device (i.e., the control device) sends the first calibration map to the imaging screen corresponding to the target camera; the imaging screen displays the first calibration map; the target camera shoots the imaging screen to obtain a second calibration map; the device (i.e., the control device) obtains the second calibration map and determines the second pixel coordinates of the calibration point in the second calibration map; the device (i.e., the control device) obtains a calibration result of the AR device based on the first pixel coordinates, the second pixel coordinates and the first transformation parameter, and the calibration result is the projection matrix required when the three-dimensional coordinate point in the first three-dimensional coordinate system is mapped to the two-dimensional coordinate system where the imaging screen is located.

[0203] In one embodiment, the system may further include a fixing frame on which the AR device, the left camera, and the right camera are all fixed.

[0204] In one embodiment, the AR device is AR glasses.

[0205] It should be noted that in actual applications, the system for calibrating AR devices can include both left and right cameras.

[0206] It is understandable that technical details not described in the system for calibrating an AR device may refer to the relevant description in the method for calibrating an AR device, and have corresponding functions and beneficial effects.

[0207] In addition, an embodiment of the present application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the method for calibrating an AR device provided in any embodiment of the present application, and have corresponding functions and beneficial effects.

[0208] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0209] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0210] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in a computer-readable medium, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0211] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0212] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0213] Note that the above are only preferred embodiments of the present application and the technical principles employed. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present application. The scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for calibrating an augmented reality device, characterized in that: The augmented reality device includes: an imaging screen and an observation sensor; The method comprises: Obtaining a first transformation parameter between a first three-dimensional coordinate system where the observation sensor is located and a second three-dimensional coordinate system where the target camera is located, where the target camera is a left camera or a right camera, and the left camera and the right camera are respectively located within a left eye activity range and a right eye activity range of the augmented reality device; generating a first calibration map, wherein the first calibration map and the imaging screen have the same resolution parameters; Obtaining first pixel coordinates of each calibration point in the first calibration image; Sending the first calibration map to an imaging screen corresponding to the target camera, so that the imaging screen displays the first calibration map; Obtaining a second calibration image obtained after the target camera captures the imaging screen, and determining a second pixel coordinate of the calibration point in the second calibration image; Based on the first pixel coordinates, the second pixel coordinates and the first transformation parameters, a calibration result of the augmented reality device is obtained, and the calibration result is the projection matrix required when the three-dimensional coordinate points in the first three-dimensional coordinate system are mapped to the two-dimensional coordinate system where the imaging screen is located.

2. The method according to claim 1, characterized in that Obtaining a calibration result of the augmented reality device according to the first pixel coordinates, the second pixel coordinates, and the first transformation parameter includes: Obtaining, based on a resolution parameter and a field of view angle parameter of the imaging screen and a focal length of a virtual camera, a first three-dimensional coordinate of the calibration point in a third three-dimensional coordinate system where the imaging screen is located, wherein the virtual camera is a virtual camera, and an imaging plane of the virtual camera is a plane where the imaging screen is located; Obtaining, according to a second intrinsic parameter matrix of the virtual camera, the first three-dimensional coordinates, and the first pixel coordinates, second transformation parameters between the third three-dimensional coordinate system and a fourth three-dimensional coordinate system where the virtual camera is located; Obtaining third transformation parameters between the third three-dimensional coordinate system and the second three-dimensional coordinate system according to a first intrinsic parameter matrix of the target camera, the first three-dimensional coordinates, and the second pixel coordinates; Obtaining fourth transformation parameters between the second three-dimensional coordinate system and the fourth three-dimensional coordinate system according to the second transformation parameters and the third transformation parameters; Obtaining a transformation matrix according to the focal length, the distance between the imaging screen and the origin of the second three-dimensional coordinate system, and the fourth transformation parameter; A calibration result of the augmented reality device is obtained according to the conversion matrix, the second intrinsic parameter matrix of the virtual camera, the fourth transformation parameter and the first transformation parameter.

3. The method according to claim 2, characterized in that The expression of the calibration result is: in, represents the projection matrix, Image_Screen represents the two-dimensional coordinate system where the imaging screen is located, and Tracker represents the first three-dimensional coordinate system where the observation sensor is located. Render K represents the second intrinsic parameter matrix of the virtual camera, Render represents the fourth three-dimensional coordinate system where the virtual camera is located, represents the transformation matrix, represents the rotation matrix in the fourth transformation parameter, Eye represents the second three-dimensional coordinate system where the target camera is located, represents the first transformation parameter; The expression of the conversion matrix is: in, represents the transformation matrix, f represents the focal length of the virtual camera, d Eye represents the distance between the imaging screen and the origin of the second three-dimensional coordinate system, represents the displacement vector in the fourth transformation parameter, represents the fourth transformation parameter.

4. The method according to claim 2 or 3, characterized in that The distance between the imaging screen and the origin of the second three-dimensional coordinate system is expressed as: Among them, d Eye represents the distance between the imaging screen and the origin of the second three-dimensional coordinate system, Eye represents the second three-dimensional coordinate system, and Screen represents the third three-dimensional coordinate system where the imaging screen is located. Screen n represents the normal vector of the plane where the imaging screen is located in the third three-dimensional coordinate system, Screen n = [0, 0, 1] T , represents the rotation matrix in the third transformation parameter, represents the displacement vector in the third transformation parameter, represents the third transformation parameter.

5. The method according to claim 2, characterized in that Obtaining the first three-dimensional coordinates of the calibration point in the third three-dimensional coordinate system where the imaging screen is located according to the resolution parameter, the field of view parameter, and the focal length of the virtual camera includes: Obtaining a conversion coefficient between the size of the imaging screen and pixels according to a resolution parameter and a field of view angle parameter of the imaging screen and a focal length of the virtual camera; The first three-dimensional coordinates of the calibration point in the third three-dimensional coordinate system where the imaging screen is located are determined according to the conversion coefficient.

6. The method according to claim 5, characterized in that The expression of the conversion coefficient is: Wherein, ρ represents the conversion coefficient, f represents the focal length of the virtual camera, vfov is the vertical field of view angle parameter in the field of view angle parameter, h Screen The resolution parameter indicates the height resolution.

7. The method according to claim 5, characterized in that The first calibration diagram is composed of a plurality of square grid patterns of the same size, and the calibration points are the corner points of each square grid in the first calibration diagram. The expression of the first three-dimensional coordinate is: Screen X i，j =[ρw grid j,ρw grid i,0] T in, Screen X i，j represents the first three-dimensional coordinate of the corner point in the i-th row and j-th column, i=1, ..., M, M represents the total number of rows of corner points, j=1, ..., N, N represents the total number of columns of corner points, ρ represents the conversion coefficient, w grid Indicates the pixel width of the square grid.

8. The method according to claim 2, characterized in that A Z axis of the fourth three-dimensional coordinate system is perpendicular to the plane where the imaging screen is located and intersects with the center point of the imaging screen.

9. The method according to claim 2, characterized in that The obtaining, according to the second intrinsic parameter matrix of the virtual camera, the first three-dimensional coordinates, and the first pixel coordinates, of a second transformation parameter between the third three-dimensional coordinate system and the fourth three-dimensional coordinate system where the virtual camera is located comprises: Determine, using a PNP algorithm, a second transformation parameter between the third three-dimensional coordinate system and a fourth three-dimensional coordinate system where the virtual camera is located, using the second intrinsic parameter matrix of the virtual camera, the first three-dimensional coordinates, and the first pixel coordinates as input; Obtaining third transformation parameters between the third three-dimensional coordinate system and the second three-dimensional coordinate system according to the first intrinsic parameter matrix of the target camera, the first three-dimensional coordinates, and the second pixel coordinates includes: A PNP algorithm is used to determine third transformation parameters between the third three-dimensional coordinate system and the second three-dimensional coordinate system using the first intrinsic parameter matrix of the target camera, the first three-dimensional coordinates, and the second pixel coordinates as input.

10. The method according to claim 1, characterized in that The observation sensor is a sensor using a camera, and the observation sensor and the target camera form a binocular vision system.

11. The method according to claim 1, wherein The augmented reality device is augmented reality glasses.

12. A device for calibrating an augmented reality device, characterized in that: The augmented reality device includes: an imaging screen and an observation sensor; The device comprises: a first acquisition unit, configured to acquire a first transformation parameter between a first three-dimensional coordinate system where the observation sensor is located and a second three-dimensional coordinate system where the target camera is located, wherein the target camera is a left camera or a right camera, and the left camera and the right camera are respectively located within an activity range of a left eye and an activity range of a right eye of the augmented reality device; a generating unit, configured to generate a first calibration map, wherein the first calibration map has the same resolution parameters as the imaging screen; a second acquiring unit, configured to acquire first pixel coordinates of each calibration point in the first calibration image; a sending unit, configured to send the first calibration map to an imaging screen corresponding to the target camera, so that the imaging screen displays the first calibration map; a third acquiring unit, configured to acquire a second calibration image obtained after the target camera shoots the imaging screen, and determine a second pixel coordinate of the calibration point in the second calibration image; A result determination unit is used to obtain a calibration result of the augmented reality device based on the first pixel coordinates, the second pixel coordinates and the first transformation parameters, wherein the calibration result is the projection matrix required when the three-dimensional coordinate points in the first three-dimensional coordinate system are mapped to the two-dimensional coordinate system where the imaging screen is located.

13. A device for calibrating an augmented reality device, characterized in that: The augmented reality device includes: an imaging screen and an observation sensor; The device comprises: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for calibrating an augmented reality device as described in any one of claims 1-11.

14. A system for calibrating an augmented reality device, characterized in that: include: An augmented reality device, a target camera, and a device for calibrating the augmented reality device as claimed in claim 13; The augmented reality device comprises: an imaging screen and an observation sensor; the target camera is a left camera or a right camera, and the left camera and the right camera are respectively located within the left eye activity range and the right eye activity range of the augmented reality device; The device obtains a first transformation parameter between a first three-dimensional coordinate system where the observation sensor is located and a second three-dimensional coordinate system where the target camera is located; The device generates a first calibration map, wherein the first calibration map has the same resolution parameters as the imaging screen; The device obtains first pixel coordinates of each calibration point in the first calibration image; The device sends the first calibration image to an imaging screen corresponding to the target camera; The imaging screen displays the first calibration image; The target camera shoots the imaging screen to obtain a second calibration image; The device obtains the second calibration image and determines a second pixel coordinate of the calibration point in the second calibration image; The device obtains a calibration result of the augmented reality device based on the first pixel coordinates, the second pixel coordinates and the first transformation parameters. The calibration result is the projection matrix required when the three-dimensional coordinate points in the first three-dimensional coordinate system are mapped to the two-dimensional coordinate system where the imaging screen is located.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for calibrating an augmented reality device described in any one of claims 1 to 11 is implemented.

Images