Optical perspective calibration method and system, electronic equipment and storage medium

Abstract

The invention discloses an optical perspective calibration method and system, electronic equipment and a storage medium, and relates to the field of augmented reality. In the method, the position and posture of a virtual calibration object on a display screen are adjusted according to received first user voice, so that the virtual calibration object coincides with an actual calibration object in a physical space; obtaining an image of an actual calibration object according to the received second user voice, obtaining a first two-dimensional pixel coordinate of the first target corner point from the image, and determining a three-dimensional coordinate of the first target corner point according to the first two-dimensional pixel coordinate; acquiring a second two-dimensional pixel coordinate of a second target angular point of the virtual calibration object on the display screen, and calculating a projection matrix according to the three-dimensional coordinate and the second two-dimensional pixel coordinate; the projection matrix is used to transform the points in the virtual space onto the screen of the augmented reality headset. By implementing the technical scheme provided by the invention, personalized optical perspective calibration is realized in a low-cost manner.

Description

Technical Field

[0001] The present application relates to the technical field of augmented reality, and in particular to a method, system, electronic device and storage medium for optical perspective calibration. Background Art

[0002] With the advancement of science and technology, augmented reality (AR) technology has received widespread attention and application. As an important carrier of AR technology, augmented reality head-mounted devices can integrate virtual images with the real world and provide an immersive interactive experience. In order to achieve accurate integration of virtual images and the real world, it is necessary to perform precise optical perspective calibration on the display system of the head-mounted device.

[0003] Traditional OST (Optical See Through) calibration uses a pinhole imaging model to simulate the relationship between the human eye and the screen by comparing the standard pupil distance with the standard wearing method, and manually aligns the marks on the screen with the objects in the real space and constructs a calibration matrix. Manual OST calibration needs to be combined with automatic OST calibration to effectively adapt to different people's eyes. For example, Microsoft's Halolens glasses need to be equipped with a pupil tracking camera inside the glasses to automatically adjust the eye-screen internal parameters in order to achieve automatic OST calibration. This method of achieving pupil tracking by adding two tracking cameras will increase the power consumption cost of the device.

[0004] Therefore, how to perform optical perspective calibration at low cost has become an urgent problem to be solved in the current development of augmented reality head-mounted device technology. Summary of the invention

[0005] The present application provides a method, system, electronic device and storage medium for optical perspective calibration, which solves the high cost problem brought by the existing automatic optical perspective (OST) calibration technology. By using a large language model to construct OST automatic calibration, the OST internal parameters can be automatically adapted to different users through fuzzy command input.

[0006] In a first aspect of the present application, a method for optical perspective calibration is provided, which is applied to an augmented reality head mounted device, wherein the augmented reality head mounted device includes a display screen, and the method includes: Adjusting the position and posture of the virtual calibration object on the display screen according to the received first user voice so that the virtual calibration object coincides with the actual calibration object in the physical space; Acquire an image of the actual calibration object according to the received second user voice, acquire first two-dimensional pixel coordinates of a first target corner point from the image, and determine three-dimensional coordinates of the first target corner point according to the first two-dimensional pixel coordinates; Acquire second two-dimensional pixel coordinates of a second target corner point of the virtual calibration object on the display screen, calculate a projection matrix according to the three-dimensional coordinates and the second two-dimensional pixel coordinates, wherein the first target corner point and the second target corner point correspond to the same point in the actual calibration object; The projection matrix is ​​used to transform points in virtual space onto a screen of the augmented reality head mounted device.

[0007] By adopting the above technical solution, the position and posture of the virtual calibration object are adjusted through voice control, so that the user does not need to directly operate the device or physical interface, which greatly improves the convenience and comfort of use. This non-contact control method is particularly suitable for operation when wearing a head-mounted device, reducing the limitations and interference of hand operations. The user's voice is used to indicate the coincidence of the actual calibration object and the virtual calibration object, ensuring high-precision alignment during the calibration process. At the same time, by extracting the two-dimensional pixel coordinates of the corner points from the image of the actual calibration object and calculating the three-dimensional coordinates based on this, the accuracy of the calibration is further improved. The present application can automatically calculate the projection matrix according to the two-dimensional and three-dimensional coordinates of the corresponding corner points of the virtual calibration object and the actual calibration object. The projection matrix is ​​a key parameter connecting the virtual world and the physical world, and its accurate calculation is crucial for the correct rendering and display of subsequent virtual content. Using the calculated projection matrix to convert points in the virtual space to the screen of the augmented reality head-mounted device ensures that the virtual content can be accurately and seamlessly integrated into the physical environment. This not only enhances the user's sense of immersion, but also makes the display of virtual information in the physical space more natural and intuitive. This application is applicable to different types of actual calibration objects and virtual calibration objects, and has strong flexibility and adaptability. The calibration process can be completed quickly through simple voice commands and image recognition without complicated settings or adjustments. As an innovative calibration method, it promotes the further development of augmented reality technology. By optimizing the calibration process and improving the calibration accuracy, it provides more stable and reliable technical support for AR applications, which helps promote the application and popularization of AR technology in more fields.

[0008] Optionally, the method comprises: Using a first detection algorithm to identify an actual calibration object in a physical space, and obtaining first data of each point in a camera coordinate system, wherein the first data includes a position and a direction; Converting the first data into second data in a display screen coordinate system, and positioning the virtual calibration object according to the second data; rendering the virtual calibration object using OpenGL drawing commands, and presenting the rendering result on the display screen; The position of the actual calibration object in the physical space is located through PnP.

[0009] By adopting the above technical solution, a first detection algorithm (such as an algorithm based on image processing or a machine learning model) is used to automatically identify the actual calibration object in the physical space, and obtain its position and orientation data (first data) in the camera coordinate system. This reduces the need for manual intervention and improves the degree of automation of the calibration process. The first data (position and orientation in the camera coordinate system) is converted into second data in the display screen coordinate system, and the virtual calibration object is accurately positioned according to these second data. This conversion ensures that the virtual calibration object can accurately correspond to the actual calibration object in the physical space, thereby improving the accuracy of the calibration. The virtual calibration object is rendered using OpenGL drawing commands, and the rendering results are presented in real time on the display screen of the augmented reality head-mounted device. OpenGL provides powerful graphics rendering capabilities, which can ensure that the virtual calibration object is displayed to the user with high-quality visual effects. The precise position of the actual calibration object in the physical space is further located by the PnP (Perspective-n-Point) algorithm. The PnP algorithm is an effective camera posture estimation algorithm that can estimate the position and posture of the camera using a small number of point correspondences. This helps to track and locate the actual calibration object more accurately in a dynamic or complex environment. These methods significantly optimize the user experience by reducing manual intervention, improving calibration accuracy, and rendering high-quality virtual calibration objects in real time. Users only need to wear a headset and issue voice commands to quickly complete the calibration and start enjoying the immersive experience brought by augmented reality.

[0010] Optionally, locating the position of the actual calibration object in the physical space by using PnP includes: Calibrate the camera of the augmented reality head mounted device to obtain an intrinsic parameter matrix of the camera; Matching feature points in the image with three-dimensional feature points on the actual calibration object, and using a PnP algorithm to calculate a first posture of the camera based on the matched feature point pairs, wherein the first posture includes a first rotation matrix and a first translation vector; The position of the calibration object in the physical space is determined according to the posture.

[0011] By adopting the above technical solution, the camera of the augmented reality head-mounted device is calibrated to obtain the camera's intrinsic parameter matrix (including focal length, optical center, etc.), which is the basis for accurate three-dimensional positioning. Accurate acquisition of the intrinsic parameter matrix can significantly improve the accuracy of subsequent positioning calculations. The PnP algorithm is used to match the feature points in the image with the three-dimensional feature points on the actual calibration object, and the camera's posture (including rotation matrix and translation vector) is calculated. The PnP algorithm has good robustness when dealing with noise, partial occlusion or missing feature points, and can ensure that the camera posture can still be accurately calculated in a complex environment. The PnP algorithm is usually efficient in calculating the camera posture, and can complete the matching and posture calculation of a large number of feature points in a short time. This makes the present application have significant advantages in augmented reality applications with high real-time requirements. According to the calculated camera posture (first rotation matrix and first translation vector), the position of the calibration object in the physical space can be accurately determined. This is crucial for subsequent virtual content rendering, spatial mapping and other tasks, and can ensure the accurate alignment of virtual elements with the physical environment. Accurate calibration object positioning can enhance the sense of reality and immersion of augmented reality applications. Users can experience the seamless integration of virtual elements and the physical environment, thus obtaining a more natural and smooth experience. Since the PnP algorithm can calculate the camera posture in real time, this application is also applicable to dynamic environments. When the camera or calibration object moves, the system can quickly update the posture information and reposition the calibration object to ensure the continuous and accurate display of virtual elements.

[0012] Optionally, adjusting the position and posture of the virtual calibration object on the display screen according to the received first user voice includes: Converting the first user voice into instructions or parameters through a large language model, and calculating a new state of the virtual calibration object according to the instructions or parameters; The new state is applied to the state model of the calibration object to update the position and posture of the virtual calibration object in the virtual space; and the updated position and posture are rendered to the display screen using a graphics rendering engine.

[0013] By adopting the above technical solution, the user's voice is converted into instructions or parameters through a large language model. This step enables users to interact with the system in the form of natural language without learning specific commands or gestures. This natural interaction method not only improves the user experience, but also reduces the user's learning cost. The large language model can understand and parse complex voice instructions, including adjustment requirements in many aspects such as position, posture, and size. This requires the system to be highly intelligent and able to accurately capture the user's intentions and calculate the new state of the virtual calibration object based on this. After receiving the user's voice instruction, the system can quickly calculate the new state of the virtual calibration object and apply it to the state model of the calibration object. This process is carried out in real time, and the user can see the effect of the adjustment almost immediately, which improves the system's response speed and dynamic adjustment ability. The updated virtual calibration object position and posture are rendered on the display screen using a graphics rendering engine to ensure the accuracy and authenticity of the rendering results. The graphics rendering engine can handle complex graphic transformations and lighting effects, making the virtual calibration object more realistic and vivid on the screen. Adjusting the position and posture of the virtual calibration object through voice control greatly simplifies the calibration process. Users do not need to manually adjust the parameters on the device or screen, they can complete the calibration work through voice, which improves the efficiency and convenience of calibration. This application is not only suitable for static calibration scenes, but also can be adjusted in real time in dynamic environments. When the physical space or camera position changes, the position and posture of the virtual calibration object can be quickly updated to ensure the continuous alignment of virtual elements with the physical environment. Combining technologies such as large language models, state models, and graphics rendering engines provides a new multimodal interaction method for augmented reality applications. This interaction method is not only limited to voice, but can also be extended to gestures, eye movements and other interaction modes, providing useful exploration and reference for the future development of intelligent interaction technology.

[0014] Optionally, acquiring first two-dimensional pixel coordinates of a first target corner point from the image, and determining three-dimensional coordinates of the first target corner point according to the first two-dimensional pixel coordinates includes: Using a second detection algorithm to identify the boundary of the actual calibration object in the image, and obtaining first two-dimensional pixel coordinates of a first target corner point, wherein the first two-dimensional pixel coordinates include row coordinates and column coordinates; The solvePnP algorithm is used to obtain a second posture of the camera in the world coordinate system according to the first two-dimensional pixel coordinates and the size of the actual calibration object. The second posture includes a second rotation matrix and a second translation vector. The first two-dimensional pixel coordinates are converted into three-dimensional coordinates through the second posture.

[0015] By adopting the above technical solution, the boundary of the actual calibration object in the image is identified by using the second detection algorithm (such as edge detection, contour detection, etc.), and the two-dimensional pixel coordinates (including row coordinates and column coordinates) of the first target corner point are accurately extracted. This process ensures that the position information of the corner point in the image is accurate and provides a reliable basis for the subsequent three-dimensional coordinate calculation. Through the solvePnP algorithm, according to the known size of the actual calibration object (such as the side length and shape of the calibration plate) and the two-dimensional pixel coordinates of the corner point detected in the image, the posture of the camera in the world coordinate system can be accurately calculated, including the rotation matrix and the translation vector. This step is the key to connecting the two-dimensional image information with the three-dimensional space information, and provides the necessary transformation matrix for converting the two-dimensional coordinates into the three-dimensional coordinates. After obtaining the posture of the camera, the two-dimensional pixel coordinates in the image are converted into three-dimensional coordinates using the posture. This process realizes the accurate mapping from the two-dimensional image to the three-dimensional space, so that the corner point information that originally existed only in the image can be reflected in the three-dimensional space, which provides strong support for subsequent three-dimensional modeling, target tracking, scene reconstruction and other applications. Since the process is based on the actual calibration objects in the image for detection and calculation, it can adapt to the influence of complex environmental factors such as changes in illumination and viewing angle to a certain extent. This makes the technology more robust and adaptable in practical applications. This application is not only suitable for corner detection and 3D reconstruction of static images, but can also be extended to fields such as dynamic target tracking and 3D motion analysis in video sequences. At the same time, by combining other computer vision technologies (such as feature matching, image stitching, etc.), its application scenarios and functions can be further expanded.

[0016] Optionally, calculating a projection matrix according to the three-dimensional coordinates and the second two-dimensional pixel coordinates includes: Determine a third rotation matrix and a third translation vector of the camera by direct linear transformation using the plurality of feature points; Calculate an estimated position of a target feature point in a camera coordinate system, and project the estimated position onto an image plane to obtain estimated pixel coordinates, wherein the target feature point is any one of the multiple feature points; Calculating the reprojection error between the two-dimensional pixel coordinates of the target feature point and the estimated pixel coordinates, and taking the sum of the squares of the reprojection errors of all feature points as the objective function; Iteratively updating the parameters between the eye and the display screen by linearizing the objective function and solving a linear least squares problem until the change in the objective function is less than a threshold; A projection matrix is ​​calculated according to the updated parameters between the eye and the display screen.

[0017] By adopting the above technical solution, the third rotation matrix and the third translation vector of the camera are preliminarily determined using multiple feature points through direct linear transformation (DLT), which provides an initial estimate that is closer to the true value for subsequent calculations. Then, through the iterative optimization process, the reprojection error between the two-dimensional pixel coordinates of the target feature point and the estimated pixel coordinates is continuously reduced until a certain convergence condition is met (that is, the change in the objective function is less than the threshold). The present application can significantly improve the accuracy of the projection matrix, making it closer to the actual situation. Due to the presence of various interference factors such as illumination changes, occlusion, and noise in the actual environment, the directly acquired feature point information may have errors. However, through iterative optimization and calculation of reprojection errors, these errors can be automatically adjusted and corrected, so that the system can adapt to complex and changeable environmental conditions and improve overall robustness. In augmented reality, accurate target tracking and positioning are crucial. The projection matrix calculated by the above method can accurately map the target in the three-dimensional space to the two-dimensional image plane, and achieve high-precision target tracking and positioning. This is of great significance for improving user experience and system performance. Since the present application is based on the detection and matching of feature points, it can be easily extended to different application scenarios. Whether it is an indoor environment or an outdoor environment, as long as enough feature point information can be obtained, this application can be used to calculate the projection matrix. Although the iterative optimization process requires certain computing resources, by reasonably setting the number of iterations and the convergence threshold, the computational burden can be reduced while ensuring accuracy. In addition, with the continuous improvement of computer hardware performance, the real-time performance of this application will also be further improved.

[0018] Optionally, calculating the projection matrix according to the updated parameters between the eye and the display screen includes: Constructing projection internal parameters according to the focal length and optical center between the eye and the display screen, and constructing projection external parameters according to the rotation parameters and translation parameters from the hardware coordinate system to the eye coordinate system; The projection matrix is ​​calculated according to the projection intrinsic parameter and the projection extrinsic parameter.

[0019] By adopting the above technical solution, by constructing the internal parameters (focal length and optical center) and external parameters (rotation parameters and translation parameters from the hardware coordinate system to the eye coordinate system) of the projection respectively, the relative position and posture relationship between the eye and the display screen can be accurately described. These parameters are the key to calculating the projection matrix, and directly determine the accuracy and authenticity of the virtual content when it is presented on the display screen. An accurate projection matrix can ensure that the virtual content is presented on the display screen with the correct viewing angle, position and proportion, thereby providing high-quality visual effects. This is particularly important for applications such as augmented reality (AR) and virtual reality (VR), which can significantly enhance the user's immersion and interactive experience. Since the calculation of the projection matrix is ​​based on the physical parameters between the eye and the display screen, the present application has a wide range of applicability. Whether it is different types of display devices (such as head-mounted displays, projectors, etc.), or different users (with different eye positions and focal lengths), the applicable projection matrix can be calculated by adjusting the corresponding parameters. In practical applications, since the user may move, adjust the display device or perform other operations, the relative position and posture between the eye and the display screen change. By updating the parameters between the eye and the display screen and recalculating the projection matrix, the presentation of virtual content can be flexibly adjusted to ensure that the system is always in the best state. Although the calculation of the projection matrix involves multiple parameters and complex transformation relationships, once the initial calibration is completed and the internal and external parameters are determined, the projection matrix can be calculated through simple matrix multiplication. This helps to reduce the computational complexity of the system and improve real-time performance.

[0020] In a second aspect of the present application, a system for optical perspective calibration is provided, comprising an adjustment module, a coordinate module, a calculation module and a display module, wherein: An adjustment module, configured to adjust the position and posture of the virtual calibration object on the display screen according to the received first user voice, so that the virtual calibration object coincides with the actual calibration object in the physical space; a coordinate module configured to obtain an image of the actual calibration object according to the received second user voice, obtain a first two-dimensional pixel coordinate of a first target corner point from the image, and determine a three-dimensional coordinate of the first target corner point according to the first two-dimensional pixel coordinates; a calculation module configured to obtain a second two-dimensional pixel coordinate of a second target corner point of the virtual calibration object on the display screen, and calculate a projection matrix according to the three-dimensional coordinate and the second two-dimensional pixel coordinate, wherein the first target corner point and the second target corner point correspond to the same point in the actual calibration object; A display module is configured to use the projection matrix to transform points in the virtual space onto a screen of the augmented reality head mounted device.

[0021] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The position and posture of the virtual calibration object can be controlled by the user's voice, making the calibration process more intuitive and convenient. Users do not need complex operations or professional knowledge, and can quickly complete the calibration through voice commands, which greatly improves the user experience. 2. By ensuring that the virtual calibration object completely overlaps with the actual calibration object in the physical space, the virtual content is accurately aligned with the real world, which provides users with a more realistic and natural augmented reality experience, allowing virtual content to be seamlessly integrated into the real environment. 3. Users can use voice commands to fine-tune the virtual calibration object until the best overlap effect is achieved. This method is more accurate than traditional manual adjustment and reduces human errors. 4. By obtaining the two-dimensional pixel coordinates of the corner points from the image of the actual calibration object and combining them with the camera's posture information, the three-dimensional coordinates of the corner points can be accurately calculated, which provides accurate data support for the subsequent calculation of the projection matrix and further improves the calibration accuracy; 5. This application supports receiving user voice commands in real time and adjusting virtual calibration objects, so that the calibration process can be completed quickly without waiting, which is especially important for augmented reality applications that need to quickly adapt to different environments or scenarios; 6. No matter what environment the user is in, as long as he can clearly see the actual calibration object and issue a voice command, calibration can be performed. This makes this application highly adaptable and flexible, and can meet the needs of different users; 7. By accurately calculating the projection matrix, the presentation of virtual content on the display screen can be optimized, reducing unnecessary calculation and rendering burdens. This helps improve the overall performance and response speed of the system; 8. Using the calculated projection matrix to transform points in the virtual space to the screen of the augmented reality headset ensures that the virtual content is presented in the correct perspective, position and scale, thereby improving the display effect and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the optical perspective calibration method disclosed in the embodiment of the present application; Figure 2 is a schematic diagram of the AprilTag disclosed in the embodiments of the present application; Figure 3 It is a module schematic diagram of the optical perspective calibration system disclosed in the embodiment of the present application; Figure 4 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0025] Explanation of the reference numerals: 301, adjustment module; 302, coordinate module; 303, calculation module; 304, display module; 401, processor; 402, communication bus; 403, user interface; 404, network interface; 405, memory. DETAILED DESCRIPTION

[0026] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0027] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0028] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0029] This embodiment discloses an optical perspective calibration method, which is applied to an augmented reality head mounted device. Figure 1 is a flow chart of the optical perspective calibration method disclosed in the embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps: S110, adjusting the position and posture of the virtual calibration object on the display screen according to the received first user voice, so that the virtual calibration object coincides with the actual calibration object in the physical space; The user's voice (i.e., the first user's voice) is obtained, such as "move the calibration object away", "the calibration object is too far away, let it come back a little" or "rotate backward". The augmented reality head-mounted device converts it into a text command through a large voice model. The augmented reality head-mounted device uses natural language processing technology to parse the converted text commands, identify the user's intention (such as moving, rotating) and specific action parameters (such as moving away, coming back, rotating backward). According to the parsing results, the direction and amplitude of the spatial transformation (translation or rotation) required for the virtual calibration object are calculated. Based on the calculated spatial transformation information, the position and posture of the virtual calibration object displayed on the screen are dynamically adjusted through the three-dimensional graphics rendering engine. This process is carried out in real time to ensure that the changes of the virtual calibration object are almost synchronized with the issuance of the user's voice command. The augmented reality head-mounted device provides visual feedback on the display screen, allowing the user to intuitively see the changes of the virtual calibration object on the screen. The user can compare the image on the display screen with the actual calibration object, and further fine-tune the position of the virtual calibration object through voice commands until the two are completely overlapped visually. The user can continue to fine-tune the virtual calibration object through voice commands until a satisfactory calibration effect is achieved. Augmented reality headsets can also record users' operating habits, optimize the accuracy of voice recognition and natural language processing, and improve the efficiency and authenticity of three-dimensional graphics rendering.

[0030] Optionally, the method comprises: Using a first detection algorithm to identify an actual calibration object in a physical space, and obtaining first data of each point in a camera coordinate system, wherein the first data includes a position and a direction; Converting the first data into second data in a display screen coordinate system, and positioning the virtual calibration object according to the second data; rendering the virtual calibration object using OpenGL drawing commands, and presenting the rendering result on the display screen; The position of the actual calibration object in the physical space is located through PnP.

[0031] Figure 2It is a schematic diagram of the AprilTag disclosed in the embodiment of the present application. In the embodiment of the present application, the AprilTag is used as the actual calibration object. Print out the AprilTag, a two-dimensional marker with a specific code, and place it precisely at a specific position in the fixed physical space that needs to be calibrated. Use the camera integrated on the head-mounted device to capture the image in the physical space, and run a specific detection algorithm (such as the AprilTag detection algorithm based on OpenCV) to identify the AprilTag calibration object in the image. The algorithm can parse the encoding information of each calibration object and its position and direction in the camera coordinate system, that is, the first data. The first data (including position and direction) in the camera coordinate system is converted into the second data in the display screen coordinate system through the coordinate transformation matrix. This step is critical because it ensures that the virtual calibration object can be displayed on the screen of the head display with the correct viewing angle and position, corresponding to the physical calibration object. According to the converted second data, the rendering position and direction of the virtual calibration object are determined on the display screen of the head-mounted device. In this way, the user can see a virtual image corresponding to the actual calibration object in the physical space in the VR environment. Using OpenGL's graphics drawing commands, the virtual calibration object is rendered on the display screen of the head-mounted device according to the position, direction and size information of the virtual calibration object. OpenGL provides a wealth of graphics rendering functions, which can efficiently generate high-quality images to ensure the smoothness and authenticity of the user's visual experience. The rendered virtual calibration object is presented in real time on the screen of the head-mounted device, allowing users to intuitively see and interact with it in the VR environment. While the virtual calibration object is rendered and displayed, the PnP (Perspective-n-Point) algorithm is used to further accurately calculate the absolute position and direction of the actual calibration object in the physical space. The PnP algorithm uses the geometric relationship between multiple feature points on the calibration object (in this case, the encoding points of AprilTag) and camera imaging to achieve high-precision spatial positioning. This step not only improves the accuracy of positioning, but also enhances the stability and reliability of the VR system.

[0032] The actual calibration object in the physical space is accurately identified by the first detection algorithm, and its precise position and direction information (i.e., the first data) in the camera coordinate system is obtained. This provides a reliable basis for the subsequent spatial transformation and positioning of the virtual calibration object. The PnP algorithm is used to further accurately calculate the absolute position and direction of the actual calibration object in the physical space. The PnP algorithm achieves high-precision spatial positioning and enhances the stability and reliability of the system by matching the geometric relationship between multiple feature points on the calibration object and the camera imaging. The first data in the camera coordinate system is converted into the second data in the display screen coordinate system in real time, and the virtual calibration object is quickly positioned according to the second data. This process ensures that the virtual calibration object can be displayed on the head display screen with the correct perspective and position, and keeps in sync with the physical calibration object. The virtual calibration object is efficiently rendered using OpenGL's drawing commands, and the rendering results are presented in real time on the head display screen. OpenGL provides powerful graphics rendering capabilities, which can generate high-quality images and ensure the smoothness and authenticity of the user's visual experience. By displaying virtual calibration objects corresponding to the actual calibration objects in the physical space in the VR environment, users can intuitively see and interact with them. This intuitive display method reduces the user's learning cost and improves the convenience and naturalness of interaction. Users can adjust the position and posture of the virtual calibration object through voice or other interactive methods to adapt to different scenarios and needs. This flexibility enables the system to better adapt to different application scenarios and user habits.

[0033] Optionally, locating the position of the actual calibration object in the physical space by using PnP includes: Calibrate the camera of the augmented reality head mounted device to obtain an intrinsic parameter matrix of the camera; Matching feature points in the image with three-dimensional feature points on the actual calibration object, and using a PnP algorithm to calculate a first posture of the camera based on the matched feature point pairs, wherein the first posture includes a first rotation matrix and a first translation vector; The position of the calibration object in the physical space is determined according to the posture.

[0034] Calibrate the camera of the AR headset. This is usually done by taking a series of images of a known pattern (such as a checkerboard) and using these images to calculate the internal parameters of the camera (intrinsic matrix). The intrinsic matrix contains key information such as the focal length and optical center of the camera, which is the basis for subsequent 3D reconstruction and positioning. Obtain the intrinsic matrix of the camera, which will be used in subsequent PnP calculations to convert the 2D pixel coordinates in the image to 3D coordinates in the camera coordinate system. Use an image processing algorithm (such as the AprilTag detector in OpenCV) to detect the AprilTag in the image and extract its code and corner points as feature points. At the same time, according to the known size and pattern of the AprilTag, generate the corresponding 3D feature points in memory. These 3D feature points represent the actual position of the calibration object in physical space. Match the 2D feature points in the image with the 3D feature points. Since AprilTag has a unique code and a fixed pattern, this matching process is usually direct and accurate. The matched 2D feature point coordinates and the corresponding 3D feature point coordinates are used as input to the PnP algorithm. Use a PnP algorithm (such as EPnP, UPnP, etc.) to calculate the first pose of the camera based on these matched feature point pairs. This pose includes the camera's rotation matrix (indicating the camera's orientation) and translation vector (indicating the camera's position relative to a reference coordinate system). Obtain the camera's first rotation matrix and first translation vector, which together describe the camera's pose when capturing images. Since the absolute position of the AprilTag calibrated object in physical space is known (represented by its three-dimensional feature points) and the camera's pose relative to these feature points has been calculated, the position of the calibrated object relative to the camera can be easily calculated. Using the camera's pose (rotation matrix and translation vector) and the calibrated object's three-dimensional feature points, the position of the calibrated object in the camera coordinate system or the world coordinate system can be obtained through a simple coordinate transformation. Finally, the precise position of the calibrated object in physical space is obtained, and this information can be used for subsequent AR rendering, spatial positioning, or interactive operations.

[0035] The PnP algorithm can accurately calculate the camera's posture (including rotation matrix and translation vector) by matching the two-dimensional feature points in the image with the three-dimensional feature points on the calibration object. This matching and calculation method based on multiple feature points has higher positioning accuracy than the method of single feature points or simple geometric relationships. Because multiple feature points are used for matching and calculation, the PnP algorithm is highly robust to interference factors such as image noise, partial occlusion or slight deformation. This makes the algorithm more reliable and stable in practical applications. With the continuous development of computer vision and computer hardware technology, the computational efficiency of the PnP algorithm has been significantly improved. Modern algorithms can complete feature point matching and posture calculation in a very short time, thereby achieving real-time or near real-time spatial positioning. Since the PnP algorithm can calculate the camera's posture in real time, it can dynamically respond to the movement and rotation of the camera and update the position information of the calibration object in the physical space in a timely manner. This is particularly important for application scenarios that require high-precision dynamic tracking. In augmented reality (AR) applications, the PnP algorithm can accurately overlay virtual information onto real-world scenes. This is of great significance for improving user experience and enhancing the sense of reality.

[0036] Optionally, adjusting the position and posture of the virtual calibration object on the display screen according to the received first user voice includes: Converting the first user voice into instructions or parameters through a large language model, and calculating a new state of the virtual calibration object according to the instructions or parameters; The new state is applied to the state model of the calibration object to update the position and posture of the virtual calibration object in the virtual space; and the updated position and posture are rendered to the display screen using a graphics rendering engine.

[0037] The received first user voice is passed to the voice recognition unit of the head-mounted device, and the voice recognition unit converts the voice signal into text. The converted text is then sent to a large language model (such as the GPT series, BERT, etc.) for processing. The large language model uses deep learning technology to understand the meaning of the text and parses it into specific instructions or parameters. These instructions or parameters may include the distance of movement, the angle of rotation, the scale of scaling, etc., which are used to describe the new state of the virtual calibration object. The parsed instructions or parameters are sent to a state calculation unit. The state calculation unit calculates the new state of the virtual calibration object based on the current state of the virtual calibration object (such as position, posture, size, etc.) and the instructions or parameters entered by the user. For example, if the user says "move the calibration object forward 5 cm and rotate 30 degrees clockwise", the state calculation unit calculates the new position and posture of the calibration object based on these parameters. The calculated new state is applied to the state model of the virtual calibration object. The state model is a data structure used to store and represent all state information of the virtual calibration object (such as position, posture, size, color, etc.). After the state model is updated, the representation of the virtual calibration object in the virtual space changes accordingly. The updated state of the virtual calibration object is passed to the graphics rendering engine. The graphics rendering engine is responsible for converting the three-dimensional model in the virtual space (including the updated virtual calibration object) into a two-dimensional image that can be displayed on the display screen. During the rendering process, the rendering engine considers multiple factors such as lighting, shadows, and textures to generate realistic visual effects. Finally, the position and posture of the updated virtual calibration object are presented on the display screen in the form of an image for users to view and interact with.

[0038] Users can control objects in the virtual environment through natural voice commands without manually operating the mouse, keyboard or other physical controllers. This interactive mode is more intuitive and convenient, enhancing the user's immersion and interactive experience. For complex or sophisticated virtual environment operations, voice control can quickly and accurately convey the user's intentions, avoiding the cumbersomeness and errors that may be caused by manual operation. Especially in scenarios where the position and posture of virtual calibration objects need to be adjusted frequently, voice control can significantly improve operational efficiency. By converting voice into instructions or parameters through a large language model, the present application can understand and respond to a variety of voice commands to meet the personalized needs of different users. Whether it is a simple position movement or a complex posture adjustment, it can be achieved through voice commands. In applications such as virtual reality (VR) or augmented reality (AR), the position and posture adjustment of virtual calibration objects is a common interactive requirement. Implementing this function through voice control can further enhance the interactivity and immersion of virtual reality, allowing users to interact with the virtual environment more naturally.

[0039] S120, acquiring an image of the actual calibration object according to the received second user voice, acquiring first two-dimensional pixel coordinates of a first target corner point from the image, and determining three-dimensional coordinates of the first target corner point according to the first two-dimensional pixel coordinates; The user's voice (i.e., the second user's voice) is obtained. The second user's voice may be to obtain an image of the actual calibration object, or more specifically, to request the three-dimensional coordinates of a specific corner point on the actual calibration object. Through a large language model or a specialized voice recognition technology, the system converts the second user's voice into an executable command or parameter. According to the parsed voice instruction, the image of the actual calibration object is obtained through the camera of the head-mounted device. After obtaining the image, the head-mounted device uses image processing technology to identify and analyze the image content. This usually includes steps such as edge detection, feature extraction, and corner point detection. In this process, the head-mounted device pays special attention to each target corner point (illustrated with the first target corner point). Through the image processing algorithm, the head-mounted device can identify the position of the first target corner point in the image and record its two-dimensional pixel coordinates (i.e., the x and y coordinates in the image coordinate system). After obtaining the two-dimensional pixel coordinates of the first target corner point, the head-mounted device needs to further use other information to determine the coordinates of the first target corner point in three-dimensional space. This usually involves the internal and external parameters of the camera (such as focal length, optical center position, rotation and translation relationship between the camera and the world coordinate system, etc.), as well as possible depth information (such as obtained through binocular stereo vision, structured light, ToF sensor, etc.). Through the camera model and 3D reconstruction algorithm, the head-mounted device can convert the two-dimensional pixel coordinates into coordinates in the three-dimensional world coordinate system. This conversion process may involve mathematical methods such as the inverse process of perspective projection and matrix operations. Finally, the head-mounted device will output the calculated three-dimensional coordinates of the first target corner point to the user or for subsequent processing.

[0040] Optionally, acquiring first two-dimensional pixel coordinates of a first target corner point from the image, and determining three-dimensional coordinates of the first target corner point according to the first two-dimensional pixel coordinates includes: Using a second detection algorithm to identify the boundary of the actual calibration object in the image, and obtaining first two-dimensional pixel coordinates of a first target corner point, wherein the first two-dimensional pixel coordinates include row coordinates and column coordinates; The solvePnP algorithm is used to obtain a second posture of the camera in the world coordinate system according to the first two-dimensional pixel coordinates and the size of the actual calibration object. The second posture includes a second rotation matrix and a second translation vector. The first two-dimensional pixel coordinates are converted into three-dimensional coordinates through the second posture.

[0041] The boundary of the actual calibration object in the image is identified using a second detection algorithm. The second detection algorithm may be based on edge detection, contour extraction, template matching or deep learning. After identifying the boundary of the calibration object, the head-mounted device further analyzes the boundary to locate the first target corner point. This usually involves performing corner detection on the boundary or inferring the position of the first target corner point based on the known geometric shape of the calibration object (such as rectangle, circle, etc.). When the first target corner point is found, the head-mounted device records its two-dimensional pixel coordinates in the image, which include row coordinates and column coordinates. SolvePnP (Perspective-n-Point) is a computer vision algorithm used to calculate the pose of a camera by matching n two-dimensional points in an image to their corresponding points in three-dimensional space. This pose includes the rotation matrix and translation vector of the camera, which describe the position and orientation of the camera in the world coordinate system. In this scenario, the input of the solvePnP algorithm includes two main parts: one is the first two-dimensional pixel coordinates of the first target corner point obtained from the image, and the other is the three-dimensional model of the actual calibration object or at least its size information. The latter allows the algorithm to know the position of the corresponding corner point in three-dimensional space. Through the solvePnP algorithm, the head-mounted device can calculate the second posture of the camera in the world coordinate system based on the input two-dimensional pixel coordinates and three-dimensional size information. This posture is expressed in the form of a second rotation matrix and a second translation vector, which together describe the precise position and orientation of the camera when capturing the image. With the second posture of the camera, the head-mounted device can use this posture to convert the two-dimensional pixel coordinates in the image (i.e., the coordinates of the first target corner point) into coordinates in three-dimensional space. This conversion process usually involves the inverse operation of perspective projection, which takes into account the camera's intrinsic parameter matrix (focal length, optical center, etc.) and extrinsic parameter matrix (rotation matrix and translation vector). Finally, the head-mounted device outputs the coordinates of the first target corner point in three-dimensional space. The three-dimensional coordinates of the first target corner point in this application are (x, y, z).

[0042] The second detection algorithm (such as edge detection, corner detection, etc.) is used to accurately identify the boundaries of the actual calibration object in the image and the position of the first target corner point, and obtain the first two-dimensional pixel coordinates (including row coordinates and column coordinates) with high precision. This provides an accurate data basis for the subsequent three-dimensional coordinate calculation. Through the solvePnP algorithm, combined with the known actual calibration object size and the two-dimensional pixel coordinates in the image, the second posture of the camera in the world coordinate system (including the second rotation matrix and the second translation vector) can be calculated. This posture information reflects the precise position and orientation of the camera relative to the world coordinate system, thereby ensuring the accuracy of the three-dimensional coordinate conversion. The entire process realizes automated processing from image acquisition to three-dimensional coordinate calculation without manual intervention. This greatly improves processing efficiency and reduces human errors. In real-time application scenarios (such as augmented reality, robot navigation, etc.), the process can quickly respond and output three-dimensional coordinate information to meet real-time requirements. This process is not only applicable to specific actual calibration objects, but can also adapt to calibration objects of different shapes, sizes and materials by adjusting the parameters of the detection algorithm and the solvePnP algorithm. This increases the flexibility and versatility of the process. At the same time, since the solvePnP algorithm is one of the widely used algorithms in the field of computer vision, it has strong compatibility and scalability and can be seamlessly integrated with other computer vision algorithms or systems. In applications such as augmented reality, by accurately calculating the three-dimensional coordinates of the actual calibration object, a more realistic and natural fusion of virtual and reality can be achieved. This helps to enhance the user's immersion and experience quality. In scenarios such as robot navigation, accurate three-dimensional coordinate information can help robots perceive and understand the environment more accurately, so as to make more reasonable decisions and actions.

[0043] S130, obtaining second two-dimensional pixel coordinates of a second target corner point of the virtual calibration object on the display screen, and calculating a projection matrix according to the three-dimensional coordinates and the second two-dimensional pixel coordinates, wherein the first target corner point and the second target corner point correspond to the same point in the actual calibration object; Find the second target corner point corresponding to the first target corner point on the virtual calibration object, and the specific position of the second target corner point on the screen, that is, the second two-dimensional pixel coordinate, is usually calculated by the resolution of the graphics rendering engine and the display device. The projection matrix is ​​a matrix that converts three-dimensional coordinates into two-dimensional screen coordinates. In AR / MR / VR applications, in order to accurately place virtual elements at specific locations in the real world, it is necessary to calculate a projection matrix that can reflect information such as camera viewing angle, focal length, and position. Through the known three-dimensional coordinates of the actual calibration object (the three-dimensional coordinates of the first target corner point) and the second two-dimensional pixel coordinates of the virtual calibration object on the display screen, mathematical methods (such as the inverse process of perspective projection) can be used to solve the projection matrix. This matrix can ensure that when the virtual calibration object is rendered on the display screen, the second target corner point on it can accurately correspond to the position of the first target corner point of the actual calibration object. The second two-dimensional pixel coordinates of the second target corner point in this application are (u, v).

[0044] Optionally, calculating a projection matrix according to the three-dimensional coordinates and the second two-dimensional pixel coordinates includes: Determine a third rotation matrix and a third translation vector of the camera by direct linear transformation using the plurality of feature points; Calculate an estimated position of a target feature point in a camera coordinate system, and project the estimated position onto an image plane to obtain estimated pixel coordinates, wherein the target feature point is any one of the multiple feature points; Calculating the reprojection error between the two-dimensional pixel coordinates of the target feature point and the estimated pixel coordinates, and taking the sum of the squares of the reprojection errors of all feature points as the objective function; Iteratively updating the parameters between the eye and the display screen by linearizing the objective function and solving a linear least squares problem until the change in the objective function is less than a threshold; A projection matrix is ​​calculated according to the updated parameters between the eye and the display screen.

[0045] Select multiple corresponding feature points from the actual calibration objects and virtual calibration objects. These feature points can be corner points, edge intersections, and other points that are easy to identify and track. Using the three-dimensional coordinates of these feature points and their two-dimensional pixel coordinates on the display screen, the third rotation matrix and third translation vector of the camera can be preliminarily estimated through the direct linear transformation (DLT) algorithm. These parameters describe the position and posture of the camera relative to a certain world coordinate system or reference coordinate system. Using the rotation matrix and translation vector obtained in the previous step, the three-dimensional coordinates of each target feature point are converted to an estimated position in the camera coordinate system. Then, these estimated positions are projected onto the image plane through the camera's projection model (such as perspective projection) to obtain the corresponding estimated pixel coordinates. For each feature point, calculate the difference between its actual two-dimensional pixel coordinates and the estimated pixel coordinates obtained by projection, that is, the reprojection error. The sum of the squares of the reprojection errors of all feature points is used as the objective function. The value of this function reflects the accuracy of the parameters between the current eye and the display screen. In order to solve the minimum value of the objective function, it is usually necessary to linearize the objective function. This involves making small adjustments to the camera parameters and observing how these adjustments affect the objective function. In each iteration, a linear least squares problem is solved to find the parameter update between the eye and the display screen that can minimize the change of the objective function. These updates are applied to update the rotation matrix, translation vector and other parameters between the eye and the display screen, and the objective function is recalculated. The above process is repeated until the change of the objective function is less than a preset threshold or the maximum number of iterations is reached. This indicates that the parameters between the eye and the display screen have converged to a stable state. After the iterative optimization process is completed, the updated parameters between the eye and the display screen are used to calculate the projection matrix. The projection matrix is ​​a matrix that converts three-dimensional coordinates into two-dimensional screen coordinates. It contains intrinsic parameters (such as focal length, optical center, etc.) and extrinsic parameters (i.e., rotation matrix and translation vector) between the eye and the display screen.

[0046] By using multiple feature points for direct linear transformation, the rotation matrix and translation vector of the camera can be preliminarily estimated, which provides the basis for subsequent high-precision alignment. The iterative optimization process gradually optimizes the parameters between the eye and the display screen by continuously reducing the reprojection error, that is, the difference between the actual pixel coordinates and the estimated pixel coordinates, thereby achieving high-precision alignment between the virtual elements and the real world. Using multiple feature points for calculation increases the robustness of the system. Even if some feature points are blocked or misidentified, the system can still rely on other feature points to maintain a stable alignment effect. The iterative optimization process avoids the alignment failure caused by inaccurate initial parameters by gradually approaching the optimal solution, thereby improving the stability of the system. Although the iterative optimization process involves complex mathematical calculations, it can be carried out in real time with the support of modern computer hardware. This means that when the user moves his head or changes the perspective, the head-mounted device can immediately update the projection matrix to keep the virtual elements continuously aligned with the real world.

[0047] Optionally, the calculating of the projection matrix according to the updated parameters between the eye and the display screen includes: constructing a projection internal parameter according to the focal length and the optical center between the eye and the display screen, and constructing a projection external parameter according to the rotation parameter and the translation parameter from the hardware coordinate system to the eye coordinate system; The projection matrix is ​​calculated according to the projection intrinsic parameter and the projection extrinsic parameter.

[0048] The conversion relationship between real space and digital space can be expressed as: Among them, E k represents the internal parameter of the projection between the eye and the screen, Indicates the rotation parameters from the hardware coordinate system to the eye coordinate system. Indicates the translation parameter from the hardware coordinate system to the eye coordinate system.

[0049] Among them, P k represents the projection matrix, E k It can be expressed as: Among them, f u and f v represents the focal distance between the eye and the screen, c u and c v Indicates the optical center between the eye and the screen.

[0050] The projection matrix is ​​a 3×4 matrix with a total of 12 unknowns. One feature point can give two sets of equations about [u, v] and [x, y, z]. If there are 6 sets of feature points, an analytical solution can be obtained.

[0051] For example, the two sets of equations can be as follows: By constructing projection internal parameters based on the focal length and optical center between the eye and the display screen, the head-mounted device can accurately simulate the optical characteristics of the camera lens, including how the focal length affects the magnification and reduction of the image, and how the optical center determines the center point of the image. This makes the rendering of virtual elements on the display screen more accurate and in line with actual optical laws. At the same time, by constructing projection external parameters based on the rotation parameters and translation parameters from the hardware coordinate system to the eye coordinate system, the head-mounted device can accurately transform the virtual elements from the position in the virtual space to the position under the user's perspective. This conversion process takes into account factors such as the user's head movement and perspective changes, thereby achieving precise alignment of virtual elements with the user's perspective. An accurate projection matrix can ensure the rendering quality of virtual elements on the display screen. By taking into account the optical characteristics of the camera and the user's perspective changes, virtual elements can present a more realistic and natural visual effect when rendering, reducing adverse factors such as distortion and blur. By updating the parameters between the eye and the display screen through an iterative optimization process and calculating the projection matrix based on this, the overall efficiency of the system can be improved. This is because the iterative optimization process can gradually approach the optimal solution and avoid unnecessary computational waste. At the same time, an accurate projection matrix can reduce errors and repeated calculations in the rendering process, thereby improving the rendering speed and responsiveness of the system.

[0052] S140: Use the projection matrix to convert points in the virtual space to the screen of the augmented reality head mounted device.

[0053] After the projection matrix transformation, the points in the virtual space are mapped to two-dimensional coordinates on the screen plane. These two-dimensional coordinates are then used to render virtual elements on the screen of the AR headset. The rendering process may also involve advanced graphics processing techniques such as texture mapping, lighting calculation, and shadow generation to further enhance the realism and immersion of virtual elements. In AR applications, the user's head movement and perspective changes are continuous. Therefore, the projection matrix and related parameters need to be updated in real time to ensure that the virtual elements can always be correctly aligned with the user's perspective. This is usually done by tracking the user's head movement through sensors in the headset (such as gyroscopes, accelerometers, etc.), and updating the projection matrix and related parameters accordingly.

[0054] This embodiment also discloses an optical perspective calibration system. Figure 3 is a schematic diagram of a module of an optical perspective calibration system disclosed in an embodiment of the present application, such as Figure 3 As shown, the system includes an adjustment module 301, a coordinate module 302, a calculation module 303 and a display module 304, wherein: An adjustment module 301 is configured to adjust the position and posture of the virtual calibration object on the display screen according to the received first user voice, so that the virtual calibration object coincides with the actual calibration object in the physical space; A coordinate module 302 is configured to obtain an image of the actual calibration object according to the received second user voice, obtain a first two-dimensional pixel coordinate of a first target corner point from the image, and determine a three-dimensional coordinate of the first target corner point according to the first two-dimensional pixel coordinates; A calculation module 303 is configured to obtain a second two-dimensional pixel coordinate of a second target corner point of the virtual calibration object on the display screen, and calculate a projection matrix according to the three-dimensional coordinate and the second two-dimensional pixel coordinate, wherein the first target corner point and the second target corner point correspond to the same point in the actual calibration object; The display module 304 is configured to use the projection matrix to transform points in the virtual space to the screen of the augmented reality head mounted device.

[0055] Optionally, the system includes a rendering module, and the rendering module is configured to: Using a first detection algorithm to identify an actual calibration object in a physical space, and obtaining first data of each point in a camera coordinate system, wherein the first data includes a position and a direction; Converting the first data into second data in a display screen coordinate system, and positioning the virtual calibration object according to the second data; rendering the virtual calibration object using OpenGL drawing commands, and presenting the rendering result on the display screen; The position of the actual calibration object in the physical space is located through PnP.

[0056] Optionally, the rendering module is configured to: Calibrate the camera of the augmented reality head mounted device to obtain an intrinsic parameter matrix of the camera; Matching feature points in the image with three-dimensional feature points on the actual calibration object, and using a PnP algorithm to calculate a first posture of the camera based on the matched feature point pairs, wherein the first posture includes a first rotation matrix and a first translation vector; The position of the calibration object in the physical space is determined according to the posture.

[0057] Optionally, the adjustment module 301 is configured to: Converting the first user voice into instructions or parameters through a large language model, and calculating a new state of the virtual calibration object according to the instructions or parameters; The new state is applied to the state model of the calibration object to update the position and posture of the virtual calibration object in the virtual space; and the updated position and posture are rendered to the display screen using a graphics rendering engine.

[0058] Optionally, the coordinate module 302 is configured to: Using a second detection algorithm to identify the boundary of the actual calibration object in the image, and obtaining first two-dimensional pixel coordinates of a first target corner point, wherein the first two-dimensional pixel coordinates include row coordinates and column coordinates; The solvePnP algorithm is used to obtain a second posture of the camera in the world coordinate system according to the first two-dimensional pixel coordinates and the size of the actual calibration object. The second posture includes a second rotation matrix and a second translation vector. The first two-dimensional pixel coordinates are converted into three-dimensional coordinates through the second posture.

[0059] Optionally, the calculation module 303 is configured to: Determine a third rotation matrix and a third translation vector of the camera by direct linear transformation using the plurality of feature points; Calculate an estimated position of a target feature point in a camera coordinate system, and project the estimated position onto an image plane to obtain estimated pixel coordinates, wherein the target feature point is any one of the multiple feature points; Calculating the reprojection error between the two-dimensional pixel coordinates of the target feature point and the estimated pixel coordinates, and taking the sum of the squares of the reprojection errors of all feature points as the objective function; Iteratively updating the parameters between the eye and the display screen by linearizing the objective function and solving a linear least squares problem until the change in the objective function is less than a threshold; A projection matrix is ​​calculated according to the updated parameters between the eye and the display screen.

[0060] Optionally, the calculation module 303 is configured to: Constructing projection internal parameters according to the focal length and optical center between the eye and the display screen, and constructing projection external parameters according to the rotation parameters and translation parameters from the hardware coordinate system to the eye coordinate system; The projection matrix is ​​calculated according to the projection intrinsic parameter and the projection extrinsic parameter.

[0061] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0062] This embodiment also discloses an electronic device, referring to Figure 4The electronic device may include: at least one processor 401 , at least one communication bus 402 , a user interface 403 , a network interface 404 , and at least one memory 405 .

[0063] The communication bus 402 is used to realize the connection and communication between these components.

[0064] The user interface 403 may include a display screen (Display) and a camera (Camera), and the optional user interface 403 may also include a standard wired interface and a wireless interface.

[0065] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0066] Among them, the processor 401 may include one or more processing cores. The processor 401 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 405, and calling data stored in the memory 405. Optionally, the processor 401 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 401 can integrate one or more combinations of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 401, and it can be implemented by a single chip.

[0067] Among them, the memory 405 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 405 includes a non-transitory computer-readable storage medium. The memory 405 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 405 may optionally be at least one storage device located away from the aforementioned processor 401. As Figure 4 As shown, the memory 405 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of the optical perspective calibration method.

[0068] exist Figure 4 In the electronic device shown, the user interface 403 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 401 can be used to call the application program of the optical perspective calibration method stored in the memory 405. When executed by one or more processors 401, the electronic device executes one or more methods in the above-mentioned embodiments.

[0069] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0070] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0072] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0074] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory 405, including a number of instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory 405 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0075] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for optical perspective calibration, characterized in that: Applied to an augmented reality head mounted device, the augmented reality head mounted device includes a display screen, and the method includes: Adjusting the position and posture of the virtual calibration object on the display screen according to the received first user voice so that the virtual calibration object coincides with the actual calibration object in the physical space; Acquire an image of the actual calibration object according to the received second user voice, acquire first two-dimensional pixel coordinates of a first target corner point from the image, and determine three-dimensional coordinates of the first target corner point according to the first two-dimensional pixel coordinates; Acquire second two-dimensional pixel coordinates of a second target corner point of the virtual calibration object on the display screen, calculate a projection matrix according to the three-dimensional coordinates and the second two-dimensional pixel coordinates, wherein the first target corner point and the second target corner point correspond to the same point in the actual calibration object; The projection matrix is ​​used to transform points in virtual space onto a screen of the augmented reality head mounted device.

2. The optical perspective calibration method according to claim 1, characterized in that: The method comprises: Using a first detection algorithm to identify an actual calibration object in a physical space, and obtaining first data of each point in a camera coordinate system, wherein the first data includes a position and a direction; Converting the first data into second data in a display screen coordinate system, and positioning the virtual calibration object according to the second data; Rendering the virtual calibration object using OpenGL drawing commands, and presenting the rendering result on the display screen; The position of the actual calibration object in the physical space is located through PnP.

3. The optical perspective calibration method according to claim 1, characterized in that: The locating the position of the actual calibration object in the physical space by PnP includes: Calibrate the camera of the augmented reality head mounted device to obtain an intrinsic parameter matrix of the camera; Matching feature points in the image with three-dimensional feature points on the actual calibration object, and using a PnP algorithm to calculate a first posture of the camera based on the matched feature point pairs, wherein the first posture includes a first rotation matrix and a first translation vector; The position of the calibration object in the physical space is determined according to the posture.

4. The optical perspective calibration method according to claim 1, characterized in that: The step of adjusting the position and posture of the virtual calibration object on the display screen according to the received first user voice comprises: Converting the first user voice into instructions or parameters through a large language model, and calculating a new state of the virtual calibration object according to the instructions or parameters; Applying the new state to the state model of the calibration object to update the position and posture of the virtual calibration object in the virtual space; The updated position and posture are rendered to the display screen using a graphics rendering engine.

5. The optical perspective calibration method according to claim 1, characterized in that: The acquiring the first two-dimensional pixel coordinates of the first target corner point from the image, and determining the three-dimensional coordinates of the first target corner point according to the first two-dimensional pixel coordinates comprises: Using a second detection algorithm to identify the boundary of the actual calibration object in the image, and obtaining first two-dimensional pixel coordinates of a first target corner point, wherein the first two-dimensional pixel coordinates include row coordinates and column coordinates; The solvePnP algorithm is used to obtain a second posture of the camera in the world coordinate system according to the first two-dimensional pixel coordinates and the size of the actual calibration object. The second posture includes a second rotation matrix and a second translation vector. The first two-dimensional pixel coordinates are converted into three-dimensional coordinates through the second posture.

6. The optical perspective calibration method according to claim 1, characterized in that: The calculating of the projection matrix according to the three-dimensional coordinates and the second two-dimensional pixel coordinates comprises: Determine a third rotation matrix and a third translation vector of the camera by direct linear transformation using the plurality of feature points; Calculate an estimated position of a target feature point in a camera coordinate system, and project the estimated position onto an image plane to obtain estimated pixel coordinates, wherein the target feature point is any one of the multiple feature points; Calculating the reprojection error between the two-dimensional pixel coordinates of the target feature point and the estimated pixel coordinates, and taking the sum of the squares of the reprojection errors of all feature points as the objective function; Iteratively updating the parameters between the eye and the display screen by linearizing the objective function and solving a linear least squares problem until the change in the objective function is less than a threshold; A projection matrix is ​​calculated according to the updated parameters between the eye and the display screen.

7. The optical perspective calibration method according to claim 6, characterized in that: Calculating the projection matrix according to the updated parameters between the eye and the display screen comprises: Constructing projection internal parameters according to the focal length and optical center between the eye and the display screen, and constructing projection external parameters according to the rotation parameters and translation parameters from the hardware coordinate system to the eye coordinate system; The projection matrix is ​​calculated according to the projection intrinsic parameter and the projection extrinsic parameter.

8. An optical perspective calibration system, characterized in that: It includes an adjustment module, a coordinate module, a calculation module and a display module, wherein: An adjustment module, configured to adjust the position and posture of the virtual calibration object on the display screen according to the received first user voice, so that the virtual calibration object coincides with the actual calibration object in the physical space; a coordinate module configured to obtain an image of the actual calibration object according to the received second user voice, obtain a first two-dimensional pixel coordinate of a first target corner point from the image, and determine a three-dimensional coordinate of the first target corner point according to the first two-dimensional pixel coordinates; a calculation module configured to obtain a second two-dimensional pixel coordinate of a second target corner point of the virtual calibration object on the display screen, and calculate a projection matrix according to the three-dimensional coordinate and the second two-dimensional pixel coordinate, wherein the first target corner point and the second target corner point correspond to the same point in the actual calibration object; A display module is configured to use the projection matrix to transform points in the virtual space onto a screen of the augmented reality head mounted device.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images