Method for determining coordinate conversion relation, MR glasses, storage medium and product

Abstract

The invention discloses a method for determining a coordinate conversion relation, MR glasses, a storage medium and a product, and relates to the technical field of mixed reality, the method for determining the coordinate conversion relation is applied to the MR glasses configured with a display screen, the MR glasses and a human eye camera are in communication connection, and the method specifically comprises the steps of controlling the display screen to display a preset calibration grating, controlling a human eye camera to shoot a preset calibration grating plate to obtain a target checkerboard image; based on the target checkerboard image, determining each piece of effective checkerboard identification information and each corner pixel coordinate corresponding to a preset calibration grid plate; according to each piece of effective checkerboard identification information, determining each target angular point display screen coordinate corresponding to a preset calibration lattice plate; and determining a coordinate conversion relationship between the display screen and the human eye camera based on the pixel coordinates of the corners and the coordinates of the display screen of the target corners. By adopting the MR glasses, the technical effect that the coordinate conversion relation between the human eye camera and the display screen can be accurately recognized is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mixed reality, and particularly relates to a coordinate conversion relationship determination method, MR glasses, a storage medium and a computer program product. BACKGROUND

[0002] With the continuous development of MR (Mixed Reality) technology, MR glasses are becoming more and more popular among users.

[0003] In the related art, the MR glasses usually use a built-in environment camera to capture objects in the surrounding environment, and present the captured objects to the user through a display module, so that the object position seen by the user through the MR glasses is consistent with the actual position of the object in the real space.

[0004] However, if there is an error in the coordinate conversion relationship between the display screen and the user's eyes in the MR glasses, the MR glasses will not be able to accurately display the captured objects on the display screen, thereby causing the object position seen by the user through the MR glasses to be inconsistent with the actual position of the object in the real space. SUMMARY

[0005] The main purpose of the present application is to provide a coordinate conversion relationship determination method, MR glasses, a storage medium and a computer program product, which aims to solve the technical problem that the MR glasses in the related art cannot accurately display the captured objects on the display screen.

[0006] To achieve the above purpose, the present application provides a coordinate conversion relationship determination method, which is applied to MR glasses provided with a display screen, the MR glasses and an eye camera are in communication connection, and the method comprises the following steps:

[0007] controlling the display screen to display a preset calibration grid plate, and controlling the eye camera to capture the preset calibration grid plate to obtain a target checkerboard image;

[0008] determining, based on the target checkerboard image, each effective checkerboard identification information and each corner pixel coordinate corresponding to the preset calibration grid plate;

[0009] determining, according to each effective checkerboard identification information, each target corner display screen coordinate corresponding to the preset calibration grid plate;

[0010] determining, based on each corner pixel coordinate and each target corner display screen coordinate, the coordinate conversion relationship between the display screen and the eye camera.

[0011] In an embodiment, the step of determining the valid grid identification information corresponding to the preset calibration grid board based on the target grid image comprises:

[0012] identifying each first grid included in the target grid image, and determining each clear grid included in the each first grid;

[0013] reading a first label image included in each of the clear grids, and determining the valid grid identification information corresponding to the preset calibration grid board based on each of the first label images.

[0014] In an embodiment, the step of determining the target corner display screen coordinates corresponding to the preset calibration grid board according to the valid grid identification information comprises:

[0015] determining each second grid included in the display screen, and reading a second label image included in each of the second grids;

[0016] determining first to-be-screened grid identification information corresponding to each of the second grids based on each of the second label images;

[0017] determining the target corner display screen coordinates corresponding to the preset calibration grid board according to the valid grid identification information and the first to-be-screened grid identification information.

[0018] In an embodiment, the step of determining the target corner display screen coordinates corresponding to the preset calibration grid board according to the valid grid identification information and the first to-be-screened grid identification information comprises:

[0019] screening each of the first to-be-screened grid identification information according to the valid grid identification information to determine each first target grid identification information matched with the valid grid identification information;

[0020] determining each target second grid based on each of the first target grid identification information, and determining the target corner display screen coordinates corresponding to the preset calibration grid board according to each of the target second grid.

[0021] In an embodiment, the step of determining the coordinate conversion relationship between the display screen and the human eye camera based on the corner pixel coordinates and the target corner display screen coordinates comprises:

[0022] determining a rotation vector between the human eye camera and the display screen based on the corner pixel coordinates and the target corner display screen coordinates;

[0023] transforming the rotation vector to determine the coordinate conversion relationship between the human eye camera and the display screen.

[0024] In one embodiment, after the step of controlling the human eye camera to capture an image of the preset calibration grid to obtain a target checkerboard image, the method further includes:

[0025] If distortion is detected in the target chessboard image, a first target chessboard cell is determined to be contained within the target chessboard image, wherein the first target chessboard cell is located at the center of the target chessboard image.

[0026] The pixel coordinates of each corner point are determined based on the target first chessboard grid, and the display screen coordinates of each corner point corresponding to the preset calibration grid are also determined.

[0027] Based on the pixel coordinates of each corner point and the display screen coordinates of each corner point, the coordinate transformation relationship between the display screen and the human eye camera is determined.

[0028] In one embodiment, the step of determining the target first chessboard square contained within the target chessboard image includes:

[0029] Determine each first chessboard grid contained within the target chessboard grid image, and determine the second chessboard grid identifier information corresponding to each first chessboard grid.

[0030] Determine preset chessboard grid identifier information, and filter multiple second chessboard grid identifier information to be filtered based on the preset chessboard grid identifier information to determine the second target chessboard grid identifier information;

[0031] The target first chessboard grid is determined within the target chessboard grid image based on the second target chessboard grid identification information.

[0032] In one embodiment, the step of determining the pixel coordinates of each corner point based on the target first checkerboard grid includes:

[0033] Determine the pixel coordinates of the target corner points corresponding to each of the first chessboard squares of the target;

[0034] The chessboard size parameters corresponding to the first chessboard square of the target are determined based on the pixel coordinates of each target corner point;

[0035] Based on the chessboard size parameters and the identification information of multiple second chessboard grids to be screened, the pixel coordinates of each corner point corresponding to the target chessboard grid image are determined.

[0036] In addition, to achieve the above objectives, this application also proposes an MR glasses, which includes: a display screen, a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method for determining the coordinate transformation relationship as described above.

[0037] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the method for determining the coordinate transformation relationship as described above.

[0038] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the method for determining coordinate transformation relationships as described above.

[0039] The coordinate transformation relationship determination method, MR glasses, storage medium, and computer program product provided in this application embodiment are applied to MR glasses equipped with a display screen. The MR glasses are communicatively connected to a human eye camera. The display screen is controlled to show a preset calibration grid, and the human eye camera is controlled to capture images of the preset calibration grid to obtain target checkerboard images. Based on the target checkerboard images, the identification information of each valid checkerboard grid and the pixel coordinates of each corner point corresponding to the preset calibration grid are determined. The display screen coordinates of each target corner point corresponding to the preset calibration grid are determined according to the identification information of each valid checkerboard grid. Based on the pixel coordinates of each corner point and the display screen coordinates of each target corner point, the coordinate transformation relationship between the display screen and the human eye camera is determined.

[0040] In this embodiment, during operation, the MR glasses first control their own display screen to show a preset calibration grid. Simultaneously, the MR glasses control a human eye camera, which is connected to them and serves as a substitute for the user's eyes, to capture images of the display screen, thereby obtaining a target checkerboard image containing the preset calibration grid. Then, the MR glasses process the target checkerboard image to determine the identification information of each valid checkerboard grid and the pixel coordinates of each corner point corresponding to the preset calibration grid. Next, based on the identification information of each valid checkerboard grid, the MR glasses filter the display screen coordinates of each corner point to determine the display screen coordinates of each target corner point that match the pixel coordinates of each corner point. Finally, the MR glasses process the pixel coordinates of each corner point and the display screen coordinates of each target corner point to determine the coordinate transformation relationship between the human eye camera and the display screen.

[0041] Thus, this application solves the technical problem in related technologies where MR glasses cannot accurately display the captured objects on the display screen. Specifically, this application removes the display screen coordinates of the corner points of the chessboard that are not displayed on the target chessboard image, ensuring that the remaining display screen coordinates of each target corner point have corresponding pixel coordinates within the target chessboard image. Based on the pixel coordinates of each corner point and the display screen coordinates of each target corner point, the coordinate transformation relationship between the human eye camera and the display screen is determined. This enables the MR glasses to accurately identify the coordinate transformation relationship between the human eye camera and the display screen, thereby allowing the MR glasses to accurately display the captured objects on the display screen based on the obtained coordinate transformation relationship. This ensures that the position of the object seen by the user through the MR glasses is consistent with the actual position of the object in real space. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating the method for determining coordinate transformation relationships in this application (Example 1).

[0045] Figure 2 This is a schematic diagram of an MR glasses calibration system according to an embodiment of the method for determining coordinate transformation relationships in this application;

[0046] Figure 3 This is a schematic diagram of a conventional calibration grid plate involved in an embodiment of the method for determining coordinate transformation relationships in this application;

[0047] Figure 4 This is a schematic diagram of a preset calibration grid plate involved in an embodiment of the method for determining coordinate transformation relationships in this application;

[0048] Figure 5 This is a schematic diagram of the target chessboard image involved in an embodiment of the method for determining coordinate transformation relationships in this application;

[0049] Figure 6 This is a schematic diagram of the corner display screen coordinates involved in one embodiment of the method for determining coordinate transformation relationships in this application;

[0050] Figure 7This is an enlarged schematic diagram of the corner display screen coordinates involved in one embodiment of the method for determining coordinate transformation relationships in this application;

[0051] Figure 8 This is a schematic diagram of corner pixel coordinates and effective checkerboard grid identification information involved in an embodiment of the method for determining coordinate transformation relationships in this application;

[0052] Figure 9 This is an enlarged schematic diagram of the target chessboard image involved in one embodiment of the method for determining coordinate transformation relationships in this application;

[0053] Figure 10 This is a schematic diagram of a freeform surface scheme involved in an embodiment of the method for determining coordinate transformation relationships in this application;

[0054] Figure 11 This is a schematic diagram of image distortion involved in an embodiment of the method for determining coordinate transformation relationships in this application;

[0055] Figure 12 This is a simplified flowchart illustrating the method for determining the coordinate transformation relationship in this application;

[0056] Figure 13 This is a schematic diagram of the device structure of the hardware operating environment involved in the method for determining coordinate transformation relationships in the embodiments of this application;

[0057] Figure 14 This is a schematic diagram of the module structure of the coordinate transformation relationship determination device according to an embodiment of this application.

[0058] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0059] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0060] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0061] In this embodiment, for ease of description, the following description uses MR glasses with an internal display screen that can communicate with a human eye camera, or a mobile terminal, data storage control terminal, PC, or other terminal connected to an electronic control unit configured on the MR glasses, as the execution subject.

[0062] Specifically, please refer to Figure 2 ,in, Figure 2 This is a schematic diagram of an MR glasses calibration system according to an embodiment of the method for determining coordinate transformation relationships in this application. Figure 2As shown, the MR glasses verification system includes MR glasses and a human eye camera used to replace the user, and the MR glasses and the human eye camera are connected in communication. Furthermore, as... Figure 2 As shown, the MR glasses contain two adjustable-pitch displays. They also contain an environment camera for capturing images of the surrounding environment, which can be understood to be an RGB camera.

[0063] Based on the aforementioned MR glasses, this application presents an overall concept for determining the coordinate transformation relationship.

[0064] With the continuous development of MR (Mixed Reality) technology, MR glasses are becoming increasingly popular. In this technology, MR glasses typically use a built-in environmental camera to photograph objects in the surrounding environment and then display these images on a screen to ensure that the position of the object seen through the MR glasses matches its actual position in real space. However, if there is an error in the coordinate transformation between the display screen and the user's eyes, the MR glasses may fail to accurately display the photographed objects on the screen, leading to a discrepancy between the position of the object seen through the MR glasses and its actual position in real space.

[0065] To address the above issues, this application provides a method for determining coordinate transformation relationships. This method is applied to MR glasses equipped with a display screen, and the MR glasses and a human eye camera are connected via communication. Specifically, it includes: controlling the display screen to show a preset calibration grid, and controlling the human eye camera to capture images of the preset calibration grid to obtain target checkerboard images; determining the identification information of each valid checkerboard grid and the pixel coordinates of each corner point corresponding to the preset calibration grid based on the target checkerboard images; determining the display screen coordinates of each target corner point corresponding to the preset calibration grid based on the identification information of each valid checkerboard grid; and determining the coordinate transformation relationship between the display screen and the human eye camera based on the pixel coordinates of each corner point and the display screen coordinates of each target corner point.

[0066] Thus, this application solves the technical problem in related technologies where MR glasses cannot accurately display the captured objects on the display screen. Specifically, this application removes the display screen coordinates of the corner points of the chessboard that are not displayed on the target chessboard image, ensuring that the remaining display screen coordinates of each target corner point have corresponding pixel coordinates within the target chessboard image. Based on the pixel coordinates of each corner point and the display screen coordinates of each target corner point, the coordinate transformation relationship between the human eye camera and the display screen is determined. This enables the MR glasses to accurately identify the coordinate transformation relationship between the human eye camera and the display screen, thereby allowing the MR glasses to accurately display the captured objects on the display screen based on the obtained coordinate transformation relationship. This ensures that the position of the object seen by the user through the MR glasses is consistent with the actual position of the object in real space.

[0067] Based on the overall concept of the coordinate transformation relationship determination method of this application, the embodiments of this application provide a method for determining coordinate transformation relationships, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the method for determining coordinate transformation relationships in this application.

[0068] In this embodiment, the coordinate transformation relationship determination method of this application is applied to MR glasses equipped with a display screen. The MR glasses and the human eye camera are connected by communication. The coordinate transformation relationship determination method of this application includes steps S10 to S40:

[0069] Step S10: Control the display screen to show the preset calibration grid, and control the human eye camera to take a picture of the preset calibration grid to obtain the target chessboard image;

[0070] In this embodiment, when the MR glasses are running, they first call their own configured display screen to show a preset calibration grid consisting of multiple checkerboard patterns containing labeled images. At the same time, the MR glasses call the human eye camera, which is connected to them and is used to replace the user's eyes, to take pictures of the display screen, thereby capturing a target checkerboard pattern image including the preset calibration grid.

[0071] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram of a preset calibration grid plate involved in an embodiment of the method for determining coordinate transformation relationships in this application. When the MR glasses are running, they first read their own configured storage module to obtain, for example... Figure 4As shown, a preset calibration grid is composed of multiple checkerboard images containing QR code images with checkerboard ID information. The preset calibration grid is input to the display screen configured by the glasses, thereby controlling the display screen to display the preset calibration grid. At the same time, the MR glasses call the human eye camera, which is connected to the glasses and is used to replace the user's eyes, to take pictures of the display screen, thereby capturing the target checkerboard image including the preset calibration grid.

[0072] In this way, MR glasses can control the display screen to show a preset calibration grid consisting of multiple checkerboard patterns containing labeled images, and can control the human eye camera to take pictures of the display screen, thereby capturing target checkerboard patterns containing the preset calibration grid.

[0073] Step S20: Determine the identification information of each valid chessboard grid and the pixel coordinates of each corner point corresponding to the preset calibration grid based on the target chessboard grid image;

[0074] It should be noted that, please refer to Figure 8 and Figure 9 ,in, Figure 8 This is a schematic diagram illustrating the corner pixel coordinates and valid checkerboard grid identification information involved in one embodiment of the method for determining coordinate transformation relationships in this application. Figure 9 This is an enlarged schematic diagram of the target chessboard image involved in one embodiment of the method for determining coordinate transformation relationships in this application, as shown below. Figure 8 and Figure 9 As shown, the effective checkerboard identification information refers to the checkerboard identification information corresponding to each clear checkerboard square contained within the target checkerboard image. Similarly, the corner pixel coordinates are the coordinates of the corner point of each clear checkerboard square in the pixel coordinate system. Furthermore, a clear checkerboard square is one whose four corner points are all recognized by the MR glasses. Additionally, the pixel coordinate system is a two-dimensional coordinate system constructed based on the target checkerboard image.

[0075] Understandably, please refer to Figure 5 and Figure 8 ,in, Figure 5 This is a schematic diagram of a target checkerboard image related to an embodiment of the coordinate transformation relationship determination method of this application. When the MR glasses recognize the target checkerboard image, the checkerboard is considered clear only when all four corner points of the checkerboard are recognized by the MR glasses. Thus, as... Figure 5 and Figure 8As shown, when the human eye camera only captures a portion of the preset calibration grid during the shooting process, or when the preset calibration grid captured by the human eye camera is partially blurred, the MR glasses are prone to failing to recognize all the checkerboard corner points. Consequently, the MR glasses cannot determine the correspondence between each checkerboard corner point on the target checkerboard image and each checkerboard corner point on the display screen, and are prone to obtaining incorrect coordinate transformation relationships.

[0076] In this embodiment, after acquiring the target checkerboard image, the MR glasses further process the target checkerboard image to determine each recognizable clear checkerboard grid contained in the target checkerboard image, and then determine the effective checkerboard grid identification information and the pixel coordinates of each corner point corresponding to each clear checkerboard grid.

[0077] For example, after acquiring a target chessboard image, the MR glasses identify the target chessboard to determine the corner points of each chessboard in the target chessboard image. The MR glasses then determine, based on the identified corner points, the clear chessboards in the target chessboard image where all corner points have been identified, and determine the valid chessboard ID information corresponding to each clear chessboard, and determine the corner point pixel coordinates corresponding to each clear chessboard in the pixel coordinate system.

[0078] In this way, by recognizing the target chessboard image, MR glasses can identify clear chessboards in the target chessboard image where all corner points are identified, and determine the valid chessboard identification information and corner pixel coordinates that can be used to calibrate coordinate transformation relationships based on each clear chessboard.

[0079] In one feasible implementation, the step of "determining the identification information of each valid chessboard grid corresponding to the preset calibration grid based on the target chessboard grid image" in step S20 above may specifically include steps S201 to S202:

[0080] Step S201: Identify each first chessboard square contained within the target chessboard image, and determine each clear chessboard square contained within each first chessboard square;

[0081] Step S202: Read the first label image contained in each clear chessboard square, and determine the identification information of each valid chessboard square corresponding to the preset calibration board based on each first label image.

[0082] It should be noted that the clear checkerboard pattern refers to a checkerboard pattern contained within the target checkerboard image where all four corner points of that pattern are recognized by the MR glasses. Furthermore, the label image is an image contained within the checkerboard pattern that includes checkerboard identification information, such as a QR code image. Please refer to [the relevant documentation / reference]. Figure 3 ,Figure 3 This is a schematic diagram of a conventional calibration grid plate involved in an embodiment of the method for determining coordinate transformation relationships in this application, as shown below. Figure 3 As shown, if the MR glasses use a standard preset calibration grid that does not contain QR code images, even if the MR glasses can filter out clear grids during the processing of the target grid image, they cannot determine the grid corresponding to each clear grid on the display screen, and thus are still prone to obtaining incorrect coordinate transformation relationships.

[0083] In this embodiment, after acquiring the target chessboard image, the MR glasses first process the target chessboard image to determine each first chessboard grid contained within the target chessboard image. Simultaneously, the MR glasses identify the corner points of each clear chessboard grid contained within the target chessboard image. The MR glasses then filter each first chessboard grid based on each clear chessboard grid corner point to determine each clear chessboard grid contained within each first chessboard grid. Afterward, the MR glasses identify the first label image contained in each clear chessboard grid and determine the corresponding valid chessboard grid identification information based on each first label image.

[0084] For example, after acquiring the target chessboard image, the MR glasses first identify the target chessboard image to determine each first chessboard grid contained within the target chessboard image. Simultaneously, the MR glasses identify each clearly identifiable chessboard grid corner point within the target chessboard image, and based on each clear chessboard grid corner point in each first chessboard grid, determine that all four of its own chessboard grid corner points are clear chessboard grid corner points. Then, the MR glasses read the QR code images contained within each clear chessboard grid and determine the chessboard grid ID information bound to each QR code image. The MR glasses then determine the identified chessboard grid ID information as valid chessboard grid ID information for filtering the preset calibration grid on the display screen.

[0085] In this way, by recognizing the target chessboard image, MR glasses can identify clear chessboards in the target chessboard image where all corner points are identified, and determine the valid chessboard identification information and corner pixel coordinates that can be used to calibrate coordinate transformation relationships based on each clear chessboard.

[0086] Step S30: Determine the display coordinates of each target corner point corresponding to the preset calibration grid based on the identification information of each valid chessboard grid;

[0087] In this embodiment, after determining the valid checkerboard grid identification information and the pixel coordinates of each corner point, the MR glasses further filter the display screen coordinates of each corner point contained on the display screen based on the valid checkerboard grid identification information to determine the display screen coordinates of each target corner point that match the pixel coordinates of each corner point.

[0088] For example, please refer to Figure 6 and Figure 7 ,in, Figure 6 This is a schematic diagram of the corner display screen coordinates involved in one embodiment of the method for determining coordinate transformation relationships in this application. Figure 7 This is an enlarged schematic diagram of the corner display screen coordinates involved in one embodiment of the method for determining coordinate transformation relationships of this application, as shown below. Figure 6 and Figure 7 As shown, after determining the ID information of each valid checkerboard grid, the MR glasses further identify the display screen to determine the corresponding second checkerboard grids in the display screen coordinate system constructed based on the display screen. The MR glasses then filter the second checkerboard grids in each display screen based on the ID information of each valid checkerboard grid, thereby determining the target second checkerboard grids that match the aforementioned clear checkerboard grids. The MR glasses then determine the display screen coordinates of the corner points corresponding to each target second checkerboard grid as the target corner point display screen coordinates that match the aforementioned corner point pixel coordinates.

[0089] In this way, based on the clear checkerboard pattern information obtained, the MR glasses can filter the display coordinates of each corner point on the display screen, thereby determining the display coordinates of each target corner point that match the pixel coordinates of each corner point.

[0090] In one feasible implementation, step S30 above may specifically include steps S301 to S303:

[0091] Step S301: Determine each second chessboard square contained in the display screen, and read the second label image contained in each second chessboard square;

[0092] Step S302: Determine the first chessboard grid identifier information corresponding to each second chessboard grid based on each second label image;

[0093] Step S303: Based on the identification information of each valid chessboard grid and the identification information of each first chessboard grid to be screened, determine the display coordinates of each target corner point corresponding to the preset calibration grid.

[0094] In this embodiment, after determining the clear checkerboard grid identification information and the pixel coordinates of each corner point, the MR glasses first identify the preset calibration grid displayed on the screen, thereby determining each second checkerboard grid within the screen. Simultaneously, the MR glasses read the second label image contained in each second checkerboard grid. Then, the MR glasses identify each second label image, thereby determining the first checkerboard grid identification information corresponding to each second checkerboard grid. Finally, the MR glasses filter the first checkerboard grid identification information based on the acquired clear checkerboard grid identification information, thereby determining the display coordinates of each target corner point within the preset calibration grid that can be used.

[0095] For example, such as Figure 6 and Figure 7 As shown, after determining the ID information of each clear checkerboard grid and the pixel coordinates of each corner point, the MR glasses first identify each preset calibration grid displayed on the screen, thereby determining each second checkerboard grid displayed on the screen by the preset calibration grid. At the same time, the MR glasses read the QR code image contained in each second checkerboard grid. Then, the MR glasses identify each QR code image, thereby determining the first checkerboard grid ID information bound to each QR code image. Finally, the MR glasses filter the first checkerboard grid ID information based on the acquired clear checkerboard grid ID information to determine the display coordinates of each usable target corner point corresponding to the preset calibration grid on the screen.

[0096] In this way, based on the clear checkerboard pattern information obtained, the MR glasses can filter the display coordinates of each corner point on the display screen, thereby determining the display coordinates of each target corner point that match the pixel coordinates of each corner point.

[0097] In one feasible implementation, step S303 above may specifically include steps S3031 to S3032:

[0098] Step S3031: Filter the first chessboard grid identifier information according to the valid chessboard grid identifier information to determine the first target chessboard grid identifier information that matches the valid chessboard grid identifier information.

[0099] Step S3032: Determine the second chessboard grid of each target based on the identification information of each first target chessboard grid, and determine the display screen coordinates of each target corner point corresponding to the preset calibration grid according to the second chessboard grid of each target.

[0100] In this embodiment, after determining the identification information of each first chessboard grid to be filtered, the MR glasses first filter the identification information of each first chessboard grid to be filtered according to the identification information of each valid chessboard grid, thereby determining the identification information of each first target chessboard grid that matches the identification information of the valid chessboard grid. Then, the MR glasses filter the second chessboard grids based on the identification information of each first target chessboard grid, thereby determining the target second chessboard grids. The MR glasses then determine the display coordinates of the corner points of each target second chessboard grid as the display coordinates of the target corner points that match the pixel coordinates of each corner point.

[0101] For example, after determining the ID information of each first chessboard grid to be filtered, the MR glasses first filter the ID information of each first chessboard grid to be filtered according to the ID information of each valid chessboard grid to be selected, thereby determining the ID information of each first target chessboard grid that is consistent with the ID information of the valid chessboard grid. Then, the MR glasses filter each second chessboard grid of the preset calibration grid on the display screen based on the ID information of each first target chessboard grid to determine the target second chessboard grid corresponding to each first target chessboard grid ID information. Finally, the MR glasses determine the display coordinates of each corner point on the display screen, and filter the display coordinates of each corner point based on each target second chessboard grid, thereby determining the display coordinates of each target corner point corresponding to each target second chessboard grid.

[0102] In this way, based on the clear checkerboard pattern information obtained, the MR glasses can filter the display coordinates of each corner point on the display screen, thereby determining the display coordinates of each target corner point that match the pixel coordinates of each corner point.

[0103] Step S40: Based on the pixel coordinates of each corner point and the display screen coordinates of each target corner point, determine the coordinate transformation relationship between the display screen and the human eye camera;

[0104] In this embodiment, after determining the display screen coordinates of each target corner point, the MR glasses further process the display screen coordinates of each target corner point and the pixel coordinates of each corner point to determine the coordinate transformation relationship between the display screen and the human eye camera.

[0105] For example, after determining the display coordinates of each target corner point, the MR glasses further process the display coordinates and pixel coordinates of each target corner point using a preset solvePNP function, thereby obtaining the coordinate transformation relationship M between the display screen and the human eye camera. Display2Eyes .

[0106] In this way, MR glasses can determine the coordinate transformation relationship between the display screen and the human eye camera by using the display screen coordinates and pixel coordinates of each target corner point.

[0107] In one feasible implementation, step S40 above may specifically include steps S401 to S402:

[0108] Step S401: Based on the pixel coordinates of each corner point and the display screen coordinates of each target corner point, determine the rotation vector between the human eye camera and the display screen;

[0109] Step S402: Transform the rotation vector to determine the coordinate transformation relationship between the human eye camera and the display screen.

[0110] In this embodiment, after determining the display screen coordinates of each target corner point, the MR glasses first read the storage module configured on its own to obtain the camera intrinsic parameters corresponding to the human eye camera. Based on the camera intrinsic parameters, the display screen coordinates of each target corner point and the pixel coordinates of each corner point are processed to determine the rotation vector generated between the human eye camera and the display screen. Then, the MR glasses obtain a preset Rodriguez function and transform the rotation vector through the Rodriguez function to obtain the coordinate transformation relationship between the human eye camera and the display screen.

[0111] For example, after determining the display coordinates of each target corner point, the MR glasses first read the storage module configured to them to obtain the intrinsic parameters of the human eye camera corresponding to the human eye camera. Then, the intrinsic parameters of the human eye camera, the display coordinates of each target corner point, and the pixel coordinates of each corner point are input into the preset solvePNP function. The solvePNP function outputs the rotation vector between the human eye camera and the display screen. After that, the MR glasses read the preset Rodriguez function and transform the rotation vector based on the Rodriguez function to obtain the coordinate transformation relationship between the human eye camera and the display screen.

[0112] In this embodiment, when the MR glasses are running, they first use their own display screen to show a preset calibration grid consisting of multiple checkerboard patterns containing labeled images. Simultaneously, the MR glasses use a human eye camera, which is connected to them and acts as a substitute for the user's eyes, to capture an image of the target checkerboard pattern, including the preset calibration grid. The MR glasses then process the target checkerboard image to identify the clearly identifiable checkerboard patterns within it, and further determine the valid checkerboard pattern identifiers and corner pixel coordinates for each clearly identifiable checkerboard pattern. Next, based on the valid checkerboard pattern identifiers, the MR glasses filter the display screen coordinates of each corner point to determine the target corner point display screen coordinates that match the corner pixel coordinates. Finally, the MR glasses process the target corner point display screen coordinates and the corner pixel coordinates to determine the coordinate transformation relationship between the display screen and the human eye camera.

[0113] Thus, this application solves the technical problem in related technologies where MR glasses cannot accurately display the captured objects on the display screen. Specifically, this application removes the display screen coordinates of the corner points of the chessboard that are not displayed on the target chessboard image, ensuring that the remaining display screen coordinates of each target corner point have corresponding pixel coordinates within the target chessboard image. Based on the pixel coordinates of each corner point and the display screen coordinates of each target corner point, the coordinate transformation relationship between the human eye camera and the display screen is determined. This enables the MR glasses to accurately identify the coordinate transformation relationship between the human eye camera and the display screen, thereby allowing the MR glasses to accurately display the captured objects on the display screen based on the obtained coordinate transformation relationship. This ensures that the position of the object seen by the user through the MR glasses is consistent with the actual position of the object in real space.

[0114] Based on the first embodiment of this application, a second embodiment of this application is proposed herein. In this second embodiment, content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, after step S10, the method for determining the coordinate transformation relationship of this application may further include steps A10 to A30:

[0115] Step A10: If distortion is detected in the target chessboard image, determine the first target chessboard cell contained in the target chessboard image, wherein the first target chessboard cell is located at the center of the target chessboard image.

[0116] Step A20: Determine the pixel coordinates of each corner point based on the target first chessboard grid, and determine the display screen coordinates of each corner point corresponding to the preset calibration grid.

[0117] Step A30: Determine the coordinate transformation relationship between the display screen and the human eye camera based on the pixel coordinates of each corner point and the display screen coordinates of each corner point.

[0118] In this embodiment, after acquiring the target checkerboard image, the MR glasses first detect the target checkerboard image to determine whether there is distortion. If the MR glasses detect distortion, they filter the first checkerboard squares within the target checkerboard image to determine the target first checkerboard square located in the middle of the target checkerboard image. Then, the MR glasses determine the pixel coordinates of each corner point in the target checkerboard image based on the target first checkerboard square. At the same time, the MR glasses identify the preset calibration grid displayed on the screen to determine the display screen coordinates of each corner point of the preset calibration grid displayed on the screen. Finally, the MR glasses process the pixel coordinates and display screen coordinates of each corner point to determine the coordinate transformation relationship between the human eye camera and the display screen.

[0119] For example, please refer toFigure 11 , Figure 11 This is a schematic diagram of image distortion in an embodiment of the method for determining coordinate transformation relationships in this application. After acquiring the target checkerboard image, the MR glasses first detect the target checkerboard image using their configured OpenCV tools to determine whether the target checkerboard image has distortion. If the MR glasses determine that the target checkerboard image has distortion, then... Figure 11 The distortion shown indicates that each first chessboard square displayed within the target chessboard image is identified, and within each first chessboard square, the target first chessboard square located in the center region of the target chessboard image is determined. Then, the MR glasses determine the pixel coordinates of each corner point of the target chessboard image when it is not distorted, based on the target first chessboard square. Simultaneously, the MR glasses recognize the display screen to determine the display screen coordinates of each corner point corresponding to the preset calibration grid displayed on the screen. Finally, the MR glasses input the acquired pixel coordinates and display screen coordinates of each corner point into a preset solvePNP function. The solvePNP function processes the pixel coordinates and display screen coordinates of each corner point to obtain the coordinate transformation relationship M between the display screen and the human eye camera. Display2Eyes .

[0120] It should be noted that, please refer to Figure 10 , Figure 10 This is a schematic diagram of a freeform surface scheme involved in an embodiment of the method for determining coordinate transformation relationships in this application, as shown below. Figure 10 As shown, during the production of AR glasses, some technicians use freeform surfaces as the display solution. Therefore, due to the optical characteristics of freeform surfaces, the images displayed on the screen will exhibit... Figure 11 The distortion shown leads to distortion in the target checkerboard image obtained when the human eye camera captures the display screen. When MR glasses process the distorted target checkerboard image, they are prone to failing to match the corner pixel coordinates with the corner display screen coordinates, resulting in the MR glasses obtaining an incorrect coordinate transformation relationship between the human eye camera and the display screen.

[0121] Thus, MR glasses, upon detecting distortion in the target checkerboard image, identify the target first checkerboard square located at the exact center of the image among the first checkerboard squares displayed on the target checkerboard image. By utilizing the optical properties of freeform surfaces, they filter out the target first checkerboard square that has almost no distortion at its center position. Furthermore, they deduce the pixel coordinates of each corner point in the undistorted state of the target checkerboard image from the target first checkerboard square, and then determine the coordinate transformation relationship between the human eye camera and the display screen based on the pixel coordinates of each corner point and the display screen coordinates of each corner point.

[0122] In one feasible implementation, the step of "determining the target first chessboard grid contained within the target chessboard grid image" in step A10 above may specifically include steps A101 to A103:

[0123] Step A101: Determine each first chessboard grid contained within the target chessboard grid image, and determine the identification information of the second chessboard grid to be filtered corresponding to each first chessboard grid.

[0124] Step A102: Determine the preset chessboard grid identification information, and filter multiple second chessboard grid identification information based on the preset chessboard grid identification information to determine the second target chessboard grid identification information;

[0125] Step A103: Determine the target first chessboard grid contained within the target chessboard grid image based on the second target chessboard grid identification information.

[0126] In this embodiment, when the MR glasses detect distortion in the target checkerboard image, they first identify the target checkerboard image to determine each first checkerboard grid contained within it, and further determine the second checkerboard grid identifier information corresponding to each first checkerboard grid. Then, the MR glasses read their configured storage module to obtain preset checkerboard grid identifier information, and filter multiple second checkerboard grid identifier information based on the preset checkerboard grid identifier information to determine the second target checkerboard grid identifier information that matches the preset checkerboard grid identifier information. Finally, the MR glasses determine the target first checkerboard grid located at the center within the target checkerboard image based on the second target checkerboard grid identifier information.

[0127] For example, when the MR glasses determine that the target chessboard image has distortion using the OpenCV tool, they first identify the target chessboard image to determine each first chessboard square displayed in the target chessboard image. They then read the QR code images contained in each first chessboard image and identify each QR code image to determine the second chessboard square ID information corresponding to each first chessboard square. Next, the MR glasses read their configured storage module to obtain the preset chessboard square ID information corresponding to the chessboard square at the center of the preset calibration grid. Based on the preset chessboard square ID information, they analyze multiple second chessboard square ID information to determine the second target chessboard square ID information that matches the preset chessboard square ID information. Finally, the MR glasses filter out the target first chessboard square at the center of the target chessboard image based on the second target chessboard square ID information.

[0128] Thus, MR glasses, upon detecting distortion in the target checkerboard image, identify the target first checkerboard square located at the exact center of the image among the first checkerboard squares displayed on the target checkerboard image. By utilizing the optical properties of freeform surfaces, they filter out the target first checkerboard square that has almost no distortion at its center position. Furthermore, they deduce the pixel coordinates of each corner point in the undistorted state of the target checkerboard image from the target first checkerboard square, and then determine the coordinate transformation relationship between the human eye camera and the display screen based on the pixel coordinates of each corner point and the display screen coordinates of each corner point.

[0129] In one feasible implementation, the step of "determining the pixel coordinates of each corner point based on the first chessboard grid" in step A20 above may specifically include steps A201 to A203:

[0130] Step A201: Determine the pixel coordinates of the target corner points corresponding to each of the first chessboard squares;

[0131] Step A202: Determine the chessboard size parameters corresponding to the first chessboard square of the target based on the pixel coordinates of each target corner point;

[0132] Step A203: Based on the checkerboard size parameters and the identification information of multiple second checkerboard grids to be filtered, determine the pixel coordinates of each corner point corresponding to the target checkerboard grid image.

[0133] In this embodiment, after the MR glasses filter out the target first chessboard grid, they first determine the target corner point pixel coordinates corresponding to each corner point of the chessboard grid within the target first chessboard grid. Then, the MR glasses calculate the chessboard grid size parameters corresponding to the target first chessboard grid based on the target corner point pixel coordinates. Finally, based on the chessboard grid size parameters and the identification information of each of the aforementioned second chessboard grids to be filtered, the MR glasses deduce the corner point pixel coordinates corresponding to each corner point of the chessboard grid contained in the target chessboard grid image in the undistorted state.

[0134] For example, after the MR glasses filter out the target first checkerboard grid located at the center of the image, they first determine the pixel coordinates of the target corner points of each checkerboard grid as (x1, y1), (x2, y2), (x3, y3), and (x4, y4). Then, the MR glasses calculate the pixel coordinates of each target corner point to obtain the checkerboard grid size parameter d corresponding to the target first checkerboard grid.

[0135]

[0136] Finally, based on the size parameter d of each chessboard and the id information of each of the aforementioned second chessboard grids to be screened, the MR glasses calculate the pixel coordinates of each corner point of each first chessboard grid.

[0137] Thus, MR glasses, upon detecting distortion in the target checkerboard image, identify the target first checkerboard square located at the exact center of the image among the first checkerboard squares displayed on the target checkerboard image. By utilizing the optical properties of freeform surfaces, they filter out the target first checkerboard square that has almost no distortion at its center position. Furthermore, they deduce the pixel coordinates of each corner point in the undistorted state of the target checkerboard image from the target first checkerboard square, and then determine the coordinate transformation relationship between the human eye camera and the display screen based on the pixel coordinates of each corner point and the display screen coordinates of each corner point.

[0138] For example, to help understand the implementation flow of the method for determining the coordinate transformation relationship obtained by combining this embodiment with the above embodiments, please refer to... Figure 12 , Figure 12 This is a simplified flowchart illustrating the method for determining the coordinate transformation relationship in this application. Specifically:

[0139] In this embodiment, the MR glasses first read their own configured storage module to obtain a preset calibration grid composed of multiple checkerboard patterns containing QR code images. This preset calibration grid is then displayed on their own screen. Simultaneously, the MR glasses use a human eye camera connected to them to capture an image of the target checkerboard pattern containing the preset calibration grid. The MR glasses then detect the target checkerboard pattern to determine if it is distorted. If no distortion is found, the MR glasses first identify each first checkerboard pattern within the target checkerboard image, and then identify clear checkerboard patterns within each first checkerboard pattern where all corners are recognized by the MR glasses. The MR glasses then read the QR code images contained in each clear checkerboard pattern to determine the valid checkerboard pattern ID information corresponding to each clear checkerboard pattern. Simultaneously, the MR glasses determine the QR code images contained in each clear checkerboard pattern. The MR glasses first determine the pixel coordinates of the corner points of each checkerboard grid. Then, they read the second checkerboard grids displayed on the screen and the QR code images contained in each second checkerboard grid to determine the first checkerboard grid ID information corresponding to each second checkerboard grid. The MR glasses then filter the first checkerboard grid ID information based on the valid checkerboard grid ID information to determine the first target checkerboard grid ID information that matches the valid checkerboard grid ID information. The MR glasses then determine the target second checkerboard grids displayed on the screen based on the first target checkerboard grid ID information, and determine the display coordinates of the corner points of each target second checkerboard grid as the display coordinates of each target corner point that match the pixel coordinates of each corner point. The MR glasses input the camera intrinsic parameters corresponding to the human eye camera, the pixel coordinates of each corner point, and the display coordinates of each target corner point into the preset solvePNP function to obtain the coordinate transformation relationship between the human eye camera and the display screen.

[0140] Similarly, when the MR glasses detect distortion in the target chessboard image, they first identify each first chessboard cell within the target chessboard image and read the QR code image contained in each first chessboard cell to determine the corresponding second chessboard cell ID information. Simultaneously, the MR glasses read from the aforementioned storage module to obtain preset chessboard cell ID information. The MR glasses then filter the second chessboard cell ID information based on the preset chessboard cell ID information, thereby identifying the target first chessboard cell located at the exact center of the target chessboard image. Afterward, the MR glasses recognize the target first chessboard cell... The system calculates the pixel coordinates of each target corner point corresponding to the grid, and then calculates the grid size corresponding to the first target grid based on the pixel coordinates of each target corner point. The MR glasses then calculate the pixel coordinates of each corner point of the first grid when the target grid image is not distorted, based on the grid size and the ID information of each second grid to be screened. Finally, the MR glasses read the display pixels of each corner point of each second grid on the display screen, and input the camera intrinsic parameters, the pixel coordinates of each corner point, and the display coordinates of each corner point into the preset solvePNP function to obtain the coordinate transformation relationship between the human eye camera and the display screen.

[0141] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the method for determining the coordinate transformation relationship of this application. Any simple transformations based on this technical concept are all within the protection scope of this application.

[0142] This application also provides a device for determining coordinate transformation relationships, please refer to... Figure 14 The coordinate transformation relationship determination device of this application is applied to MR glasses equipped with a display screen, and the MR glasses and the human eye camera are connected in communication, specifically including:

[0143] The grid shooting module 10 is used to control the display screen to show the preset calibration grid and control the human eye camera to shoot the preset calibration grid to obtain the target chessboard grid image;

[0144] The coordinate traversal module 20 is used to determine the identification information of each valid chessboard grid and the pixel coordinates of each corner point corresponding to the preset calibration grid based on the target chessboard grid image.

[0145] The coordinate filtering module 30 is used to determine the display coordinates of each target corner point corresponding to the preset calibration grid based on the identification information of each valid chessboard grid.

[0146] The transformation calculation module 40 is used to determine the coordinate transformation relationship between the display screen and the human eye camera based on the pixel coordinates of each corner point and the display screen coordinates of each target corner point.

[0147] In one feasible implementation, the coordinate traversal module 20 is further configured to:

[0148] Identify each first chessboard square contained within the target chessboard image, and determine each clear chessboard square contained within each first chessboard square;

[0149] Read the first label image contained in each clear chessboard square, and determine the identification information of each valid chessboard square corresponding to the preset calibration board based on each first label image.

[0150] In one feasible implementation, the coordinate filtering module 30 is further configured to:

[0151] Identify each second chessboard square contained in the display screen, and read the second label image contained in each second chessboard square;

[0152] Based on each second label image, determine the identification information of the first chessboard grid to be filtered corresponding to each second chessboard grid;

[0153] Based on the identification information of each valid chessboard grid and the identification information of each first chessboard grid to be screened, determine the display coordinates of each target corner point corresponding to the preset calibration grid.

[0154] In one feasible implementation, the coordinate filtering module 30 is further configured to:

[0155] Based on the valid chessboard grid identifier information, the identifier information of each first chessboard grid to be screened is filtered to determine the first target chessboard grid identifier information that matches the valid chessboard grid identifier information.

[0156] Based on the identification information of each first target chessboard grid, determine the second target chessboard grid, and determine the display coordinates of each target corner point corresponding to the preset calibration grid according to the second target chessboard grid.

[0157] In one feasible implementation, the conversion calculation module 40 is further configured to:

[0158] Based on the pixel coordinates of each corner point and the display screen coordinates of each target corner point, determine the rotation vector between the human eye camera and the display screen;

[0159] The rotation vector is transformed to determine the coordinate transformation relationship between the human eye camera and the display screen.

[0160] In one feasible implementation, the coordinate traversal module 20 is further configured to:

[0161] If distortion is detected in the target chessboard image, the first target chessboard cell contained within the target chessboard image is determined, wherein the first target chessboard cell is located at the center of the target chessboard image.

[0162] The pixel coordinates of each corner point are determined based on the first chessboard grid of the target, and the display screen coordinates of each corner point corresponding to the preset calibration grid are also determined.

[0163] Based on the pixel coordinates of each corner point and the display screen coordinates of each corner point, determine the coordinate transformation relationship between the display screen and the human eye camera.

[0164] In one feasible implementation, the coordinate traversal module 20 is further configured to:

[0165] Identify each first chessboard grid contained within the target chessboard grid image, and determine the identification information of the second chessboard grid to be filtered corresponding to each first chessboard grid.

[0166] Determine the preset chessboard grid identifier information, and filter multiple second chessboard grid identifier information based on the preset chessboard grid identifier information to determine the second target chessboard grid identifier information;

[0167] The target first chessboard grid is determined within the target chessboard grid image based on the second target chessboard grid identification information.

[0168] In one feasible implementation, the coordinate traversal module 20 is further configured to:

[0169] Determine the pixel coordinates of the target corner points for each of the first squares of the target chessboard.

[0170] Determine the chessboard size parameters corresponding to the first chessboard square of the target based on the pixel coordinates of each target corner point;

[0171] Based on the chessboard size parameters and the identification information of multiple second chessboard grids to be screened, the pixel coordinates of each corner point corresponding to the target chessboard grid image are determined.

[0172] The coordinate transformation relationship determination device provided in this application, employing the coordinate transformation relationship determination method in the above embodiments, can solve the technical problem in related technologies where MR glasses cannot accurately display captured objects on the display screen. Compared with the prior art, the beneficial effects of the coordinate transformation relationship determination device provided in this application are the same as those of the coordinate transformation relationship determination method provided in the above embodiments, and other technical features in the coordinate transformation relationship determination device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0173] This application provides an MR glasses, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the coordinate transformation relationship determination method in the above embodiment 1.

[0174] The following is for reference. Figure 13The diagram illustrates a structural schematic suitable for implementing MR glasses in the embodiments of this application. The MR glasses in the embodiments of this application may include, but are not limited to, MR glasses with an internal display screen capable of communicating with a human eye camera, or terminals such as mobile terminals, data storage control terminals, and PCs connected to an electronic control unit configured on the MR glasses. Figure 13 The MR glasses shown are merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0175] like Figure 13 As shown, the MR glasses may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the MR glasses. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the MR glasses to communicate wirelessly or wiredly with other devices to exchange data. Although MR glasses with various systems are shown in the figures, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0176] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0177] The MR glasses provided in this application, employing the coordinate transformation relationship determination method described in the above embodiments, can solve the technical problem in related technologies where MR glasses cannot accurately display captured objects on the display screen. Compared with the prior art, the beneficial effects of the MR glasses provided in this application are the same as those of the coordinate transformation relationship determination method provided in the above embodiments, and other technical features of the MR glasses are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0178] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0179] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0180] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the method for determining coordinate transformation relationships in the above embodiments.

[0181] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0182] The aforementioned computer-readable storage medium may be included in the MR glasses; or it may exist independently and not be assembled into the MR glasses.

[0183] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the MR glasses, the MR glasses cause the following: the display screen to show a preset calibration grid, and the human eye camera to capture images of the preset calibration grid to obtain target checkerboard images; based on the target checkerboard images, the glasses determine the identification information of each valid checkerboard grid and the pixel coordinates of each corner point corresponding to the preset calibration grid; based on the identification information of each valid checkerboard grid, the glasses determine the display screen coordinates of each target corner point corresponding to the preset calibration grid; and based on the pixel coordinates of each corner point and the display screen coordinates of each target corner point, the glasses determine the coordinate transformation relationship between the display screen and the human eye camera.

[0184] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0185] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0186] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0187] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for performing the coordinate transformation relationship determination method described above. This solves the technical problem in related technologies where MR glasses cannot accurately display captured objects on a screen. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the coordinate transformation relationship determination method provided in the above embodiments, and will not be repeated here.

[0188] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of determining the coordinate transformation relationship as described above.

[0189] The computer program product provided in this application can solve the technical problem in related technologies that MR glasses cannot accurately display the captured objects on the display screen. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the coordinate transformation relationship determination method provided in the above embodiments, and will not be repeated here.

[0190] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method of determining a coordinate transformation relationship, characterized by, The method for determining the coordinate conversion relationship is applied to an MR glasses provided with a display screen, the MR glasses are in communication connection with a human eye camera, and the method comprises the following steps: controlling the display screen to display a preset calibration checkerboard, and controlling the human eye camera to capture the preset calibration checkerboard to obtain a target checkerboard image; determining respective effective checkerboard identification information and respective corner pixel coordinates corresponding to the preset calibration checkerboard based on the target checkerboard image; determining respective target corner display screen coordinates corresponding to the preset calibration checkerboard according to the respective effective checkerboard identification information; determining the coordinate conversion relationship between the display screen and the human eye camera based on the respective corner pixel coordinates and the respective target corner display screen coordinates.

2. The method of claim 1, wherein The step of determining the respective effective checkerboard identification information corresponding to the preset calibration checkerboard based on the target checkerboard image comprises the following steps: identifying respective first checkerboards contained in the target checkerboard image, and determining respective clear checkerboards contained in the respective first checkerboards; reading respective first label images contained in the respective clear checkerboards, and determining the respective effective checkerboard identification information corresponding to the preset calibration checkerboard based on the respective first label images.

3. The method of claim 1, wherein The step of determining the respective target corner display screen coordinates corresponding to the preset calibration checkerboard according to the respective effective checkerboard identification information comprises the following steps: determining respective second checkerboards contained in the display screen, and reading respective second label images contained in the respective second checkerboards; determining respective first to-be-screened checkerboard identification information corresponding to the respective second checkerboards based on the respective second label images; determining the respective target corner display screen coordinates corresponding to the preset calibration checkerboard according to the respective effective checkerboard identification information and the respective first to-be-screened checkerboard identification information.

4. The method of claim 3, wherein The step of determining the respective target corner display screen coordinates corresponding to the preset calibration checkerboard according to the respective effective checkerboard identification information and the respective first to-be-screened checkerboard identification information comprises the following steps: screening the respective first to-be-screened checkerboard identification information according to the respective effective checkerboard identification information to determine respective first target checkerboard identification information matched with the respective effective checkerboard identification information; determining respective target second checkerboards based on the respective first target checkerboard identification information, and determining the respective target corner display screen coordinates corresponding to the preset calibration checkerboard according to the respective target second checkerboards.

5. The method of claim 1, wherein The step of determining the coordinate conversion relationship between the display screen and the human eye camera based on the respective corner pixel coordinates and the respective target corner display screen coordinates comprises the following steps: determining a rotation vector between the human eye camera and the display screen based on the respective corner pixel coordinates and the respective target corner display screen coordinates; transforming the rotation vector to determine the coordinate conversion relationship between the human eye camera and the display screen.

6. The method of claim 1, wherein After the step of controlling the human eye camera to capture the preset calibration checkerboard to obtain a target checkerboard image, the method further comprises the following steps: In a case where it is detected that the target chessboard image has distortion, a target first chessboard contained in the target chessboard image is determined, wherein the target first chessboard is located at a center position of the target chessboard image; Based on the target first chessboard, each corner pixel coordinate is determined, and each corner display screen coordinate corresponding to the preset calibration grid plate is determined; According to each corner pixel coordinate and each corner display screen coordinate, a coordinate conversion relationship between the display screen and the human eye camera is determined.

7. The method of claim 6, wherein The step of determining the target first chessboard contained in the target chessboard image comprises: Each first chessboard contained in the target chessboard image is determined, and each second to-be-screened chessboard identification information corresponding to each first chessboard is determined; Preset chessboard identification information is determined, and a plurality of second to-be-screened chessboard identification information is screened based on the preset chessboard identification information to determine second target chessboard identification information; The target first chessboard contained in the target chessboard image is determined based on the second target chessboard identification information.

8. The method of claim 6, wherein, The step of determining each corner pixel coordinate based on the target first chessboard comprises: Each target corner pixel coordinate corresponding to the target first chessboard is determined; According to each target corner pixel coordinate, a chessboard size parameter corresponding to the target first chessboard is determined; Based on the chessboard size parameter and a plurality of second to-be-screened chessboard identification information, each corner pixel coordinate corresponding to the target chessboard image is determined.

9. MR glasses, characterized in that The MR glasses comprise a display screen, a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the coordinate conversion relationship determination method according to any one of claims 1 to 8.

10. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, wherein the computer program is executed by the processor to implement the steps of the coordinate conversion relationship determination method according to any one of claims 1 to 8.

11. A computer program product, characterised in that, The computer program product comprises a computer program, wherein the computer program is executed by the processor to implement the steps of the coordinate conversion relationship determination method according to any one of claims 1 to 8.

Images