Virtual display control method, device and near-eye display device of a near-eye display device

Abstract

An embodiment of the present application discloses a virtual display control method, device, and near-eye display device for a near-eye display device, which relates to the technical field of near-eye displays. The method includes: displaying a reference target, determining an ideal conversion relationship between an ideal simulation camera with ideal position parameters and the reference target, and determining an actual conversion relationship between an actual simulation camera with actual position parameters and the reference target. A target conversion relationship is determined based on the ideal conversion relationship and the actual conversion relationship, and the near-eye display device is controlled to display a target virtual image according to the target conversion relationship. By determining the target conversion relationship between the ideal simulation camera and the actual simulation camera, and compensating and adjusting the display of the near-eye display device through the target conversion relationship, the accuracy of device adjustment is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of near-eye display, and specifically relate to a virtual display control method, device and near-eye display device for a near-eye display device. Background Art

[0002] A near-eye display device is a display technology that directly projects images in front of the user's eyes, and is widely used in fields such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality). Its core goal is to provide a high-definition, large field of view (FOV), and low-latency visual experience at a very short distance (usually <50 mm).

[0003] A near-eye display device usually adjusts by taking the image content captured by a simulated human eye camera as a reference. For example, for adjustments such as display alignment, thus, the position accuracy of the simulated human eye camera, that is, the information accuracy provided by the simulated human eye camera, directly determines the accuracy of image display.

[0004] However, in the actual production and assembly process, due to factors such as machining errors and human errors during the assembly process, the positions of the binocular simulated human eye cameras usually deviate from the designed positions. In the prior art, the actual assembled positions of the simulated human eye cameras are usually directly used as the designed ideal positions, that is, installed at the actual assembled positions, but still perform operations such as display alignment based on the designed ideal positions. This ignores the deviation between the actual position and the designed ideal position, resulting in the inability to adjust the near-eye display device based on the actual installation position. Summary of the Invention

[0005] The embodiments of the present application provide a virtual display control method, device and near-eye display device for a near-eye display device, which can improve the accuracy of device adjustment when adjusting the near-eye display device based on a simulated camera.

[0006] In a first aspect, the embodiments of the present application provide a virtual display control method for a near-eye display device, and the method includes:

[0007] Display a reference target;

[0008] Determine the ideal conversion relationship between the ideal simulated camera with ideal position parameters and the reference target, and determine the actual conversion relationship between the actual simulated camera with actual position parameters and the reference target;

[0009] Determine a target conversion relationship according to the ideal conversion relationship and the actual conversion relationship;

[0010] Control the near-eye display device to display a target virtual image according to the target conversion relationship.

[0011] In a second aspect, an embodiment of the present application further provides a virtual display control device for a near-eye display device, the device comprising:

[0012] A display module for displaying a reference target;

[0013] A first determination module for determining an ideal conversion relationship between an ideal simulated camera with ideal position parameters and the reference target, and determining an actual conversion relationship between an actual simulated camera with actual position parameters and the reference target;

[0014] A second determination module for determining a target conversion relationship according to the ideal conversion relationship and the actual conversion relationship;

[0015] A control module for controlling the near-eye display device to display a target virtual image according to the target conversion relationship.

[0016] Optionally, in some embodiments of the present application, the near-eye display device includes an optical engine and an optical lens;

[0017] The controlling the near-eye display device to display a target virtual image according to the target conversion relationship includes:

[0018] Adjusting the positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target positional relationship;

[0019] Displaying the target virtual image based on the target positional relationship.

[0020] Optionally, in some embodiments of the present application, the adjusting the positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target positional relationship includes:

[0021] Displaying a target target image through the virtual display screen of the near-eye display device;

[0022] Determining a first projection coordinate of a target reference object in the target target image under an ideal virtual display screen corresponding to the ideal simulated camera;

[0023] Determining a second projection coordinate of the target reference object in the ideal virtual display screen corresponding to the actual simulated camera according to the first projection coordinate and the target conversion relationship;

[0024] Adjusting the positional relationship between the optical engine and the optical lens according to the target reference object and the second projection coordinate in the target target image under the virtual display screen to obtain a target positional relationship.

[0025] Optionally, in some embodiments of the present application, determining the second projection coordinates of the target reference object corresponding to the ideal virtual display screen in the actual simulated camera according to the first projection coordinates and the target conversion relationship includes:

[0026] Calculating a homography matrix between the ideal simulated camera and the actual simulated camera according to the target conversion relationship, the normal vector coordinates of the ideal virtual display screen corresponding to the ideal simulated camera, and the internal parameters of the actual simulated camera;

[0027] Transforming the first projection coordinates into the second projection coordinates corresponding to the actual simulated camera according to the homography matrix.

[0028] Optionally, in some embodiments of the present application, the ideal simulated camera includes an ideal left-eye simulated camera and an ideal right-eye simulated camera, and the actual simulated camera includes an actual left-eye simulated camera and an actual right-eye simulated camera;

[0029] The target conversion relationship includes a target left-eye conversion relationship and a target right-eye conversion relationship, where the target left-eye conversion relationship is determined based on the left-eye virtual display screen, the ideal left-eye simulated camera, and the actual left-eye simulated camera, and the target right-eye conversion relationship is determined based on the right-eye virtual display screen, the ideal right-eye simulated camera, and the actual right-eye simulated camera;

[0030] The target position relationship includes a target left-eye position relationship and a target right-eye position relationship, the optical engine includes a left-eye optical engine and a right-eye optical engine, and the optical lens includes a left-eye optical lens and a right-eye optical lens;

[0031] Adjusting the position relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target position relationship includes:

[0032] Adjusting the position relationship between the left-eye optical engine and the left-eye optical lens according to the target left-eye conversion relationship to obtain the target left-eye position relationship, and adjusting the position relationship between the right-eye optical engine and the right-eye optical lens according to the target right-eye conversion relationship to obtain the target right-eye position relationship.

[0033] Optionally, in some embodiments of the present application, the virtual display screen of the near-eye display device includes a left-eye virtual display screen and a right-eye virtual display screen, and the ideal virtual display screen of the near-eye display device includes an ideal left-eye virtual display screen and an ideal right-eye virtual display;

[0034] Before displaying the target virtual image based on the target position relationship, the method further includes:

[0035] The test target images are respectively displayed through the left-eye virtual display screen and the right-eye virtual display screen of the near-eye display device, wherein the test target images contain target corner points;

[0036] Determine the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulated camera, and determine the actual three-dimensional coordinates of the target corner point corresponding to the left-eye virtual display screen in the actual left-eye simulated camera;

[0037] Determine the theoretical three-dimensional coordinates of the ideal three-dimensional coordinates corresponding to the left-eye simulated camera according to the target left-eye conversion relationship;

[0038] Verify the target position relationship according to the actual three-dimensional coordinates and the theoretical three-dimensional coordinates to obtain a position verification result;

[0039] If the position verification result indicates that the binocular image combination of the left-eye virtual display screen and the right-eye virtual display screen meets the preset conditions, display the target virtual images through the left-eye virtual display screen and the right-eye virtual display screen respectively.

[0040] Optionally, in some embodiments of the present application, the determination of the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulated camera includes:

[0041] Calculate the first test projection coordinates of the target corner point in the ideal left-eye virtual display screen corresponding to the ideal left-eye simulated camera;

[0042] Calculate the second test projection coordinates of the target corner point in the ideal right-eye virtual display screen corresponding to the ideal right-eye simulated camera;

[0043] Perform triangulation processing on the first test projection coordinates and the second test projection coordinates according to the internal parameters of the ideal left-eye simulated camera, the internal parameters of the ideal right-eye simulated camera, and the external parameters between the ideal left-eye simulated camera and the ideal right-eye simulated camera to obtain the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye simulated camera.

[0044] In a third aspect, an embodiment of the present application further provides a near-eye display device, which includes the near-eye display device in the virtual display control method of the near-eye display device described above. Wherein, the near-eye display device displays a target target image through a virtual display screen, and the near-eye display device controls the output of a target virtual image according to a target conversion relationship.

[0045] Fourthly, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the virtual display control method of the near-eye display device described above are implemented.

[0046] Fifthly, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the virtual display control method of the near-eye display device described above are implemented.

[0047] Sixthly, an embodiment of the present application further provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementation manners described in the embodiments of the present application.

[0048] The embodiment of the present application displays a reference target, determines an ideal conversion relationship between an ideal simulation camera for determining ideal position parameters and the reference target, and determines an actual conversion relationship between an actual simulation camera for determining actual position parameters and the reference target. A target conversion relationship is determined according to the ideal conversion relationship and the actual conversion relationship, and the near-eye display device is controlled to display a target virtual image according to the target conversion relationship.

[0049] Among them, since there are differences between the ideal position parameters of the ideal simulation camera and the actual position parameters of the actual simulation camera, and by determining the target conversion relationship between the ideal simulation camera and the actual simulation camera, the display of the near-eye display device can be compensated and adjusted through the target conversion relationship, improving the accuracy of device adjustment. Description of the Drawings

[0050] In order to more clearly illustrate the technical solutions in the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0051] Figure 1 is a schematic diagram of a scenario where an electronic device provided by an embodiment of the present application executes the virtual display control method of the near-eye display device;

[0052] Figure 2 is a schematic flowchart of the virtual display control method of the near-eye display device provided by an embodiment of the present application;

[0053] Figure 3 It is a schematic diagram of the target image provided by an embodiment of the present application;

[0054] Figure 4 It is a schematic diagram of the positional relationship between the binocular ideal simulation cameras and the reference target provided by an embodiment of the present application;

[0055] Figure 5 It is a schematic diagram of the structure of the near-eye display device provided by an embodiment of the present application;

[0056] Figure 6 It is a schematic diagram of the structure of the virtual display control device of the near-eye display device provided by an embodiment of the present application;

[0057] Figure 7 It is a schematic diagram of the structure of the electronic device provided by an embodiment of the present application.

[0058] Explanation of the reference numerals in the drawings:

[0059] 10 - Electronic device; 11 - Near-eye display device; 111 - Left optical lens; 112 - Right optical lens; 113 - Left optical engine; 201 - Display module; 202 - First determination module; 203 - Second determination module; 204 - Control module; 301 - Processor; 302 - Memory; 303 - Power supply; 304 - Input unit. Detailed implementation manners

[0060] Next, the technical solutions in the present application will be clearly and completely described in conjunction with the accompanying drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0061] An embodiment of the present application provides a virtual display control method, device, near-eye display device, electronic device and computer-readable storage medium for a near-eye display device. Specifically, an embodiment of the present application provides a virtual display control device for a near-eye display device applicable to an electronic device, which is used to improve the accuracy of device adjustment when adjusting the near-eye display device based on a simulation camera. Specifically, the electronic device includes computer devices such as laptop computers, desktop computers or industrial control computers.

[0062] Please refer to Figure 1 , Figure 1 It is a schematic diagram of the scenario in which the electronic device executes the virtual display control method for the near-eye display device provided by an embodiment of the present application. Among them, the specific execution process of the electronic device executing the virtual display control method for the near-eye display device is as follows:

[0063] The electronic device 10 displays a reference target, determines an ideal conversion relationship between an ideal simulated camera with ideal position parameters and the reference target, and determines an actual conversion relationship between an actual simulated camera with actual position parameters and the reference target. The electronic device 10 determines a target conversion relationship based on the ideal conversion relationship and the actual conversion relationship, and controls the near-eye display device 11 to display a target virtual image according to the target conversion relationship.

[0064] For example, the electronic device 10 controls the display of the reference target, determines the ideal conversion relationship and the actual conversion relationship, calculates the target conversion relationship based on the ideal conversion relationship and the actual conversion relationship, and then adjusts the near-eye display device based on the target conversion relationship. In addition, the near-eye display device 11 displays the target virtual image according to the target conversion relationship.

[0065] In summary, due to the difference between the ideal position parameters of the ideal simulated camera and the actual position parameters of the actual simulated camera, in the embodiments of the present application, by determining the target conversion relationship between the ideal simulated camera and the actual simulated camera, the display of the near-eye display device can be compensated and adjusted through the target conversion relationship, improving the accuracy of device adjustment.

[0066] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the priority order of the embodiments.

[0067] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of a virtual display control method for a near-eye display device provided by an embodiment of the present application. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order from that shown in the flowchart. Specifically, the execution subject of the virtual display control method for the near-eye display device is an electronic device, which includes computer devices such as a laptop computer, a desktop computer, or an industrial control computer. Specifically, the process of the electronic device executing the virtual display control method for the near-eye display device specifically includes:

[0068] 101. Display a reference target.

[0069] The reference target is a specific pattern or mark used to calibrate, position, measure, or evaluate the system performance. For example, the reference target is a checkerboard.

[0070] 102. Determine an ideal conversion relationship between an ideal simulated camera with ideal position parameters and the reference target, and determine an actual conversion relationship between an actual simulated camera with actual position parameters and the reference target.

[0071] It should be noted that an ideal simulated camera is a simulated human eye camera with ideal position parameters (i.e., ideal installation position). For example, for a near-eye display device, the ideal position parameters refer to the best position for observing the image output by the near-eye display device. For instance, for a binocular near-eye display device, the ideal simulated camera refers to a simulated human eye camera that is parallel to the virtual display screen of the near-eye display device and is located at a position where the eyes are closed to a specific distance from the near-eye display device.

[0072] Correspondingly, a real simulated camera is a simulated human eye camera with actual position parameters (i.e., actual installation position or actual assembly position). For example, when installing a simulated human eye camera according to the ideal position parameters, due to errors during the assembly process, the simulated human eye camera is installed at the actual installation position instead of the ideal installation position. Then, the simulated human eye camera at the actual installation position rather than the ideal installation position is the real simulated camera.

[0073] It should be noted that the ideal simulated camera and the real simulated camera can be simulated human eye cameras of the same type and with the same internal and external parameters. In the embodiments of the present application, they are only distinguished based on the different installation positions.

[0074] Among them, the ideal conversion relationship is the conversion relationship (i.e., position relationship or coordinate relationship) between the ideal simulated camera and the reference target calculated based on the designed ideal position parameters. Among them, this conversion relationship includes a rotation relationship and a translation relationship.

[0075] Among them, the actual conversion relationship is the conversion relationship calculated based on the actual position parameters. In the embodiments of the present application, the reference target image is obtained by the real simulated camera shooting the reference target, and then the corner points in the reference target image are extracted, and the actual conversion relationship is solved through the coordinate information of these corner points and using the PnP algorithm.

[0076] It can be understood that due to the differences between the ideal position parameters and the actual position parameters, there are also differences between the ideal conversion relationship and the actual conversion relationship, that is, there is a position difference between the ideal simulated camera and the real simulated camera.

[0077] 103. Determine the target conversion relationship based on the ideal conversion relationship and the actual conversion relationship.

[0078] It should be noted that since the ideal simulated camera is a simulated human eye camera with ideal position parameters, in the prior art, the position information provided by this ideal simulated camera is used to optimize the display of the near-eye display device. For example, by using the ideal simulated camera to shoot the content displayed by the near-eye display device and performing alignment processing between the content and the projection coordinates under this ideal simulated camera to achieve the adjustment of the near-eye display device.

[0079] However, due to reasons such as the installation error of the simulated eye camera, there is a difference between the actual installation position of the actual simulated camera and the ideal position. Therefore, there is a difference between the content captured by the actual simulated camera and the content captured by the ideal simulated camera. Furthermore, if the content captured by the actual simulated camera is directly aligned with the projection coordinates of the ideal simulated camera, there will be an alignment reference error, resulting in inaccurate device adjustment and the problem that the content cannot be displayed at the specified target position.

[0080] In addition, since the ideal conversion relationship represents the positional relationship between the reference target and the ideal simulated camera, the actual conversion relationship represents the positional relationship between the reference target and the actual simulated camera, and both the ideal conversion relationship and the actual conversion relationship are determined based on the reference target as a reference. Therefore, in the embodiments of the present application, the positional relationship (i.e., the conversion relationship) between the ideal simulated camera and the actual simulated camera can be calculated based on the aforementioned ideal conversion relationship and actual conversion relationship. That is, the target conversion relationship characterizes the positional relationship (i.e., the conversion relationship) between the ideal simulated camera and the actual simulated camera.

[0081] Based on this, in the embodiments of the present application, by calculating the ideal conversion relationship and the actual conversion relationship, and calculating the target conversion relationship based on the ideal conversion relationship and the actual conversion relationship, the determination of the positional relationship between the ideal simulated camera and the actual simulated camera is achieved. Furthermore, it helps to adjust the reference object corresponding to the captured content based on the target conversion relationship, thereby improving the accuracy of device adjustment.

[0082] 104. Control the near-eye display device to display the target virtual image according to the target conversion relationship.

[0083] For example, adjust the display parameters of the near-eye display device according to the target conversion relationship to improve the accuracy of device adjustment, so that the content output by the near-eye display device matches the information provided by the position of the actual simulated camera. Furthermore, ensure that a target virtual image with accurate display position and good display effect can be observed through the actual simulated camera.

[0084] It can be understood that the target virtual image is virtual content generated by calculation. For example, taking the augmented reality device in the near-eye display device as an example, the target virtual image is the virtual content superimposed on the real scene for augmented reality.

[0085] In summary, due to the difference between the ideal position parameters of the ideal simulated camera and the actual position parameters of the actual simulated camera, the embodiments of the present application determine the target conversion relationship between the ideal simulated camera and the actual simulated camera, and perform compensation adjustment on the display of the near-eye display device through the target conversion relationship to improve the accuracy of device adjustment.

[0086] It can be understood that, in order to ensure that the image output by the near-eye display device matches the information provided by the position of the actual simulated camera, embodiments of the present application can adjust the image output position of the near-eye display device to match the information provided by the position of the actual simulated camera. The image output position can be achieved by adjusting the light engine of the near-eye display device. For example, by adjusting the positional relationship between the light engine of the near-eye display device and the optical lens, the position of the virtual content projected by the light engine in the optical lens is changed, so that the image output by the near-eye display device matches the information provided by the position of the actual simulated camera. That is, optionally, in some embodiments of the present application, the near-eye display device includes a light engine and an optical lens. The step of "controlling the near-eye display device to display a target virtual image according to the target conversion relationship" includes:

[0087] Adjust the positional relationship between the light engine and the optical lens according to the target conversion relationship to obtain a target positional relationship;

[0088] Display the target virtual image based on the target positional relationship.

[0089] For example, adjust the position of the light engine according to the target conversion relationship, thereby changing the display position of the target virtual image.

[0090] It should be noted that, in order to ensure the accuracy of the display content of the near-eye display device, in the prior art, projection coordinates for the image plane of an ideal simulated camera are designed, and display alignment processing is performed with reference to these projection coordinates. For example, when the imaging coordinates of the target reference object (such as the entire crosshair) in the target target image on the ideal simulated camera are aligned with these projection coordinates, it is considered that the display position of the near-eye display device is accurate at this time and the display effect is good. Among them, the projection coordinates of the imaging plane of the ideal simulated camera are obtained by projecting the content displayed on the ideal virtual display screen of the near-eye display device, that is, the projection coordinates are obtained by projecting the coordinates of the display content on the ideal virtual display screen onto the imaging plane of the ideal simulated camera.

[0091] However, due to the actual installation error problem of the simulated human eye camera, the simulated human eye camera may not necessarily be installed in the ideal position, that is, the actual simulated camera corresponding to the embodiments of the present application. Therefore, the projection coordinates for the ideal simulated camera are not applicable to the alignment processing of the actual simulated camera, and it is necessary to calculate the projection coordinates for the actual simulated camera and perform alignment processing for the actual simulated camera according to the calculated projection coordinates.

[0092] In the embodiments of the present application, the projection coordinates applicable to the actual simulated camera are calculated according to the conversion relationship between the ideal simulated camera and the actual simulated camera. That is, optionally, in some embodiments of the present application, the step of "adjusting the positional relationship between the light engine and the optical lens according to the target conversion relationship to obtain a target positional relationship" includes:

[0093] Display a target target image through the virtual display screen of the near-eye display device;

[0094] Determine the first projection coordinates of the target reference object in the target target image under the ideal virtual display screen corresponding to the ideal simulation camera;

[0095] Determine the second projection coordinates of the target reference object under the ideal virtual display screen corresponding to the actual simulation camera according to the first projection coordinates and the target conversion relationship;

[0096] Adjust the positional relationship between the optical engine and the optical lens according to the target reference object and the second projection coordinates in the target target image under the virtual display screen to obtain the target positional relationship.

[0097] Among them, the near-eye display device is a display technology that directly projects images in front of the user's eyes and is widely used in fields such as VR (virtual reality), AR (augmented reality), and MR (mixed reality). For example, the near-eye display device includes extended reality devices such as augmented reality devices and virtual reality devices. The core goal of the near-eye display device is to provide a high-definition, large field of view (FOV), and low-latency visual experience at a very close distance (usually <50mm).

[0098] It should be noted that the virtual display screen is not the physical screen of the near-eye display device. After the user wears the near-eye display device, through the direction of the optical lens (such as a lens), a screen with a certain resolution (such as 1280*960) can be seen at a position close to 4m in front of the eyes sensorially. This screen with a certain resolution is the virtual display screen. For example, the optical engine projects the image of a micro display (such as Micro-OLED, LCoS, DLP, etc.) onto the optical lens through an optical system. When the user wears the near-eye display device, the eyes fixate on the optical lens and can see a virtual image in front of the eyes, and this virtual image appears on the virtual display screen.

[0099] It should be noted that the virtual display screen is the actually displayed virtual display screen, and the ideal virtual display screen is the ideally displayed virtual display screen. For example, it is strictly aligned with the optical axis in position, has no distortion, and does not require calibration, etc. The content displayed on the ideal virtual display screen meets the ideal requirements. The first projection coordinates projected based on the content displayed on the ideal virtual display screen are applicable as the alignment reference based on the ideal simulation camera. However, since the alignment reference in the embodiments of the present application is the projection coordinates in the actual simulation camera, therefore, the second projection coordinates that can be used as the alignment reference in the actual simulation camera can be calculated according to the first projection coordinates and the target conversion relationship.

[0100] Among them, in the embodiments of the present application, in order to more intuitively and accurately determine whether the coordinate positions are aligned, the target target image is a target image with a crosshair selected, such as Figure 3 , Figure 3 is a schematic diagram of the target target image provided by the embodiments of the present application. Crosshairs are respectively arranged at the four corners and the middle position of the image shown in Figure 3 . It should be noted that the target target images displayed by the virtual display screen and the ideal virtual display screen in the embodiments of the present application are the same, that is, the coordinates of the target reference object in the target target image are the same under the virtual display screen and the ideal virtual display screen.

[0101] It should be noted that the first projection coordinate is the projection coordinate for the aforementioned ideal simulation camera. That is, when the imaging coordinate of the target reference object (such as the entire crosshair) in the target target image on the image plane of the ideal simulation camera is aligned with the first projection coordinate, it is considered that at this time, the image content with accurate image position and good display effect can be observed on the optical lens of the near-eye display device through the ideal simulation camera. It can be understood that the first projection coordinate is pre-configured. For example, it is a projection coordinate that can observe accurate position and good display effect image content based on the ideal virtual display screen and the ideal position parameters.

[0102] It can be understood that the second projection coordinate is a projection coordinate that can observe accurate position and good display effect image content based on the first projection coordinate and the target conversion relationship under the actual position parameters. That is, when the imaging coordinate of the target reference object (such as the entire crosshair) in the target target image on the image plane of the actual simulation camera is aligned with the second projection coordinate, it is considered that at this time, the image content with accurate image position and good display effect can be observed on the optical lens of the near-eye display device through the actual simulation camera.

[0103] It should be noted that after the second projection coordinate is determined, the optical engine can be adjusted according to the second projection coordinate, so that the imaging coordinate of the target reference object in the target target image output by the virtual display screen (here refers to the actual virtual display screen) of the near-eye display device is aligned with the second projection coordinate. For example, the position of the optical engine is adjusted until the imaging coordinate of the target reference object in the target target image on the virtual display screen on the actual simulation camera is aligned with the second projection coordinate. Among them, this adjustment can be completed by driving a manipulator, or an instruction message can be sent to the user to guide the user to adjust the position of the optical engine to complete.

[0104] Specifically, based on the requirements of coordinate transformation, a homography matrix can be generated based on the target transformation relationship, and the first projection coordinates can be transformed into the second projection coordinates based on the homography matrix. That is, optionally, in some embodiments of the present application, the step of "determining the second projection coordinates of the target reference object corresponding to the ideal virtual display screen in the actual simulated camera according to the first projection coordinates and the target transformation relationship" includes:

[0105] Calculate the homography matrix between the ideal simulated camera and the actual simulated camera according to the target transformation relationship, the normal vector coordinates of the ideal virtual display screen corresponding to the ideal simulated camera, and the internal parameters of the actual simulated camera;

[0106] Transform the first projection coordinates into the second projection coordinates corresponding to the actual simulated camera according to the homography matrix.

[0107] It should be noted that if the ideal simulated camera and the actual simulated camera adopt simulated eye cameras with the same parameters, the internal parameters of the ideal simulated camera and the actual simulated camera are the same.

[0108] It can be understood that for a near-eye display device, it is generally binocular that conforms to the characteristics of the human eye (which corresponds to a left-eye virtual display screen and a right-eye virtual display screen, and correspondingly, there are an ideal left-eye virtual display screen and an ideal right-eye virtual display screen), that is, it includes a left-eye optical engine and a right-eye optical engine, and includes a left-eye optical lens corresponding to the left-eye optical engine and a right-eye optical lens corresponding to the right-eye optical engine. Therefore, whether the image display position is accurate directly affects the accuracy of binocular image synthesis. Therefore, in the embodiments of the present application, the ideal simulated cameras and the actual simulated cameras of binoculars are taken as examples, and whether the binocular image synthesis observed by the ideal simulated cameras and the actual simulated cameras of binoculars is aligned is used to judge whether the image display position is accurate.

[0109] That is, in the embodiments of the present application, the virtual display screen of the near-eye display device includes a left-eye virtual display screen and a right-eye virtual display screen, the ideal virtual display screen includes an ideal left-eye virtual display screen and an ideal right-eye virtual display screen, the ideal simulated camera includes an ideal left-eye simulated camera and an ideal right-eye simulated camera, and the actual simulated camera includes an actual left-eye simulated camera and an actual right-eye simulated camera;

[0110] The target transformation relationship includes a target left-eye transformation relationship and a target right-eye transformation relationship. Among them, the target left-eye transformation relationship is determined based on the ideal left-eye simulated camera and the actual left-eye simulated camera, and the target right-eye transformation relationship is determined based on the ideal right-eye simulated camera and the actual right-eye simulated camera;

[0111] The target position relationship includes a target left-eye position relationship and a target right-eye position relationship. The optical engine includes a left-eye optical engine and a right-eye optical engine, and the optical lens includes a left-eye optical lens and a right-eye optical lens;

[0112] Adjust the positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain the target positional relationship, including:

[0113] Adjust the positional relationship between the left-eye optical engine and the left-eye optical lens according to the target left-eye conversion relationship to obtain the target left-eye positional relationship, and adjust the positional relationship between the right-eye optical engine and the right-eye optical lens according to the target right-eye conversion relationship to obtain the target right-eye positional relationship.

[0114] For example, for the left-eye optical engine, it is adjusted based on the left-eye virtual display screen, the ideal left-eye virtual display screen, the ideal left-eye simulation camera, and the actual left-eye simulation camera. That is, the target left-eye conversion relationship for the left-eye optical engine is determined according to the ideal left-eye simulation camera and the actual left-eye simulation camera, and the second projection relationship for the actual left-eye simulation camera is obtained according to the target left-eye conversion relationship and the first projection coordinates for the ideal left-eye simulation camera (the first projection coordinates are calculated based on the coordinate projection of the ideal left-eye virtual display screen and the ideal left-eye simulation camera). Furthermore, the left-eye optical engine is adjusted according to the second projection relationship for the actual left-eye simulation camera and the left-eye virtual display screen, so that the imaging coordinates of the target reference object finally displayed by the left-eye optical engine on the actual left-eye simulation camera are aligned with the second projection relationship of the actual left-eye simulation camera.

[0115] Similarly, for the right-eye optical engine, it is adjusted based on the right-eye virtual display screen, the ideal right-eye virtual display screen, the ideal right-eye simulation camera, and the actual right-eye simulation camera. That is, the target right-eye conversion relationship for the right-eye optical engine is determined according to the ideal right-eye simulation camera and the actual right-eye simulation camera, and the second projection relationship for the actual right-eye simulation camera is obtained according to the target right-eye conversion relationship and the first projection coordinates for the ideal right-eye simulation camera (the first projection coordinates are calculated based on the coordinate projection of the ideal right-eye virtual display screen and the ideal right-eye simulation camera). Furthermore, the right-eye optical engine is adjusted according to the second projection relationship for the actual right-eye simulation camera and the right-eye virtual display screen, so that the imaging coordinates of the target reference object finally displayed by the right-eye optical engine on the actual right-eye simulation camera are aligned with the second projection relationship of the actual right-eye simulation camera.

[0116] Specifically, in order to more clearly illustrate the adjustment of the left-eye optical engine and the right-eye optical engine in the embodiments of the present application, the adjustment of the left-eye optical engine and the right-eye optical engine will be specifically described below in combination with specific parameters. Specifically, it includes a calibration camera module, a verification camera calibration accuracy module, a determination of compensation information module, and a binocular image synthesis optimization module. Specifically:

[0117] The calibration camera module is used to calibrate the actual simulation cameras of both eyes, specifically including:

[0118] 1.1. When the actual simulated binocular cameras are not calibrated, or the internal and external parameters of the previously calibrated actual simulated binocular cameras do not meet the requirements, calibrate the actual simulated binocular cameras.

[0119] 1.2. When calibrating the actual simulated binocular cameras, change the relative position between the actual simulated binocular cameras and the camera calibration target. The cameras acquire images of the camera calibration target at different relative positions, where all the acquired images of the camera calibration target should cover the entire camera image as evenly as possible.

[0120] 1.3. Use the camera internal and external parameter calibration algorithm to process the images of the camera calibration target acquired by the actual simulated binocular cameras, and obtain the internal parameters of the actual simulated binocular cameras and the conversion relationship between the actual left simulated camera and the actual right simulated camera.

[0121] 1.4. Calibrate the actual simulated binocular cameras based on the internal parameters of the actual simulated binocular cameras and the conversion relationship between the actual left simulated camera and the actual right simulated camera.

[0122] The camera calibration accuracy verification module is used to verify the calibration accuracy of the actual simulated binocular cameras, specifically including:

[0123] 2.1. Each of the actual simulated binocular cameras captures an image facing the camera calibration target.

[0124] 2.2. Extract corner points on the undistorted camera calibration target images.

[0125] 2.3. Classify the corner points by row and column, and use all the corner points on each row or column line. Use the fitLine algorithm in the opencv toolbox to fit the line parameters and calculate the mean distance and variance of all corner points to the corresponding horizontal and vertical lines. and variance .

[0126] 2.4. Search for the imaging coordinates of all corner points with the same id on the image plane of the actual simulated binocular cameras. Triangulate to obtain the corresponding triangulated 3D coordinates of each id corner point based on the internal and external parameters of the actual simulated binocular cameras, and calculate the mean error and variance between all the triangulated 3D coordinates and the corresponding true 3D coordinates. and variance .

[0127] 2.5. When the mean distance and variance of all corner points to the corresponding horizontal and vertical lines are both less than the threshold, and the mean error and variance When both are less than the threshold value, it is considered that the internal and external parameters of the calibrated binocular actual simulation cameras meet the requirements. Otherwise, re - execute the steps of the foregoing camera calibration module to calibrate the binocular actual simulation cameras.

[0128] The position compensation information module is used to determine the position compensation information of the binocular actual simulation cameras. Among them, in order to achieve high - precision binocular image synthesis of the near - eye display device at a specified eye - closing distance, it is required that the binocular simulated human - eye cameras be placed at the designed ideal positions, that is, to be the ideal simulated cameras of the binoculars. Taking a fixed reference target in space (that is, the image - synthesis target for binocular image synthesis) as the standard, the positions of the ideal simulated cameras of the binoculars and the near - eye display device are determined. Please refer to Figure 4 , Figure 4 is a schematic diagram of the position relationship between the ideal simulated cameras of the binoculars and the reference target provided by an embodiment of the present application. Among them, assuming that the x - axis direction of the reference target is parallel to the connection direction of the left and right optical lenses of the near - eye display device, the ideal simulated cameras of the binoculars meet the following conditions a - i. Specifically:

[0129] a. There is only a rotation around the z - axis between the optical axis of the ideal left - eye simulated camera and the normal vector of the reference target, and the included angle is degrees.

[0130] b. The optical axis of the ideal left - eye virtual display screen of the near - eye display device coincides with the optical axis of the ideal left - eye simulated camera, and this optical axis intersects the reference target at point A.

[0131] c. There is only a rotation around the z - axis between the optical axis of the ideal right - eye simulated camera and the normal vector of the reference target, and the included angle is degrees.

[0132] d. The optical axis of the ideal right - eye virtual display screen of the near - eye display device coincides with the optical axis of the ideal right - eye simulated camera, and this optical axis intersects the reference target at point B. Among them, point A and point B have the same yz - axis coordinates and different x - axis coordinates.

[0133] e. The virtual camera center corresponding to the ideal left - eye virtual display screen of the near - eye display device coincides with the center of the ideal left - eye simulated camera.

[0134] f. The virtual camera center corresponding to the ideal right - eye virtual display screen of the near - eye display device coincides with the center of the ideal right - eye simulated camera.

[0135] g. The distance from the center of the center of the ideal left - eye simulated camera and the center of the ideal right - eye simulated camera to the reference target is d, with the unit of m.

[0136] h. The distance between the center of the ideal left - eye simulated camera and the center of the ideal right - eye simulated camera is the standard inter - pupillary distance.

[0137] i. The binocular convergence distance between the center of the imaging plane of the ideal left-eye simulation camera and the center of the imaging plane of the ideal right-eye simulation camera is , with the unit of m.

[0138] However, in the actual production and assembly process, due to factors such as machining errors of the base (used to fix the near-eye display device and the actual simulation camera) and human errors during the assembly process, it is very difficult to adjust the binocular simulation human-eye cameras to the ideal positions designed. Therefore, in order to improve the accuracy of the position information of the actual binocular simulation cameras (i.e., the binocular simulation human-eye cameras installed at the actual position parameters), based on the positions of the assembled actual binocular simulation cameras, the position compensation information corresponding to the assembled positions is determined. That is, the specific modules for determining the compensation information include:

[0139] 3.1. Each of the actual binocular simulation cameras captures an image facing the reference target, obtaining two reference target images.

[0140] 3.2. Extract the corner points on the captured reference target images.

[0141] 3.3. Based on the 2D-3D matching points, using the pnp algorithm, calculate the transformation relationship between the actual left-eye simulation camera and the reference target and the transformation relationship between the actual right-eye simulation camera and the reference target , where represents the coordinate system of the actual left-eye simulation camera, represents the coordinate system of the actual right-eye simulation camera, represents the coordinate system of the reference target.

[0142] 3.4. According to the ideal position parameters of the ideal simulation cameras, calculate the transformation relationship between the ideal left-eye simulation camera and the reference target and the transformation relationship between the ideal right-eye simulation camera and the reference target , where represents the coordinate system of the ideal left-eye simulation camera, represents the coordinate system of the ideal right-eye simulation camera, represents the coordinate system of the reference target.

[0143] 3.5. Combine the transformation relationship between the actual left-eye simulation camera and the reference target , the transformation relationship between the ideal left-eye simulation camera and the reference target , and calculate the transformation relationship between the actual left-eye simulation camera and the ideal left-eye simulation camera , which is expressed as:

[0144]

[0145] And, combining the conversion relationship between the actual right-eye simulation camera and the reference target and the conversion relationship between the ideal right-eye simulation camera and the reference target to calculate the conversion relationship between the actual right-eye simulation camera and the ideal right-eye simulation camera which is expressed as:

[0146]

[0147] The binocular image fusion optimization module is used to adjust the display parameters of the near-eye display device based on the conversion relationship between the actual left-eye simulation camera and the ideal left-eye simulation camera and the conversion relationship between the actual right-eye simulation camera and the ideal right-eye simulation camera

[0148] 4.1. Display a target target image on each of the left-eye virtual display screen and the right-eye virtual display screen of the near-eye display device.

[0149] Among them, the target target image is as Figure 3 shown, and the coordinate sets of all the crosshair endpoints in the target target image under the left-eye virtual display screen and the right-eye virtual display screen are both . Among them, the coordinates of the crosshair endpoints in the virtual display screen (including the left-eye virtual display screen and the right-eye virtual display screen) and in the ideal virtual display screen (including the ideal left-eye virtual display screen and the ideal right-eye virtual display screen) are the same.

[0150] 4.2. Combining the conversion relationship between the ideal left-eye simulation camera and the virtual camera center corresponding to the ideal left-eye virtual display screen , the virtual image distance of the ideal left-eye virtual display screen of the near-eye display device, the internal parameters of the ideal left-eye virtual display screen of the near-eye display device, the internal parameters of the ideal left-eye simulation camera to calculate the projection coordinates of all the crosshair endpoints

[0151]

[0152] displayed on the ideal left-eye virtual display screen on the undistorted imaging plane of the ideal left-eye simulation camera where represents homogeneous coordinates. For example, represents the homogeneous coordinates corresponding to and represents the homogeneous coordinates corresponding to is a three-dimensional vector, , and represent variables for each dimension respectively, , represents the rotation from the ideal left-eye virtual display screen to the ideal left-eye simulated camera, represents the translation from the ideal left-eye virtual display screen to the ideal left-eye simulated camera, represents the coordinate system of the ideal left-eye virtual display screen. Among them, represents the pixel focal length of the ideal left-eye virtual display screen on the x-axis, represents the pixel focal length of the ideal left-eye virtual display screen on the y-axis, represents the principal point coordinate of the ideal left-eye virtual display screen on the x-axis, represents the principal point coordinate of the ideal left-eye virtual display screen on the y-axis. Among them, represents the pixel focal length of the ideal left-eye simulated camera on the x-axis, represents the pixel focal length of the ideal left-eye simulated camera on the y-axis, represents the principal point coordinate of the ideal left-eye simulated camera on the x-axis, represents the principal point coordinate of the ideal left-eye simulated camera on the y-axis.

[0153] 4.3. Combining the conversion relationship between the actual left-eye simulated camera and the ideal left-eye simulated camera calculated in the previous text , the virtual image distance of the ideal left-eye virtual display screen of the near-eye display device , the coordinates of the normal vector of the ideal left-eye virtual display screen of the near-eye display device under the ideal left-eye simulated camera , the internal parameters of the actual left-eye simulated camera (where the internal parameters of the ideal left-eye simulated camera and the actual left-eye simulated camera are the same), calculate the homography matrix from the ideal left-eye simulated camera to the actual left-eye simulated camera based on the ideal left-eye virtual display screen of the near-eye display device:

[0154]

[0155] Among them, = [0 0 1] T represents the coordinates of the normal vector of the ideal left-eye virtual display screen in the coordinate system of the ideal left-eye simulated camera, represents the rotation from the ideal left-eye simulated camera to the actual left-eye simulated camera, represents the translation from the ideal left-eye simulated camera to the actual left-eye simulated camera.

[0156] 4.4. Use the homography matrix from the ideal left-eye virtual display's ideal left-eye simulation camera to the actual left-eye simulation camera to transform the projection coordinates on the imaging plane of the ideal left-eye simulation camera to the projection coordinates on the imaging plane of the actual left-eye simulation camera:

[0157]

[0158] where, represents the corresponding homogeneous coordinates.

[0159] 4.5. Combine the conversion relationship between the center of the ideal right-eye simulation camera and the virtual camera center corresponding to the ideal right-eye virtual display , the virtual image distance of the ideal right-eye virtual display of the near-eye display device, the internal parameters of the ideal right-eye virtual display of the near-eye display device, the internal parameters of the ideal right-eye simulation camera (where the internal parameters of the ideal right-eye simulation camera and the actual right-eye simulation camera are the same), and calculate the projection coordinates of all the crosshair endpoints displayed on the ideal right-eye virtual display on the undistorted imaging plane of the ideal right-eye simulation camera:

[0160]

[0161] where, represents homogeneous coordinates. For example, represents the corresponding homogeneous coordinates, represents the corresponding homogeneous coordinates, is a three-dimensional vector, , and respectively represent the variables of each dimension, , represents the rotation from the ideal right-eye virtual display to the ideal right-eye simulation camera, represents the translation from the ideal right-eye virtual display to the ideal right-eye simulation camera, represents the ideal right-eye virtual display coordinate system. Where, represents the pixel focal length of the ideal right-eye virtual display on the x-axis, represents the pixel focal length of the ideal right-eye virtual display on the y-axis, represents the principal point coordinate of the ideal right-eye virtual display on the x-axis, Represents the principal point coordinates of the ideal right-eye virtual display screen on the y-axis. Among them, Represents the pixel focal length of the ideal right-eye simulated camera on the x-axis, Represents the pixel focal length of the ideal right-eye simulated camera on the y-axis, Represents the principal point coordinates of the ideal right-eye simulated camera on the x-axis, Represents the principal point coordinates of the ideal right-eye simulated camera on the y-axis.

[0162] 4.6. Combining the conversion relationship between the actual right-eye simulated camera and the ideal right-eye simulated camera obtained from the previous calculation , the virtual image distance of the ideal right-eye virtual display screen of the near-eye display device , the coordinates of the normal vector of the ideal right-eye virtual display screen of the near-eye display device under the ideal right-eye simulated camera , the internal parameters of the actual right-eye simulated camera , calculate the homography matrix from the ideal right-eye simulated camera to the actual right-eye simulated camera based on the ideal right-eye virtual display screen of the near-eye display device:

[0163]

[0164] Among them, = [0 0 1] T Represents the coordinates of the normal vector of the ideal right-eye virtual display screen in the coordinate system of the ideal right-eye simulated camera, Represents the rotation from the ideal right-eye simulated camera to the actual right-eye simulated camera, Represents the translation from the ideal right-eye simulated camera to the actual right-eye simulated camera.

[0165] 4.7. Using the homography matrix from the ideal right-eye simulated camera to the actual right-eye simulated camera based on the ideal right-eye virtual display screen , transform the projection coordinates on the imaging plane of the ideal right-eye simulated camera to the projection coordinates on the imaging plane of the actual right-eye simulated camera:

[0166]

[0167] Among them, Represents The corresponding homogeneous coordinates.

[0168] 4.8. Adjust the positional relationship between the left-eye optical engine and the left-eye lens of the near-eye display device until all the crosshair endpoints on the actual left-eye virtual display screen of the near-eye display device are aligned to the projection coordinates position.

[0169] 4.9. Adjust the positional relationship between the right optical engine and the right-eye lens of the near-eye display device until all the crosshair endpoints on the actual right-eye virtual display screen of the near-eye display device are aligned to the projection coordinates on the imaging plane of the right-eye simulated human eye camera.

[0170] For example, adjust the positional relationship between the left optical engine and the left-eye lens of the near-eye display device or adjust the positional relationship between the right optical engine and the right-eye lens of the near-eye display device through a robotic arm.

[0171] Thus, through the target left-eye conversion relationship between the actual left-eye simulated camera and the ideal left-eye simulated camera, and the target right-eye conversion relationship between the actual right-eye simulated camera and the ideal right-eye simulated camera, the adjustment of the left and right optical engines of the near-eye display device realizes the position compensation for the actual left-eye simulated camera and the actual right-eye simulated camera, and achieves high-precision binocular image synthesis of the near-eye display device.

[0172] Correspondingly, in the embodiment of the present application, after the position compensation for the actual left-eye simulated camera and the actual right-eye simulated camera, the position compensation can also be verified through coordinate errors to ensure that the binocular image synthesis accuracy meets the requirements. That is, optionally, in some embodiments of the present application, before the step of "displaying a target virtual image based on the target position relationship", the method further includes:

[0173] Display test target images through the left-eye virtual display screen and the right-eye virtual display screen of the near-eye display device respectively, where the test target images include target corner points;

[0174] Determine the ideal three-dimensional coordinates of the target corner points corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulated camera, and determine the actual three-dimensional coordinates of the target corner points corresponding to the left-eye virtual display screen in the actual left-eye simulated camera;

[0175] Determine the theoretical three-dimensional coordinates of the ideal three-dimensional coordinates corresponding to the left-eye simulated camera according to the target left-eye conversion relationship;

[0176] Verify the target position relationship according to the actual three-dimensional coordinates and the theoretical three-dimensional coordinates to obtain a position verification result;

[0177] Displaying a target virtual image based on the target position relationship includes:

[0178] If the position verification result indicates that the binocular image synthesis of the left-eye virtual display screen and the right-eye virtual display screen meets the preset conditions, display the target virtual images through the left-eye virtual display screen and the right-eye virtual display screen respectively.

[0179] Among them, the embodiments of the present application perform position compensation for the coordinate error of the left eye. In some scenarios, it is also possible to select to perform position compensation for the coordinate error of the right eye, or simultaneously select the coordinate errors of the left and right eyes for comprehensive judgment.

[0180] Among them, the theoretical three-dimensional coordinates are calculated based on the conversion relationship of the target left eye and the ideal left-eye simulated camera, and are the theoretical comparison objects for coordinate errors in the embodiments of the present application. The actual three-dimensional coordinates are triangulated based on the actual imaging coordinates of the target corner points on the actual left-eye simulated camera, and are used as the actual values of the coordinate errors. The accuracy of position compensation is judged by comparing the actual value with the theoretical comparison object.

[0181] Optionally, in the embodiments of the present application, the ideal three-dimensional coordinates can be obtained by triangulating the projection coordinates of the target corner points on the ideal left-eye simulated camera and the projection coordinates on the ideal right-eye simulated camera. That is, optionally, in some embodiments of the present application, the step of "determining the ideal three-dimensional coordinates of the target corner points corresponding to the ideal left-eye simulated camera under the ideal left-eye virtual display screen" includes:

[0182] Calculating the first test projection coordinates of the target corner points corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulated camera;

[0183] Calculating the second test projection coordinates of the target corner points corresponding to the ideal right-eye virtual display screen in the ideal right-eye simulated camera;

[0184] Triangulating the first test projection coordinates and the second test projection coordinates according to the internal parameters of the ideal left-eye simulated camera, the internal parameters of the ideal right-eye simulated camera, and the external parameters between the ideal left-eye simulated camera and the ideal right-eye simulated camera to obtain the ideal three-dimensional coordinates of the target corner points corresponding to the ideal left-eye simulated camera.

[0185] For example, the verification of position compensation specifically includes:

[0186] 5.1. Display a checkerboard target image on each of the actual left-eye virtual display screen and the actual right-eye virtual display screen of the near-eye display device.

[0187] Among them, the checkerboard target image corresponds to the test target image in the previous text.

[0188] 5.2. Combining the conversion relationship between the center of the ideal left-eye simulated camera and the virtual camera center corresponding to the ideal left-eye virtual display screen , the virtual image distance of the ideal left-eye virtual display screen of the near-eye display device , the internal parameters of the ideal left-eye virtual display screen of the near-eye display device , the internal parameters of the ideal left-eye simulated camera (where the internal parameters of the ideal left-eye simulation camera and the actual left-eye simulation camera are the same), calculate all the checkerboard corner points displayed on the ideal left-eye virtual display The projection coordinates on the imaging plane of the undistorted ideal left-eye simulation camera :

[0189]

[0190] Where represents homogeneous coordinates. For example, represents The corresponding homogeneous coordinates, represents The corresponding homogeneous coordinates, , represents the rotation from the ideal left-eye virtual display to the ideal left-eye simulation camera, represents the translation from the ideal left-eye virtual display to the ideal left-eye simulation camera. The projection coordinates are the first test projection coordinates mentioned above. Where represents the pixel focal length of the ideal left-eye virtual display on the x-axis, represents the pixel focal length of the ideal left-eye virtual display on the y-axis, represents the principal point coordinate of the ideal left-eye virtual display on the x-axis, represents the principal point coordinate of the ideal left-eye virtual display on the y-axis. Where represents the pixel focal length of the ideal left-eye simulation camera on the x-axis, represents the pixel focal length of the ideal left-eye simulation camera on the y-axis, represents the principal point coordinate of the ideal left-eye simulation camera on the x-axis, represents the principal point coordinate of the ideal left-eye simulation camera on the y-axis.

[0191] 5.3. Combine the conversion relationship between the ideal right-eye simulation camera center and the virtual camera center corresponding to the ideal right-eye virtual display , the virtual image distance of the ideal right-eye virtual display of the near-eye display device , the internal parameters of the ideal right-eye virtual display of the near-eye display device , the internal parameters of the ideal right-eye simulation camera (where the internal parameters of the ideal right-eye simulation camera and the actual right-eye simulation camera are the same), calculate all the checkerboard corner points displayed on the right-eye virtual display The projection coordinates on the imaging plane of the undistorted ideal right-eye simulation camera :

[0192]

[0193] wherein, represents homogeneous coordinates. For example, represents the corresponding homogeneous coordinates, represents the corresponding homogeneous coordinates, , represents the rotation from the ideal right-eye virtual display screen to the ideal right-eye simulated camera, represents the translation from the ideal right-eye virtual display screen to the ideal right-eye simulated camera. The projection coordinates are the second test projection coordinates mentioned above. wherein, represents the pixel focal length of the ideal right-eye virtual display screen on the x-axis, represents the pixel focal length of the ideal right-eye virtual display screen on the y-axis, represents the principal point coordinate of the ideal right-eye virtual display screen on the x-axis, represents the principal point coordinate of the ideal right-eye virtual display screen on the y-axis. wherein, represents the pixel focal length of the ideal right-eye simulated camera on the x-axis, represents the pixel focal length of the ideal right-eye simulated camera on the y-axis, represents the principal point coordinate of the ideal right-eye simulated camera on the x-axis, represents the principal point coordinate of the ideal right-eye simulated camera on the y-axis.

[0194] 5.4. Combine the internal parameters of the ideal left-eye simulated camera, the internal parameters of the ideal right-eye simulated camera, and the external parameters between the ideal left-eye simulated camera and the ideal right-eye simulated camera, and triangulate the imaging coordinates and of each checkerboard corner point on the image planes of the ideal left-eye simulated camera and the ideal right-eye simulated camera to obtain the true 3D combined image coordinates of all checkerboard corner points under the ideal left-eye simulated camera. Then, combined with the conversion relationship between the ideal left-eye simulated camera and the actual left-eye simulated camera, calculate the true 3D combined image coordinates of all checkerboard corner points under the actual left-eye simulated camera:

[0195]

[0196] wherein, represents the rotation from the ideal left-eye simulated camera to the actual left-eye simulated camera, represents the translation from the ideal left-eye simulated camera to the actual left-eye simulated camera. wherein, the true 3D combined image coordinates is the ideal three-dimensional coordinate mentioned above. The true value of this 3D combined image coordinate is the theoretical three-dimensional coordinate mentioned above.

[0197] 5.5. Each of the actual left-eye simulation camera and the actual right-eye simulation camera captures an image facing the checkerboard target, obtaining the left-eye captured image and the right-eye captured image.

[0198] 5.6. The checkerboard corner points extracted from the left-eye captured image are represented as , and the checkerboard corner points extracted from the right-eye captured image are represented as .

[0199] 5.7. Combining the internal parameters of the actual left-eye simulation camera, the internal parameters of the actual right-eye simulation camera, and the external parameters between the actual left-eye simulation camera and the actual right-eye simulation camera, the imaging coordinates and of each checkerboard corner point on the image plane of the actual left-eye simulation camera and the image plane of the actual right-eye simulation camera are triangulated to obtain the triangulated 3D combined image coordinates of all checkerboard corner points under the left-eye simulated human eye camera.

[0200] 5.8. Calculate the mean error between the triangulated 3D combined image coordinates and the true value of the 3D combined image coordinate and the variance . When both the mean and the variance are less than the threshold, it is considered that the binocular combined image accuracy meets the requirements; otherwise, the steps of the binocular combined image optimization module need to be continued.

[0201] In summary, due to the difference between the ideal position parameters of the ideal simulation camera and the actual position parameters of the actual simulation camera, in the embodiment of the present application, by determining the target conversion relationship between the ideal simulation camera and the actual simulation camera, and compensating and adjusting the display of the near-eye display device through this target conversion relationship, the accuracy of device adjustment is improved.

[0202] Among them, by separately adjusting the positions of the left and right optical machines, the accuracy of binocular combined imaging of the near-eye display device is improved.

[0203] Please refer to Figure 5 , Figure 5 which is the structural schematic diagram of the near-eye display device provided by the embodiment of the present application. This near-eye display device is the near-eye display device in the virtual display control method mentioned above, and the professional terms and meanings involved are the same, so they will not be elaborated here. For example Figure 5As shown, taking this near-eye display device as an example of augmented reality smart glasses, the smart glasses include a left optical lens 111 and a right optical lens 112. The smart glasses can output target virtual images on the virtual display screens of the left and right eyes respectively, and based on the position compensation for the left optical engine 113 and the right optical engine (not shown in the figure), the binocular imaging accuracy of the smart glasses is improved.

[0204] To facilitate better implementation of the virtual display control method for the near-eye display device of the present application, the present application also provides a virtual display control device for a near-eye display device based on the above virtual display control method for a near-eye display device. The meanings of the nouns are the same as those in the above virtual display control method for a near-eye display device, and the specific implementation details can be referred to the description in the method embodiments.

[0205] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of the virtual display control device for a near-eye display device provided by an embodiment of the present application. The virtual display control device for a near-eye display device can be specifically as follows:

[0206] A display module 201, configured to display a reference target;

[0207] A first determination module 202, configured to determine the ideal conversion relationship between the ideal simulated camera with ideal position parameters and the reference target, and determine the actual conversion relationship between the actual simulated camera with actual position parameters and the reference target;

[0208] A second determination module 203, configured to determine a target conversion relationship according to the ideal conversion relationship and the actual conversion relationship;

[0209] A control module 204, configured to control the near-eye display device to display a target virtual image according to the target conversion relationship.

[0210] Optionally, in some embodiments of the present application, the near-eye display device includes an optical engine and an optical lens;

[0211] Controlling the near-eye display device to display a target virtual image according to the target conversion relationship includes:

[0212] Adjusting the position relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target position relationship;

[0213] Displaying the target virtual image based on the target position relationship.

[0214] Optionally, in some embodiments of the present application, adjusting the position relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target position relationship includes:

[0215] Displaying a target target image through the virtual display screen of the near-eye display device;

[0216] Determine the first projection coordinates of the target reference object in the target target image under the ideal virtual display screen corresponding to the ideal simulation camera;

[0217] Determine the second projection coordinates of the target reference object in the ideal virtual display screen corresponding to the actual simulation camera according to the first projection coordinates and the target conversion relationship;

[0218] According to the target reference object and the second projection coordinates in the target target image under the virtual display screen, adjust the positional relationship between the optical engine and the optical lens to obtain the target positional relationship.

[0219] Optionally, in some embodiments of the present application, determining the second projection coordinates of the target reference object in the ideal virtual display screen corresponding to the actual simulation camera according to the first projection coordinates and the target conversion relationship includes:

[0220] Calculate the homography matrix between the ideal simulation camera and the actual simulation camera according to the target conversion relationship, the normal vector coordinates of the ideal virtual display screen corresponding to the ideal simulation camera, and the internal parameters of the actual simulation camera;

[0221] Transform the first projection coordinates into the second projection coordinates corresponding to the actual simulation camera according to the homography matrix.

[0222] Optionally, in some embodiments of the present application, the ideal simulation camera includes an ideal left-eye simulation camera and an ideal right-eye simulation camera, and the actual simulation camera includes an actual left-eye simulation camera and an actual right-eye simulation camera;

[0223] The target conversion relationship includes a target left-eye conversion relationship and a target right-eye conversion relationship, where the target left-eye conversion relationship is determined based on the ideal left-eye simulation camera and the actual left-eye simulation camera, and the target right-eye conversion relationship is determined based on the ideal right-eye simulation camera and the actual right-eye simulation camera;

[0224] The target positional relationship includes a target left-eye positional relationship and a target right-eye positional relationship, the optical engine includes a left-eye optical engine and a right-eye optical engine, and the optical lens includes a left-eye optical lens and a right-eye optical lens;

[0225] Adjusting the positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain the target positional relationship includes:

[0226] Adjust the positional relationship between the left-eye optical engine and the left-eye optical lens according to the target left-eye conversion relationship to obtain the target left-eye positional relationship, and adjust the positional relationship between the right-eye optical engine and the right-eye optical lens according to the target right-eye conversion relationship to obtain the target right-eye positional relationship.

[0227] Optionally, in some embodiments of the present application, the virtual display screen of the near-eye display device includes a left-eye virtual display screen and a right-eye virtual display screen, and the ideal virtual display screen of the near-eye display device includes an ideal left-eye virtual display screen and an ideal right-eye virtual display;

[0228] Before displaying the target virtual image based on the target position relationship, the method further includes:

[0229] Display a test target image through the left-eye virtual display screen and the right-eye virtual display screen of the near-eye display device respectively, wherein the test target image contains target corner points;

[0230] Determine the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulated camera, and determine the actual three-dimensional coordinates of the target corner point corresponding to the left-eye virtual display screen in the actual left-eye simulated camera;

[0231] Determine the theoretical three-dimensional coordinates of the ideal three-dimensional coordinates corresponding to the left-eye simulated camera according to the target left-eye conversion relationship;

[0232] Verify the target position relationship according to the actual three-dimensional coordinates and the theoretical three-dimensional coordinates to obtain a position verification result;

[0233] If the position verification result indicates that the binocular fusion of the left-eye virtual display screen and the right-eye virtual display screen meets the preset conditions, display the target virtual image through the left-eye virtual display screen and the right-eye virtual display screen respectively.

[0234] Optionally, in some embodiments of the present application, determining the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulated camera includes:

[0235] Calculate the first test projection coordinates of the target corner point in the ideal left-eye virtual display screen corresponding to the ideal left-eye simulated camera;

[0236] Calculate the second test projection coordinates of the target corner point in the ideal right-eye virtual display screen corresponding to the ideal right-eye simulated camera;

[0237] Triangulate the first test projection coordinates and the second test projection coordinates according to the internal parameters of the ideal left-eye simulated camera, the internal parameters of the ideal right-eye simulated camera, and the external parameters between the ideal left-eye simulated camera and the ideal right-eye simulated camera to obtain the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye simulated camera.

[0238] In an embodiment of the present application, a reference target is displayed by a display module 201, an ideal conversion relationship between an ideal simulation camera for determining an ideal position parameter and the reference target is determined by a first determination module 202, and an actual conversion relationship between an actual simulation camera for determining an actual position parameter and the reference target is determined. A second determination module 203 determines a target conversion relationship based on the ideal conversion relationship and the actual conversion relationship, and a control module 204 controls a near-eye display device to display a target virtual image based on the target conversion relationship.

[0239] In summary, in the embodiment of the present application, by determining the target conversion relationship between the ideal simulation camera and the actual simulation camera, and compensating and adjusting the display of the near-eye display device through the target conversion relationship, the accuracy of device adjustment is improved.

[0240] In addition, the present application also provides an electronic device, as Figure 7 shown, which shows a schematic structural diagram of the electronic device involved in the present application. Specifically:

[0241] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art can understand that Figure 7 the structural diagram of the electronic device shown in does not constitute a limitation on the electronic device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements. Among them:

[0242] The processor 301 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 302, and invoking data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modulation and demodulation processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, etc., and the modulation and demodulation processor mainly processes wireless communication. It can be understood that the above modulation and demodulation processor may not be integrated into the processor 301.

[0243] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and virtual display control of the near-eye display device by running the software programs and modules stored in the memory 302. The memory 302 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 302 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 302 can also include a memory controller to provide the processor 301 with access to the memory 302.

[0244] The electronic device further includes a power supply 303 for supplying power to each component. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, a recharge system, a power device debugging circuit, a power converter or an inverter, and a power status indicator.

[0245] The electronic device may further include an input unit 304, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0246] Although not shown, the electronic device may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, so as to implement the steps in any virtual display control method of the near-eye display device provided in the embodiments of the present application.

[0247] The electronic device in the embodiment of the present application controls the display of a reference target. The electronic device determines the ideal conversion relationship between the ideal analog camera with ideal position parameters and the reference target, and determines the actual conversion relationship between the actual analog camera with actual position parameters and the reference target. The target conversion relationship is determined according to the ideal conversion relationship and the actual conversion relationship, and the near-eye display device is controlled to display a target virtual image according to the target conversion relationship.

[0248] Among them, since there are differences between the ideal position parameters of the ideal simulation camera and the actual position parameters of the actual simulation camera, by determining the target conversion relationship between the ideal simulation camera and the actual simulation camera, the display of the near-eye display device can be compensated and adjusted through this target conversion relationship, improving the accuracy of device adjustment.

[0249] For the specific implementation of each of the above operations, reference can be made to the previous embodiments and will not be elaborated here.

[0250] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling relevant hardware through instructions. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0251] Therefore, this application provides a computer-readable storage medium on which a computer program is stored. This computer program can be loaded by a processor to execute the steps in any virtual display control method of the near-eye display device provided by this application.

[0252] For the specific implementation of each of the above operations, reference can be made to the previous embodiments and will not be elaborated here.

[0253] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0254] Since the instructions stored in the computer-readable storage medium can execute the steps in any virtual display control method of the near-eye display device provided by this application, the beneficial effects achievable by any virtual display control method of the near-eye display device provided by this application can be realized. For details, reference can be made to the previous embodiments and will not be elaborated here.

[0255] The virtual display control method, device, electronic device, and computer-readable storage medium provided by this application have been introduced in detail above. Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A virtual display control method for a near-eye display device, characterized in that: The method comprises: Display reference targets; Determining an ideal conversion relationship between an ideal simulated camera having ideal position parameters and the reference target, and determining an actual conversion relationship between an actual simulated camera having actual position parameters and the reference target; Determining a target conversion relationship according to the ideal conversion relationship and the actual conversion relationship; Controlling the near-eye display device to display a target virtual image according to the target conversion relationship; The near-eye display device includes an optical engine and an optical lens, and controlling the near-eye display device to display a target virtual image according to the target conversion relationship includes: Displaying a target image via a virtual display screen of the near-eye display device; Determine the first projection coordinates of the target reference object in the target image on the ideal virtual display screen corresponding to the ideal simulated camera; Determine, based on the first projection coordinates and the target conversion relationship, the second projection coordinates of the target reference object corresponding to the actual simulation camera under the ideal virtual display screen; According to the target reference object in the target image under the virtual display screen and the second projection coordinates, adjusting the positional relationship between the optical machine and the optical lens to obtain a target positional relationship; The target virtual image is displayed based on the target position relationship.

2. The virtual display control method of the near-eye display device according to claim 1, characterized in that: The determining, based on the first projection coordinates and the target conversion relationship, the second projection coordinates of the target reference object corresponding to the actual simulation camera on the ideal virtual display screen includes: Calculating a homography matrix between the ideal simulated camera and the actual simulated camera according to the target transformation relationship, the normal vector coordinates of the ideal virtual display screen corresponding to the normal vector of the ideal simulated camera, and the intrinsic parameters of the actual simulated camera; The first projection coordinates are transformed into the second projection coordinates corresponding to the actual simulated camera according to the homography matrix.

3. The virtual display control method of the near-eye display device according to claim 1, characterized in that: The ideal simulated camera includes an ideal left-eye simulated camera and an ideal right-eye simulated camera, and the actual simulated camera includes an actual left-eye simulated camera and an actual right-eye simulated camera; The target conversion relationship includes a target left-eye conversion relationship and a target right-eye conversion relationship, wherein the target left-eye conversion relationship is determined based on the ideal left-eye simulation camera and the actual left-eye simulation camera, and the target right-eye conversion relationship is determined based on the ideal right-eye simulation camera and the actual right-eye simulation camera; The target position relationship includes a target left eye position relationship and a target right eye position relationship, the optical machine includes a left eye machine and a right eye machine, and the optical lens includes a left eye optical lens and a right eye optical lens; The adjusting the positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target positional relationship includes: The target left-eye position relationship is obtained by adjusting the positional relationship between the left-eye motor and the left-eye optical lens according to the target left-eye conversion relationship, and the target right-eye position relationship is obtained by adjusting the positional relationship between the right-eye motor and the right-eye optical lens according to the target right-eye conversion relationship.

4. The virtual display control method of the near-eye display device according to claim 3, characterized in that: The virtual display screen of the near-eye display device includes a left-eye virtual display screen and a right-eye virtual display screen, and the ideal virtual display screen of the near-eye display device includes an ideal left-eye virtual display screen and an ideal right-eye virtual display screen; Before displaying the target virtual image based on the target position relationship, the method further includes: Displaying a test target image on the left-eye virtual display screen and the right-eye virtual display screen of the near-eye display device, respectively, wherein the test target image includes a target corner point; Determine the ideal three-dimensional coordinates of the target corner point on the ideal left-eye virtual display screen corresponding to the ideal left-eye simulated camera, and determine the actual three-dimensional coordinates of the target corner point on the left-eye virtual display screen corresponding to the actual left-eye simulated camera; Determine the theoretical three-dimensional coordinates of the left-eye simulation camera corresponding to the ideal three-dimensional coordinates according to the target left-eye conversion relationship; Verifying the target position relationship according to the actual three-dimensional coordinates and the theoretical three-dimensional coordinates to obtain a position verification result; The displaying of the target virtual image based on the target position relationship includes: If the position verification result indicates that the binocular combined image of the left-eye virtual display screen and the right-eye virtual display screen meets a preset condition, the target virtual image is displayed through the left-eye virtual display screen and the right-eye virtual display screen respectively.

5. The virtual display control method of the near-eye display device according to claim 4, characterized in that: Determining the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye virtual display screen on the ideal left-eye simulated camera includes: Calculate the first test projection coordinates of the target corner point in the ideal left-eye virtual display screen corresponding to the ideal left-eye simulation camera; Calculate the second test projection coordinates of the target corner point in the ideal right-eye virtual display screen corresponding to the ideal right-eye simulated camera; The first test projection coordinates and the second test projection coordinates are triangulated according to the intrinsic parameters of the ideal left-eye simulated camera, the intrinsic parameters of the ideal right-eye simulated camera, and the extrinsic parameters between the ideal left-eye simulated camera and the ideal right-eye simulated camera to obtain the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye simulated camera.

6. A virtual display control device for a near-eye display device, characterized in that: The device comprises: A display module, used for displaying a reference target; A first determining module is configured to determine an ideal conversion relationship between an ideal simulated camera of ideal position parameters and the reference target, and to determine an actual conversion relationship between an actual simulated camera of actual position parameters and the reference target; A second determining module is configured to determine a target conversion relationship based on the ideal conversion relationship and the actual conversion relationship; a control module, configured to control the near-eye display device to display a target virtual image according to the target conversion relationship; The near-eye display device includes an optical engine and an optical lens, and controlling the near-eye display device to display a target virtual image according to the target conversion relationship includes: Displaying a target image via a virtual display screen of the near-eye display device; Determine the first projection coordinates of the target reference object in the target image on the ideal virtual display screen corresponding to the ideal simulated camera; Determine, based on the first projection coordinates and the target conversion relationship, the second projection coordinates of the target reference object corresponding to the actual simulation camera under the ideal virtual display screen; According to the target reference object in the target image under the virtual display screen and the second projection coordinates, adjusting the positional relationship between the optical machine and the optical lens to obtain a target positional relationship; The target virtual image is displayed based on the target position relationship.

7. A near-eye display device, characterized in that: The near-eye display device is the near-eye display device in the virtual display control method for a near-eye display device according to any one of claims 1 to 5.

8. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the virtual display control method of the near-eye display device according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the virtual display control method of the near-eye display device according to any one of claims 1 to 5 are implemented.

Images