Virtual display control method and device of near-to-eye display equipment and near-to-eye display equipment

Abstract

The embodiment of the invention discloses a virtual display control method and device of near-to-eye display equipment and the near-to-eye display equipment, and relates to the technical field of near-to-eye display, and the method comprises the steps: displaying a reference target, determining an ideal conversion relation between an ideal simulation camera of an ideal position parameter and the reference target, and determining an ideal conversion relation between the ideal simulation camera and the reference target; and determining an actual conversion relation between the actual simulation camera of the actual position parameter and the reference target, determining a target conversion relation according to the ideal conversion relation and the actual conversion relation, and controlling the near-to-eye display device to display a target virtual image according to the target conversion relation. The target conversion relation between the ideal simulation camera and the actual simulation camera is determined, and the display of the near-to-eye display equipment is subjected to compensation adjustment through the target conversion relation, so that the equipment adjustment accuracy is improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of near-eye display, and particularly to a virtual display control method, device, and near-eye display device for a near-eye display device. Background Art

[0002] Near-eye display devices are a display technology that directly projects images in front of the user's eyes and are widely used in fields such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality). Their core goal is to provide a high-definition, large field of view (FOV), and low-latency visual experience at a very close distance (usually <50mm).

[0003] Near-eye display devices usually adjust by referring to the image content captured by a simulated human eye camera. For example, adjustments such as display alignment are performed. Thus, the position accuracy of the simulated human eye camera, that is, the accuracy of the information provided by the simulated human eye camera, directly determines the accuracy of image display.

[0004] However, in the actual production and assembly process, due to factors such as machining errors and human errors during the assembly process, the positions of the binocular simulated human eye cameras usually deviate from the designed positions. In the prior art, the actual assembled positions of the simulated human eye cameras are usually directly used as the designed ideal positions, that is, installed at the actual assembled positions, but still display alignment and other operations are performed based on the designed ideal positions. This ignores the deviation between the actual position and the designed ideal position, resulting in the inability to adjust the near-eye display device based on the actual installation position. Summary of the Invention

[0005] Embodiments of the present application provide a virtual display control method, device, and near-eye display device for a near-eye display device, which can improve the accuracy of device adjustment when adjusting the near-eye display device based on a simulated camera.

[0006] In a first aspect, embodiments of the present application provide a virtual display control method for a near-eye display device, the method including: Display a reference target; Determine the ideal conversion relationship between an ideal simulated camera with ideal position parameters and the reference target, and determine the actual conversion relationship between an actual simulated camera with actual position parameters and the reference target; Determine a target conversion relationship based on the ideal conversion relationship and the actual conversion relationship; Control the near-eye display device to display a target virtual image based on the target conversion relationship.

[0007] In a second aspect, embodiments of the present application further provide a virtual display control device for a near-eye display device, the device including: A display module, configured to display a reference target; A first determination module, configured to determine an ideal conversion relationship between an ideal simulated camera with ideal position parameters and the reference target, and determine an actual conversion relationship between an actual simulated camera with actual position parameters and the reference target; A second determination module, configured to determine a target conversion relationship according to the ideal conversion relationship and the actual conversion relationship; A control module, configured to control the near-eye display device to display a target virtual image according to the target conversion relationship.

[0008] Optionally, in some embodiments of the present application, the near-eye display device includes an optical engine and an optical lens; The controlling the near-eye display device to display a target virtual image according to the target conversion relationship includes: Adjusting a positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target positional relationship; Displaying the target virtual image based on the target positional relationship.

[0009] Optionally, in some embodiments of the present application, the adjusting the positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target positional relationship includes: Displaying a target target image through a virtual display screen of the near-eye display device; Determining a first projection coordinate of a target reference object in the target target image under an ideal virtual display screen corresponding to the ideal simulated camera; Determining a second projection coordinate of the target reference object in the target target image under the ideal virtual display screen corresponding to the actual simulated camera according to the first projection coordinate and the target conversion relationship; Adjusting the positional relationship between the optical engine and the optical lens according to the target reference object and the second projection coordinate in the target target image under the virtual display screen to obtain a target positional relationship.

[0010] Optionally, in some embodiments of the present application, the determining the second projection coordinate of the target reference object in the target target image under the ideal virtual display screen corresponding to the actual simulated camera according to the first projection coordinate and the target conversion relationship includes: Calculating a homography matrix between the ideal simulated camera and the actual simulated camera according to the target conversion relationship, a normal vector coordinate of the normal vector of the ideal virtual display screen corresponding to the ideal simulated camera, and an internal parameter of the actual simulated camera; Transforming the first projection coordinate into the second projection coordinate corresponding to the actual simulated camera according to the homography matrix.

[0011] Optionally, in some embodiments of the present application, the ideal simulation camera includes an ideal left-eye simulation camera and an ideal right-eye simulation camera, and the actual simulation camera includes an actual left-eye simulation camera and an actual right-eye simulation camera; The target conversion relationship includes a target left-eye conversion relationship and a target right-eye conversion relationship. Among them, the target left-eye conversion relationship is determined based on the left-eye virtual display screen, the ideal left-eye simulation camera, and the actual left-eye simulation camera, and the target right-eye conversion relationship is determined based on the right-eye virtual display screen, the ideal right-eye simulation camera, and the actual right-eye simulation camera; The target position relationship includes a target left-eye position relationship and a target right-eye position relationship. The optical engine includes a left-eye optical engine and a right-eye optical engine, and the optical lens includes a left-eye optical lens and a right-eye optical lens; Adjusting the positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain the target position relationship includes: Adjusting the positional relationship between the left-eye optical engine and the left-eye optical lens according to the target left-eye conversion relationship to obtain the target left-eye position relationship, and adjusting the positional relationship between the right-eye optical engine and the right-eye optical lens according to the target right-eye conversion relationship to obtain the target right-eye position relationship.

[0012] Optionally, in some embodiments of the present application, the virtual display screen of the near-eye display device includes a left-eye virtual display screen and a right-eye virtual display screen, and the ideal virtual display screen of the near-eye display device includes an ideal left-eye virtual display screen and an ideal right-eye virtual display screen; Before displaying the target virtual image based on the target position relationship, the method further includes: Displaying a test target image through the left-eye virtual display screen and the right-eye virtual display screen of the near-eye display device respectively, where the test target image contains target corner points; Determining the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulation camera, and determining the actual three-dimensional coordinates of the target corner point corresponding to the left-eye virtual display screen in the actual left-eye simulation camera; Determining the theoretical three-dimensional coordinates of the ideal three-dimensional coordinates corresponding to the left-eye simulation camera according to the target left-eye conversion relationship; Verifying the target position relationship according to the actual three-dimensional coordinates and the theoretical three-dimensional coordinates to obtain a position verification result; If the position verification result indicates that the binocular image combination of the left-eye virtual display screen and the right-eye virtual display screen meets a preset condition, then display the target virtual image through the left-eye virtual display screen and the right-eye virtual display screen respectively.

[0013] Optionally, in some embodiments of the present application, determining the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulation camera includes: Calculating the first test projection coordinates of the target corner point corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulation camera; Calculating the second test projection coordinates of the target corner point corresponding to the ideal right-eye virtual display screen in the ideal right-eye simulation camera; Triangulating the first test projection coordinates and the second test projection coordinates according to the internal parameters of the ideal left-eye simulation camera, the internal parameters of the ideal right-eye simulation camera, and the external parameters between the ideal left-eye simulation camera and the ideal right-eye simulation camera to obtain the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye simulation camera.

[0014] In a third aspect, an embodiment of the present application further provides a near-eye display device. The near-eye display device includes the near-eye display device in the virtual display control method of the near-eye display device described above. Wherein, the near-eye display device displays a target target image through a virtual display screen, and the near-eye display device controls the output of a target virtual image according to a target conversion relationship.

[0015] In a fourth aspect, an embodiment of the present application further provides an electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the virtual display control method of the near-eye display device described above are implemented.

[0016] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps in the virtual display control method of the near-eye display device described above are implemented.

[0017] In a sixth aspect, an embodiment of the present application further provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementation manners described in the embodiments of the present application.

[0018] An embodiment of the present application displays a reference target, determines an ideal conversion relationship between an ideal simulation camera for determining ideal position parameters and the reference target, and determines an actual conversion relationship between an actual simulation camera for determining actual position parameters and the reference target. A target conversion relationship is determined based on the ideal conversion relationship and the actual conversion relationship, and the near-eye display device is controlled to display a target virtual image according to the target conversion relationship.

[0019] Among them, since there are differences between the ideal position parameters of the ideal simulation camera and the actual position parameters of the actual simulation camera, by determining the target conversion relationship between the ideal simulation camera and the actual simulation camera, the display of the near-eye display device can be compensated and adjusted through the target conversion relationship, improving the accuracy of device adjustment. Description of the Drawings

[0020] In order to more clearly illustrate the technical solutions in the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 is a schematic diagram of the scenario where an electronic device executes the virtual display control method of the near-eye display device provided by the embodiment of the present application; Figure 2 is a schematic flowchart of the virtual display control method of the near-eye display device provided by the embodiment of the present application; Figure 3 is a schematic diagram of the target target image provided by the embodiment of the present application; Figure 4 is a schematic diagram of the positional relationship between the binocular ideal simulation camera and the reference target provided by the embodiment of the present application; Figure 5 is a schematic diagram of the structure of the near-eye display device provided by the embodiment of the present application; Figure 6 is a schematic diagram of the structure of the virtual display control device of the near-eye display device provided by the embodiment of the present application; Figure 7 is a schematic diagram of the structure of the electronic device provided by the embodiment of the present application.

[0022] Explanation of the Reference Numerals in the Drawings: 10 - Electronic device; 11 - Near-eye display device; 111 - Left optical lens; 112 - Right optical lens; 113 - Left optical engine; 201 - Display module; 202 - First determination module; 203 - Second determination module; 204 - Control module; 301 - Processor; 302 - Memory; 303 - Power supply; 304 - Input unit. Detailed Embodiments

[0023] The technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in this application. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts fall within the protection scope of the present invention.

[0024] The embodiments of this application provide a virtual display control method, device, near-eye display device, electronic device, and computer-readable storage medium for a near-eye display device. Specifically, the embodiments of this application provide a virtual display control device for a near-eye display device applicable to an electronic device, which is used to improve the accuracy of device adjustment when adjusting the near-eye display device based on an analog camera. Specifically, the electronic device includes computer devices such as a laptop computer, a desktop computer, or an industrial control computer.

[0025] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the scenario where the electronic device provided by the embodiments of this application executes the virtual display control method for the near-eye display device. Among them, the specific execution process of the electronic device executing the virtual display control method for the near-eye display device is as follows: The electronic device 10 displays a reference target, determines the ideal conversion relationship between the ideal analog camera with the ideal position parameters and the reference target, and determines the actual conversion relationship between the actual analog camera with the actual position parameters and the reference target. The electronic device 10 determines the target conversion relationship according to the ideal conversion relationship and the actual conversion relationship, and the electronic device 10 controls the near-eye display device 11 to display the target virtual image according to the target conversion relationship.

[0026] For example, the electronic device 10 controls the display of the reference target, determines the ideal conversion relationship and the actual conversion relationship, calculates the target conversion relationship according to the ideal conversion relationship and the actual conversion relationship, and then the electronic device 10 adjusts the near-eye display device based on the target conversion relationship. And the near-eye display device 11 displays the target virtual image according to the target conversion relationship.

[0027] In summary, due to the difference between the ideal position parameters of the ideal analog camera and the actual position parameters of the actual analog camera, the embodiments of this application can compensate and adjust the display of the near-eye display device through the determined target conversion relationship between the ideal analog camera and the actual analog camera, thereby improving the accuracy of device adjustment.

[0028] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the priority order of the embodiments.

[0029] Please refer to Figure 2 , Figure 2 , which is a schematic flowchart of the virtual display control method for the near-eye display device provided by the embodiment of the present application. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order from that shown in the flowchart. Specifically, the execution subject of the virtual display control method for the near-eye display device is an electronic device, and the electronic device includes computer devices such as laptop computers, desktop computers, or industrial control computers. Specifically, the process of the electronic device executing the virtual display control method for the near-eye display device specifically includes: 101. Display a reference target.

[0030] Among them, the reference target is a specific pattern or mark used for calibrating, positioning, measuring, or evaluating the system performance. For example, the reference target is a checkerboard.

[0031] 102. Determine the ideal conversion relationship between the ideal simulation camera with ideal position parameters and the reference target, and determine the actual conversion relationship between the actual simulation camera with actual position parameters and the reference target.

[0032] It should be noted that the ideal simulation camera is a simulated human eye camera with ideal position parameters (i.e., ideal installation position). For example, for a near-eye display device, the ideal position parameters refer to the best position for observing the image output by the near-eye display device. For example, for a binocular near-eye display device, the ideal simulation camera refers to a simulated human eye camera that is parallel to the virtual display screen of the near-eye display device and is located at a specific distance from the near-eye display device when the eyes are closed.

[0033] Correspondingly, the actual simulation camera is a simulated human eye camera with actual position parameters (i.e., actual installation position or actual assembly position). For example, when installing the simulated human eye camera according to the ideal position parameters, due to errors during the assembly process, the simulated human eye camera is installed at the actual installation position instead of the ideal installation position. Then, the simulated human eye camera at the actual installation position instead of the ideal installation position is the actual simulation camera.

[0034] It should be noted that the ideal simulation camera and the actual simulation camera can be simulated human eye cameras of the same type and with the same internal and external parameters. In the embodiments of the present application, they are only distinguished based on the different installation positions.

[0035] Among them, the ideal conversion relationship is the conversion relationship (i.e., position relationship or coordinate relationship) between the ideal simulation camera and the reference target calculated based on the designed ideal position parameters, and the conversion relationship includes a rotation relationship and a translation relationship.

[0036] Among them, the actual conversion relationship is the conversion relationship calculated based on the actual position parameters. In the embodiment of the present application, the reference target image is obtained by actually simulating the camera shooting the reference target, and then the corner points in the reference target image are extracted, and the actual conversion relationship is obtained by solving using the coordinate information of the corner points and the pnp algorithm.

[0037] It can be understood that since there are differences between the ideal position parameters and the actual position parameters, there are also differences between the ideal conversion relationship and the actual conversion relationship, that is, there are position differences between the ideal simulation camera and the actual simulation camera.

[0038] 103. Determine the target conversion relationship according to the ideal conversion relationship and the actual conversion relationship.

[0039] It should be noted that since the ideal simulation camera is a simulated human eye camera with ideal position parameters, in the prior art, the position information provided by the ideal simulation camera is used to optimize the display of the near-eye display device. For example, the content displayed by the near-eye display device is photographed by the ideal simulation camera, and the content is aligned with the projection coordinates under the ideal simulation camera to realize the adjustment of the near-eye display device.

[0040] However, due to reasons such as the installation error of the simulated human eye camera, there is a difference between the actual installation position of the actual simulation camera and the ideal position. Therefore, there are differences between the content photographed based on the actual simulation camera and the content photographed based on the ideal simulation camera. Furthermore, if the content photographed based on the actual simulation camera is directly aligned with the projection coordinates of the ideal simulation camera, there will be an error in the alignment reference, resulting in inaccurate device adjustment and the problem that the content cannot be displayed at the specified target position.

[0041] In addition, since the ideal conversion relationship represents the position relationship between the reference target and the ideal simulation camera, the actual conversion relationship represents the position relationship between the reference target and the actual simulation camera, and both the ideal conversion relationship and the actual conversion relationship are determined based on the reference target as a reference. Therefore, in the embodiment of the present application, the position relationship (i.e., the conversion relationship) between the ideal simulation camera and the actual simulation camera can be calculated according to the foregoing ideal conversion relationship and actual conversion relationship, that is, the target conversion relationship characterizes the position relationship (i.e., the conversion relationship) between the ideal simulation camera and the actual simulation camera.

[0042] Based on this, in the embodiment of the present application, by calculating the ideal conversion relationship and the actual conversion relationship, and calculating the target conversion relationship based on the ideal conversion relationship and the actual conversion relationship, the position relationship between the ideal simulation camera and the actual simulation camera is determined. Furthermore, it helps to adjust the reference object corresponding to the photographed content based on the target conversion relationship, thereby improving the accuracy of device adjustment.

[0043] 104. Control the near-eye display device to display a target virtual image according to the target conversion relationship.

[0044] For example, adjust the display parameters of the near-eye display device according to the target conversion relationship to improve the accuracy of device adjustment, so that the content output by the near-eye display device matches the information provided by the position of the actual simulated camera, thereby ensuring that a target virtual image with accurate display position and good display effect can be observed through the actual simulated camera.

[0045] It can be understood that the target virtual image is virtual content generated by calculation. For example, taking the augmented reality device in the near-eye display device as an example, the target virtual image is the virtual content for augmented reality superimposed on the real scene.

[0046] In summary, due to the difference between the ideal position parameters of the ideal simulated camera and the actual position parameters of the actual simulated camera, the embodiments of the present application determine the target conversion relationship between the ideal simulated camera and the actual simulated camera, and perform compensation adjustment on the display of the near-eye display device through the target conversion relationship to improve the accuracy of device adjustment.

[0047] It can be understood that in order to ensure that the image output by the near-eye display device matches the information provided by the position of the actual simulated camera, the embodiments of the present application can adjust the image output position of the near-eye display device to match the information provided by the position of the actual simulated camera, and the image output position can be achieved by adjusting the optical engine of the near-eye display device. For example, adjust the positional relationship between the optical engine and the optical lens of the near-eye display device, so as to change the position of the virtual content projected by the optical engine in the optical lens, so that the image output by the near-eye display device matches the information provided by the position of the actual simulated camera. That is, optionally, in some embodiments of the present application, the near-eye display device includes an optical engine and an optical lens. The step of "controlling the near-eye display device to display a target virtual image according to the target conversion relationship" includes: Adjust the positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target positional relationship; Display the target virtual image based on the target positional relationship.

[0048] For example, adjust the position of the optical engine according to the target conversion relationship, thereby changing the display position of the target virtual image.

[0049] It should be noted that, in order to ensure the accuracy of the displayed content of the near-eye display device, in the prior art, the projection coordinates of the image plane of the ideal simulation camera are designed, and the display alignment process is carried out with these projection coordinates as a reference. For example, when the imaging coordinates of the target reference object (such as the entire crosshair) in the target reticle image on the ideal simulation camera are aligned with these projection coordinates, it is considered that the display position of the near-eye display device is accurate at this time and the display effect is good. Among them, the projection coordinates of the imaging plane of the ideal simulation camera are obtained by projecting the content displayed on the ideal virtual display screen of the near-eye display device, that is, the projection coordinates are obtained by projecting the coordinates of the display content on the ideal virtual display screen onto the imaging plane of the ideal simulation camera.

[0050] However, due to the actual installation error of the simulated human eye camera, the simulated human eye camera may not necessarily be installed at the ideal position, that is, the actual simulation camera in the embodiments of the present application. Therefore, the projection coordinates for the ideal simulation camera are not applicable to the alignment process of the actual simulation camera, and it is necessary to calculate the projection coordinates for the actual simulation camera and perform the alignment process for the actual simulation camera according to the calculated projection coordinates.

[0051] In the embodiments of the present application, the projection coordinates applicable to the actual simulation camera are calculated according to the conversion relationship between the ideal simulation camera and the actual simulation camera. That is, optionally, in some embodiments of the present application, the step of "adjusting the position relationship between the optical engine and the optical lens according to the target conversion relationship to obtain the target position relationship" includes: Displaying a target reticle image through the virtual display screen of the near-eye display device; Determining the first projection coordinates of the target reference object in the target reticle image under the ideal virtual display screen on the ideal simulation camera; Determining the second projection coordinates of the target reference object under the ideal virtual display screen on the actual simulation camera according to the first projection coordinates and the target conversion relationship; Adjusting the position relationship between the optical engine and the optical lens according to the target reference object and the second projection coordinates in the target reticle image under the virtual display screen to obtain the target position relationship.

[0052] Among them, the near-eye display device is a display technology that directly projects images in front of the user's eyes and is widely used in fields such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality). For example, the near-eye display device includes extended reality devices such as augmented reality devices and virtual reality devices. The core goal of the near-eye display device is to provide a high-definition, large field of view (FOV), and low-latency visual experience at a very short distance (usually <50 mm).

[0053] It should be noted that the virtual display screen is not the physical screen of the near-eye display device. After the user wears the near-eye display device, through the direction of the optical lens (such as a lens), a screen with a certain resolution (such as 1280*960) can be visually seen at a position close to 4m in front of the eyes. This screen with a certain resolution is the virtual display screen. For example, the optical engine projects the image of the micro display (such as Micro-OLED, LCoS, DLP, etc.) onto the optical lens through the optical system. When the user wears the near-eye display device and looks at the optical lens, a virtual image can be seen in front of the eyes, and this virtual image is presented on the virtual display screen.

[0054] It should be noted that the virtual display screen is the actually displayed virtual display screen, and the ideal virtual display screen is the ideally displayed virtual display screen. For example, it is strictly aligned with the optical axis in position, has no distortion, and does not require calibration, etc. The content displayed on the ideal virtual display screen meets the ideal requirements. The first projection coordinates obtained by projecting the content displayed on the ideal virtual display screen are applicable as the alignment reference based on the ideal simulated camera. However, since the alignment reference in the embodiments of the present application is the projection coordinates in the actual simulated camera, the second projection coordinates that can be used as the alignment reference in the actual simulated camera can be calculated according to the first projection coordinates and the target conversion relationship.

[0055] Among them, in the embodiments of the present application, in order to more intuitively and accurately determine whether the coordinate positions are aligned, the target target image is a target image with a crosshair selected, such as Figure 3 , Figure 3 is a schematic diagram of the target target image provided by the embodiments of the present application. Crosshairs are respectively configured at the four corners and the middle position of the image shown in Figure 3 . It should be noted that the target target images displayed on the virtual display screen and the ideal virtual display screen in the embodiments of the present application are the same, that is, the coordinates of the target reference object in the target target image are the same under the virtual display screen and the ideal virtual display screen.

[0056] It should be noted that the first projection coordinates are the projection coordinates for the ideal simulated camera as described above. That is, when the imaging coordinates of the target reference object (such as the entire crosshair) in the target target image on the image plane of the ideal simulated camera are aligned with the first projection coordinates, it is considered that at this time, through the ideal simulated camera, image content with accurate position and good display effect can be observed on the optical lens of the near-eye display device. It can be understood that the first projection coordinates are pre-configured. For example, they are the projection coordinates that can observe image content with accurate position and good display effect based on the ideal virtual display screen and the ideal position parameters.

[0057] It can be understood that the second projection coordinate is a projection coordinate calculated based on the first projection coordinate and the target conversion relationship, and it is the projection coordinate of the image content with accurate position and good display effect that can be observed under the actual position parameter. That is, when the imaging coordinate of the target reference object (such as the entire crosshair) in the target target image on the image plane of the actual simulated camera is aligned with this second projection coordinate, it is considered that at this time, through the actual simulated camera, the image content with accurate image position and good display effect can be observed on the optical lens of the near-eye display device.

[0058] It should be noted that after determining the second projection coordinate, the optical engine can be adjusted according to this second projection coordinate, so that the imaging coordinate of the target reference object in the target target image output by the virtual display screen (here refers to the actual virtual display screen) of the near-eye display device is aligned with this second projection coordinate. For example, adjust the position of the optical engine until the imaging coordinate of the target reference object in the target target image on the virtual display screen on the actual simulated camera is aligned with the second projection coordinate. Among them, this adjustment can be completed by driving a robotic arm, or an instruction message can be sent to the user to guide the user to adjust the position of the optical engine to complete.

[0059] Specifically, based on the requirement of coordinate transformation, a homography matrix can be generated based on the target conversion relationship, and the first projection coordinate can be transformed into the second projection coordinate based on this homography matrix. That is, optionally, in some embodiments of the present application, the step "determine the second projection coordinate corresponding to the target reference object in the actual simulated camera under the ideal virtual display screen according to the first projection coordinate and the target conversion relationship" includes: Calculate the homography matrix between the ideal simulated camera and the actual simulated camera according to the target conversion relationship, the normal vector coordinate of the ideal virtual display screen corresponding to the normal vector of the ideal simulated camera, and the internal parameters of the actual simulated camera; Transform the first projection coordinate into the second projection coordinate corresponding to the actual simulated camera according to the homography matrix.

[0060] It should be noted that if the ideal simulated camera and the actual simulated camera use simulated human eye cameras with the same parameters, the internal parameters of the ideal simulated camera and the actual simulated camera are the same.

[0061] It can be understood that for a near-eye display device, it is generally binocular that conforms to the characteristics of the human eye (which corresponds to a left-eye virtual display screen and a right-eye virtual display screen, and correspondingly, there are an ideal left-eye virtual display screen and an ideal right-eye virtual display screen), that is, it includes a left-eye optical engine and a right-eye optical engine, and includes a left-eye optical lens corresponding to the left-eye optical engine and a right-eye optical lens corresponding to the right-eye optical engine. Therefore, whether the image display position is accurate directly affects the accuracy of binocular image synthesis. Therefore, in the embodiments of the present application, taking the ideal analog cameras of the binocular and the actual analog cameras of the binocular as examples, it is determined whether the image display position is accurate by whether the binocular image synthesis observed by the ideal analog cameras of the binocular and the actual analog cameras of the binocular is aligned.

[0062] That is, in the embodiments of the present application, the virtual display screen of the near-eye display device includes a left-eye virtual display screen and a right-eye virtual display screen, the ideal virtual display screen includes an ideal left-eye virtual display screen and an ideal right-eye virtual display screen, the ideal analog camera includes an ideal left-eye analog camera and an ideal right-eye analog camera, and the actual analog camera includes an actual left-eye analog camera and an actual right-eye analog camera; The target conversion relationship includes a target left-eye conversion relationship and a target right-eye conversion relationship, where the target left-eye conversion relationship is determined based on the ideal left-eye analog camera and the actual left-eye analog camera, and the target right-eye conversion relationship is determined based on the ideal right-eye analog camera and the actual right-eye analog camera; The target position relationship includes a target left-eye position relationship and a target right-eye position relationship, the optical engine includes a left-eye optical engine and a right-eye optical engine, and the optical lens includes a left-eye optical lens and a right-eye optical lens; Adjusting the position relationship between the optical engine and the optical lens according to the target conversion relationship to obtain the target position relationship includes: Adjusting the position relationship between the left-eye optical engine and the left-eye optical lens according to the target left-eye conversion relationship to obtain the target left-eye position relationship, and adjusting the position relationship between the right-eye optical engine and the right-eye optical lens according to the target right-eye conversion relationship to obtain the target right-eye position relationship.

[0063] For example, for the left-eye optical engine, it is adjusted based on the left-eye virtual display screen, the ideal left-eye virtual display screen, the ideal left-eye analog camera, and the actual left-eye analog camera. That is, the target left-eye conversion relationship for the left-eye optical engine is determined according to the ideal left-eye analog camera and the actual left-eye analog camera, and the second projection relationship for the actual left-eye analog camera is obtained according to the target left-eye conversion relationship and the first projection coordinates for the ideal left-eye analog camera (the first projection coordinates are calculated according to the coordinate projection of the ideal left-eye virtual display screen and the ideal left-eye analog camera). Furthermore, the left-eye optical engine is adjusted according to the second projection relationship for the actual left-eye analog camera and the left-eye virtual display screen, so that the imaging coordinates of the target reference object finally displayed by the left-eye optical engine on the actual left-eye analog camera are aligned with the second projection relationship of the actual left-eye analog camera.

[0064] Similarly, for the right-eye optical machine, adjustments are made based on the right-eye virtual display screen, the ideal right-eye virtual display screen, the ideal right-eye simulated camera, and the actual right-eye simulated camera. That is, the target right-eye conversion relationship for the right-eye optical machine is determined according to the ideal right-eye simulated camera and the actual right-eye simulated camera, and the second projection relationship for the actual right-eye simulated camera is obtained based on this target right-eye conversion relationship and the first projection coordinates for the ideal right-eye simulated camera (the first projection coordinates are calculated based on the coordinate projection of the ideal right-eye virtual display screen and the ideal right-eye simulated camera). Furthermore, the right-eye optical machine is adjusted according to the second projection relationship for the actual right-eye simulated camera and the right-eye virtual display screen, so that the imaging coordinates of the target reference object finally displayed by the right-eye optical machine on the actual right-eye simulated camera are aligned with the second projection relationship of the actual right-eye simulated camera.

[0065] Specifically, to more clearly illustrate the adjustments for the left-eye optical machine and the right-eye optical machine in the embodiments of the present application, the adjustments for the left-eye optical machine and the right-eye optical machine will be specifically described below in combination with specific parameters. Specifically, it includes a calibration camera module, a verification camera calibration accuracy module, a determination of compensation information module, and a binocular image synthesis optimization module. Specifically: The calibration camera module is used to calibrate the actual simulated cameras of both eyes, specifically including: 1.1. When the actual simulated cameras of both eyes have not been calibrated, or the internal and external parameters of the previously calibrated actual simulated cameras of both eyes do not meet the requirements, calibrate the actual simulated cameras of both eyes.

[0066] 1.2. When calibrating the actual simulated cameras of both eyes, change the relative position between the actual simulated cameras of both eyes and the camera calibration target. The cameras acquire images of the camera calibration target at different relative positions, and ensure that all the captured images of the camera calibration target cover the entire camera image as evenly as possible.

[0067] 1.3. Use the camera internal and external parameter calibration algorithm to process the images of the camera calibration target collected by the actual simulated cameras of both eyes, and obtain the internal parameters of the actual simulated cameras of both eyes, as well as the conversion relationship between the actual left-eye simulated camera and the actual right-eye simulated camera.

[0068] 1.4. Calibrate the actual simulated cameras of both eyes with the internal parameters of the actual simulated cameras of both eyes and the conversion relationship between the actual left-eye simulated camera and the actual right-eye simulated camera.

[0069] The verification camera calibration accuracy module is used to verify the calibration accuracy of the actual simulated cameras of both eyes, specifically including: 2.1. Each of the actual simulated cameras of both eyes captures an image facing the camera calibration target.

[0070] 2.2. Extract corner points from the undistorted camera calibration target image.

[0071] 2.3. Classify the corner points by row and column, and use all the corner points on each row or column line. Utilize the fitLine algorithm in the opencv toolbox to fit the line parameters and calculate the mean distance and variance of all corner points to the corresponding horizontal and vertical lines. and variance 。

[0072] 2.4. Search for the imaging coordinates of all corner points with the same id on the image plane of the binocular actual simulation camera. Based on the internal and external parameters of the binocular actual simulation camera, perform triangulation to obtain the triangulated 3D coordinates corresponding to each id corner point, and calculate the mean error and variance between all triangulated 3D coordinates and the corresponding true 3D coordinates. and variance 。

[0073] 2.5. When the mean distance and variance of all corner points to the corresponding horizontal and vertical lines and variance are both less than the threshold, and the mean error and variance between all triangulated 3D coordinates and the true 3D coordinates and variance are both less than the threshold, it is considered that the internal and external parameters of the calibrated binocular actual simulation camera meet the requirements. Otherwise, re - execute the steps of the aforementioned camera calibration module to calibrate the binocular actual simulation camera.

[0074] The position compensation information determination module is used to determine the position compensation information of the binocular actual simulation camera. Among them, in order to achieve high - precision binocular image fusion of the near - eye display device at a specified convergence distance, it is required that the binocular simulated human - eye cameras be placed at the designed ideal positions, that is, to be the ideal simulated cameras for binoculars. Taking a fixed reference target in space (i.e., the image - fusion target for binocular image fusion) as a reference, the positions of the ideal simulated cameras for binoculars and the near - eye display device are determined. Please refer to Figure 4 , Figure 4 which is a schematic diagram of the position relationship between the ideal simulated cameras for binoculars and the reference target provided by the embodiment of the present application. Among them, assuming that the x - axis direction of the reference target is parallel to the connection direction of the left and right optical lenses of the near - eye display device, the ideal simulated cameras for binoculars meet the following conditions a - i, specifically: a. There is only a rotation around the z - axis between the optical axis of the ideal left - eye simulated camera and the normal vector of the reference target, and the included angle is degrees.

[0075] b. The optical axis of the ideal left - eye virtual display screen of the near - eye display device coincides with the optical axis of the ideal left - eye simulated camera, and this optical axis intersects the reference target at point A.

[0076] c. There is only a rotation around the z-axis between the optical axis of the ideal right-eye simulation camera and the normal vector of the reference target, and the included angle is degrees.

[0077] d. The optical axis of the ideal right-eye virtual display screen of the near-eye display device coincides with the optical axis of the ideal right-eye simulation camera, and this optical axis intersects the reference target at point B. Among them, point A and point B have the same yz-axis coordinates and different x-axis coordinates.

[0078] e. The virtual camera center corresponding to the ideal left-eye virtual display screen of the near-eye display device coincides with the ideal left-eye simulation camera center.

[0079] f. The virtual camera center corresponding to the ideal right-eye virtual display screen of the near-eye display device coincides with the ideal right-eye simulation camera center.

[0080] g. The distance from the center of the center of the ideal left-eye simulation camera center and the ideal right-eye simulation camera center to the reference target is d, with the unit of m.

[0081] h. The distance between the ideal left-eye simulation camera center and the ideal right-eye simulation camera center is the standard interpupillary distance.

[0082] i. The binocular fusion distance between the imaging plane center of the ideal left-eye simulation camera and the imaging plane center of the ideal right-eye simulation camera is , with the unit of m.

[0083] However, in the actual production and assembly process, due to factors such as machining errors of the base (used to fix the near-eye display device and the actual simulation camera) and human errors during the assembly process, it is very difficult to adjust the binocular simulation human eye cameras to the ideal positions designed. Therefore, in order to improve the accuracy of the position information of the binocular actual simulation cameras (that is, the binocular simulation human eye cameras installed at the actual position parameters), based on the positions of the binocular actual simulation cameras during assembly, determine the position compensation information corresponding to the positions of this assembly. That is, the specific steps for determining the compensation information module include: 3.1. Each of the binocular actual simulation cameras captures an image of the reference target, obtaining two reference target images.

[0084] 3.2. Extract corner points on the captured reference target images.

[0085] 3.3. Based on the 2D-3D matching points, use the pnp algorithm to calculate the transformation relationship between the actual left-eye simulation camera and the reference target and the transformation relationship between the actual right-eye simulation camera and the reference target , where represents the coordinate system of the actual left-eye simulation camera, represents the coordinate system of the actual right-eye simulation camera, Indicates the reference target coordinate system.

[0086] 3.4. Calculate the transformation relationship between the ideal left-eye simulated camera and the reference target based on the ideal position parameters of the ideal simulated camera and the transformation relationship between the ideal right-eye simulated camera and the reference target , where represents the ideal left-eye simulated camera coordinate system, represents the ideal right-eye simulated camera coordinate system, represents the reference target coordinate system.

[0087] 3.5. Combine the transformation relationship between the actual left-eye simulated camera and the reference target , the transformation relationship between the ideal left-eye simulated camera and the reference target , and calculate the transformation relationship between the actual left-eye simulated camera and the ideal left-eye simulated camera , which is expressed as:

[0088] And, combine the transformation relationship between the actual right-eye simulated camera and the reference target , the transformation relationship between the ideal right-eye simulated camera and the reference target , and calculate the transformation relationship between the actual right-eye simulated camera and the ideal right-eye simulated camera , which is expressed as:

[0089] The binocular image synthesis optimization module is used to adjust the display parameters of the near-eye display device based on the transformation relationship between the actual left-eye simulated camera and the ideal left-eye simulated camera and the transformation relationship between the actual right-eye simulated camera and the ideal right-eye simulated camera to improve the accuracy of the binocular image synthesis output by the near-eye display device when observed from the actual simulated cameras of both eyes. Specifically, it includes: 4.1. Display a target target image on each of the left-eye virtual display screen and the right-eye virtual display screen of the near-eye display device.

[0090] Among them, the target target image is as Figure 3 shown, and record the coordinate sets of all the crosshair endpoints in the target target image under the left-eye virtual display screen and the right-eye virtual display screen as . Among them, the coordinates of the crosshair endpoints in the target target image on the virtual display screen (including the left-eye virtual display screen and the right-eye virtual display screen) are the same as those on the ideal virtual display screen (including the ideal left-eye virtual display screen and the ideal right-eye virtual display screen).

[0091] 4.2. Combine the conversion relationship between the ideal left-eye simulation camera and the virtual camera center corresponding to the ideal left-eye virtual display screen , the virtual image distance of the ideal left-eye virtual display screen of the near-eye display device , the internal parameters of the ideal left-eye virtual display screen of the near-eye display device , the internal parameters of the ideal left-eye simulation camera , calculate the projection coordinates of all the crosshair endpoints displayed on the ideal left-eye virtual display screen on the undistorted imaging plane of the ideal left-eye simulation camera :

[0092] wherein, represents homogeneous coordinates. For example, represents the corresponding homogeneous coordinates, represents the corresponding homogeneous coordinates, is a three-dimensional vector, , and respectively represent the variables of each dimension, , represents the rotation from the ideal left-eye virtual display screen to the ideal left-eye simulation camera, represents the translation from the ideal left-eye virtual display screen to the ideal left-eye simulation camera, represents the coordinate system of the ideal left-eye virtual display screen. Among them, represents the pixel focal length of the ideal left-eye virtual display screen on the x-axis, represents the pixel focal length of the ideal left-eye virtual display screen on the y-axis, represents the principal point coordinate of the ideal left-eye virtual display screen on the x-axis, represents the principal point coordinate of the ideal left-eye virtual display screen on the y-axis. Among them, represents the pixel focal length of the ideal left-eye simulation camera on the x-axis, represents the pixel focal length of the ideal left-eye simulation camera on the y-axis, represents the principal point coordinate of the ideal left-eye simulation camera on the x-axis, represents the principal point coordinate of the ideal left-eye simulation camera on the y-axis.

[0093] 4.3. Combine the conversion relationship between the actual left-eye simulation camera and the ideal left-eye simulation camera obtained from the previous calculation , the virtual image distance of the ideal left-eye virtual display screen of the near-eye display device , the coordinates of the normal vector of the ideal left-eye virtual display screen of the near-eye display device under the ideal left-eye simulation camera , the internal parameters of the actual left-eye simulation camera (wherein, the internal parameters of the ideal left-eye simulation camera and the actual left-eye simulation camera are the same), calculate the homography matrix from the ideal left-eye simulation camera to the actual left-eye simulation camera of the ideal left-eye virtual display screen based on the near-eye display device:

[0094] Among them, = [0 0 1] T represents the coordinates of the normal vector of the ideal left-eye virtual display screen in the coordinate system of the ideal left-eye simulation camera, represents the rotation from the ideal left-eye simulation camera to the actual left-eye simulation camera, represents the translation from the ideal left-eye simulation camera to the actual left-eye simulation camera.

[0095] 4.4. Use the homography matrix from the ideal left-eye simulation camera to the actual left-eye simulation camera of the ideal left-eye virtual display screen , and transform the projection coordinates on the imaging plane of the ideal left-eye simulation camera to the projection coordinates on the imaging plane of the actual left-eye simulation camera:

[0096] Among them, represents the corresponding homogeneous coordinates.

[0097] 4.5. Combine the conversion relationship between the ideal right-eye simulation camera center and the virtual camera center corresponding to the ideal right-eye virtual display screen , the virtual image distance of the ideal right-eye virtual display screen of the near-eye display device, the internal parameters of the ideal right-eye virtual display screen of the near-eye display device, the internal parameters of the ideal right-eye simulation camera (wherein, the internal parameters of the ideal right-eye simulation camera and the actual right-eye simulation camera are the same), calculate the projection coordinates of all the crosshair endpoints

[0098] displayed on the ideal right-eye virtual display screen on the undistorted imaging plane of the ideal right-eye simulation camera: represents the homogeneous coordinates. For example, represents the corresponding homogeneous coordinates, represents the corresponding homogeneous coordinates, is a three - dimensional vector, , and represent the variables of each dimension respectively, , represents the rotation from the ideal right - eye virtual display to the ideal right - eye simulated camera, represents the translation from the ideal right - eye virtual display to the ideal right - eye simulated camera, represents the ideal right - eye virtual display coordinate system. Among them, represents the pixel focal length of the ideal right - eye virtual display on the x - axis, represents the pixel focal length of the ideal right - eye virtual display on the y - axis, represents the principal point coordinate of the ideal right - eye virtual display on the x - axis, represents the principal point coordinate of the ideal right - eye virtual display on the y - axis. Among them, represents the pixel focal length of the ideal right - eye simulated camera on the x - axis, represents the pixel focal length of the ideal right - eye simulated camera on the y - axis, represents the principal point coordinate of the ideal right - eye simulated camera on the x - axis, represents the principal point coordinate of the ideal right - eye simulated camera on the y - axis.

[0099] 4.6. Combining the transformation relationship between the actual right - eye simulated camera and the ideal right - eye simulated camera calculated from the previous text , the virtual image distance of the ideal right - eye virtual display of the near - eye display device , the coordinates of the normal vector of the ideal right - eye virtual display of the near - eye display device under the ideal right - eye simulated camera , the internal parameters of the actual right - eye simulated camera , calculate the homography matrix from the ideal right - eye simulated camera to the actual right - eye simulated camera based on the ideal right - eye virtual display of the near - eye display device:

[0100] Among them, = [0 0 1] T represents the coordinates of the normal vector of the ideal right - eye virtual display in the ideal right - eye simulated camera coordinate system, represents the rotation from the ideal right - eye simulated camera to the actual right - eye simulated camera, represents the translation from the ideal right - eye simulated camera to the actual right - eye simulated camera.

[0101] 4.7. Using the homography matrix from the ideal right - eye simulated camera to the actual right - eye simulated camera based on the ideal right - eye virtual display transform the projection coordinates on the imaging plane of the ideal right-eye simulation camera to the projection coordinates on the imaging plane of the actual right-eye simulation camera :

[0102] wherein, represents the corresponding homogeneous coordinates.

[0103] 4.8. Adjust the positional relationship between the left optical engine and the left lens of the near-eye display device until all the crosshair endpoints on the actual left-eye virtual display screen of the near-eye display device are aligned to the projection coordinates

[0104] on the imaging plane of the left-eye simulation human eye camera. 4.9. Adjust the positional relationship between the right optical engine and the right lens of the near-eye display device until all the crosshair endpoints on the actual right-eye virtual display screen of the near-eye display device

[0105] are aligned to the projection coordinates

[0106] on the imaging plane of the right-eye simulation human eye camera.

[0107] For example, adjust the positional relationship between the left optical engine and the left lens of the near-eye display device or adjust the positional relationship between the right optical engine and the right lens of the near-eye display device through a robotic arm. Thus, through the target left-eye conversion relationship between the actual left-eye simulation camera and the ideal left-eye simulation camera, and the target right-eye conversion relationship between the actual right-eye simulation camera and the ideal right-eye simulation camera, the adjustment of the left and right optical engines of the near-eye display device realizes the position compensation for the actual left-eye simulation camera and the actual right-eye simulation camera, and achieves high-precision binocular image synthesis of the near-eye display device.

[0107] Correspondingly, in the embodiment of the present application, after the position compensation for the actual left-eye simulation camera and the actual right-eye simulation camera, the position compensation can also be verified through coordinate errors to ensure that the binocular image synthesis accuracy meets the requirements, that is, optionally, in some embodiments of the present application, before the step of "displaying a target virtual image based on the target position relationship", the method further includes: displaying a test target image through the left-eye virtual display screen and the right-eye virtual display screen of the near-eye display device respectively, wherein the test target image includes target corner points; determining the ideal three-dimensional coordinates of the target corner points corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulation camera, and determining the actual three-dimensional coordinates of the target corner points corresponding to the left-eye virtual display screen in the actual left-eye simulation camera; Determine the theoretical three-dimensional coordinates corresponding to the ideal three-dimensional coordinates in the left-eye simulated camera according to the target left-eye conversion relationship; Verify the target position relationship based on the actual three-dimensional coordinates and the theoretical three-dimensional coordinates to obtain a position verification result; Display the target virtual image based on the target position relationship, including: If the position verification result indicates that the binocular fusion of the left-eye virtual display screen and the right-eye virtual display screen meets the preset conditions, display the target virtual image through the left-eye virtual display screen and the right-eye virtual display screen respectively.

[0108] Among them, in the embodiments of the present application, position compensation is performed based on the coordinate error of the left eye. In some scenarios, position compensation can also be selected for the coordinate error of the right eye, or the coordinate errors of both the left eye and the right eye can be selected simultaneously for comprehensive judgment.

[0109] Among them, the theoretical three-dimensional coordinates are calculated based on the target left-eye conversion relationship and the ideal left-eye simulated camera, and are the theoretical comparison objects for coordinate errors in the embodiments of the present application. The actual three-dimensional coordinates are triangulated based on the actual imaging coordinates of the target corner points on the actual left-eye simulated camera and are used as the actual values of the coordinate errors. The accuracy of the position compensation is judged by comparing the actual value with the theoretical comparison object.

[0110] Optionally, in the embodiments of the present application, the ideal three-dimensional coordinates can be obtained through triangulation of the projection coordinates of the target corner points on the ideal left-eye simulated camera and the projection coordinates on the ideal right-eye simulated camera. That is, optionally, in some embodiments of the present application, the step of "determining the ideal three-dimensional coordinates corresponding to the target corner points under the ideal left-eye virtual display screen in the ideal left-eye simulated camera" includes: Calculate the first test projection coordinates corresponding to the target corner points in the ideal left-eye virtual display screen in the ideal left-eye simulated camera; Calculate the second test projection coordinates corresponding to the target corner points in the ideal right-eye virtual display screen in the ideal right-eye simulated camera; Perform triangulation on the first test projection coordinates and the second test projection coordinates according to the internal parameters of the ideal left-eye simulated camera, the internal parameters of the ideal right-eye simulated camera, and the external parameters between the ideal left-eye simulated camera and the ideal right-eye simulated camera to obtain the ideal three-dimensional coordinates corresponding to the target corner points in the ideal left-eye simulated camera.

[0111] For example, the verification of the position compensation specifically includes: 5.1. Display a checkerboard target image on each of the actual left-eye virtual display screen and the actual right-eye virtual display screen of the near-eye display device.

[0112] Among them, the checkerboard target image corresponds to the test target image in the previous text.

[0113] 5.2. Combine the conversion relationship between the ideal left-eye simulated camera center and the virtual camera center corresponding to the ideal left-eye virtual display screen , the virtual image distance of the ideal left-eye virtual display screen of the near-eye display device , the internal parameters of the ideal left-eye virtual display screen of the near-eye display device , the internal parameters of the ideal left-eye simulated camera (wherein, the internal parameters of the ideal left-eye simulated camera and the actual left-eye simulated camera are the same), calculate all the checkerboard corner points displayed on the ideal left-eye virtual display screen The projection coordinates on the undistorted imaging plane of the ideal left-eye simulated camera :

[0114] Among them, represents homogeneous coordinates. For example, represents The corresponding homogeneous coordinates, represents The corresponding homogeneous coordinates, , represents the rotation from the ideal left-eye virtual display screen to the ideal left-eye simulated camera, represents the translation from the ideal left-eye virtual display screen to the ideal left-eye simulated camera. The projection coordinates are the first test projection coordinates mentioned above. Among them, represents the pixel focal length of the ideal left-eye virtual display screen on the x-axis, represents the pixel focal length of the ideal left-eye virtual display screen on the y-axis, represents the principal point coordinate of the ideal left-eye virtual display screen on the x-axis, represents the principal point coordinate of the ideal left-eye virtual display screen on the y-axis. Among them, represents the pixel focal length of the ideal left-eye simulated camera on the x-axis, represents the pixel focal length of the ideal left-eye simulated camera on the y-axis, represents the principal point coordinate of the ideal left-eye simulated camera on the x-axis, represents the principal point coordinate of the ideal left-eye simulated camera on the y-axis.

[0115] 5.3. Combine the conversion relationship between the ideal right-eye simulated camera center and the virtual camera center corresponding to the ideal right-eye virtual display screen , the virtual image distance of the ideal right-eye virtual display screen of the near-eye display device , the internal parameters of the ideal right-eye virtual display screen of the near-eye display device , the internal parameters of the ideal right-eye simulated camera (where the internal parameters of the ideal right-eye simulation camera and the actual right-eye simulation camera are the same), calculate all the checkerboard corner points displayed on the virtual display screen of the right eye The projection coordinates on the imaging plane of the undistorted ideal right-eye simulation camera :

[0116] Where represents homogeneous coordinates. For example, represents The corresponding homogeneous coordinates, represents The corresponding homogeneous coordinates, , represents the rotation from the ideal right-eye virtual display screen to the ideal right-eye simulation camera, represents the translation from the ideal right-eye virtual display screen to the ideal right-eye simulation camera. These projection coordinates are the second test projection coordinates mentioned above. Where represents the pixel focal length of the ideal right-eye virtual display screen on the x-axis, represents the pixel focal length of the ideal right-eye virtual display screen on the y-axis, represents the principal point coordinate of the ideal right-eye virtual display screen on the x-axis, represents the principal point coordinate of the ideal right-eye virtual display screen on the y-axis. Where represents the pixel focal length of the ideal right-eye simulation camera on the x-axis, represents the pixel focal length of the ideal right-eye simulation camera on the y-axis, represents the principal point coordinate of the ideal right-eye simulation camera on the x-axis, represents the principal point coordinate of the ideal right-eye simulation camera on the y-axis.

[0117] 5.4. Combine the internal parameters of the ideal left-eye simulation camera, the internal parameters of the ideal right-eye simulation camera, and the external parameters between the ideal left-eye simulation camera and the ideal right-eye simulation camera, and calculate the imaging coordinates and of each checkerboard corner point on the image plane of the ideal left-eye simulation camera and the image plane of the ideal right-eye simulation camera, and obtain the true 3D combined image coordinates of all checkerboard corner points under the ideal left-eye simulation camera. Then, combined with the conversion relationship between the ideal left-eye simulation camera and the actual left-eye simulation camera, calculate the true 3D combined image coordinates of all checkerboard corner points under the actual left-eye simulation camera:

[0118] Among them, represents the rotation from the ideal left-eye simulation camera to the actual left-eye simulation camera, represents the translation from the ideal left-eye simulation camera to the actual left-eye simulation camera. Among them, the true value of this 3D combined image coordinate is the ideal three-dimensional coordinate mentioned above. The true value of this 3D combined image coordinate is the theoretical three-dimensional coordinate mentioned above.

[0119] 5.5. Each of the actual left-eye simulation camera and the actual right-eye simulation camera captures an image facing the checkerboard target, obtaining the left-eye captured image and the right-eye captured image.

[0120] 5.6. The checkerboard corner points extracted from the left-eye captured image are represented as , and the checkerboard corner points extracted from the right-eye captured image are represented as .

[0121] 5.7. Combining the internal parameters of the actual left-eye simulation camera, the internal parameters of the actual right-eye simulation camera, and the external parameters between the actual left-eye simulation camera and the actual right-eye simulation camera, the imaging coordinates and of each checkerboard corner point on the image planes of the actual left-eye simulation camera and the actual right-eye simulation camera are triangulated to obtain the triangulated 3D combined image coordinates of all checkerboard corner points under the left-eye simulated human eye camera.

[0122] 5.8. Calculate the mean error between the triangulated 3D combined image coordinates and the true value of the 3D combined image coordinate and the variance . When both the mean and the variance are less than the threshold, it is considered that the binocular combined image accuracy meets the requirements; otherwise, the steps of the binocular combined image optimization module need to be continued.

[0123] In summary, due to the difference between the ideal position parameters of the ideal simulation camera and the actual position parameters of the actual simulation camera, in the embodiments of the present application, by determining the target conversion relationship between the ideal simulation camera and the actual simulation camera, and compensating and adjusting the display of the near-eye display device through this target conversion relationship, the accuracy of device adjustment is improved.

[0124] Among them, by separately adjusting the positions of the left and right opto-mechanisms, the accuracy of binocular combined imaging of the near-eye display device is improved.

[0125] Please refer to Figure 5 , Figure 5FIG. 0 is a schematic structural diagram of a near-eye display device provided by an embodiment of the present application. The near-eye display device is the near-eye display device in the virtual display control method described above. The professional terms and their meanings involved are the same and will not be elaborated here. For example, Figure 5 As shown, taking the near-eye display device as an augmented reality-based smart glasses as an example, the smart glasses include a left optical lens 111 and a right optical lens 112. The smart glasses can output target virtual images on the virtual display screens of the left and right eyes respectively, and improve the binocular imaging accuracy of the smart glasses based on the position compensation for the left optical engine 113 and the right optical engine (not shown in the figure).

[0126] To facilitate better implementation of the virtual display control method of the near-eye display device of the present application, the present application also provides a virtual display control device for the near-eye display device based on the above virtual display control method of the near-eye display device. The meanings of the nouns are the same as those in the virtual display control method of the above near-eye display device, and the specific implementation details can refer to the description in the method embodiment.

[0127] Please refer to Figure 6 , Figure 6 FIG. 12 is a schematic structural diagram of a virtual display control device for a near-eye display device provided by an embodiment of the present application. The virtual display control device for the near-eye display device can be specifically as follows: A display module 201 for displaying a reference target; A first determination module 202 for determining the ideal conversion relationship between the ideal simulation camera with ideal position parameters and the reference target, and determining the actual conversion relationship between the actual simulation camera with actual position parameters and the reference target; A second determination module 203 for determining a target conversion relationship according to the ideal conversion relationship and the actual conversion relationship; A control module 204 for controlling the near-eye display device to display a target virtual image according to the target conversion relationship.

[0128] Optionally, in some embodiments of the present application, the near-eye display device includes an optical engine and an optical lens; Controlling the near-eye display device to display a target virtual image according to the target conversion relationship includes: Adjusting the positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target positional relationship; Displaying the target virtual image based on the target positional relationship.

[0129] Optionally, in some embodiments of the present application, adjusting the positional relationship between the optical engine and the optical lens according to the target conversion relationship to obtain a target positional relationship includes: Displaying a target target image through the virtual display screen of the near-eye display device; Determine the first projection coordinates of the target reference object in the target target image under the ideal virtual display screen corresponding to the ideal simulation camera; Determine the second projection coordinates of the target reference object in the ideal virtual display screen corresponding to the actual simulation camera according to the first projection coordinates and the target conversion relationship; According to the target reference object and the second projection coordinates in the target target image under the virtual display screen, adjust the positional relationship between the light engine and the optical lens to obtain the target positional relationship.

[0130] Optionally, in some embodiments of the present application, determining the second projection coordinates of the target reference object in the ideal virtual display screen corresponding to the actual simulation camera according to the first projection coordinates and the target conversion relationship includes: Calculate the homography matrix between the ideal simulation camera and the actual simulation camera according to the target conversion relationship, the normal vector coordinates of the ideal virtual display screen corresponding to the ideal simulation camera, and the internal parameters of the actual simulation camera; Transform the first projection coordinates into the second projection coordinates corresponding to the actual simulation camera according to the homography matrix.

[0131] Optionally, in some embodiments of the present application, the ideal simulation camera includes an ideal left-eye simulation camera and an ideal right-eye simulation camera, and the actual simulation camera includes an actual left-eye simulation camera and an actual right-eye simulation camera; The target conversion relationship includes a target left-eye conversion relationship and a target right-eye conversion relationship, wherein the target left-eye conversion relationship is determined based on the ideal left-eye simulation camera and the actual left-eye simulation camera, and the target right-eye conversion relationship is determined based on the ideal right-eye simulation camera and the actual right-eye simulation camera; The target positional relationship includes a target left-eye positional relationship and a target right-eye positional relationship, the light engine includes a left-eye light engine and a right-eye light engine, and the optical lens includes a left-eye optical lens and a right-eye optical lens; Adjusting the positional relationship between the light engine and the optical lens according to the target conversion relationship to obtain the target positional relationship includes: Adjust the positional relationship between the left-eye light engine and the left-eye optical lens according to the target left-eye conversion relationship to obtain the target left-eye positional relationship, and adjust the positional relationship between the right-eye light engine and the right-eye optical lens according to the target right-eye conversion relationship to obtain the target right-eye positional relationship.

[0132] Optionally, in some embodiments of the present application, the virtual display screen of the near-eye display device includes a left-eye virtual display screen and a right-eye virtual display screen, and the ideal virtual display screen of the near-eye display device includes an ideal left-eye virtual display screen and an ideal right-eye virtual display; Before displaying the target virtual image based on the target positional relationship, the method further includes: The left-eye virtual display screen and the right-eye virtual display screen of the near-eye display device respectively display test target images, where the test target images contain target corner points; Determine the ideal three-dimensional coordinates of the target corner points corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulation camera, and determine the actual three-dimensional coordinates of the target corner points corresponding to the left-eye virtual display screen in the actual left-eye simulation camera; Determine the theoretical three-dimensional coordinates of the ideal three-dimensional coordinates corresponding to the left-eye simulation camera according to the target left-eye conversion relationship; Verify the target position relationship according to the actual three-dimensional coordinates and the theoretical three-dimensional coordinates to obtain a position verification result; If the position verification result indicates that the binocular image combination of the left-eye virtual display screen and the right-eye virtual display screen meets the preset conditions, then display target virtual images through the left-eye virtual display screen and the right-eye virtual display screen respectively.

[0133] Optionally, in some embodiments of the present application, determining the ideal three-dimensional coordinates of the target corner points corresponding to the ideal left-eye virtual display screen in the ideal left-eye simulation camera includes: Calculate the first test projection coordinates of the target corner points in the ideal left-eye virtual display screen corresponding to the ideal left-eye simulation camera; Calculate the second test projection coordinates of the target corner points in the ideal right-eye virtual display screen corresponding to the ideal right-eye simulation camera; Perform triangulation on the first test projection coordinates and the second test projection coordinates according to the internal parameters of the ideal left-eye simulation camera, the internal parameters of the ideal right-eye simulation camera, and the external parameters between the ideal left-eye simulation camera and the ideal right-eye simulation camera to obtain the ideal three-dimensional coordinates of the target corner points corresponding to the ideal left-eye simulation camera.

[0134] In the embodiment of the present application, the display module 201 displays a reference target, the first determination module 202 determines the ideal conversion relationship between the ideal simulation camera with the ideal position parameters and the reference target, and determines the actual conversion relationship between the actual simulation camera with the actual position parameters and the reference target. The second determination module 203 determines the target conversion relationship according to the ideal conversion relationship and the actual conversion relationship, and the control module 204 controls the near-eye display device to display the target virtual image according to the target conversion relationship.

[0135] In summary, the embodiment of the present application determines the target conversion relationship between the ideal simulation camera and the actual simulation camera, and compensates and adjusts the display of the near-eye display device through this target conversion relationship, improving the accuracy of device adjustment.

[0136] In addition, the present application also provides an electronic device, as Figure 7 shown, which shows the structural schematic diagram of the electronic device involved in the present application. Specifically: The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 of one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art can understand that Figure 7 the structure of the electronic device shown in does not limit the electronic device, and it may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. Among them:

[0137] The processor 301 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 302, and by calling the data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 301.

[0138] The electronic device further includes a power supply 303 that powers each component. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby implementing functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 303 may also include any components such as one or more DC or AC power supplies, a recharge system, a power device debugging circuit, a power converter or inverter, and a power status indicator.

[0139] The electronic device may further include an input unit 304, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function controls.

[0140] Although not shown, the electronic device may further include a display unit and the like, which will not be elaborated herein. Specifically, in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, so as to implement the steps in any virtual display control method of the near-eye display device provided by the embodiments of the present application.

[0141] The electronic device in the embodiment of the present application controls the display of a reference target. The electronic device determines the ideal conversion relationship between the ideal simulation camera with ideal position parameters and the reference target, and determines the actual conversion relationship between the actual simulation camera with actual position parameters and the reference target. According to the ideal conversion relationship and the actual conversion relationship, the target conversion relationship is determined, and the near-eye display device is controlled to display the target virtual image according to the target conversion relationship.

[0142] Among them, since there are differences between the ideal position parameters of the ideal simulation camera and the actual position parameters of the actual simulation camera, by determining the target conversion relationship between the ideal simulation camera and the actual simulation camera, the display of the near-eye display device can be compensated and adjusted through the target conversion relationship, improving the accuracy of device adjustment.

[0143] For the specific implementation of the above operations, reference may be made to the previous embodiments, which will not be elaborated herein.

[0144] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0145] Therefore, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any virtual display control method of the near-eye display device provided by the present application.

[0146] For the specific implementation of the above operations, reference may be made to the previous embodiments, which will not be elaborated herein.

[0147] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0148] Since the instructions stored in the computer-readable storage medium can execute the steps in any virtual display control method of the near-eye display device provided by the present application, the beneficial effects achievable by any virtual display control method of the near-eye display device provided by the present application can be realized. For details, refer to the previous embodiments and will not be elaborated here.

[0149] The virtual display control method, device, electronic device, and computer-readable storage medium provided by the present application have been introduced in detail above. Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A virtual display control method for a near-eye display device, characterized in that: The method comprises: Display reference targets; Determine an ideal conversion relationship between an ideal simulated camera of ideal position parameters and the reference target, and determine an actual conversion relationship between an actual simulated camera of actual position parameters and the reference target; Determine a target conversion relationship according to the ideal conversion relationship and the actual conversion relationship; The near-eye display device is controlled to display a target virtual image according to the target conversion relationship.

2. The virtual display control method of the near-eye display device according to claim 1, characterized in that: The near-eye display device includes an optical machine and an optical lens; The controlling the near-eye display device to display a target virtual image according to the target conversion relationship includes: Adjust the positional relationship between the optical machine and the optical lens according to the target conversion relationship to obtain a target positional relationship; The target virtual image is displayed based on the target position relationship.

3. The virtual display control method of the near-eye display device according to claim 2, characterized in that: The step of adjusting the positional relationship between the optical machine and the optical lens according to the target conversion relationship to obtain a target positional relationship includes: Displaying a target image through a virtual display screen of the near-eye display device; Determine the first projection coordinates of the target reference object in the target image under the ideal virtual display screen corresponding to the ideal simulated camera; Determine, according to the first projection coordinates and the target conversion relationship, the second projection coordinates of the target reference object corresponding to the actual simulation camera under the ideal virtual display screen; According to the target reference object in the target image under the virtual display screen and the second projection coordinates, the positional relationship between the optical machine and the optical lens is adjusted to obtain the target positional relationship.

4. The virtual display control method of the near-eye display device according to claim 3, characterized in that: The step of determining the second projection coordinates of the target reference object on the ideal virtual display screen corresponding to the actual simulation camera according to the first projection coordinates and the target conversion relationship includes: Calculate a homography matrix between the ideal simulated camera and the actual simulated camera according to the target conversion relationship, the normal vector coordinates of the ideal virtual display screen corresponding to the normal vector of the ideal simulated camera, and the internal parameters of the actual simulated camera; The first projection coordinates are transformed into the second projection coordinates corresponding to the actual simulated camera according to the homography matrix.

5. The virtual display control method of the near-eye display device according to claim 2, characterized in that: The ideal simulated camera includes an ideal left-eye simulated camera and an ideal right-eye simulated camera, and the actual simulated camera includes an actual left-eye simulated camera and an actual right-eye simulated camera; The target conversion relationship includes a target left-eye conversion relationship and a target right-eye conversion relationship, wherein the target left-eye conversion relationship is determined based on the ideal left-eye simulation camera and the actual left-eye simulation camera, and the target right-eye conversion relationship is determined based on the ideal right-eye simulation camera and the actual right-eye simulation camera; The target position relationship includes a target left eye position relationship and a target right eye position relationship, the optical machine includes a left eye machine and a right eye machine, and the optical lens includes a left eye optical lens and a right eye optical lens; The step of adjusting the positional relationship between the optical machine and the optical lens according to the target conversion relationship to obtain a target positional relationship includes: The target left-eye position relationship is obtained by adjusting the position relationship between the left-eye motor and the left-eye optical lens according to the target left-eye conversion relationship, and the target right-eye position relationship is obtained by adjusting the position relationship between the right-eye motor and the right-eye optical lens according to the target right-eye conversion relationship.

6. The virtual display control method of the near-eye display device according to claim 5, characterized in that: The virtual display screen of the near-eye display device includes a left-eye virtual display screen and a right-eye virtual display screen, and the ideal virtual display screen of the near-eye display device includes an ideal left-eye virtual display screen and an ideal right-eye virtual display screen; Before displaying the target virtual image based on the target position relationship, the method further includes: Displaying a test target image through the left-eye virtual display screen and the right-eye virtual display screen of the near-eye display device, respectively, wherein the test target image includes a target corner point; Determine the ideal three-dimensional coordinates of the target corner point on the ideal left-eye virtual display screen corresponding to the ideal left-eye simulated camera, and determine the actual three-dimensional coordinates of the target corner point on the left-eye virtual display screen corresponding to the actual left-eye simulated camera; Determine the theoretical three-dimensional coordinates of the left-eye simulation camera corresponding to the ideal three-dimensional coordinates according to the target left-eye conversion relationship; Verifying the target position relationship according to the actual three-dimensional coordinates and the theoretical three-dimensional coordinates to obtain a position verification result; The displaying of the target virtual image based on the target position relationship comprises: If the position verification result indicates that the binocular combined image of the left-eye virtual display screen and the right-eye virtual display screen meets a preset condition, the target virtual image is displayed through the left-eye virtual display screen and the right-eye virtual display screen respectively.

7. The virtual display control method of the near-eye display device according to claim 6, characterized in that: The step of determining that the target corner point under the ideal left-eye virtual display screen corresponds to the ideal three-dimensional coordinate of the ideal left-eye simulation camera includes: Calculate the first test projection coordinates of the target corner point in the ideal left-eye virtual display screen corresponding to the ideal left-eye simulation camera; Calculate the second test projection coordinates of the target corner point in the ideal right-eye virtual display screen corresponding to the ideal right-eye simulation camera; The first test projection coordinates and the second test projection coordinates are triangulated according to the intrinsic parameters of the ideal left-eye simulation camera, the intrinsic parameters of the ideal right-eye simulation camera, and the extrinsic parameters between the ideal left-eye simulation camera and the ideal right-eye simulation camera to obtain the ideal three-dimensional coordinates of the target corner point corresponding to the ideal left-eye simulation camera.

8. A virtual display control device for a near-eye display device, characterized in that: The device comprises: A display module, used for displaying a reference target; A first determination module is used to determine an ideal conversion relationship between an ideal simulated camera of ideal position parameters and the reference target, and to determine an actual conversion relationship between an actual simulated camera of actual position parameters and the reference target; A second determining module, configured to determine a target conversion relationship according to the ideal conversion relationship and the actual conversion relationship; A control module is used to control the near-eye display device to display a target virtual image according to the target conversion relationship.

9. A near-eye display device, characterized in that: The near-eye display device is a near-eye display device in the virtual display control method for a near-eye display device according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the virtual display control method of the near-eye display device according to any one of claims 1 to 7 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the virtual display control method of the near-eye display device according to any one of claims 1 to 7 are implemented.

Images