Virtual picture display method and device, split AR glasses and readable storage medium

Abstract

This application provides a virtual image display method, device, split-type AR glasses, and readable storage medium. After receiving a trigger signal to initiate reprojection, the split-type AR glasses reads a virtual image in response to the trigger signal, reprojects the virtual image according to the refresh mode of the glasses' screen, and displays the reprojected virtual image at the moment of display. The refresh mode indicates the process of pixels being lit and turned off within one refresh cycle of the screen. By employing this scheme, combined with the refresh mode of the optical screen to reproject the virtual image, the reprojected virtual image matches the pose of the glasses at the moment of display, thereby improving the reprojection quality and ultimately improving the display quality of the virtual image.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to a virtual screen display method, device, split-type AR glasses, and readable storage medium. Background Technology

[0002] Currently, Augmented Reality (AR) glasses are gradually developing in fields such as gaming, entertainment, and education.

[0003] Common AR glasses include all-in-one AR glasses and split-type AR glasses. All-in-one AR glasses are relatively heavy because the computing unit is located on the main body of the glasses, which seriously affects the user experience. Split-type AR glasses, on the other hand, connect the computing unit and the main body of the glasses via a cable, making the glasses lighter and thus more popular. However, split-type AR glasses suffer from both data transmission and rendering latency. The pose and velocity used to generate the virtual image are different from the pose and velocity of the glasses themselves at the moment of display. Therefore, the virtual image needs to be reprojected before being displayed to match the pose and velocity of the glasses themselves at the moment of display, i.e., reprojection reduces latency. For example, split-type AR glasses include a rendering engine and a reprojection module. The rendering engine generates the virtual image and stores it in a buffer. The reprojection module reads the virtual image from the buffer, predicts the pose of the glasses themselves at the moment of display, and reprojects the virtual image based on the predicted pose. At the moment of display, the reprojected virtual image is displayed on the optical screen of the glasses.

[0004] However, the split-type AR glasses predict the pose when projecting onto the screen based on the pose and speed of the glasses themselves. The accuracy of the predicted pose is poor, resulting in poor reprojection quality. The latency compensation is not accurate enough, which in turn leads to poor optical screen display quality. Summary of the Invention

[0005] This application provides a virtual image display method, device, split AR glasses, and readable storage medium. By combining the refresh method of the optical screen to reproject the virtual image, the reprojection quality is improved, thereby improving the display quality of the virtual image.

[0006] In a first aspect, embodiments of this application provide a virtual screen display method applied to split-type AR glasses, the split-type AR glasses including a glasses body and a computing unit, the method comprising:

[0007] Obtain the trigger signal used to initiate reprojection;

[0008] In response to the trigger signal, read the virtual screen;

[0009] The virtual image is reprojected according to the refresh mode of the screen of the glasses body, wherein the refresh mode is used to indicate the process of pixels being lit and turned off within one refresh cycle of the screen;

[0010] The virtual image after reprojection is displayed on the screen.

[0011] Secondly, embodiments of this application provide a virtual screen display device, which is integrated into a split-type AR glasses. The split-type AR glasses include a glasses body and a computing unit. The device includes:

[0012] The acquisition module is used to acquire the trigger signal used to trigger reprojection;

[0013] The response module is used to read the virtual screen in response to the trigger signal;

[0014] The processing module is used to reproject the virtual image according to the refresh mode of the screen of the glasses body, wherein the refresh mode is used to indicate the process of pixels being lit and turned off within one refresh cycle of the screen;

[0015] The display module is used to display the reprojected virtual image when the screen is on.

[0016] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it causes the electronic device to implement the method described in the first aspect or various possible implementations of the first aspect.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the method described in the first aspect or various possible implementations of the first aspect.

[0018] Fifthly, embodiments of this application provide a computer program product comprising a computing program, wherein when the computer program is executed by a processor, it implements the method described in the first aspect or various possible implementations of the first aspect.

[0019] The virtual image display method, apparatus, split-type AR glasses, and readable storage medium provided in this application embodiment allow the split-type AR glasses to read a virtual image in response to a trigger signal for reprojection after receiving the trigger signal. The virtual image is then reprojected according to the refresh mode of the glasses' screen, and the reprojected virtual image is displayed at the moment of screen display. The refresh mode indicates the process of pixels being lit and turned off within one refresh cycle of the screen. This approach, combining the refresh mode of the optical screen with the reprojection of the virtual image, ensures that the reprojected virtual image matches the pose of the glasses at the moment of screen display, thereby improving the reprojection quality and ultimately enhancing the virtual image display quality. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1A This is a schematic diagram of the virtual screen display process of traditional split-type AR glasses;

[0022] Figure 1B yes Figure 1A Corresponding timeline diagram;

[0023] Figure 2 This is a schematic diagram of the structure of the split AR glasses provided in the embodiments of this application;

[0024] Figure 3A This is a flowchart illustrating the virtual screen display method provided in the embodiments of this application;

[0025] Figure 3B yes Figure 3A Corresponding timeline diagram;

[0026] Figure 4 This is a schematic diagram of the screen refresh process in the virtual screen display method provided in the embodiments of this application;

[0027] Figure 5 This is a timeline diagram illustrating the process of receiving a trigger signal and reading a virtual image in the virtual image display method provided in this application embodiment;

[0028] Figure 6 This is a schematic diagram of determining the reprojected quadrilateral region in the virtual screen display method provided in the embodiments of this application;

[0029] Figure 7 This is a schematic diagram illustrating the determination of target depth in the virtual screen display method provided in this application embodiment;

[0030] Figure 8 This is a schematic diagram of a virtual screen display device provided in an embodiment of this application;

[0031] Figure 9 This is a schematic diagram of the structure of the split AR glasses provided in the embodiments of this application. Detailed Implementation

[0032] AR glasses are wearable devices that allow users to see virtual objects superimposed onto their physical environment. AR glasses include all-in-one and split-type models. Compared to all-in-one AR glasses, split-type AR glasses connect the glasses and computing unit via a data cable. The computing unit can be placed in the user's pocket or worn on their waist, and the glasses themselves are lighter.

[0033] The modular AR glasses house sensors for environmental perception and a display for visual representation. Sensors include, but are not limited to, cameras and inertial measurement units (IMUs), while the display, also known as a screen or optical screen, can be based on organic light-emitting diodes (OLEDs). The computing unit, similar to a mobile device, calculates the posture of the glasses based on data from the sensors and renders the corresponding virtual images.

[0034] Since split-type AR glasses have both data transmission latency and image rendering latency, the virtual image needs to be reprojected before it is displayed on the screen to reduce latency. Figure 1A This is a schematic diagram of the virtual screen display process of traditional split-type AR glasses. Figure 1B yes Figure 1A The corresponding timeline diagram is shown below. Please refer to it. Figure 1A and Figure 1B The computing unit contains a rendering engine and a reprojection module. The rendering engine and reprojection module run in their respective threads. The rendering engine reads the pose and velocity of the glasses at a specific frequency, in t x Virtual images are generated and stored in a buffer at constant intervals. The speed includes linear velocity and angular velocity. The buffer is a specific memory area capable of simultaneously storing several virtual images generated at different times. Before refreshing, the optical screen sends a vertical sync (vsync) signal to the computing unit. The frequency of the vsync signal is the same as the screen refresh rate. Taking the optical screen sending a vsync signal at time t1 as an example, after receiving the vsync signal, the reprojection module of the computing unit generates a task and calculates an execution time t. 目标When t 目标 At that time, a suitable virtual image is read from the buffer, and according to t 目标 Given the pose and velocity of the glasses at time t2, predict the pose of the glasses at time t2, and reproject the virtual image based on the predicted pose. The virtual image is one of multiple candidate virtual images stored in a buffer, for example, t... x The virtual image generated by the rendering engine is displayed at time t2. At time t2, the computing unit will display the reprojected virtual image on the screen. x The time is before time t1, t x There is a certain duration between time t1 and time t2.

[0035] In the above screen display method, t1 and t2 are two adjacent screen display moments, and the duration between t1 and t2 is one refresh cycle of the optical screen. 目标 It is a time point between t1 and t2, where t x It may or may not be the moment it appears on the screen.

[0036] However, the split-type AR glasses do not take into account the screen refresh rate when predicting pose, resulting in poor pose accuracy, poor reprojection quality, and inaccurate latency compensation, which in turn leads to poor optical screen display quality.

[0037] Based on this, embodiments of this application provide a virtual image display method, apparatus, device, and readable storage medium, which combines the refresh method of the AR glasses optical screen to reproject the virtual image, thereby accurately compensating for delay, improving the reprojection quality, and ultimately improving the display quality of the virtual image.

[0038] Figure 2 This is a schematic diagram of the structure of the split-type AR glasses provided in the embodiments of this application. Please refer to... Figure 2 The split-type AR glasses provided in this application embodiment include a glasses body 21, a computing unit 22, and a connecting cable 23. The glasses body 21 is connected to the computing unit 22 via the connecting cable 23. The glasses body 21 includes two lenses 211, a housing 213, and temples 212. The temples 212 and the housing 213 can be an integral structure or detachably connected. The lenses 211 are also referred to as the display, screen, optical screen, etc. of the split-type AR glasses, and can be OLED-based displays, LED-based displays, micro LED-based displays, etc., and this application embodiment is not limited to these.

[0039] Sensors are housed in the temple 212 and / or housing 213, including but not limited to IMU, camera, accelerometer, gyroscope, geomagnetic sensor, etc.

[0040] The computing unit 22 is equipped with a rendering engine and a reprojection module. The rendering engine reads the pose and velocity of the glasses body 21 at a specific frequency, generates a virtual image, and stores it in a buffer. Each time the reprojection module receives a trigger signal, it responds to the trigger signal and reprojects the virtual image in conjunction with the screen refresh method. When the time for displaying the image is reached, the reprojected virtual image is displayed on the screen.

[0041] Below, based on Figure 2 The illustrated split-type AR glasses will be used to describe in detail the virtual screen display method described in this application embodiment. For example, please refer to... Figure 3A and Figure 3B .

[0042] Figure 3A This is a flowchart illustrating the virtual screen display method provided in this application embodiment. The execution subject of this embodiment is a split-type AR glasses, and this embodiment includes:

[0043] 301. Obtain the trigger signal used to trigger reprojection.

[0044] In this embodiment, due to limitations in image transmission speed, the screen refresh of the split-type AR glasses often takes more than 10 milliseconds. Each time the screen of the split-type AR glasses refreshes, a trigger signal is sent, such as a vertical sync (vsync) signal. The frequency of the trigger signal is consistent with the screen's refresh rate. The screen refresh rate, also known as the refresh frequency, refers to the number of times the screen refreshes its image per second, including but not limited to 60 Hz, 120 Hz, 144 Hz, 240 Hz, etc. The refresh period refers to the duration of one screen refresh, which is the reciprocal of the refresh rate. Taking an OLED optical screen with a refresh rate of 60 Hz as an example, the refresh period T = 16.6 milliseconds, meaning it takes 16.6 milliseconds for the screen to refresh from the first line to the last line. Therefore, the screen sends a trigger signal every 16.6 milliseconds. Correspondingly, the computing unit receives a trigger signal every 16.6 milliseconds.

[0045] 302. In response to the trigger signal, read the virtual screen.

[0046] In this embodiment of the application, the virtual screen is a screen used to present virtual image information. By superimposing the virtual screen onto the real world, users wearing AR glasses can see the combination of the virtual screen and the real world in the same field of vision.

[0047] The rendering engine and reprojection module are deployed on the computing unit of the split-type AR glasses. The rendering engine reads the pose and velocity of the glasses body at a specific frequency, generates candidate virtual images, and saves them in a buffer. For example, the rendering engine at t x Candidate virtual images are generated and stored in a buffer at any time. The reprojection module responds to the trigger signal at t p The virtual image is read from the buffer at any time; this virtual image is, for example, the rendering engine at t. x Candidate virtual images generated in real time. x The time is before time t1, t x There is a certain duration between time t1 and time t2. For example, if a buffer stores three candidate virtual images, the reprojection module responds to the trigger signal at time t1. p A candidate virtual image is randomly read from the buffer at any time.

[0048] 303. Reproject the virtual image according to the refresh mode of the screen of the glasses body, wherein the refresh mode is used to indicate the process of pixels being lit and turned off within one refresh cycle of the screen.

[0049] Upon receiving a trigger signal, the computing unit generates a task based on this signal. This task involves reading a virtual image, predicting its pose, and then reprojecting it based on the predicted pose. The computing unit does not execute the task immediately upon receiving the trigger signal; instead, it executes the task at a future time. This future time will be referred to as the reading time. The moment when the computing unit receives the trigger signal will be referred to as the acquisition time.

[0050] In this embodiment of the application, reprojection refers to distorting the acquired virtual image to produce deformation, so that the deformed virtual image matches the pose of the glasses body at the moment of screen display.

[0051] 304. Display the virtual image after reprojection at the moment of screen loading.

[0052] In this embodiment, the "screen-on" moment is the moment when the reprojected virtual image is displayed on the screen. The duration between two adjacent screen-on moments is one screen refresh cycle. Each screen-on moment is also the moment when a trigger signal is received. The process by which the computing unit reads the virtual image from the buffer, reprojects the virtual image, and displays the reprojected virtual image on the screen is called screen-on.

[0053] Figure 3B yes Figure 3A The corresponding timeline diagram is shown below. Please refer to it. Figure 3Bt1, t2, and t3 are three consecutive screen-on moments, and also the moments when the computing unit of the split-type AR glasses receives the trigger signal three times in a row. After receiving the trigger signal from the optical screen at time t1, the computing unit of the split-type AR glasses... p A suitable virtual image is read from the buffer at any time, and the virtual image is reprojected according to the screen refresh method. Time t1 is the time when the trigger signal is acquired, t p This is the reading time. When the next screen display time is reached, i.e., time t2, the electronic device will complete the reprojection of the virtual image onto the screen.

[0054] In this embodiment, reprojection is a computational process that takes a certain amount of time. The packaging task after the computation unit receives the trigger signal also takes a certain amount of time. Therefore, it is essential to ensure sufficient time for both reprojection and packaging tasks. Thus, the reading time t... p Located between the acquisition time and the display time, the split-type AR glasses can flexibly determine the reading time t. p In one approach, the split-type AR glasses use any time between the acquisition time and the display time as the reading time t. p .

[0055] In another approach, the split-type AR glasses use the midpoint between the acquisition time and the display time as a reference time. A time period is determined based on this reference time, where the difference between any time within this period and the reference time is less than a first preset duration. The split-type AR glasses then use any time within this time period as the reading time t. p The first preset duration is, for example, 2 milliseconds, 1 millisecond, etc.

[0056] In another approach, the split-type AR glasses determine the reading time based on the acquisition time of the trigger signal, and read the virtual image based on the reading time. For example, the duration between the acquisition time and the display time is one refresh cycle. Taking a refresh cycle of 16.6 milliseconds as an example, after determining the acquisition time, the split-type AR glasses use the time that is a second preset duration away from the acquisition time as the reading time. The second preset duration is, for example, 10 milliseconds, 12 milliseconds, etc., and this application embodiment is not limited thereto.

[0057] Similarly, the split-type AR glasses can determine the reading time based on the time it is displayed on the screen. Continuing with the example of a refresh cycle of 16.6 milliseconds, the split-type AR glasses can use the time 5 milliseconds after the time it is displayed on the screen as the reading time.

[0058] Using this approach, the split-type AR glasses determine the reading time based on the acquisition time of the trigger signal, ensuring sufficient time for reprojection and packaging tasks, thereby improving the quality of reprojection.

[0059] In this application embodiment, the screen refresh methods of different split-type AR glasses are different. The refresh method is used to indicate the process of pixels being lit and turned off within one refresh cycle of the screen. For example, the screen refresh method of one type of split-type AR glasses is: the pixels of the screen are refreshed line by line, and the lighting duration of each row of pixels in one refresh cycle is positively correlated with the duty cycle.

[0060] For example, another type of split-type AR glasses uses a screen refresh method of 16.6 milliseconds. All pixels are refreshed within 0-3 milliseconds, and all pixels remain lit within the remaining 3-16.6 milliseconds.

[0061] For example, some split-type AR glasses have a screen refresh method as follows: taking a refresh cycle of 16.6 milliseconds as an example, all pixels are refreshed within 0 to 3 milliseconds, all pixels remain lit within 3 to 6 milliseconds, and all pixels are turned off within 6 to 16.6 milliseconds.

[0062] In this embodiment, the split-type AR glasses reproject the virtual image by combining the screen refresh method. On one hand, it considers time, specifically the pose of the glasses at the midpoint of the time interval when the first row of pixels on the screen is lit, and the pose of the glasses at the midpoint of the time interval when the last row of pixels on the screen is lit. On the other hand, it considers posture, using rendering compensation and position compensation to establish the image projection relationship for reprojection, avoiding the drawback of projection relationships established solely by rotation compensation failing to compensate for positional delays. In other words, this embodiment addresses both time and posture aspects for more accurate reprojection, thereby compensating for data transmission and image rendering delays.

[0063] The virtual image display method provided in this application is applied to a split-type AR glasses system including a glasses body and a computing unit. After receiving a trigger signal to initiate reprojection, the split-type AR glasses reads a virtual image in response to the trigger signal, reprojects the virtual image according to the refresh mode of the glasses body's screen, and displays the reprojected virtual image at the moment of screen display. The refresh mode indicates the process of pixels being lit and turned off within one refresh cycle of the screen. By employing this scheme, combined with the refresh mode of the optical screen for reprojecting the virtual image, the reprojected virtual image matches the pose of the glasses body at the moment of screen display, thereby improving the reprojection quality and ultimately improving the virtual image display quality.

[0064] In this embodiment, to reduce ghosting, the virtual image displayed in each frame is not kept lit throughout the entire display cycle. The lighting duration of each row of pixels in one refresh cycle is related to the duty cycle. The duty cycle is the ratio of the lighting duration of a row of pixels in one refresh cycle to the refresh cycle itself. If the refresh cycle is T and the duty cycle is λ, then the lighting duration of each row of pixels in one refresh cycle is: λ × T.

[0065] Below, using a 50% duty cycle as an example, we will explain in detail the refresh method where "the screen pixels are refreshed line by line, and the illumination duration of each row of pixels in one refresh cycle is positively correlated with the duty cycle." For an example, please refer to... Figure 4 . Figure 4 This is a schematic diagram of the screen refresh process in the virtual screen display method provided in the embodiments of this application.

[0066] Please refer to Figure 4 The screen refresh rate is T, and the duty cycle is 50%. Therefore, the duration each row of pixels is lit for one refresh cycle is 0.5T. At time t0, the last row of pixels in frame i0 is just lit. Each row of pixels lights up sequentially from top to bottom, with each row lit for 0.5T. After 0.25T, at time t0+0.25T, one-quarter of the screen pixels are lit from top to bottom, displaying the virtual image of frame i0+1. Half of the screen pixels are turned off, and the last quarter of the screen pixels are lit, displaying the virtual image of frame i0.

[0067] At time t0+0.5T, from top to bottom, half of the screen pixels are lit up to display the virtual image of frame i0+1, while the remaining half of the screen pixels are turned off.

[0068] At time t0+0.75T, from top to bottom, one-quarter of the screen pixels are turned off, no longer displaying the virtual image of frame i0+1; half of the screen pixels are turned on, displaying the virtual image of frame i0+1; and the remaining one-quarter of the screen pixels are turned off, no longer displaying the virtual image of frame i0.

[0069] At time t0+T, from top to bottom, half of the screen pixels are turned off and no longer display the virtual image of frame i0+1, while the remaining half of the screen pixels remain lit and display the virtual image of frame i0+1.

[0070] Then, the next cycle displays the virtual image of frame i0+2 in the same refresh method.

[0071] Figure 4In this diagram, the white areas represent the areas formed by illuminated pixels, also known as the refresh areas, while the gray areas represent the areas formed by de-illuminated pixels. Since the duty cycle is 50%, approximately 50% of the pixels on the screen are illuminated at any given time. That is, each row of pixels remains illuminated for half of a refresh cycle and remains de-illuminated for the remaining half. It can be understood that the smaller the duty cycle, the smaller the total area of ​​white pixels on the screen; conversely, the larger the duty cycle, the larger the total area of ​​white pixels on the screen.

[0072] The split-type AR glasses reproject the virtual image according to the refresh mode of the screen on the glasses body, including a pose prediction stage and a reprojection stage. In the pose prediction stage, the split-type AR glasses predict an initial pose, a first pose, and a second pose. The initial pose is the pose of the glasses body at the midpoint of the next refresh cycle of the screen. The first pose is the pose of the glasses body at the midpoint of the time period during which the first row of pixels on the screen is lit. The second pose is the pose of the glasses body at the midpoint of the time period during which the last row of pixels on the screen is lit. Taking a refresh cycle T = 16.6 milliseconds and a duty cycle λ = 50% as an example, the lighting duration of one row of pixels in one refresh cycle is: 16.6 × 50% = 8.3 milliseconds. The midpoint refers to the midpoint of the lighting time period. Assuming the start and end times of a cycle are 0 seconds and 16.6 milliseconds respectively, and a row of pixels is lit for a period of time from 0 milliseconds to 8.3 milliseconds, and the row of pixels is off for the rest of the cycle, then the midpoint of the time period during which the row of pixels is lit is 4.15 milliseconds.

[0073] After predicting the pose, the computing unit determines and reads the target virtual image from multiple candidate virtual images in its buffer based on the initial pose. The target virtual image is the virtual image whose generated pose has the smallest error compared to the initial pose among the multiple candidate virtual images, and the generated pose is the pose of the glasses body when the computing unit generates the candidate virtual image.

[0074] For example, the rendering engine of the computing unit reads the pose and velocity of the glasses body at a specific frequency, generates candidate virtual images, and stores them in a buffer. Clearly, each candidate virtual image has a corresponding generated pose. The initial pose is used to determine a reasonable target virtual image from the buffer. For instance, if the buffer stores three candidate virtual images, the computing unit reads the target virtual image from the buffer based on the initial pose. The read target virtual image is the one among the three candidate virtual images that best matches the initial pose. That is, the error between the generated pose and the initial pose of the target virtual image is minimized.

[0075] Using this approach, the target virtual image is the candidate virtual image with the smallest error between the generated pose and the initial pose in the buffer. During reprojection, the deformation of the target virtual image is small, which can improve the reprojection quality.

[0076] After the split-type AR glasses determine the target virtual image based on the initial pose, during the reprojection phase, the split-type AR glasses can reproject the target virtual image based on the first pose (pose1) and the second pose (pose2). The reprojection process is the process of transferring the pixel values ​​in the target virtual image to another blank image. pose1 and pose2 correspond to the poses of the glasses body at the midpoint when the first row of pixels is lit and the midpoint when the last row of pixels is lit, respectively. Therefore, through these two poses, the position (x, y) of each pixel in the target virtual image can be accurately determined to correspond to the position (x', y') of the pixel in the blank image, and the value of the pixel corresponding to position (x, y) can be assigned to the pixel corresponding to position (x', y').

[0077] In this embodiment, the screen refresh method characterizes the time each row of pixels is illuminated and the duration it remains illuminated. Since line-by-line compensation, prediction, and reprojection are impractical in real-world operation, only the first and last rows of pixels are characterized in terms of time. For the middle rows of pixels, interpolation is performed based on the first and last rows.

[0078] This approach considers the screen refresh rate of the split-type AR glasses. Based on the refresh rate, the initial pose, first pose, and second pose are predicted. Based on the initial pose, the most suitable virtual image to be reprojected can be read from the buffer. Based on the first pose and second pose, the corresponding pixel position after reprojection of each position on the virtual image can be accurately determined. Therefore, by combining pose prediction with the screen refresh rate, the delay can be accurately compensated, thereby improving the reprojection quality of the virtual image.

[0079] The two stages will be explained in detail below.

[0080] First, the pose prediction stage.

[0081] In this stage, the split-type AR glasses determine the midpoint of the time period during which each row of pixels on the screen of the glasses body is lit up, based on the duty cycle, the refresh cycle, and the acquisition time of the trigger signal. Then, for each row of pixels, the split-type AR glasses predict the rotation and position of the glasses body at the midpoint of the time period based on the motion parameters of the glasses body at the time of virtual image reading, the reading time, and the midpoint of the time period during which the pixels in that row are lit. The rotation and position of the glasses body at the midpoint of the time period constitute the pose of the glasses body at that midpoint of the time period.

[0082] Figure 5 This is a timeline diagram illustrating the receiving of trigger signals and reading of virtual images in the virtual image display method provided in this application embodiment. Please refer to... Figure 5 , with the computing unit at t k-1 The screen receives the (k-1)th trigger signal at time t, the screen refresh period is T, the duty cycle is λ, the screen's vertical resolution is H, and at time t... p Taking the real-time reading of the virtual image as an example, the split AR glasses determine t according to the following formula (1). p The midpoint of the time interval during which the m-th row of pixels in the virtual image is illuminated is read. The range of m is 1 to H, where H is the maximum number of rows in a pixel, such as 1080. When m = 1, This represents the midpoint of the time period during which the pixels in the first row are lit; when m = H, This represents the midpoint of the time interval during which the pixels in the H-th row, i.e., the last row of pixels, are lit. The following will refer to t... p The virtual image that is constantly read is called the virtual image to be reprojected.

[0083]

[0084] Calculation unit t k-1 After receiving the (k-1)th trigger signal at time t p The virtual image to be reprojected is read from the buffer at any time, and after reprojection, the virtual image is displayed at time t. k The time is displayed on the screen. The calculation unit predicts t according to formula (1). p The midpoint of the time interval during which the m-th row of pixels in the virtual image is illuminated. At the same time, t p The split-type AR glasses obtain motion parameters based on sensor data, including but not limited to the rotation R, position M, angular velocity w, linear velocity v, angular acceleration g, and linear acceleration a of the glasses body.

[0085] For the m-th row of pixels, the split AR glasses determine the midpoint of the time interval during which the m-th row of pixels is illuminated. t p Motion parameters at time t p Time, predicting intermediate time At that time, the rotation R of the glasses body m and position M m That is, the intermediate time The pose of the glasses body. For example, please refer to the following formulas (2) to (4).

[0086]

[0087] R m =R×exp(wΔt+0.5gΔt) 2 ) Formula (3)

[0088] M m =M + vΔt + 0.5aΔt 2 Formula (4)

[0089] Based on formulas (1) to (4) above, the rotation and position of the glasses body can be determined at the midpoint of the time interval during which each row of pixels in the virtual image to be reprojected is illuminated. For example, when m = 1 in formula (1), the rotation and position of the glasses body at the midpoint of the time interval during which the first row of pixels is illuminated can be determined. The midpoint of the time interval during which the first row of pixels is illuminated is as follows: Figure 5 middle As shown. Similarly, when m = H in formula (1), the rotation and position of the glasses body at the midpoint of the time period during which the last row of pixels is lit can be determined. The midpoint of the time period during which the last row of pixels is lit is as follows: Figure 5 middle As shown. H can be, for example, 1080, etc., and the embodiments of this application are not limited thereto.

[0090] In this approach, the first and second poses include rotation and position, enabling the split AR glasses to use rotation and position to create a mapping relationship between the virtual image and the reprojection plane. This allows the virtual image to compensate for position delays, avoiding the drawback that projection relationships established solely through rotation compensation cannot compensate for position delays, thus improving the quality of reprojection.

[0091] In this embodiment, the initial pose refers to the predicted pose of the glasses body at the midpoint of the next refresh cycle of the screen on the glasses body. The split-type AR glasses at t k-1 After receiving the (k-1)th trigger signal at time t p Time based on t k-1 The time to predict the screen display time is t. k-1 +T, i.e., t k At time, the intermediate time corresponding to the initial pose is as follows: Figure 5 t in k +0.5T.

[0092] Please refer to Figure 3B When t k-1 At time t1, the initial pose refers to the pose of the glasses body at the midpoint between time t2 and t3, and the midpoint corresponding to the initial pose is t2 + 0.5T. The midpoint of the time period during which the first row of pixels is illuminated can be determined according to formula (1). And the midpoint of the time period during which the last row of pixels is lit.

[0093] Afterwards, the initial pose of the split AR glasses can be determined according to formulas (2) to (4).

[0094] Secondly, the reprojection stage.

[0095] After predicting the initial pose, the first pose, and the second pose, the split-type AR glasses project the virtual imaging plane corresponding to the initial pose onto the target depth to obtain a depth plane. The upper edge pixels of the depth plane are projected onto the virtual imaging plane corresponding to the first pose to obtain a first line segment, and the lower edge pixels of the depth plane are projected onto the virtual imaging plane corresponding to the second pose to obtain a second line segment. Then, a reprojection quadrilateral region is determined based on the first and second line segments, and the virtual image is reprojected based on the reprojection quadrilateral region. The virtual imaging plane corresponding to the initial pose refers to the imaging plane corresponding to the virtual camera in the computing unit when the pose of the glasses body is the initial pose. The virtual imaging plane corresponding to the first pose refers to the imaging plane corresponding to the virtual camera in the computing unit when the pose of the glasses body is the first pose. The virtual imaging plane corresponding to the second pose refers to the imaging plane corresponding to the virtual camera in the computing unit when the pose of the glasses body is the second pose.

[0096] Figure 6 This is a schematic diagram illustrating the determination of the reprojected quadrilateral region in the virtual screen display method provided in this application embodiment. Please refer to... Figure 6 To compensate for both rotation and position simultaneously, the virtual imaging plane corresponding to the initial pose needs to be projected onto the target depth to obtain a depth plane. The rendering engine generates virtual images, each with a virtual camera coordinate system and a virtual imaging plane. When the pose of the glasses body differs, the virtual camera coordinate system and virtual imaging plane of the virtual images also differ. Figure 6 The initial pose (pose0), the first pose (pose1), and the second pose (pose2) correspond to different virtual imaging planes.

[0097] The virtual imaging plane is perpendicular to the z-axis. The distance between the virtual imaging plane and the origin of the camera coordinate system is called the focal length, which is measured in pixels and can also be converted to distance. In this embodiment, a target depth is calculated in meters. The virtual imaging plane is enlarged based on the initial pose (pose0), i.e., projected onto the target depth to obtain the depth plane. Clearly, the depth plane refers to the plane obtained by enlarging the virtual plane based on the target depth from the initial pose (pose0). The distance between the virtual imaging plane of the initial pose (pose0) and the depth plane is called the target depth. Figure 6 The diagram illustrates the projection relationship of the virtual camera in the initial pose (pose0), the first pose (pose1), and the second pose (pose2). The gray-filled quadrilateral represents the depth plane, and the three unfilled quadrilaterals to the left of the arrow represent the virtual imaging planes corresponding to the initial pose (pose0), the first pose (pose1), and the second pose (pose2), respectively. After obtaining the depth plane, the split AR glasses project the upper edge pixel 61 of the depth plane onto the virtual imaging plane corresponding to the first pose (pose1), resulting in the first line segment 63, and project the lower edge pixel 62 of the depth plane onto the virtual imaging plane corresponding to the second pose (pose2), resulting in the second line segment 64. The upper edge pixel 61 refers to the first row of pixels in the depth plane, and the lower edge pixel 62 refers to the last row of pixels in the depth plane.

[0098] As shown to the right of the arrow, after obtaining the first line segment 63 and the second line segment 64, the split-type AR glasses project the first line segment 63 and the second line segment 64 onto the reprojection plane to obtain a reprojection quadrilateral region 65. This reprojection quadrilateral region 65 is used to indicate the projection range of the virtual image on the reprojection plane. The reprojection plane can be the virtual imaging plane corresponding to the first pose 1 or the virtual imaging plane corresponding to the second pose 2. When the reprojection plane is the virtual imaging plane corresponding to the first pose 1, the second line segment 64 is reprojected onto the plane containing the first line segment 63 to obtain the reprojection quadrilateral region 65; when the reprojection plane is the virtual imaging plane corresponding to the second pose 2, the first line segment 63 is reprojected onto the plane containing the second line segment 64 to obtain the reprojection quadrilateral region 65. Figure 6 The example below uses the reprojection plane as the virtual imaging plane corresponding to the second pose pose2.

[0099] After obtaining the reprojection quadrilateral region 65, the split AR glasses assign the value (such as color) of each pixel in the virtual image to the corresponding pixel in the reprojection quadrilateral region 65, thereby completing the reprojection of the virtual image.

[0100] This approach involves projecting the virtual imaging plane of the initial pose onto the target depth to determine the position of the reprojected virtual image, i.e., the reprojected quadrilateral region. This achieves compensation for position and rotation, thereby improving the quality of the reprojection.

[0101] Optionally, in the above embodiments, during the process of reprojecting the virtual image onto the reprojection quadrilateral region using the split AR glasses, a projection matrix is ​​determined based on the reprojection quadrilateral region, and a mapping relationship between the virtual image and the reprojection plane is established based on the projection matrix. Then, the split AR glasses determine the position of each pixel in the virtual image within the reprojection quadrilateral region based on the mapping relationship, and assign the value of each pixel in the virtual image to the corresponding pixel in the reprojection quadrilateral region, thereby reprojecting the virtual image onto the reprojection quadrilateral region.

[0102] For example, the target depth is d h The intrinsic parameter matrix of the virtual camera is given by formula (5):

[0103]

[0104] Where fx and fy are focal lengths, cx and cy are principal point coordinates, and s is the coordinate axis tilt parameter, which is 0 in an ideal case.

[0105] In the pose prediction stage described above, the split-type AR glasses can predict the first pose (pose1) and the second pose (pose2). The split-type AR glasses transform the first pose (pose1) into a pose coordinate system to obtain the first relative pose, including a first rotation R. s and the first position M s Similarly, the split-type AR glasses transform the second pose pose2 into a pose coordinate system to obtain the second relative pose, including the second rotation R. e Second position M e Assume the homogeneous coordinates p of the first row of pixels in the virtual image. t =(u t ,v t ,1) T Then the homogeneous coordinates p t The projection onto the imaging plane of the virtual camera in the first relative pose is shown in the following formula (6):

[0106]

[0107] Where, p s Let R be the homogeneous coordinates on the imaging plane of the virtual camera in the first relative pose, and let R be the first rotation. s It is a 3×3 matrix, with the first position M. s It is a 3-dimensional vector, u t,v t These are the horizontal and vertical coordinates, respectively.

[0108] Similarly, by projecting the homogeneous coordinates of the last row of pixels in the virtual image onto the imaging plane of the virtual camera in the second relative pose, we can obtain the projection of the last row of pixels onto the imaging plane of the virtual camera in the second relative pose.

[0109] After establishing the above projection relationship, the split AR glasses obtain the target depth d. h Below, the virtual image is projected onto the complete reprojection quadrilateral region of the reprojection plane. Based on the four vertices of the reprojection quadrilateral region, a projection matrix is ​​calculated; this projection matrix is ​​a 3×3 homography matrix. Then, the split-type AR glasses establish a mapping relationship between the virtual image and the reprojection plane based on the projection matrix. This mapping relationship is shown in formula (7):

[0110] p tex =Hp prj Formula (7)

[0111] Where, p tex p prj These are the homogeneous pixel coordinates on the virtual screen and the homogeneous pixel coordinates on the reprojection plane, respectively. The split-type AR glasses use this mapping relationship to reproject the virtual screen.

[0112] Using this approach, the split-type AR glasses determine the projection matrix based on the reprojection quadrilateral area, thereby establishing a mapping relationship between the virtual image and the reprojection plane, achieving the goal of quickly and accurately reprojecting the virtual image.

[0113] The following section provides a detailed explanation of how to determine the target depth.

[0114] Figure 7 This is a schematic diagram illustrating the determination of target depth in the virtual image display method provided in this application embodiment. Please refer to... Figure 7 To determine the target depth, the split AR glasses need to be worn on a human head model. The human head model includes a first industrial camera and a second industrial camera to simulate human eyes. The first industrial camera and the second industrial camera are used to take pictures of the first screen and the second screen of the split AR glasses, respectively. A physical calibration plate is placed about 3 centimeters in front of the split AR glasses.

[0115] The split-type AR glasses acquire a first image and a second image. The first image is obtained by a first industrial camera capturing a calibration board through the first screen, and the second image is obtained by a second industrial camera capturing the calibration board through the second screen. During this process, both screens of the split-type AR glasses are initially kept off. Two industrial cameras capture the first and second images of the calibration board through the AR glasses' screens, and the pose of the calibration board is determined based on the first and second images.

[0116] Subsequently, the split-type AR glasses determine multiple errors based on the pose of the calibration board, and then determine the target depth based on these multiple errors. Each of the multiple errors corresponds one-to-one with an angle among multiple angles, and these errors indicate the error in the vertex of the calibration board reconstructed from the pose before and after the first and second industrial cameras rotate by the corresponding angles.

[0117] For example, the human head model is rotated. Given the rotation angle, the vertices of the calibration board are projected onto a first screen based on the board's pose, and the positions of the projected points are detected using a first industrial camera. Similarly, the vertices of the calibration board are projected onto a second screen based on the board's pose, and the positions of the projected points are detected using a second industrial camera. The projection points captured by the first and second industrial cameras are combined to perform binocular reconstruction of the virtual calibration board, obtaining the reconstructed vertices of the calibration board. Then, the screen is turned off, and the actual calibration board is photographed again. The positions of each vertex are recorded, and the error between the reconstructed calibration board and the corresponding vertices of the physical calibration board is calculated.

[0118] The human head model is rotated multiple times, and the error is calculated after each rotation, resulting in multiple errors, i.e., several sets of data at different rotation angles. The split-type AR glasses use a least-squares method based on these multiple errors to calculate the target depth, minimizing the error.

[0119] This approach uses a simulated user wearing split-type AR glasses to determine the target depth, which allows for accurate projection of the virtual imaging plane of the initial pose onto the target depth. This simultaneously compensates for position and rotation, thereby improving the display quality of the virtual image.

[0120] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0121] Figure 8 This is a schematic diagram of a virtual screen display device provided in an embodiment of this application. The virtual screen display device 800 includes: an acquisition module 81, a response module 82, a processing module 83, and a display module 84.

[0122] The acquisition module 81 is used to acquire the trigger signal used to trigger reprojection;

[0123] Response module 82 is used to read the virtual screen in response to the trigger signal;

[0124] The processing module 83 is used to reproject the virtual image according to the refresh mode of the screen of the glasses body, wherein the refresh mode is used to indicate the process of pixels being lit and turned off within one refresh cycle of the screen;

[0125] Display module 84 is used to display the virtual image after reprojection when the screen is on.

[0126] In one feasible implementation, the refresh method is as follows: the pixels of the screen are refreshed line by line, and the illumination duration of each row of pixels in one refresh cycle is positively correlated with the duty cycle. The processing module 83 is used to predict an initial pose, a first pose, and a second pose. The initial pose is the pose of the glasses body at the midpoint of the next refresh cycle of the screen of the glasses body. The first pose is the pose of the glasses body at the midpoint of the time period when the first row of pixels of the screen of the glasses body is illuminated. The second pose is the pose of the glasses body at the midpoint of the time period when the last row of pixels of the screen of the glasses body is illuminated. The virtual image is reprojected based on the initial pose, the first pose, and the second pose.

[0127] In one feasible implementation, when the processing module 83 reprojects the virtual image based on the initial pose, the first pose, and the second pose, it projects the virtual imaging plane corresponding to the initial pose onto a preset target depth to obtain a depth plane; projects the upper edge pixels of the depth plane onto the virtual imaging plane corresponding to the first pose to obtain a first line segment; projects the lower edge pixels of the depth plane onto the virtual imaging plane corresponding to the second pose to obtain a second line segment; determines a reprojection quadrilateral region based on the first line segment and the second line segment; and reprojects the virtual image based on the reprojection quadrilateral region. The virtual imaging plane corresponding to the initial pose refers to the imaging plane corresponding to the virtual camera in the computing unit when the pose of the glasses body is the initial pose; the virtual imaging plane corresponding to the first pose refers to the imaging plane corresponding to the virtual camera in the computing unit when the pose of the glasses body is the first pose; and the virtual imaging plane corresponding to the second pose refers to the imaging plane corresponding to the virtual camera in the computing unit when the pose of the glasses body is the second pose.

[0128] In one feasible implementation, when the processing module 83 reprojects the virtual image according to the reprojection quadrilateral region, it is used to determine a projection matrix according to the reprojection quadrilateral region; establish a mapping relationship between the virtual image and the reprojection plane according to the projection matrix, wherein the reprojection plane is the virtual imaging plane corresponding to the first pose; or, the reprojection plane is the virtual imaging plane corresponding to the second pose; determine the position of each pixel in the virtual image in the reprojection quadrilateral region according to the mapping relationship, and assign the value of each pixel in the virtual image to the pixel at the corresponding position in the reprojection quadrilateral region, so as to reproject the virtual image onto the reprojection quadrilateral region.

[0129] In one feasible implementation, the response module 82 is configured to respond to the trigger signal, determine a target virtual image from a plurality of candidate virtual images located in the buffer of the computing unit according to the initial pose, wherein the target virtual image is the virtual image with the smallest error between the generated pose and the initial pose among the plurality of candidate virtual images, and the generated pose is the pose of the glasses body when the computing unit generates the candidate virtual image, and read the target virtual image.

[0130] In one feasible implementation, when the processing module 83 predicts the first pose, it is used to determine the midpoint of the time period during which the first row of pixels on the screen of the glasses body is lit up, based on the duty cycle, the refresh cycle, and the acquisition time of the trigger signal; and to predict the rotation and position of the glasses body at the midpoint of the time period during which the first row of pixels is lit up, based on the midpoint of the time period during which the first row of pixels is lit up, the motion parameters of the glasses body at the time of reading the virtual image, and the reading time, wherein the first pose includes the rotation and the position.

[0131] In one feasible implementation, the response module 82 is used to determine the reading time based on the acquisition time of the trigger signal, wherein the reading time is located between the acquisition time and the screen display time; and to read the virtual screen based on the reading time.

[0132] The virtual screen display device provided in this application embodiment can perform the actions of the split AR glasses in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0133] Figure 9 This is a schematic diagram of the structure of the split-type AR glasses provided in the embodiments of this application. Please refer to... Figure 9The split-type AR glasses include a glasses body 91, a computing unit 92, and a connecting cable 93. The connecting cable 93 connects the glasses body 91 and the computing unit 92. The glasses body 91 includes a sensor 911 and a display 912. The sensor 911 is used to acquire the motion parameters of the glasses body, including but not limited to rotation R, position M, angular velocity w, linear velocity v, angular acceleration g, and linear acceleration a. The display 912 consists of two lenses of the split-type AR glasses and is used to display the reprojected virtual image.

[0134] The computing unit 92 includes a processor 921 and a memory 922. The memory 922 stores a computer program that can run on the processor 921. When the processor executes the computer program, it causes the split AR device to reproject the virtual image and display the reprojected virtual image on the display 912.

[0135] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the virtual screen display method described above.

[0136] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the virtual screen display method described above.

[0137] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0139] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0140] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0141] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0142] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0143] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0144] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0145] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for displaying a virtual image, characterized in that, The method, applied to split-type AR glasses, which includes a glasses body and a computing unit, includes: Obtain the trigger signal used to initiate reprojection; In response to the trigger signal, read the virtual screen; The virtual image is reprojected according to the refresh mode of the screen of the glasses body, wherein the refresh mode is used to indicate the process of pixels being lit and turned off within one refresh cycle of the screen; The virtual image after reprojection is displayed on the screen.

2. The method according to claim 1, characterized in that, The refresh method includes: the screen pixels are refreshed line by line, and the illumination duration of each row of pixels in a refresh cycle is positively correlated with the duty cycle; the reprojection of the virtual image according to the refresh method of the screen of the glasses body includes: Predict the initial pose, which is the pose of the glasses body at the midpoint of the next refresh cycle of the screen of the glasses body; Predict the first pose, which is the pose of the glasses body when the first row of pixels on the screen of the glasses body is lit at the middle of the time period. Predict the second pose, which is the pose of the glasses body when the last row of pixels on the screen of the glasses body is lit up at the middle of the time period. The virtual image is reprojected based on the initial pose, the first pose, and the second pose.

3. The method according to claim 2, characterized in that, The step of reprojecting the virtual image based on the initial pose, the first pose, and the second pose includes: The virtual imaging plane corresponding to the initial pose is projected onto the preset target depth to obtain the depth plane. The virtual imaging plane corresponding to the initial pose refers to the imaging plane corresponding to the virtual camera in the computing unit when the pose of the glasses body is the initial pose. The upper edge pixels of the depth plane are projected onto the virtual imaging plane corresponding to the first pose to obtain a first line segment, and the lower edge pixels of the depth plane are projected onto the virtual imaging plane corresponding to the second pose to obtain a second line segment. The virtual imaging plane corresponding to the first pose refers to the imaging plane corresponding to the virtual camera in the computing unit when the pose of the glasses body is the first pose; the virtual imaging plane corresponding to the second pose refers to the imaging plane corresponding to the virtual camera in the computing unit when the pose of the glasses body is the second pose. The reprojected quadrilateral region is determined based on the first line segment and the second line segment; The virtual image is reprojected based on the reprojection quadrilateral region.

4. The method according to claim 3, characterized in that, The reprojection of the virtual image based on the reprojection quadrilateral region includes: The projection matrix is ​​determined based on the reprojected quadrilateral region; A mapping relationship between the virtual image and the reprojection plane is established based on the projection matrix, wherein the reprojection plane is the virtual imaging plane corresponding to the first pose; or, the reprojection plane is the virtual imaging plane corresponding to the second pose. The position of each pixel in the virtual image in the reprojection quadrilateral region is determined according to the mapping relationship, and the value of each pixel in the virtual image is assigned to the corresponding pixel in the reprojection quadrilateral region so as to reproject the virtual image onto the reprojection quadrilateral region.

5. The method according to any one of claims 2-4, characterized in that, The step of reading the virtual screen in response to the trigger signal includes: In response to the trigger signal, a target virtual image is determined from a plurality of candidate virtual images in the buffer of the computing unit according to the initial pose. The target virtual image is the virtual image with the smallest error between the generated pose and the initial pose among the plurality of candidate virtual images. The generated pose is the pose of the glasses body when the computing unit generates the candidate virtual image. Read the target virtual image.

6. The method according to any one of claims 2 to 4, characterized in that, The predicted first pose includes: Based on the duty cycle, the refresh cycle, and the acquisition time of the trigger signal, determine the midpoint of the time period during which the first row of pixels on the screen of the glasses body is lit. Based on the midpoint of the time period during which the first row of pixels is lit, the motion parameters of the glasses body at the time of reading the virtual image, and the reading time, the rotation and position of the glasses body at the midpoint of the time period during which the first row of pixels is lit are predicted, and the first pose includes the rotation and the position.

7. The method according to any one of claims 1 to 4, characterized in that, The step of reading the virtual screen in response to the trigger signal includes: The reading time is determined based on the acquisition time of the trigger signal, and the reading time is located between the acquisition time and the screen display time; The virtual image is read according to the specified reading time.

8. A virtual screen display device, characterized in that, The device is integrated into a split-type AR glasses, which includes a glasses body and a computing unit. The device includes: The acquisition module is used to acquire the trigger signal used to trigger reprojection; The response module is used to read the virtual screen in response to the trigger signal; The processing module is used to reproject the virtual image according to the refresh mode of the screen of the glasses body, wherein the refresh mode is used to indicate the process of pixels being lit and turned off within one refresh cycle of the screen; The display module is used to display the reprojected virtual image when the screen is on.

9. A split-type AR device, comprising glasses body, computing unit, and connecting cable, wherein the connecting cable is used to connect the computing unit and the glasses body, characterized in that, The computing unit includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it causes the split-type AR device to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images