A synchronization display method of an AR split glasses and the AR split glasses

Abstract

The application discloses a synchronous display method of AR split glasses and the AR split glasses. The AR split glasses comprise glasses, a control box and a data line. The data line is used for connecting the glasses and the control box. The method is applied to the control box. The method comprises the following steps: detecting that the glasses are connected with the control box through the data line in the case that the glasses are disconnected with the control box; receiving a plurality of synchronization signals periodically sent from the glasses; calculating a first time difference between a first time at which the glasses send a first synchronization signal and a second time at which the control box receives the first synchronization signal; the synchronization signal is any one of the plurality of synchronization signals; the first time difference is a time difference corresponding to the first synchronization signal; applying a set algorithm to optimize the first time difference, and adjusting the first time by using the optimized first time difference to obtain a target time; and sending an image frame corresponding to the target time to the glasses, so that the glasses display the image frame. The picture tearing problem is avoided, and the picture synchronous display is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of AR device, and particularly relates to a synchronization display method of AR split glasses and AR split glasses. BACKGROUND

[0002] AR glasses can be divided into split AR glasses (referred to as AR split glasses) and integrated AR glasses. The integrated AR glasses have a built-in processor, storage space and battery, and have a certain computing power, which can process part of AR applications and scenes independently, and can run independently without connecting any device. The glasses of the split AR glasses usually only have a display function, and the glasses are connected with a control box through a data line. The control box is a terminal with data processing function.

[0003] By pulling out the data line, the connection between the glasses and the control box can be disconnected. During the plugging and unplugging process, the time stamps of the glasses and the control box are inconsistent, so that the picture display refresh rate of the glasses is not synchronized with the rendering cache picture refresh rate of the control box, which may cause the phenomenon of picture tearing. SUMMARY

[0004] In the exemplary embodiments of the present application, a synchronization display method of AR split glasses and AR split glasses are provided to avoid the problem of picture tearing and realize synchronized display of pictures.

[0005] According to a first aspect in the exemplary embodiments, a synchronization display method of AR split glasses is provided, the AR split glasses comprising glasses, a control box and a data line; the data line is used to connect the glasses and the control box; the method is applied to the control box. The method comprises:

[0006] In the case that the glasses and the control box are disconnected, it is detected that the glasses are connected with the control box through the data line;

[0007] A plurality of synchronization signals periodically sent from the glasses are received;

[0008] A first time difference between a first time at which the glasses send a first synchronization signal and a second time at which the control box receives the first synchronization signal is calculated; wherein the first synchronization signal is any one of the plurality of synchronization signals; each synchronization signal corresponds to a time difference; the first time difference is the time difference corresponding to the first synchronization signal;

[0009] A set algorithm is applied to optimize the first time difference, and the first time is adjusted by using the optimized first time difference to obtain a target time;

[0010] An image frame corresponding to the target time is sent to the glasses, so that the glasses display the image frame.

[0011] According to a second aspect in the exemplary embodiments, there is provided a synchronization display device for an AR split glasses, the AR split glasses comprising a glasses, a control box and a data line, the data line being used to connect the glasses and the control box. The device is applied to the control box, and the device comprises:

[0012] a detection unit configured to detect that the glasses are connected to the control box through the data line in a case that the glasses are disconnected from the control box;

[0013] a transmission unit configured to receive a plurality of synchronization signals periodically transmitted from the glasses;

[0014] a processing unit configured to calculate a first time difference between a first time at which the glasses transmit a first synchronization signal and a second time at which the control box receives the first synchronization signal, wherein the first synchronization signal is any one of the plurality of synchronization signals, each synchronization signal corresponds to a time difference, and the first time difference is the time difference corresponding to the first synchronization signal;

[0015] the processing unit is further configured to optimize the first time difference by using a set algorithm, and adjust the first time by using the optimized first time difference to obtain a target time;

[0016] the transmission unit is further configured to transmit an image frame corresponding to the target time to the glasses, so that the glasses display the image frame.

[0017] According to a third aspect in the exemplary embodiments, there is provided an AR split glasses, the AR split glasses comprising a glasses, a control box and a data line, the data line being used to connect the glasses and the control box. The method is applied to the control box, and the control box comprises a processor and a memory.

[0018] the processor is configured to perform:

[0019] detect that the glasses are connected to the control box through the data line in a case that the glasses are disconnected from the control box;

[0020] receive a plurality of synchronization signals periodically transmitted from the glasses;

[0021] calculate a first time difference between a first time at which the glasses transmit a first synchronization signal and a second time at which the control box receives the first synchronization signal, wherein the first synchronization signal is any one of the plurality of synchronization signals, each synchronization signal corresponds to a time difference, and the first time difference is the time difference corresponding to the first synchronization signal;

[0022] optimize the first time difference by using a set algorithm, and adjust the first time by using the optimized first time difference to obtain a target time;

[0023] transmit an image frame corresponding to the target time to the glasses, so that the glasses display the image frame.

[0024] The memory is configured to perform:

[0025] store a plurality of image frames.

[0026] According to a fourth aspect in the exemplary embodiments, a computer storage medium is provided, and the computer storage medium stores computer program instructions, when the instructions are run on a computer, the computer executes the synchronization display method of the AR split glasses according to the first aspect.

[0027] In the embodiments of the present application, when the glasses are disconnected from the control box, the control box detects that the glasses are connected to the control box through the data line, and receives a plurality of synchronization signals periodically sent from the glasses. The first time difference between the first time when the glasses send the first synchronization signal (any one of the plurality of synchronization signals) and the second time when the control box receives the first synchronization signal is calculated, and the first time difference is the time difference corresponding to the first synchronization signal. The first time difference is optimized by applying a set algorithm, and the first time is adjusted by applying the optimized first time difference to obtain a target time. The image frame corresponding to the target time is sent to the glasses to make the glasses display the image frame. Such design can ensure the time synchronization of the glasses and the control box, thereby avoiding the tearing problem of the picture caused by the insertion and removal of the glasses, and realizing the synchronous display of the picture. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Figure 1 An exemplary structure diagram of AR split glasses provided by the embodiments of the present application is shown;

[0030] Figure 2 An exemplary flowchart of a synchronization display method of AR split glasses provided by the embodiments of the present application is shown;

[0031] Figure 3 An exemplary flowchart of a method for calculating the first time difference provided by the embodiments of the present application is shown;

[0032] Figure 4 An exemplary diagram of the time delay of each stage in the process of receiving the first synchronization signal by the control box provided by the embodiments of the present application is shown;

[0033] Figure 5 An exemplary diagram of the time delay of each stage in the process of receiving the first synchronization signal by the control box provided by the embodiments of the present application is shown;

[0034] Figure 6 An example shows a method flow chart of the optimization process of the first time difference provided by the embodiment of the application.

[0035] Figure 7 An example shows a schematic diagram of sending N image frames to the glasses display provided by the embodiment of the application.

[0036] Figure 8 An example shows a signaling diagram of the synchronization display method of the AR split glasses provided by the embodiment of the application.

[0037] Figure 9 An example shows a structural schematic diagram of the synchronization display device of the AR split glasses provided by the embodiment of the application.

[0038] Figure 10 An example shows a structural block diagram of the AR split glasses provided by the embodiment of the application. DETAILED DESCRIPTION

[0039] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme of the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application.

[0040] In the embodiments of the application, the control box of the AR split glasses is a terminal with data processing function, and in addition, operation buttons are arranged on the control box. Generally, the control box is an Android system. To facilitate understanding, the terms involved in the embodiments of the application are explained as follows.

[0041] (1) Picture tearing phenomenon refers to the situation that in a game or a video, the screen appears to be unable to switch between two frames neatly, that is, the two frames appear out of synchronization. Generally, this situation will cause the game or video picture to appear shaking or flickering.

[0042] (2) The Android system architecture is divided into an application layer, a Java framework layer, a native C / C++ library layer and a Linux kernel layer. In the Android system, the native C / C++ library layer is also called the Native layer. In the Native layer, developers can use C / C++ language to write code to access the underlying hardware, implement high-performance computing and other operations. At the same time, the Native layer is also responsible for converting the requests delivered by the Java framework layer into instructions that can be recognized by the underlying operating system.

[0043] (3) Android application package (APK) is a kind of application package file format used by Android operating system, which is used for distributing and installing mobile applications and middleware. The code of an Android application wants to run on an Android device, it is first compiled, and then packaged into a file that can be recognized by the Android system before it can be run, and this file format that can be recognized and run by the Android system is APK.

[0044] (4) Software Development Kit (SDK) is generally a collection of development tools established by software engineers for specific software packages, software frameworks, hardware platforms, operating systems, etc. when developing application software.

[0045] (5) Human Interface Device (HID) refers to any input or output device used to interact with computers or other electronic devices. Common HID devices include mice, keyboards, game controllers, digital pens, scanners, etc. In the embodiments of the present application, HID can refer to the glasses in AR glasses.

[0046] (6) HIDRAW refers to the driver in the Linux system for handling HID devices. Through HIDRAW, users can read and write operations with HID devices to achieve data input and output.

[0047] (7) Vertical Synchronization (VSync) refers to a technique used in computer graphics processing to synchronize the screen refresh rate and graphics rendering. Through VSync, the frame rate of graphics rendering is synchronized with the refresh rate of the display to eliminate screen tearing and provide smoother visuals.

[0048] (8) Data line is a necessary component for connecting hard disk and motherboard, and its interface classification includes Cluster Communication (COM) interface, Universal Serial Bus (USB) interface, Type-C interface and Micro-USB interface, etc. In the embodiments of the present application, the data line can be a Type-C interface data line.

[0049] AR split glasses as a display screen, the processor is located in the control box, so the screen and the processor processing is in a state of separation. In the process of normal use of AR glasses, the glasses are connected with the control box through the data line. When the user needs to rest, the connection between the two can be disconnected. The way to disconnect can be to disconnect the end of the data line connected with the glasses and / or the end of the data line connected with the control box, while the control box may still be in the on state. When the AR split glasses are plugged in and out of the control box through the data line, the time stamp transmitted between the glasses and the control box is inconsistent. After detecting that the glasses are connected with the control box through the data line, the timing of the control box is from the time when the control box is turned on, while the timing of the glasses is from the time when the connection between the data line and the control box is established again, which leads to the difference between the display refresh frequency and the rendering refresh frequency. For example, the glasses are timers starting from 0, and the control box is from the time when the control box is turned on. For example, the control box is turned on at 8 o'clock and connected with the glasses through the data line, but after 50 minutes of use, the user wants to pause and rest, and can disconnect the data line, at this time the control box is not turned off. When the user plugs in the data line again at 9 o'clock, the use continues. At this time, the time stamp of the control box has been calculated according to 8 o'clock all the time, while the time stamp of the glasses is calculated according to 9 o'clock after the data line is plugged in again.

[0050] Due to the inconsistency of the time stamp, the picture display refresh rate of the glasses is different from the picture rendering buffer refresh rate of the control box, which further causes the problem of picture asynchronization. For example, if the picture display refresh rate is lower than the picture rendering buffer refresh rate, tearing of the picture will occur; if the picture display refresh rate is higher than the picture rendering buffer refresh rate, the picture will be stuck. In actual application, repeated plugging and unplugging usually leads to picture tearing.

[0051] Therefore, the embodiment of the present application provides a synchronization display method of AR split glasses, in which, in the case where the glasses are disconnected with the control box, the time difference (time delay) between the time when the glasses send a synchronization signal and the time when the control box receives the synchronization signal is considered, and a set algorithm is applied to optimize the time difference, and then the optimized time difference is applied to optimize the time when the glasses send the synchronization signal, to obtain the accurate time when the control box receives the synchronization signal. The image frame matched with the control time is sent to the glasses for display, to overcome the problem of picture tearing.

[0052] After introducing the design idea of the embodiment of the present application, the application scenarios to which the technical scheme of the embodiment of the present application can be applied will be briefly introduced, and it should be noted that the following introduction of the application scenarios is only used to illustrate the embodiment of the present application but not to limit. In specific implementation, the technical scheme provided by the embodiment of the present application can be flexibly applied according to actual needs.

[0053] ReferenceFigure 1 Fig. 11 shows a structural schematic diagram of an AR split glasses, wherein 11 is glasses, 12 is a control box, and 13 is a data line.

[0054] To further illustrate the technical solutions provided by the embodiments of the present application, the following will describe in detail in conjunction with the accompanying drawings and specific embodiments. Although the embodiments of the present application provide the following method operation steps as shown in the embodiments or the accompanying drawings, more or less operation steps can be included in the method based on conventional or non-creative labor. The execution order of the steps is not limited to the execution order provided by the embodiments of the present application in the logical sense.

[0055] The following will be described in conjunction with the application scenario shown in Fig. 1, and the flow chart of the synchronization display method of the AR split glasses shown in Fig. 2, which is applied to the control box. In conjunction with Fig. 3, the technical solutions provided by the embodiments of the present application will be described. Figure 1 Figure 2 The following will be described in conjunction with the application scenario shown in Fig. 1, and the flow chart of the synchronization display method of the AR split glasses shown in Fig. 2, which is applied to the control box. In conjunction with Fig. 3, the technical solutions provided by the embodiments of the present application will be described. Figure 2

[0056] S201: In the case of disconnecting the glasses from the control box, it is detected that the glasses are connected to the control box through the data line.

[0057] S202: Receive a plurality of synchronization signals periodically sent from the glasses.

[0058] S203: Calculate the first time difference between the first time when the glasses send the first synchronization signal (any one of the plurality of synchronization signals) and the second time when the control box receives the first synchronization signal.

[0059] S204: Optimize the first time difference by applying a set algorithm, and adjust the first time by applying the optimized first time difference to obtain the target time.

[0060] S205: Send the image frame corresponding to the target time to the glasses to make the glasses display the image frame.

[0061] In the embodiments of the present application, in the case of disconnecting the glasses from the control box, the control box detects that the glasses are connected to the control box through the data line, and receives a plurality of synchronization signals periodically sent from the glasses. The first time difference between the first time when the glasses send the first synchronization signal (any one of the plurality of synchronization signals) and the second time when the control box receives the first synchronization signal is calculated, which is the time difference corresponding to the first synchronization signal. The first time difference is optimized by applying a set algorithm, and the first time is adjusted by applying the optimized first time difference to obtain the target time. The image frame corresponding to the target time is sent to the glasses to make the glasses display the image frame. Such design can ensure the time synchronization of the glasses and the control box, and further avoid the tearing problem of the picture caused by the insertion and removal of the glasses, and realize the synchronous display of the picture.

[0062] ​​Involving S201, in the case of glasses and control box disconnected, it is detected that the glasses are connected with the control box through the data line, at this time, the glasses periodically send multiple synchronization signals to the control box. The synchronization signal is a Vsync signal.

[0063] Involving S202, after the glasses are connected with the control box through the data line, the glasses periodically send multiple synchronization signals to the control box periodically. The control box receives multiple synchronization signals from the glasses. Among them, the number of synchronization signals is represented by k, which can take 0, 1, 2, 3, 4.

[0064] Involving S203, for multiple synchronization signals, multiple synchronization signals can be calculated respectively corresponding to the time difference, which can represent a time delay of the control box receiving the synchronization signal and the glasses sending the synchronization signal. Next, take any one synchronization signal as an example to explain the calculation process of the time difference. Among them, any one synchronization signal can be represented by the first synchronization signal, and the corresponding time difference is represented by the first time difference; or, any one synchronization signal is represented by synchronization signal k, and the corresponding time difference is represented by △T k .

[0065] For example, the first time difference is the time difference between the first time when the glasses send the first synchronization signal and the second time when the control box receives the second synchronization signal.

[0066] Among them, the two ends of the data line are respectively called the first end and the second end, and the end of the data line connected with the glasses is the first end. The time when the first synchronization signal is received by the serial port signal receiver at the first end of the data line can be recorded as the first time. For example, the time when the first synchronization signal is received by the rendering tool kit is recorded as the second time.

[0067] For example, k=1, then Teye1=Teye0+t; k=2, then Teye2=Teye0+2t; k=3, then Teye3=Teye0+3t; k=4, then Teye4=Teye0+4t. Among them, Teye0 may be 1000ms, that is, the first synchronization signal is sent at 1000ms when the timer starts timing. The transmission period of the synchronization signal is recorded as t, which may be 6ms, for example.

[0068] Based on the way of calculating the first time difference, the time difference corresponding to each of the multiple synchronization signals can be obtained. As above, the five time differences obtained are △T0, △T1, △T2, △T3, △T4, which can take values of 0.1ms, 0.12ms, 0.11ms, 0.15ms and 0.14ms, respectively.

[0069] In the above embodiment, the second time at which the control box receives the first synchronization signal is subtracted from the first time at which the glasses send the first synchronization signal to obtain the first time difference. In addition, the first time difference can also be calculated by the steps in Figure 3

[0070] S301: Record the time Teye at which the first synchronization signal is received by the serial port signal receiver through the first end of the data line k .

[0071] S302: Record the time Tbox at which the first synchronization signal is received by the control box driving interface HIDRAW k .

[0072] S303: Calculate the first time delay △box k =Tbox k -Teye k .

[0073] The first time delay represents the control box driving time delay.

[0074] S304: Record the time Tapk at which the first synchronization signal is received by the framework layer data receiving service or application of the Android system k .

[0075] The file format of the framework layer data receiving service or application of the Android system is APK.

[0076] S305: Calculate the second time delay △apk k =Tapk k -Tbox k .

[0077] The second time delay represents the APK service or application receiving time delay.

[0078] S306: Calculate the time Tsdk at which the first synchronization signal is received through the SDK k .

[0079] S307: Calculate the third time delay △sdk k =Tsdk k - Tapk k .

[0080] The third time delay represents the SDK receiving time delay.

[0081] Figure 4 A schematic diagram of the time delays in the process in which the control box receives the first synchronization signal is provided in the embodiment of the application.

[0082] Therefore, the total time delay △T k =△box k ​+△apk k +△sdk k The total time delay is the first time difference. By calculating the first time difference in this way, the time delay of each stage in the transmission process of the first synchronization signal can be known. Figure 5 A schematic diagram of the time delay of each stage in the transmission process of a synchronization signal k is provided for the embodiments of the present application.

[0083] Involving S204, in order to better improve the tearing phenomenon, after obtaining the first time difference, a setting algorithm is applied to optimize the first time difference. The setting algorithm is an improved Kalman filter algorithm, that is, the Kalman filter algorithm is applied to estimate each time difference, each time difference can be referred to as an observation or an observation value, a set of each time difference is referred to as an observation data set, and the result of estimation is referred to as an optimized time difference.

[0084] In the Kalman filter algorithm, for each observation (for example, observation △Tk), the prior estimate X k|k-1 , the posterior estimate X k|k , the Kalman filter gain K k , the prior error covariance P k|k-1 , and the posterior error covariance P k|k are calculated, and the posterior estimate is the result of estimation. In a specific example, the observation data set is [△T0, △T1, △T2, △T3, △T4], for example, the values are [0.1 ms, 0.12 ms, 0.11 ms, 0.15 ms, and 0.14 ms].

[0085] In the process of calculating the above posterior estimate, the process noise covariance and the observation noise covariance are also used. In one way, the process noise covariance and the observation noise covariance are both fixed values, which are irrelevant to the value of k, for example, the fixed value of the process noise covariance (referred to as the set process noise covariance) is Q=0.00001; the fixed value of the observation noise covariance (referred to as the set observation noise covariance) is R=0.01. In another way, the process noise covariance and the observation noise covariance are dynamically updated, for example, when k=0 and k=1, the process noise covariance Q0=Q, Q1=Q, the observation noise covariance R0=R, R1=R; starting from k=2, Q2 is obtained by updating Q1.

[0086] Next, taking the first synchronization signal as an example, the optimization process of the first time difference is described. The first synchronization signal is represented by synchronization signal k, and the first time difference is △T k .

[0087] The first case is k=0, that is, the first synchronization signal is the first synchronization signal in the plurality of synchronization signals.

[0088] In this case, since there is no historical observation value for reference, the first time difference can be directly optimized, that is, the first time difference is taken as the optimized first time difference. In addition, in order to provide data support for the case where k>0, the prior estimation value, the prior error covariance and the posterior error covariance in this case can be calculated in a set manner. And

[0089] For example, the process noise covariance Q0=Q=0.00001 and the observation noise covariance R0=R=0.01; the prior estimation value X 0|-1 = X 0|0 0=0.1; the prior error covariance P 0|-1 =P 0|0 +Q0=1.0+0.00001 =1.00001; the posterior error covariance P 0|0 is usually taken as 1. The set value of the process noise covariance is Q=0.00001.

[0090] The second case is that k is greater than 0, that is, the first synchronization signal is not the first synchronization signal in the multiple synchronization signals.

[0091] In this case, the optimization process can be implemented through steps S204-1 to S204-5 in the following. Figure 6

[0092] S204-1: taking the posterior estimation value of the second time difference as the prior estimation value of the first time difference.

[0093] wherein the second time difference is the time difference corresponding to the last period of the first time difference, the current period synchronization signal is the synchronization signal k, and the last period synchronization signal is the synchronization signal k-1. For example, the prior estimation value of the first time difference can be calculated according to the following formula: X k|k-1 = X k-1|k-1 . Wherein X k|k-1 is the prior estimation value of the first time difference, and X k-1|k-1 is the posterior estimation value of the second time difference.

[0094] For example, when k=1, X 1|0 = X 0|0 0=0.1; when k=2, X 2|1 = X 1|1 .

[0095] S204-2: calculating the prior error covariance of the first time difference according to the posterior error covariance of the second time difference and the process noise covariance of the first time difference.

[0096] For example, the prior error covariance of the first time difference can be calculated according to the following formula: P k|k-1 ​= P k-1|k-1 +Q k . Wherein, P k|k-1 is the prior error covariance of the first time difference, P k-1|k-1 is the posterior error covariance of the second time difference, Q k is the process noise covariance of the first time difference.

[0097] For example, k = 1, P 1|0 = P 0|0 +Q1=1.0+0.00001=1.00001.

[0098] S204-3: Calculate the Kalman gain of the first time difference according to the prior error covariance of the first time difference and the observation noise covariance of the first time difference.

[0099] For example, the Kalman gain of the first time difference can be calculated according to the following formula: K k = P k|k-1 / (P k|k-1 +R k ). Wherein, K k is the Kalman gain of the first time difference, P k|k-1 is the prior error covariance of the first time difference, R k is the observation noise covariance of the first time difference.

[0100] For example, k = 1, K1= P 1|0 / P 1|0 +R1) = 1.00001 / 1.00001+0.01≈0.9901.

[0101] S204-4: Obtain the posterior estimate of the first time difference according to the Kalman gain of the first time difference, the prior estimate of the first time difference and the first time difference.

[0102] For example, the posterior estimate of the first time difference can be calculated according to the following formula: X k|k = X k|k-1 +△T k . Wherein, X k|k is the posterior estimate of the first time difference,△T k is the first time difference.

[0103] For example, k = 1, X 1|1 = X 1|0 +K1(△T1- X 1|0 ) = 0.1+0.9901 (0.12-0.1) = 0.1198.

[0104] In addition, the posterior error covariance of the first time difference can also be determined according to the Kalman gain of the first time difference and the prior error covariance of the first time difference. For example, the posterior error covariance of the second time difference can be calculated by the following formula: P k|k = (1-K k )P k|k-1 . Wherein, P k|k is the posterior error covariance of the first time difference, K k is the Kalman gain of the first time difference, and P k|k-1 is the prior error covariance of the first time difference.

[0105] S204-5: determining the posterior estimation value of the first time difference as the optimization result of the first time difference.

[0106] In the embodiments of the present application, the dynamic updating of the process noise covariance and the observation noise covariance can better improve the optimization effect. Therefore, the updating process of the two is described respectively.

[0107] The updating process of the process noise covariance is as follows:

[0108] As described in the above embodiments, when k = 1, Q1 = Q0 is used, that is, no updating is performed, and the set value is taken. When k is greater than 1, the process noise covariance of the first time difference can be calculated according to the first set parameter α, the process noise covariance Q k-1 of the second time difference, the prior estimation value and the posterior estimation value X k|k-1 = X k-1|k-1 , that is, Q k = αQ k-1 + (1-α)(X k-1|k-1 –X k-1|k-2 ) (X k-1|k-1 –X k-1|k-2 ).

[0109] For example, when k = 2, Q2 = αQ1+ (1-α)(X 1|1 – X 1|0) ) (X 1|1 – X 1|0 ) = 0.9 0.00001+0.1 (0.1198–0.1) (0.1198–0.1) =4.8601 0.00001.

[0110] For example, when k = 3, Q3 = αQ2+ (1-α)( X 2|2 –X 2|1 ) (X 2|2 –X 2|1)= 0.9 4.8601 0.00001 + 0.1 (0.1143-0.1198) (0.1143-0.1198)= 4.67659 0.00001.

[0111] The updating process of the observation noise covariance is as follows:

[0112] As in the above embodiment, when k = 1, R1 = R0 is used, that is, no updating is performed, and a set value is taken.

[0113] When k is greater than 1, the observation noise covariance of the first time difference can be calculated according to the second set parameter β, the observation noise covariance R k-1 of the second time difference, the posteriori estimation value of the first time difference △T1 and the second time difference, that is, R k = βR k-1 + (1-β) (△T k - X k-1|k-2 ) (△T k - X k-1|k-2 ).

[0114] For example, when k = 2, R2 = βR0 + (1-β) (△T1-X 1|0 ) (△T1-X 1|0 )= 0.9 0.01+0.1 (0.12-0.1) (0.12-0.1)= 0.009+ 0.00004 = 0.00904.

[0115] For example, when k = 3, R3 = βR2 + (1-β) (△T2-X 2|1 ) (△T2-X 2|1 )= 0.9 0.00904+0.1 (0.11-0.1198) (0.11-0.1198)= 0.0081456.

[0116] Wherein, α and β are both smoothing factors. The value of α ranges between (0, 1). The larger α is, the stronger the dependence on the historical process noise covariance is, and the weaker the adaptability to the observation data to be estimated is; the value of β ranges between (0, 1). The larger β is, the stronger the dependence on the historical observation noise covariance is, and the weaker the adaptability to the observation data to be estimated is. On the contrary, the smaller α and β are, the stronger the adaptability to the new data is, but it may lead to larger fluctuations of the noise covariance.

[0117] In the actual application process, the initial values of a and β can be selected as 0.95, which means that the historical process noise covariance and the historical observation noise covariance are highly dependent. The final a and β can be determined by trial adjustment and cross-validation, for example, 0.9. The trial adjustment can be to run the filtering algorithm, observe the filtering results (especially the noise level and the delay response), and gradually reduce a and β when it is found that the filter reacts too slowly to new data. In cross-validation, the data set is divided, cross-validation is performed, the error under different a and β is calculated, and the parameter combination with the minimum error is selected.

[0118] In summary, in order to better illustrate the embodiments of the present application, the calculation processes of the estimated values and the covariances for k=0, 1, and 2 are summarized respectively.

[0119] (1) k=0

[0120] Process noise covariance Q0=Q=0.00001;

[0121] Observation noise covariance R0=R=0.01;

[0122] Prior estimated value X 0|-1 = X 0|0 =△T0=0.1;

[0123] Prior error covariance P 0|-1 =P 0|0 +Q0=1.0+0.00001 =1.00001;

[0124] Posterior error covariance P 0|0 is usually 1.

[0125] (2) k=1

[0126] Process noise covariance Q1=Q=0.00001;

[0127] Observation noise covariance R1=R=0.01;

[0128] Prior estimated value X 1|0 = X 0|0 = 0.1;

[0129] Prior error covariance P 1|0 = P 0|0 +Q1= 1.0+0.00001 =1.00001;

[0130] Kalman gain K1=P 1|0 / P 1|0 +R1=1.00001 / 1.00001+0.01≈0.9901;

[0131] Posterior estimate X 1|1 = X 1|0 +K1(△T1-X 1|0 ) = 0.1+0.9901 (0.12-0.1) = 0.1198;

[0132] Posterior error covariance P 1|1 =(1-K1)P 1|0 = (1-0.9901) 1.00001≈0.0099。

[0133] (3)k=2

[0134] The process noise covariance and the observation noise covariance can be fixed or updated.

[0135] If the process noise covariance and the observation noise covariance are fixed, the process noise covariance Q2=Q=0.00001 and the observation noise covariance R2=R=0.01.

[0136] Prior estimate X 2|1 = X 1|1 = 0.1198;

[0137] Prior error covariance P 2|1 =P 1|1 +Q1= 0.0099+0.00001 = 0.00991;

[0138] Kalman gain K2=P 2|1 / P 2|1 +R1=0.00991 / 0.00991+0.01≈0.4978;

[0139] Posterior estimate X 2|2 = X 2|1 +K2(△T2-X 2|1 ) =0.1198+0.4978 (0.11-0.1198) =0.1149;

[0140] Posterior error covariance P 2|2 =(1-K2)P 2|1 =(1- 0.4978) 0.00991≈0.00498。

[0141] The estimate values in the case of fixed process noise covariance and observation noise covariance are shown in Table 1.

[0142] Table 1 The estimated values under the condition of fixed process noise covariance and observation noise covariance

[0143]

[0144] If the dynamic updated process noise covariance and observation noise covariance are used, the process noise covariance Q2=4.8601 0.00001, and the observation noise covariance R2=0.00904.

[0145] Prior estimated value X 2|1 = X 1|1 = 0.1198.

[0146] Prior error covariance P 2|1 = P 1|1 + Q1= 0.0099+4.8601 0.00001 = 0.009948601; Kalman gain K2= P 2|1 / P 2|1 + R2= 0.00991 / 0.00991+0.00904≈0.5241.

[0147] Posterior estimated value X 2|2 = X 2|1 + K2(△T2- X 2|1 ) = 0.1198+0.5241 (0.11-0.1198) =0.1143.

[0148] Posterior error covariance P 2|2 =(1-K2)P 2|1 = (1- 0.5241) 0.009948601≈0.004986.

[0149] Optionally, the estimated values under the condition of updated process noise covariance and observation noise covariance can be seen from Table 2.

[0150] Table 2 The estimated values under the condition of updated process noise covariance and observation noise covariance

[0151]

[0152] S204 is involved, the first time is adjusted by using the optimized first time difference to obtain a target time.

[0153] Taking k=2 as an example, under the condition of fixed process noise covariance and observation noise covariance, the optimized first time difference is 0.1149, and the target time obtained is Teye2+2t+0.1149. Taking t as an example, the target time is 1006.1149.

[0154] Taking k=2 as an example, under the condition of updated process noise covariance and observation noise covariance, the optimized first time difference is 0.00143, and the target time obtained is Teye2+2t+0.1143. Taking t as an example, the target time is 1006.1143.

[0155] S205 is involved, and the image frame corresponding to the target time is sent to the glasses for display.

[0156] The target time is the calibrated timestamp. Under the condition of updated process noise covariance and observation noise covariance, the image frame corresponding to 1006.1143 ms can be sent to the glasses for display.

[0157] Figure 7 A schematic diagram of sending N image frames to the glasses for display is provided for the embodiments of the application, where N is an integer greater than 1, and the specific value is not limited.

[0158] The above embodiments are described by taking one synchronization signal as an example. In the application process, the improved Kalman filter algorithm calibrates in real time when new observation data arrives, without waiting for multiple data points. Through the improved filter algorithm and the addition of smoothing processing, the parameters of the filter can be dynamically adjusted to adapt to different delays and jitter conditions, reduce the fluctuation of the timestamp offset, improve the estimation accuracy of the timestamp, reduce the picture tearing phenomenon, and thus improve the user experience of the AR glasses and achieve more accurate time synchronization.

[0159] In order to make the technical solutions of the application more perfect, Figure 8 A signaling diagram of a synchronous display method of an AR split glasses is provided for the embodiments of the application, Figure 8 at least including the following steps:

[0160] S801: In the case where the glasses are disconnected from the control box, the control box detects that the glasses are connected to itself through the data line.

[0161] S802: The glasses periodically send multiple synchronization signals to the control box.

[0162] S803: The control box calculates the first time difference between the first time when the glasses send the first synchronization signal and the second time when the control box receives the first synchronization signal.

[0163] S804: The control box applies a setting algorithm to optimize the first time difference, and adjusts the first time by using the optimized first time difference to obtain a target time.

[0164] S805: The control box sends the image frame corresponding to the target time to the glasses.

[0165] S806: The glasses display the image frame.

[0166] The implementation of each step can refer to the foregoing embodiments, which will not be described here.

[0167] The embodiments of the present application improve the original Android scheme time signal transmission mode and timestamp calculation method to adapt to the AR split glasses signal transmission mode, thereby significantly reducing the synchronization delay and improving the real-time performance and accuracy of display. It can ensure that the AR glasses can quickly synchronize the timestamp signal after the plug-in operation and maintain the stability of the picture.

[0168] As shown in Figure 9 Based on the same inventive concept, the embodiments of the present application provide a synchronous display device of AR split glasses, which comprises a detection unit 91, a transmission unit 92 and a processing unit 93.

[0169] The detection unit 91 is configured to detect that the glasses are connected to the control box through the data line when the glasses are disconnected from the control box.

[0170] The transmission unit 92 is configured to receive a plurality of synchronization signals periodically sent by the glasses.

[0171] The processing unit 93 is configured to calculate a first time difference between a first time at which the glasses send a first synchronization signal and a second time at which the control box receives the first synchronization signal; wherein the synchronization signal is any one of the plurality of synchronization signals; each synchronization signal corresponds to a time difference; and the first time difference is the time difference corresponding to the first synchronization signal.

[0172] The processing unit 93 is further configured to apply a setting algorithm to optimize the first time difference, and adjust the first time by using the optimized first time difference to obtain a target time.

[0173] The transmission unit 92 is further configured to send the image frame corresponding to the target time to the glasses, so that the glasses display the image frame.

[0174] In an optional embodiment, the processing unit 93 is further configured to:

[0175] The time at which the serial port signal receiver of the first end of the data line receives the first synchronization signal is the first time; wherein the first end of the data line is one end of the two ends of the data line connected to the glasses.

[0176] In an alternative embodiment, the processing unit 93 is further configured to:

[0177] record a second time at which the first synchronization signal is received by the rendering toolkit.

[0178] In an alternative embodiment, the setting algorithm is a modified Kalman filter algorithm; and if the first synchronization signal is not the first synchronization signal in the plurality of synchronization signals, the processing unit 93 is specifically configured to:

[0179] use a posteriori estimation of the second time difference as a priori estimation of the first time difference; wherein the second time difference is a time difference of a previous cycle of the first time difference.

[0180] calculate a priori error covariance of the first time difference according to a posteriori error covariance of the second time difference and process noise covariance of the first time difference; wherein the a posteriori error covariance of the second time difference is determined according to a Kalman gain of the second time difference and a priori error covariance of the second time difference.

[0181] calculate a Kalman gain of the first time difference according to the a priori error covariance of the first time difference and observation noise covariance of the first time difference.

[0182] obtain a posteriori estimation of the first time difference according to the Kalman gain of the first time difference, the a priori estimation of the first time difference and the first time difference.

[0183] determine the a posteriori estimation of the first time difference as an optimization result of the first time difference.

[0184] In an alternative embodiment, if the first synchronization signal is not the first synchronization signal in the plurality of synchronization signals and is not the second synchronization signal, the processing unit 93 is further configured to:

[0185] calculate the process noise covariance of the first time difference according to the first setting parameter, the process noise covariance of the second time difference, the a priori estimation of the second time difference and the a posteriori estimation of the second time difference.

[0186] In an alternative embodiment, if the first synchronization signal is the second synchronization signal in the plurality of synchronization signals, the processing unit 93 is further configured to:

[0187] determine the setting process noise covariance as the process noise covariance of the first time difference.

[0188] In an alternative embodiment, if the first synchronization signal is not the first synchronization signal in the plurality of synchronization signals and is not the second synchronization signal, the processing unit 93 is further configured to:

[0189] The observation noise covariance of the first time difference is calculated according to the second setting parameter, the observation noise covariance of the second time difference, the first time difference, and the posterior estimation value of the second time difference.

[0190] In an optional implementation, if the first synchronization signal is a second synchronization signal in the plurality of synchronization signals, the processing unit 93 is further configured to:

[0191] determine the setting observation noise covariance as the observation noise covariance of the first time difference.

[0192] In an optional implementation, if the first synchronization signal is a first synchronization signal in the plurality of first synchronization signals, the processing unit 93 is further configured to:

[0193] determine that the prior estimation value and the posterior estimation value of the first synchronization signal are both the first time difference;

[0194] determine that the posterior error covariance of the first synchronization signal is the setting error covariance;

[0195] determine that the prior error covariance of the first synchronization signal is a sum of the error covariance and the setting process noise covariance.

[0196] Since the apparatus is the apparatus in the method in the embodiments of the present application, and the principle of solving the problem of the apparatus is similar to that of the method, the implementation of the apparatus can be referred to the implementation of the method, and the repeated parts will not be described here.

[0197] As shown in Figure 10 based on the same inventive concept, the embodiments of the present application provide an AR split glasses, the AR split glasses comprising glasses, a control box and a data line; the data line is used to connect the glasses and the control box; the method is applied to the control box; the control box comprises a processor 101 and a memory 102.

[0198] The processor 101 is configured to perform:

[0199] In the case that the glasses and the control box are disconnected, it is detected that the glasses are connected with the control box through the data line;

[0200] receiving a plurality of synchronization signals periodically sent from the glasses;

[0201] calculating a first time difference between a first time at which the glasses sends a first synchronization signal and a second time at which the control box receives the first synchronization signal; wherein the synchronization signal is any one of the plurality of synchronization signals; each synchronization signal corresponds to a time difference; the first time difference is the time difference corresponding to the first synchronization signal;

[0202] applying a setting algorithm to optimize the first time difference, and adjusting the first time by using the optimized first time difference to obtain a target time;

[0203] sending the image frame corresponding to the target time to the glasses to make the glasses display the image frame;

[0204] the memory 102 is configured to perform:

[0205] store a plurality of image frames.

[0206] In an optional embodiment, the processor 101 is further configured to:

[0207] record a time when the serial port signal receiver of the first end of the data line receives the first synchronization signal as the first time; wherein the first end of the data line is one end of the data line connected to the glasses.

[0208] In an optional embodiment, the processor 101 is further configured to:

[0209] record a time when the rendering toolkit receives the first synchronization signal as the second time.

[0210] In an optional embodiment, the setting algorithm is an improved Kalman filter algorithm; if the first synchronization signal is not the first synchronization signal in the plurality of synchronization signals, the processor 101 is specifically configured to:

[0211] take the posterior estimate value of the second time difference as the prior estimate value of the first time difference; wherein the second time difference is the time difference of the previous cycle of the first time difference;

[0212] calculate the prior error covariance of the first time difference according to the posterior error covariance of the second time difference and the process noise covariance of the first time difference; wherein the posterior error covariance of the second time difference is determined according to the Kalman gain of the second time difference and the prior error covariance of the second time difference;

[0213] calculate the Kalman gain of the first time difference according to the prior error covariance of the first time difference and the observation noise covariance of the first time difference;

[0214] obtain the posterior estimate value of the first time difference according to the Kalman gain of the first time difference, the prior estimate value of the first time difference and the first time difference;

[0215] determine the posterior estimate value of the first time difference as the optimization result of the first time difference.

[0216] In an optional embodiment, if the first synchronization signal is not the first synchronization signal in the plurality of synchronization signals and is not the second synchronization signal, the processor 101 is further configured to:

[0217] calculate the process noise covariance of the first time difference according to the first setting parameter, the process noise covariance of the second time difference, the prior estimate value and the posterior estimate value of the second time difference.

[0218] In an optional implementation, if the first synchronization signal is the second synchronization signal in the plurality of synchronization signals, the processor 101 is further configured to:

[0219] determine to set the process noise covariance as the process noise covariance of the first time difference.

[0220] In an optional implementation, if the first synchronization signal is neither the first synchronization signal nor the second synchronization signal in the plurality of synchronization signals, the processor 101 is further configured to:

[0221] calculate the observation noise covariance of the first time difference according to the second setting parameter, the observation noise covariance of the second time difference, the first time difference, and the posteriori estimation value of the second time difference.

[0222] In an optional implementation, if the first synchronization signal is the second synchronization signal in the plurality of synchronization signals, the processor 101 is further configured to:

[0223] determine to set the observation noise covariance as the observation noise covariance of the first time difference.

[0224] In an optional implementation, if the first synchronization signal is the first synchronization signal in the plurality of first synchronization signals, the processor 101 is further configured to:

[0225] determine that the priori estimation value and the posteriori estimation value of the first synchronization signal are both the first time difference;

[0226] determine that the posteriori error covariance of the first synchronization signal is the setting error covariance;

[0227] determine that the priori error covariance of the first synchronization signal is the sum of the error covariance and the setting process noise covariance.

[0228] The embodiment of the present application further provides a computer storage medium, which stores computer program instructions, and when the instructions are run on a computer, the computer executes the steps of the synchronization display method of the AR split glasses.

[0229] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0230] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0231] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0232] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0233] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A synchronous display method for AR split glasses, characterized in that: The AR split glasses include glasses, a control box and a data cable; the data cable is used to connect the glasses and the control box; The method is applied to the control box; the method comprises: When the glasses are disconnected from the control box, detecting that the glasses are connected to the control box via the data cable; receiving a plurality of synchronization signals periodically sent from the glasses; Calculating a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal; wherein the first synchronization signal is any one of the plurality of synchronization signals; each synchronization signal corresponds to a time difference; and the first time difference is the time difference corresponding to the first synchronization signal; Applying a set algorithm to optimize the first time difference, and applying the optimized first time difference to adjust the first time to obtain a target time; The image frame corresponding to the target time is sent to the glasses, so that the glasses display the image frame.

2. The method according to claim 1, characterized in that The method further comprises: The time when the serial port signal receiver at the first end of the data line receives the first synchronization signal is recorded as the first time; wherein the first end of the data line is the end connected to the glasses among the two ends of the data line.

3. The method according to claim 1, characterized in that The method further comprises: The time when the first synchronization signal is received by the rendering toolkit is recorded as the second time.

4. The method according to claim 1, wherein The setting algorithm is an improved Kalman filter algorithm; if the first synchronization signal is not the first synchronization signal among the multiple synchronization signals, then optimizing the first time difference includes: Using the posterior estimated value of the second time difference as the prior estimated value of the first time difference; wherein the second time difference is the time difference of the previous cycle of the first time difference; Calculating an a priori error covariance of the first time difference based on the a posteriori error covariance of the second time difference and the process noise covariance of the first time difference; wherein the a posteriori error covariance of the second time difference is determined based on the Kalman gain of the second time difference and the a priori error covariance of the second time difference; Calculating a Kalman gain of the first time difference according to a priori error covariance of the first time difference and an observation noise covariance of the first time difference; Obtaining a posterior estimate of the first time difference according to the Kalman gain of the first time difference, the prior estimate of the first time difference, and the first time difference; Determining the posterior estimated value of the first time difference is an optimization result of the first time difference.

5. The method according to claim 4, characterized in that If the first synchronization signal is not the first synchronization signal and not the second synchronization signal among the plurality of synchronization signals, the method further includes: The process noise covariance of the first time difference is calculated according to the first setting parameter, the process noise covariance of the second time difference, the priori estimate value and the a posteriori estimate value of the second time difference.

6. The method according to claim 4, characterized in that If the first synchronization signal is the second synchronization signal among the multiple synchronization signals, the method further includes: A process noise covariance is determined and set to be the process noise covariance of the first time difference.

7. The method according to claim 4, characterized in that If the first synchronization signal is not the first synchronization signal and not the second synchronization signal among the plurality of synchronization signals, the method further includes: The observation noise covariance of the first time difference is calculated according to the second setting parameter, the observation noise covariance of the second time difference, the first time difference, and the posterior estimate of the second time difference.

8. The method according to claim 4, characterized in that If the first synchronization signal is the second synchronization signal among the multiple synchronization signals, the method further includes: The observation noise covariance is determined and set to be the observation noise covariance of the first time difference.

9. The method according to any one of claims 1 to 8, characterized in that If the first synchronization signal is the first synchronization signal among the multiple first synchronization signals, the method further includes: Determining that a priori estimated value and a posteriori estimated value of the first synchronization signal are both the first time difference; Determining a posterior error covariance of the first synchronization signal as a set error covariance; A priori error covariance of the first synchronization signal is determined to be the sum of the error covariance and a setting process noise covariance.

10. A pair of AR split glasses, characterized in that: The AR split glasses include glasses, a control box and a data cable; the data cable is used to connect the glasses and the control box; the control box includes a processor and a memory; The processor is configured to execute: When the glasses are disconnected from the control box, detecting that the glasses are connected to the control box via the data cable; receiving a plurality of synchronization signals periodically sent from the glasses; Calculating a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal; wherein the first synchronization signal is any one of the plurality of synchronization signals; each synchronization signal corresponds to a time difference; and the first time difference is the time difference corresponding to the first synchronization signal; Applying a set algorithm to optimize the first time difference, and applying the optimized first time difference to adjust the first time to obtain a target time; sending the image frame corresponding to the target time to the glasses, so that the glasses display the image frame; The memory is configured to execute: Store multiple image frames.

Images