Synchronous display method of AR split glasses and AR split glasses

Abstract

The invention discloses a synchronous display method of AR split glasses and the AR split glasses. The AR split glasses comprise glasses, a control box and a data line. The data line is used for connecting the glasses and the control box. The method is applied to the control box. The method comprises the following steps: detecting that the glasses are connected with the control box through a data line under the condition that the glasses are disconnected from the control box; receiving a plurality of synchronization signals periodically sent by the glasses; calculating a first time difference between a first time when the glasses send the first synchronization signal and a second time when the control box receives the first synchronization signal; the synchronization signal is any one of a plurality of synchronization signals; the first time difference is a time difference corresponding to the first synchronization signal; optimizing the first time difference by using a setting algorithm, and adjusting the first time by using the optimized first time difference to obtain target time; and sending the image frame corresponding to the target time to the glasses, so that the glasses display the image frame. The picture tearing problem is avoided, and picture synchronous display is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of AR devices, and in particular to a synchronous display method of AR split glasses and AR split glasses. Background Art

[0002] AR glasses can be divided into split AR glasses (referred to as AR split glasses) and integrated AR glasses. Among them, integrated AR glasses have built-in processors, storage space and batteries, and have certain computing capabilities. They can process some AR applications and scenarios by themselves and can run independently without connecting to any equipment. Split AR glasses usually only have display functions. The glasses are connected to the control box through a data cable. The control box is a terminal with data processing functions.

[0003] The connection between the glasses and the control box can be disconnected by unplugging the data cable. During the plugging and unplugging process, due to the inconsistent timestamps between the glasses and the control box, the refresh rate of the glasses' screen display is out of sync with the rendering cache screen refresh rate of the control box, which may cause screen tearing. Summary of the invention

[0004] In an exemplary embodiment of the present application, a synchronous display method of AR split glasses and AR split glasses are provided to avoid the problem of screen tearing and achieve synchronous display of the screen.

[0005] According to a first aspect of an exemplary embodiment, a synchronous display method of AR split glasses is provided, wherein the AR split glasses include glasses, a control box and a data cable; the data cable is used to connect the glasses and the control box; the method is applied to the control box. The method includes:

[0006] When the glasses are disconnected from the control box, it is detected that the glasses are connected to the control box via a data cable;

[0007] Receiving multiple synchronization signals periodically sent from the glasses;

[0008] Calculate a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal; wherein the first synchronization signal is any one of a plurality of synchronization signals; each synchronization signal corresponds to a time difference; and the first time difference is the time difference corresponding to the first synchronization signal;

[0009] Applying a set algorithm to optimize the first time difference, and applying the optimized first time difference to adjust the first time to obtain a target time;

[0010] The image frame corresponding to the target time is sent to the glasses so that the glasses display the image frame.

[0011] According to a second aspect of an exemplary embodiment, a synchronous display device for AR split glasses is provided, wherein the AR split glasses include glasses, a control box, and a data cable; the data cable is used to connect the glasses and the control box. The device is applied to the control box; the device includes:

[0012] The detection unit is used to detect that the glasses are connected to the control box via the data line when the glasses are disconnected from the control box;

[0013] The transmission unit is used to: receive a plurality of synchronization signals periodically sent from the glasses;

[0014] The processing unit is used to calculate a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal; wherein the first synchronization signal is any one of a plurality of synchronization signals; each synchronization signal corresponds to a time difference; and the first time difference is the time difference corresponding to the first synchronization signal;

[0015] The processing unit is further used to: optimize the first time difference by applying a set algorithm, and adjust the first time by applying the optimized first time difference to obtain a target time;

[0016] The transmission unit is further used to: send the image frame corresponding to the target time to the glasses, so that the glasses display the image frame.

[0017] According to a third aspect of an exemplary embodiment, there is provided an AR split glasses, the AR split glasses comprising glasses, a control box and a data cable; the data cable is used to connect the glasses and the control box; the method is applied to the control box; the control box comprises a processor and a memory;

[0018] A processor configured to execute:

[0019] When the glasses are disconnected from the control box, it is detected that the glasses are connected to the control box via a data cable;

[0020] Receiving multiple synchronization signals periodically sent from the glasses;

[0021] Calculate a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal; wherein the first synchronization signal is any one of a plurality of synchronization signals; each synchronization signal corresponds to a time difference; and the first time difference is the time difference corresponding to the first synchronization signal;

[0022] Applying a set algorithm to optimize the first time difference, and applying the optimized first time difference to adjust the first time to obtain a target time;

[0023] sending the image frame corresponding to the target time to the glasses so that the glasses display the image frame;

[0024] Memory, configured to execute:

[0025] Store multiple image frames.

[0026] According to a fourth aspect of the exemplary implementation, a computer storage medium is provided, in which computer program instructions are stored. When the instructions are executed on a computer, the computer executes the synchronous display method of AR split glasses as in the first aspect.

[0027] In an embodiment of the present application, when the glasses are disconnected from the control box, the control box detects that the glasses are connected to the control box via a data cable, and receives multiple synchronization signals periodically sent from the glasses. The first time difference between the first time when the glasses send a first synchronization signal (any one of the multiple synchronization signals) and the second time when the control box receives the first synchronization signal is calculated, and the first time difference is the time difference corresponding to the first synchronization signal. The set algorithm is applied to optimize the first time difference, and the optimized first time difference is applied to adjust the first time to obtain the target time. The image frame corresponding to the target time is sent to the glasses so that the glasses display the image frame. Such a design can ensure the time synchronization of the glasses and the control box, thereby avoiding the problem of screen tearing caused by plugging and unplugging the glasses, and realizing synchronous display of the screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 The schematic diagram of the structure of an AR split glasses provided by an embodiment of the present application is exemplarily shown;

[0030] Figure 2 A flowchart of a synchronous display method of AR split glasses provided in an embodiment of the present application is exemplarily shown;

[0031] Figure 3 A flowchart of a method for calculating a first time difference provided in an embodiment of the present application is exemplarily shown;

[0032] Figure 4 A schematic diagram exemplarily shows various time delays in a process in which a control box receives a first synchronization signal provided by an embodiment of the present application;

[0033] Figure 5 A schematic diagram exemplarily shows the delay of a synchronization signal k at each stage in the transmission process provided by an embodiment of the present application;

[0034] Figure 6 A flowchart of a method for optimizing a first time difference provided in an embodiment of the present application is exemplarily shown;

[0035] Figure 7 A schematic diagram of sending N image frames to glasses for display provided by an embodiment of the present application is exemplarily shown;

[0036] Figure 8 A signaling diagram of a synchronous display method of AR split glasses provided in an embodiment of the present application is exemplarily shown;

[0037] Fig. 9 The schematic diagram of the structure of a synchronous display device of AR split glasses provided in an embodiment of the present application is exemplarily shown;

[0038] Fig.10 The structural block diagram of an AR split glasses provided in an embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0040] In the embodiment of the present application, the control box of the AR split glasses is a terminal with data processing function. In addition, operation buttons are also provided on the control box. Usually, the control box is an Android system. For the convenience of understanding, the terms involved in the embodiment of the present application are explained below:

[0041] (1) Screen tearing refers to the situation in which two frames in a game or video are not switched neatly on the screen, that is, the two frames are out of sync. Generally speaking, this situation will cause the game or video screen to jitter or flicker.

[0042] (2) The Android system architecture is divided into the application layer, Java framework layer, native C / C++ library layer and Linux kernel layer. In the Android system, the native C / C++ library layer is also called the Native layer. In the Native layer, developers can use C / C++ to write code to access the underlying hardware and implement high-performance computing operations. At the same time, the Native layer is also responsible for converting the requests passed by the Java framework layer into instructions that can be recognized by the underlying operating system.

[0043] (3) Android application package (APK) is an application package file format used by the Android operating system to distribute and install mobile applications and middleware. If an Android application code is to be run on an Android device, it must first be compiled and then packaged into a file that can be recognized by the Android system before it can be run. The file format that can be recognized and run by the Android system is APK.

[0044] (4) Software Development Kit (SDK) is generally a collection of development tools used by software engineers to build application software for a specific software package, software framework, hardware platform, operating system, etc.

[0045] (5) Human Interface Device (HID) refers to any input or output device used to interact with a computer or other electronic device. Common HID devices include mice, keyboards, game controllers, digital pens, scanners, etc. In the embodiments of the present application, HID may refer to glasses in AR human glasses.

[0046] (6) HIDRAW refers to the driver used to process HID devices in the Linux system. Through HIDRAW, users can perform read and write operations with HID devices to achieve data input and output.

[0047] (7) Vertical Synchronization (VSync) refers to the technology used to synchronize the screen refresh rate and graphics rendering in computer graphics processing. Through VSync, the frame rate of graphics rendering is synchronized with the refresh rate of the display to eliminate screen tearing and provide smoother vision.

[0048] (8) Data cable, which is a necessary component for connecting the hard disk and the motherboard. Its interface classification includes a cluster communication port (COM) interface, a universal serial bus (USB) interface, a Type-C interface, and a Micro-USB interface. In the embodiment of the present application, the data cable can be a data cable of a Type-C interface.

[0049] The glasses of AR split glasses serve as display screens, and the processor is located inside the control box. Therefore, the screen and processor are separated. During the normal use of AR glasses, the glasses are connected to the control box through a data cable. When the user needs to take a break, the connection between the two can be disconnected. The disconnection method can be to disconnect the end of the data cable connected to the glasses, and / or the end of the data cable connected to the control box, while the control box may still be turned on. When the glasses of AR split glasses are plugged in and out of the control box through the data cable, the timestamps transmitted by the glasses and the control box are inconsistent. After detecting that the glasses are connected to the control box through the data cable, the timing of the control box is counted from the power-on time, while the timing of the glasses is counted after the data cable is plugged in again to establish a connection with the control box, which causes the display refresh frequency to be out of sync with the rendering refresh frequency. For example, the glasses are timers starting from 0, and the control box is timed from the power-on time. For example, the control box is turned on at 8 o'clock and connected to the glasses through a data cable, but after 50 minutes of use, the user wants to pause and rest, so the data cable can be disconnected, and the control box is not turned off at this time. When the user plugs in the data cable again at 9 o'clock, it continues to be used. At this time, the timestamp of the control box has been calculated according to 8 o'clock. After the glasses are disconnected from the control box, the timer is reset. After the data cable is plugged in again, the timestamp of the glasses starts to be calculated according to 9 o'clock.

[0050] Due to the inconsistent timestamps, the refresh rate of the glasses' screen display is out of sync with the refresh rate of the control box's screen rendering cache, which leads to the problem of screen asynchrony. For example, if the screen display refresh rate is lower than the screen rendering cache refresh rate, screen tearing will occur; if the screen display refresh rate is higher than the screen rendering cache refresh rate, screen freeze will occur. In actual application, repeated plugging and unplugging usually leads to screen tearing.

[0051] To this end, the embodiment of the present application provides a synchronous display method for AR split glasses, in which, when the glasses are disconnected from the control box, it is detected that the glasses are connected to the control box via a data cable, and the time difference (delay) between the time when the glasses send the synchronization signal and the time when the control box receives the synchronization signal is taken into account, and the set algorithm is applied to optimize the time difference, and then the optimized time difference is applied to optimize the time when the glasses send the synchronization signal, and the accurate time when the control box receives the synchronization signal is obtained. The image frame matching the control time is sent to the glasses for display, so as to overcome the problem of display screen tearing.

[0052] After introducing the design ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0053] refer to Figure 1 , shows a schematic structural diagram of an AR split glasses, wherein 11 is the glasses, 12 is the control box, and 13 is the data cable.

[0054] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiment of the present application.

[0055] Combine the following Figure 1 The application scenario shown is shown in Figure 2 A flowchart of a synchronous display method of AR split glasses is shown, and the method is applied to a control box. Figure 2 , the technical solution provided in the embodiments of the present application is described.

[0056] S201: When the glasses are disconnected from the control box, it is detected that the glasses are connected to the control box via a data cable.

[0057] S202: Receive multiple synchronization signals periodically sent from the glasses.

[0058] S203: Calculate a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal.

[0059] S204: Apply a set algorithm to optimize the first time difference, and apply the optimized first time difference to adjust the first time to obtain a target time.

[0060] S205: Sending the image frame corresponding to the target time to the glasses, so that the glasses display the image frame.

[0061] In an embodiment of the present application, when the glasses are disconnected from the control box, the control box detects that the glasses are connected to the control box via a data cable, and receives multiple synchronization signals periodically sent from the glasses. The first time difference between the first time when the glasses send a first synchronization signal (any one of the multiple synchronization signals) and the second time when the control box receives the first synchronization signal is calculated, and the first time difference is the time difference corresponding to the first synchronization signal. The set algorithm is applied to optimize the first time difference, and the optimized first time difference is applied to adjust the first time to obtain the target time. The image frame corresponding to the target time is sent to the glasses so that the glasses display the image frame. Such a design can ensure the time synchronization of the glasses and the control box, thereby avoiding the problem of screen tearing caused by plugging and unplugging the glasses, and realizing synchronous display of the screen.

[0062] In S201, when the glasses are disconnected from the control box, it is detected that the glasses are connected to the control box via the data line, and at this time, the glasses periodically send a plurality of synchronization signals to the control box, and the synchronization signals are Vsync signals.

[0063] In S202, after the glasses are connected to the control box via the data line, the glasses periodically send multiple synchronization signals to the control box. The control box receives the multiple synchronization signals from the glasses. The synchronization signal is numbered with k, and k can be 0, 1, 2, 3, or 4.

[0064] In S203, for multiple synchronization signals, the time differences corresponding to the multiple synchronization signals can be calculated respectively, and the time difference can represent the time delay between a control box receiving the synchronization signal and the glasses sending the synchronization signal. Next, take any synchronization signal as an example to illustrate the calculation process of the time difference. Among them, any synchronization signal can be represented by the first synchronization signal, and the corresponding time difference is represented by the first time difference; or any synchronization signal can be represented by the synchronization signal k, and the corresponding time difference is represented by △T k express.

[0065] Exemplarily, the first time difference is a time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives a second synchronization signal.

[0066] The two ends of the data line are respectively referred to as the first end and the second end, and the end of the data line connected to the glasses is the first end. The time when the serial port signal receiver at the first end of the data line receives the first synchronization signal can be recorded as the first time. For another example, the time when the first synchronization signal is received by the rendering toolkit is recorded as the second time.

[0067] For example, if k=1, then Teye1=Teye0+t; if k=2, then Teye2=Teye0+2t; if k=3, then Teye3=Teye0+3t; if k=4, then Teye4=Teye0+4t. Teye0 can be, for example, 1000ms, that is, the first synchronization signal is sent 1000ms after the timer starts counting. The transmission period of the synchronization signal is t, which can be, for example, 6ms.

[0068] Based on the method of calculating the first time difference, the time differences corresponding to the multiple synchronization signals can be obtained. As above, the five time differences obtained are △T0, △T1, △T2, △T3, and △T4, which can take values ​​of 0.1ms, 0.12ms, 0.11ms, 0.15ms, and 0.14ms respectively.

[0069] In the above embodiment, the first time difference is obtained by subtracting the second time when the control box receives the first synchronization signal from the first time when the glasses send the first synchronization signal. Figure 3 The steps in calculating the first time difference are:

[0070] S301: Recording the time when the first synchronization signal is received by the serial port signal receiver at the first end of the data line Teye k .

[0071] S302: Record the time Tbox when the control box drive interface HIDRAW receives the first synchronization signal k .

[0072] S303: Calculate the first delay as △box k =Tbox k -Teye k .

[0073] Among them, the first delay represents the control box driving delay.

[0074] S304: Record the time Tapk when the first synchronization signal is received by the framework layer data receiving service or application of the Android system k .

[0075] Among them, the file format of the framework layer data receiving service or application of the Android system is APK.

[0076] S305: Calculate the second delay as △apk k =Tapk k -Tbox k .

[0077] The second delay represents the APK service or application reception delay.

[0078] S306: Calculate the time Tsdk when the first synchronization signal is received by SDK k .

[0079] S307: Calculate the third delay as △sdk k =Tsdk k -Tapk k .

[0080] Among them, the third delay represents the SDK receiving delay.

[0081] Figure 4 A schematic diagram of various delays in the process of a control box receiving a first synchronization signal provided in an embodiment of the present application.

[0082] Therefore, the total delay △T can be obtained k =△box k+△apk k +△sdk k , the total delay is the first time difference. By calculating the first time difference in this way, the delay of the first synchronization signal at each stage in the transmission process can be understood. Figure 5 A schematic diagram of the delay of a synchronization signal k at each stage in the transmission process provided in an embodiment of the present application.

[0083] In S204, in order to better improve the tearing phenomenon, after obtaining the first time difference, a setting algorithm is applied to optimize the first time difference. The setting algorithm is an improved Kalman filter algorithm, that is, the Kalman filter algorithm is applied to estimate each time difference, each time difference can be called an observation or an observation value, the set of each time difference is called an observation data set, and the estimated result is called an optimized time difference.

[0084] In the Kalman filter algorithm, for each observation (such as observation △Tk), the prior estimate X of the observation is calculated. k|k-1 , the posterior estimate X k|k , Kalman filter gain K k , prior error covariance P k|k-1 and the posterior error covariance P k|k , and the posterior estimate is the result of the estimation. In a specific example, the observation data set is [△T0, △T1, △T2, △T3, △T4], for example, the values ​​are [0.1ms, 0.12ms, 0.11ms, 0.15ms and 0.14ms].

[0085] In the process of calculating the above-mentioned posterior estimates, the process noise covariance and the observation noise covariance are also used. In one method, the process noise covariance and the observation noise covariance are both fixed values, regardless of the value of k. For example, the fixed value of the process noise covariance (called the set process noise covariance) is Q = 0.00001; the fixed value of the observation noise covariance (called the set observation noise covariance) is R = 0.01. In another method, the process noise covariance and the observation noise covariance are dynamically updated. For example, when k = 0 and k = 1, the process noise covariance Q0 = Q, Q1 = Q, and the observation noise covariance R0 = R, R1 = R; starting from k = 2, Q2 is obtained by updating Q1.

[0086] Next, the optimization process of the first time difference is described by taking the first synchronization signal as an example. The first synchronization signal is represented by synchronization signal k, and the first time difference is △T k .

[0087] In the first case, k=0, that is, the first synchronization signal is the first synchronization signal among multiple synchronization signals.

[0088] In this case, since there is no historical observation value as a reference, the first time difference can be directly not optimized, that is, the first time difference is used as the optimized first time difference. In addition, in order to provide data support for the case of k>0, the prior estimate, prior error covariance and posterior error covariance in this case can be calculated according to the setting method.

[0089] For example, the process noise covariance Q0 = Q = 0.00001, the observation noise covariance R0 = R = 0.01; the prior estimate X 0|-1 =X 0|0 =△T0=0.1; prior error covariance P 0|-1 =P 0|0 +Q0=1.0+0.00001=1.00001; posterior error covariance P 0|0 The value is usually 1. The setting value of process noise covariance is Q = 0.00001.

[0090] In the second case, k is greater than 0, that is, the first synchronization signal is not the first synchronization signal among the multiple synchronization signals.

[0091] In this case, the optimization process can be Figure 6 Steps S204-1 to S204-5 in are implemented.

[0092] S204-1: Using the posterior estimation value of the second time difference as the prior estimation value of the first time difference.

[0093] The second time difference is the time difference corresponding to the previous cycle of the first time difference, the synchronization signal of the current cycle is synchronization signal k, and the synchronization signal of the previous cycle is synchronization signal k-1. Exemplarily, the prior estimation value of the first time difference can be calculated according to the following formula: k|k-1 =X k-1|k-1 Among them, X k|k-1 is the prior estimate of the first time difference, X k-1|k-1 is the a posteriori estimate of the second time difference.

[0094] For example, k = 1, then X 1|0 =X 0|0 =△T0=0.1; k=2, then X 2|1 =X 1|1 .

[0095] S204-2: Calculate the a priori error covariance of the first time difference according to the a posteriori error covariance of the second time difference and the process noise covariance of the first time difference.

[0096] Exemplarily, the a priori error covariance of the first time difference may be calculated according to the following formula: k|k-1 =Pk-1|k-1 +Q k Among them, P k|k-1 is the a priori error covariance of the first time difference, P k-1|k-1 is the posterior error covariance of the second time difference, Q k is the process noise covariance of the first time difference.

[0097] For example, k = 1, then P 1|0 =P 0|0 +Q1=1.0+0.00001=1.00001.

[0098] S204-3: Calculate the Kalman gain of the first time difference according to the a priori error covariance of the first time difference and the observation noise covariance of the first time difference.

[0099] Exemplarily, the Kalman gain of the first time difference can be calculated according to the following formula: K k =P k|k-1 / (P k|k-1 +R k ). Among them, K k is the Kalman gain of the first time difference, P k|k-1 is the a priori error covariance of the first time difference, R k is the observation noise covariance of the first time difference.

[0100] For example, k = 1, K1 = P 1|0 / P 1|0 +R1)=1.00001 / 1.00001+0.01≈0.9901.

[0101] S204-4: Obtain a posterior estimate of the first time difference according to the Kalman gain of the first time difference, the a priori estimate of the first time difference, and the first time difference.

[0102] Exemplarily, the posterior estimate of the first time difference may be calculated according to the following formula: k|k =X k|k-1 +△T k Among them, X k|k is the a posteriori estimate of the first time difference, △T k The first time difference.

[0103] For example, k = 1, X 1|1 =X 1|0 +K1(△T1-X 1|0 )=0.1+0.9901*(0.12-0.1)=0.1198.

[0104] In addition, the a posteriori error covariance of the first time difference may also be determined according to the Kalman gain of the first time difference and the a priori error covariance of the first time difference. Exemplarily, the a posteriori error covariance of the second time difference may be calculated by the following formula: k|k =(1-K k ) k|k-1 Among them, P k|k is the posterior error covariance of the first time difference, K k is the Kalman gain of the first time difference, P k|k-1 is the a priori error covariance of the first time difference.

[0105] S204-5: Determine that the posterior estimation value of the first time difference is an optimization result of the first time difference.

[0106] In the embodiment of the present application, the method of dynamically updating the process noise covariance and the observation noise covariance can better improve the optimization effect. Therefore, the update process is described for the two respectively.

[0107] The updating process of process noise covariance is as follows:

[0108] As in the above embodiment, when k=1, Q1=Q0 is used, that is, no update is performed and the set value is taken. If k is greater than 1, the first setting parameter α, the process noise covariance Q of the second time difference can be used. k-1 , the second time difference prior estimate and the posterior estimate X k|k-1 =X k-1|k-1 , calculate the process noise covariance of the first time difference, that is, Q k =αQ k-1 +(1-α)(X k-1|k-1 –X k-1|k-2 )(X k-1|k-1 –X k-1|k-2 ).

[0109] For example, k = 2, then Q2 = αQ1 + (1-α)(X 1|1 –X 1|0) )(X 1|1 –X 1|0 )=0.9*0.00001+0.1*(0.1198–0.1)(0.1198–0.1)=4.8601*0.00001.

[0110] For example, k = 3, then Q3 = αQ2 + (1-α)(X 2|2 –X 2|1 )(X 2|2 –X 2|1)=0.9*4.8601*0.00001+0.1*(0.1143–0.1198)(0.1143–0.1198)=4.67659*0.00001.

[0111] The updating process of the observation noise covariance is as follows:

[0112] As in the above embodiment, when k=1, R1=R0 is used, that is, no update is performed and the set value is taken.

[0113] k is greater than 1, then according to the second setting parameter β, the observation noise covariance R of the second time difference k-1 , the first time difference △T1, the posterior estimate of the second time difference, and calculate the observation noise covariance of the first time difference, that is, R k =βR k-1 +(1-β)(△T k –X k-1|k-2 )(△T k –X k-1|k-2 ).

[0114] For example, k = 2, then R2 = βR0 + (1-β)(△T1 – X 1|0 )(△T1–X 1|0 )=0.9*0.01+0.1*(0.12-0.1)(0.12-0.1)=0.009+0.00004=0.00904.

[0115] For example, k = 3, then R3 = βR2 + (1-β)(△T2 – X 2|1 )(△T2–X 2|1 )=0.9*0.00904+0.1*(0.11-0.1198)(0.11-0.1198)=0.0081456.

[0116] Among them, α and β are smoothing factors. The value range of α is between (0,1). The larger the α, the stronger the dependence on the historical process noise covariance, and the weaker the adaptability to the estimated observation data; the value range of β is between (0,1). The larger the β, the stronger the dependence on the historical observation noise covariance, and the weaker the adaptability to the estimated observation data. Conversely, the smaller the α and β, the stronger the adaptability to new data, but it may cause greater fluctuations in the noise covariance.

[0117] In the actual application process, α and β can first select an initial value of 0.95, which means that they are highly dependent on the historical process noise covariance and the historical observation noise covariance. The final α and β can be determined by experimental adjustment and cross-validation, for example, 0.9. Experimental adjustment can be to run the filtering algorithm and observe the filtering results (especially the noise level and delayed response). When it is found that the filter reacts too slowly to new data, α and β can be gradually reduced. In cross-validation, the data set is split, cross-validation is performed, and the errors under different α and β are calculated, so as to select the parameter combination with the smallest error.

[0118] In summary, in order to better illustrate the embodiments of the present application, the calculation processes of the estimated values ​​and covariances when k=0, 1, and 2 are summarized respectively.

[0119] (1) k = 0

[0120] Process noise covariance Q0 = Q = 0.00001;

[0121] Observation noise covariance R0 = R = 0.01;

[0122] A priori estimate of X 0|-1 =X 0|0 =△T0=0.1;

[0123] Prior error covariance P 0|-1 =P 0|0 +Q0=1.0+0.00001=1.00001;

[0124] Posterior error covariance P 0|0 The value is usually 1.

[0125] (2) k = 1

[0126] Process noise covariance Q1 = Q = 0.00001;

[0127] Observation noise covariance R1 = R = 0.01;

[0128] Prior estimate X 1|0 =X 0|0 =0.1;

[0129] Prior error covariance P 1|0 =P 0|0 +Q1=1.0+0.00001=1.00001;

[0130] Kalman gain K1 = P 1|0 / P 1|0 +R1=1.00001 / 1.00001+0.01≈0.9901;

[0131] The posterior estimate X1|1 =X 1|0 +K1(△T1-X 1|0 )=0.1+0.9901*(0.12-0.1)=0.1198;

[0132] Posterior error covariance P 1|1 =(1-K1)P 1|0 =(1-0.9901)*1.00001≈0.0099.

[0133] (3) k = 2

[0134] It can be divided into two cases: fixed process noise covariance and observation noise covariance, and updated process noise covariance and observation noise covariance.

[0135] If fixed process noise covariance and observation noise covariance are used, the process noise covariance is Q2=Q=0.00001, and the observation noise covariance is R2=R=0.01.

[0136] Prior estimate X 2|1 =X 1|1 =0.1198;

[0137] Prior error covariance P 2|1 =P 1|1 +Q1=0.0099+0.00001=0.00991;

[0138] Kalman gain K2 = P 2|1 / P 2|1 +R1=0.00991 / 0.00991+0.01≈0.4978;

[0139] The posterior estimate X 2|2 =X 2|1 +K2(△T2-X 2|1 )=0.1198+0.4978*(0.11-0.1198)=0.1149;

[0140] Posterior error covariance P 2|2 =(1-K2)P 2|1 =(1-0.4978)*0.00991≈0.00498.

[0141] Alternatively, estimates for the case where the process noise covariance and observation noise covariance are fixed can be found in Table 1.

[0142] Table 1 Estimated values ​​for fixed process noise covariance and observation noise covariance

[0143]

[0144] If the dynamically updated process noise covariance and observation noise covariance are used, the process noise covariance Q2=4.8601*0.00001, and the observation noise covariance R2=0.00904.

[0145] Prior estimate X 2|1 =X 1|1 =0.1198;

[0146] Prior error covariance P 2|1 =P 1|1 +Q1=0.0099+4.8601*0.00001=0.009948601;Kalman gain K2=P 2|1 / P 2|1 +R2=0.00991 / 0.00991+0.00904≈0.5241;

[0147] The posterior estimate X 2|2 =X 2|1 +K2(△T2-X 2|1 )=0.1198+0.5241*(0.11-0.1198)=0.1143;

[0148] Posterior error covariance P 2|2 =(1-K2)P 2|1 =(1-0.5241)*0.009948601≈0.004986.

[0149] Optionally, the estimated values ​​for the updated process noise covariance and observation noise covariance can be found in Table 2.

[0150] Table 2 Updated estimates of process noise covariance and observation noise covariance

[0151]

[0152] In S204, the first time is adjusted by applying the optimized first time difference to obtain the target time.

[0153] Taking k=2 as an example, under the condition of fixed process noise covariance and observation noise covariance, the first time difference after optimization is 0.1149, and the target time is Teye2+2t+0.1149. Taking t as 6ms as an example, the target time is 1006.1149.

[0154] Taking k=2 as an example, under the updated process noise covariance and observation noise covariance, the first time difference after optimization is 0.00143, and the target time is Teye2+2t+0.1143. Taking t as 6ms as an example, the target time is 1006.1143.

[0155] In S205, the image frame corresponding to the target time is sent to the glasses so that the glasses can display it.

[0156] Among them, the target time is the calibrated timestamp. Under the updated process noise covariance and observation noise covariance, the image frame corresponding to the 1006.1143ms can be sent to the glasses for display.

[0157] Figure 7 A schematic diagram of sending N image frames to glasses for display is provided in an embodiment of the present application, wherein N is an integer greater than 1 and is not specifically limited.

[0158] The above embodiment is illustrated by taking a synchronization signal as an example. In the adaptive application process, the improved Kalman filter algorithm will be calibrated instantly each time new data to be observed arrives, without waiting for multiple data points. By improving the filtering algorithm and adding smoothing processing, the parameters of the filter can be dynamically adjusted to adapt to different delays and jitter conditions, reduce the fluctuation of the timestamp offset, improve the estimation accuracy of the timestamp, and reduce the screen tearing phenomenon, thereby improving the user experience of AR glasses and achieving more accurate time synchronization.

[0159] In order to make the technical solution of the application more perfect, Figure 8 A signaling diagram of a synchronous display method of AR split glasses provided in an embodiment of the present application, Figure 8 At least the following steps are included:

[0160] S801: When the glasses are disconnected from the control box, the control box detects that the glasses are connected to itself via the data cable.

[0161] S802: The glasses periodically send multiple synchronization signals to the control box.

[0162] S803: The control box calculates a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal.

[0163] S804: The control box optimizes the first time difference by applying a set algorithm, and adjusts the first time by applying the optimized first time difference to obtain a target time.

[0164] S805: The control box sends the image frame corresponding to the target time to the glasses.

[0165] S806: The glasses display the image frame.

[0166] The implementation method of each step can be found in the above-mentioned embodiment and will not be repeated here.

[0167] The embodiment of the present application improves the native Android solution time signal transmission method and timestamp calculation method to adapt to the AR split glasses signal transmission method, thereby significantly reducing synchronization delay and improving the real-time performance and accuracy of display. It can ensure that the AR glasses can quickly synchronize the timestamp signal after the plug-in operation to maintain the stability of the picture.

[0168] like Fig. 9 As shown, based on the same inventive concept, an embodiment of the present application provides a synchronous display device for AR split glasses, which includes a detection unit 91, a transmission unit 92 and a processing unit 93.

[0169] The detection unit 91 is used to: detect that the glasses are connected to the control box via the data line when the glasses are disconnected from the control box;

[0170] The transmission unit 92 is used to: receive multiple synchronization signals periodically sent from the glasses;

[0171] The processing unit 93 is used to calculate a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal; wherein the synchronization signal is any one of a plurality of synchronization signals; each synchronization signal corresponds to a time difference; and the first time difference is a time difference corresponding to the first synchronization signal;

[0172] The processing unit 93 is further configured to: optimize the first time difference by applying a set algorithm, and adjust the first time by applying the optimized first time difference to obtain a target time;

[0173] The transmission unit 92 is further configured to send the image frame corresponding to the target time to the glasses, so that the glasses display the image frame.

[0174] In an optional implementation, the processing unit 93 is further configured to:

[0175] The time when the serial port signal receiver at the first end of the data line receives the first synchronization signal is recorded as the first time; wherein the first end of the data line is one end of the data line connected to the glasses.

[0176] In an optional implementation, the processing unit 93 is further configured to:

[0177] The time when the first synchronization signal is received through the rendering toolkit is recorded as the second time.

[0178] In an optional implementation, the algorithm is set to an improved Kalman green wave algorithm; if the first synchronization signal is not the first synchronization signal among the multiple synchronization signals, the processing unit 93 is specifically configured to:

[0179] Using the posterior estimation value of the second time difference as the prior estimation value of the first time difference; wherein the second time difference is the time difference of the previous cycle of the first time difference;

[0180] Calculating the a priori error covariance of the first time difference according to the a posteriori error covariance of the second time difference and the process noise covariance of the first time difference; wherein the a posteriori error covariance of the second time difference is determined according to the Kalman gain of the second time difference and the a priori error covariance of the second time difference;

[0181] Calculating the Kalman gain of the first time difference according to the a priori error covariance of the first time difference and the observation noise covariance of the first time difference;

[0182] Obtaining a posterior estimate of the first time difference according to the Kalman gain of the first time difference, the a priori estimate of the first time difference, and the first time difference;

[0183] Determining the posterior estimate of the first time difference is an optimization result of the first time difference.

[0184] In an optional implementation, if the first synchronization signal is not the first synchronization signal among the multiple synchronization signals and is not the second synchronization signal, the processing unit 93 is further configured to:

[0185] The process noise covariance of the first time difference is calculated according to the first setting parameter, the process noise covariance of the second time difference, the a priori estimate value and the a posteriori estimate value of the second time difference.

[0186] In an optional implementation, if the first synchronization signal is the second synchronization signal among the multiple synchronization signals, the processing unit 93 is further configured to:

[0187] A determination is made setting the process noise covariance to be the process noise covariance of the first time difference.

[0188] In an optional implementation, if the first synchronization signal is not the first synchronization signal among the multiple synchronization signals and is not the second synchronization signal, the processing unit 93 is further configured to:

[0189] The observed noise covariance of the first time difference is calculated according to the second setting parameter, the observed noise covariance of the second time difference, the first time difference, and the a posteriori estimate of the second time difference.

[0190] In an optional implementation, if the first synchronization signal is the second synchronization signal among the multiple synchronization signals, the processing unit 93 is further configured to:

[0191] It is determined to set the observation noise covariance to the observation noise covariance of the first time difference.

[0192] In an optional implementation, if the first synchronization signal is the first synchronization signal among multiple first synchronization signals, the processing unit 93 is further configured to:

[0193] Determine that the a priori estimate value and the a posteriori estimate value of the first synchronization signal are both the first time difference;

[0194] Determine a posterior error covariance of the first synchronization signal as a set error covariance;

[0195] The a priori error covariance of the first synchronization signal is determined to be the sum of the error covariance and the set process noise covariance.

[0196] Since the device is the device in the method in the embodiment of the present application, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0197] like Fig.10 As shown, based on the same inventive concept, an embodiment of the present application provides an AR split glasses, which includes glasses, a control box and a data cable; the data cable is used to connect the glasses and the control box; the method is applied to the control box; the control box includes a processor 101 and a memory 102.

[0198] The processor 101 is configured to execute:

[0199] When the glasses are disconnected from the control box, it is detected that the glasses are connected to the control box via a data cable;

[0200] Receiving multiple synchronization signals periodically sent from the glasses;

[0201] Calculate a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal; wherein the synchronization signal is any one of a plurality of synchronization signals; each synchronization signal corresponds to a time difference; and the first time difference is the time difference corresponding to the first synchronization signal;

[0202] Applying a set algorithm to optimize the first time difference, and applying the optimized first time difference to adjust the first time to obtain a target time;

[0203] sending the image frame corresponding to the target time to the glasses so that the glasses display the image frame;

[0204] The memory 102 is configured to execute:

[0205] Store multiple image frames.

[0206] In an optional implementation, the processor 101 is further configured to:

[0207] The time when the serial port signal receiver at the first end of the data line receives the first synchronization signal is recorded as the first time; wherein the first end of the data line is one end of the data line connected to the glasses.

[0208] In an optional implementation, the processor 101 is further configured to:

[0209] The time when the first synchronization signal is received through the rendering toolkit is recorded as the second time.

[0210] In an optional implementation, the algorithm is set to an improved Kalman green wave algorithm; if the first synchronization signal is not the first synchronization signal among the multiple synchronization signals, the processor 101 is specifically configured to:

[0211] Using the posterior estimation value of the second time difference as the prior estimation value of the first time difference; wherein the second time difference is the time difference of the previous cycle of the first time difference;

[0212] Calculating the a priori error covariance of the first time difference according to the a posteriori error covariance of the second time difference and the process noise covariance of the first time difference; wherein the a posteriori error covariance of the second time difference is determined according to the Kalman gain of the second time difference and the a priori error covariance of the second time difference;

[0213] Calculating the Kalman gain of the first time difference according to the a priori error covariance of the first time difference and the observation noise covariance of the first time difference;

[0214] Obtaining a posterior estimate of the first time difference according to the Kalman gain of the first time difference, the a priori estimate of the first time difference, and the first time difference;

[0215] Determining the posterior estimate of the first time difference is an optimization result of the first time difference.

[0216] In an optional implementation, if the first synchronization signal is not the first synchronization signal among the multiple synchronization signals and is not the second synchronization signal, the processor 101 is further configured to:

[0217] The process noise covariance of the first time difference is calculated according to the first setting parameter, the process noise covariance of the second time difference, the a priori estimate value and the a posteriori estimate value of the second time difference.

[0218] In an optional implementation, if the first synchronization signal is the second synchronization signal among the multiple synchronization signals, the processor 101 is further configured to:

[0219] A determination is made setting the process noise covariance to be the process noise covariance of the first time difference.

[0220] In an optional implementation, if the first synchronization signal is not the first synchronization signal among the multiple synchronization signals and is not the second synchronization signal, the processor 101 is further configured to:

[0221] The observed noise covariance of the first time difference is calculated according to the second setting parameter, the observed noise covariance of the second time difference, the first time difference, and the a posteriori estimate of the second time difference.

[0222] In an optional implementation, if the first synchronization signal is the second synchronization signal among the multiple synchronization signals, the processor 101 is further configured to:

[0223] It is determined to set the observation noise covariance to the observation noise covariance of the first time difference.

[0224] In an optional implementation, if the first synchronization signal is the first synchronization signal among multiple first synchronization signals, the processor 101 is further configured to:

[0225] Determine that the a priori estimate value and the a posteriori estimate value of the first synchronization signal are both the first time difference;

[0226] Determine a posterior error covariance of the first synchronization signal as a set error covariance;

[0227] The a priori error covariance of the first synchronization signal is determined to be the sum of the error covariance and the set process noise covariance.

[0228] An embodiment of the present application further provides a computer storage medium, in which computer program instructions are stored. When the instructions are executed on a computer, the computer executes the steps of the above-mentioned method for synchronous display of AR split glasses.

[0229] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0230] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0231] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0232] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0233] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A synchronous display method for AR split glasses, characterized in that: The AR split glasses include glasses, a control box and a data cable; the data cable is used to connect the glasses and the control box; The method is applied to the control box; the method comprises: When the glasses are disconnected from the control box, detecting that the glasses are connected to the control box via the data cable; receiving a plurality of synchronization signals periodically sent from the glasses; Calculate a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal; wherein the first synchronization signal is any one of the multiple synchronization signals; each synchronization signal corresponds to a time difference; the first time difference is the time difference corresponding to the first synchronization signal; Applying a set algorithm to optimize the first time difference, and applying the optimized first time difference to adjust the first time to obtain a target time; The image frame corresponding to the target time is sent to the glasses, so that the glasses display the image frame.

2. The method according to claim 1, characterized in that The method further comprises: The time when the serial port signal receiver at the first end of the data line receives the first synchronization signal is recorded as the first time; wherein the first end of the data line is one end of the two ends of the data line connected to the glasses.

3. The method according to claim 1, characterized in that The method further comprises: The time when the first synchronization signal is received by the rendering toolkit is recorded as the second time.

4. The method according to claim 1, characterized in that The setting algorithm is an improved Kalman green wave algorithm; if the first synchronization signal is not the first synchronization signal among the multiple synchronization signals, then optimizing the first time difference includes: Using the posterior estimation value of the second time difference as the prior estimation value of the first time difference; wherein the second time difference is the time difference of the previous cycle of the first time difference; Calculating the a priori error covariance of the first time difference according to the a posteriori error covariance of the second time difference and the process noise covariance of the first time difference; wherein the a posteriori error covariance of the second time difference is determined according to the Kalman gain of the second time difference and the a priori error covariance of the second time difference; Calculating the Kalman gain of the first time difference according to the a priori error covariance of the first time difference and the observation noise covariance of the first time difference; Obtaining a posterior estimate of the first time difference according to the Kalman gain of the first time difference, the a priori estimate of the first time difference, and the first time difference; Determining the a posteriori estimate of the first time difference is an optimization result of the first time difference.

5. The method according to claim 4, characterized in that If the first synchronization signal is not the first synchronization signal and not the second synchronization signal among the multiple synchronization signals, the method further includes: The process noise covariance of the first time difference is calculated according to the first setting parameter, the process noise covariance of the second time difference, the a priori estimate value and the a posteriori estimate value of the second time difference.

6. The method according to claim 4, characterized in that If the first synchronization signal is the second synchronization signal among the multiple synchronization signals, the method further includes: A process noise covariance is determined and set to be the process noise covariance of the first time difference.

7. The method according to claim 4, characterized in that If the first synchronization signal is not the first synchronization signal and not the second synchronization signal among the multiple synchronization signals, the method further includes: The observed noise covariance of the first time difference is calculated according to a second setting parameter, the observed noise covariance of the second time difference, the first time difference, and a posterior estimation value of the second time difference.

8. The method according to claim 4, characterized in that If the first synchronization signal is the second synchronization signal among the multiple synchronization signals, the method further includes: It is determined to set the observation noise covariance to the observation noise covariance of the first time difference.

9. The method according to any one of claims 1 to 8, characterized in that: If the first synchronization signal is the first synchronization signal among the multiple first synchronization signals, the method further includes: Determining that a priori estimated value and a posteriori estimated value of the first synchronization signal are both the first time difference; Determining a posteriori error covariance of the first synchronization signal as a set error covariance; The a priori error covariance of the first synchronization signal is determined to be the sum of the error covariance and a set process noise covariance.

10. A kind of AR split glasses, characterized in that: The AR split glasses include glasses, a control box and a data cable; the data cable is used to connect the glasses and the control box; the method is applied to the control box; the control box includes a processor and a memory; The processor is configured to execute: When the glasses are disconnected from the control box, detecting that the glasses are connected to the control box via the data cable; receiving a plurality of synchronization signals periodically sent from the glasses; Calculate a first time difference between a first time when the glasses send a first synchronization signal and a second time when the control box receives the first synchronization signal; wherein the first synchronization signal is any one of the multiple synchronization signals; each synchronization signal corresponds to a time difference; the first time difference is the time difference corresponding to the first synchronization signal; Applying a set algorithm to optimize the first time difference, and applying the optimized first time difference to adjust the first time to obtain a target time; Sending the image frame corresponding to the target time to the glasses, so that the glasses display the image frame; The memory is configured to execute: Store multiple image frames.

Images