Delay Measurement Method and System, Electronic Device, Storage Medium, Program Product

Abstract

Embodiments of the present disclosure provide a delay measurement method, a delay measurement system, an electronic device, a computer-readable storage medium, and a computer program product. The delay measurement system includes: a motion platform, a vision platform, and a processing platform. The motion platform is fixedly connected to the extended reality device to be measured and drives the extended reality device to be measured to move synchronously. The vision platform is fixedly connected to the extended reality device to be measured and acquires a plurality of consecutive images of a calibration board displayed via the extended reality device to be measured. The processing platform obtains a visual pose sequence of the extended reality device to be measured based on the plurality of consecutive images, and determines the delay of the extended reality device to be measured based on the visual pose sequence and the motion pose sequence of the motion platform.

Description

Technical Field

[0001] The present disclosure relates to the technical field of extended reality devices, and particularly to a latency measurement method, a latency measurement system, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of science and technology, higher requirements are put forward for the performance of extended reality devices such as virtual reality devices, augmented reality devices, and mixed reality devices.

[0003] Taking a head-mounted display (HMD) as an example, an extended reality device needs to respond to the user's head movement in real time. Although the user's head movement can be measured by various sensors, due to the limitations of transmission and calculation time, the HMD cannot immediately display the corresponding image to the user, which will cause image latency. This latency will cause a difference between visual motion perception and vestibular motion perception, which will not only reduce the sense of presence of the extended reality device, but also increase the burden on the brain and cause dizziness and nausea. The MTP (motion-to-photon) latency can include the time from the start of the user's movement to the corresponding image being displayed on the screen, which characterizes the delay time between the image seen by the user in an extended reality device such as an HMD and the user's head movement, and can quantitatively represent the matching degree between visual observation and the user's head movement. Therefore, for the user, the smaller the MTP latency, the better the user's immersion; the larger the MTP latency, the stronger the user's sense of dizziness.

[0004] Considering that the human head movement is mostly continuous three-dimensional (three-axis) composite movement, for an extended display device that supports a motion prediction algorithm, analyzing the delay change in the continuous composite three-dimensional movement is a key indicator for evaluating its motion prediction algorithm.

[0005] Therefore, how to accurately measure the MTP latency of an extended reality device in the continuous composite three-dimensional movement of the human body has become one of the research hotspots in the technical field of extended reality devices. Summary of the Invention

[0006] Embodiments of the present disclosure propose a latency measurement method, a latency measurement system, an electronic device, a computer-readable storage medium, and a computer program product, which realize accurate measurement of the MTP latency of an extended reality device in the continuous composite three-dimensional movement of the human body.

[0007] In a first aspect, an embodiment of the present disclosure provides a delay measurement system, which includes: a motion platform fixedly connected to the extended reality device to be measured and driving the extended reality device to be measured to move synchronously; a vision platform fixedly connected to the extended reality device to be measured and acquiring a plurality of consecutive images of a calibration board displayed by the extended reality device to be measured; and a processing platform obtaining a visual pose sequence of the extended reality device to be measured based on the plurality of consecutive images, and determining a delay of the extended reality device to be measured based on the visual pose sequence and a motion pose sequence of the motion platform.

[0008] In some embodiments of the present disclosure, the processing platform further includes: a splitting unit splitting the visual pose sequence and the motion pose sequence into a plurality of motion segments respectively according to motion characteristic parameters of the synchronous motion; a fitting unit fitting the plurality of motion segments of the visual pose sequence into a plurality of first time-pose curves respectively, and fitting the plurality of motion segments of the motion pose sequence into a plurality of second time-pose curves respectively; and a delay unit determining a delay of the extended reality device to be measured based on a difference between the first time-pose curve and the second time-pose curve corresponding to the first time-pose curve in time.

[0009] In some embodiments of the present disclosure, the motion characteristic parameters of the synchronous motion include at least one of a speed of the synchronous motion, an acceleration of the synchronous motion, and a derivative of the acceleration of the synchronous motion.

[0010] In some embodiments of the present disclosure, the delay unit further includes: a first calculation sub-unit determining a first statistical index based on a difference between the first time-pose curve and the second time-pose curve corresponding to the first time-pose curve in time, where the first statistical index includes at least one of a delay extreme value, a delay average value, a delay median value, and a delay standard deviation of the extended reality device to be measured.

[0011] In some embodiments of the present disclosure, the delay unit further includes: a second calculation sub-unit determining a second statistical index according to at least one of a mean value and a standard deviation of the first statistical index obtained by multiple measurements.

[0012] In some embodiments of the present disclosure, the processing platform further includes: a preprocessing unit performing zero-offset preprocessing on the visual pose sequence and the motion pose sequence respectively before fitting the plurality of motion segments of the visual pose sequence and the plurality of motion segments of the motion pose sequence respectively.

[0013] In some embodiments of the present disclosure, the system further includes: a time platform including a clock source, wherein the clock source is used to synchronize the time of the motion platform and the vision platform before and during the synchronous motion.

[0014] In some embodiments of the present disclosure, the processing platform further includes: a first conversion unit configured to extract pixel coordinates of feature points of the calibration board based on the plurality of consecutive images; a second conversion unit configured to convert the pixel coordinates into a sequence of preliminary pose measurements in a first coordinate system referenced to the vision platform according to a preset spatial geometric constraint relationship between the feature points; and a calibration unit configured to convert the sequence of preliminary pose measurements into the vision pose sequence in a second coordinate system referenced to the motion platform.

[0015] In some embodiments of the present disclosure, the processing platform further includes: a time correction unit configured to correct the time of the sequence of preliminary pose measurements before converting the sequence of preliminary pose measurements into the vision pose sequence.

[0016] In some embodiments of the present disclosure, the processing platform further includes: an image processing unit configured to process the plurality of consecutive images before extracting the pixel coordinates of the feature points of the calibration board based on the plurality of consecutive images, and the processing includes at least one of downsampling processing and Gaussian blur processing.

[0017] In some embodiments of the present disclosure, the calibration board is a virtual calibration board, and the virtual calibration board includes a visual marker calibration board in a visual reference library.

[0018] In some embodiments of the present disclosure, the vision pose sequence includes a plurality of first time-pose information arranged in chronological order, the first time-pose information includes first time information and a first pose angle corresponding to the first time information, and the first pose angle includes a first roll angle, a first yaw angle, and a first pitch angle; and the motion pose sequence includes a plurality of second time-pose information arranged in chronological order, the second time-pose information includes second time information and a second pose angle corresponding to the second time information, and the second pose angle includes a second roll angle, a second yaw angle, and a second pitch angle.

[0019] Second aspect, an embodiment of the present disclosure provides a latency measurement method, which includes: obtaining a sequence of motion poses of a motion platform, where the motion platform is fixedly connected to the to-be-tested extended reality device and drives the to-be-tested extended reality device to move synchronously; obtaining a plurality of consecutive images of a calibration board displayed via the to-be-tested extended reality device; obtaining a sequence of visual poses of the to-be-tested extended reality device based on the plurality of consecutive images; and determining the latency of the to-be-tested extended reality device based on the sequence of visual poses and the sequence of motion poses.

[0020] In some embodiments of the present disclosure, determining the latency of the to-be-tested extended reality device based on the sequence of visual poses and the sequence of motion poses includes: splitting the sequence of visual poses and the sequence of motion poses into a plurality of motion segments respectively according to the motion characteristic parameters of the synchronous motion; fitting the plurality of motion segments of the sequence of visual poses into a plurality of first time-pose curves respectively, and fitting the plurality of motion segments of the sequence of motion poses into a plurality of second time-pose curves respectively; and determining the latency of the to-be-tested extended reality device based on the differences between the first time-pose curves and the second time-pose curves corresponding to the first time-pose curves in time.

[0021] In some embodiments of the present disclosure, the motion characteristic parameters of the synchronous motion include at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion.

[0022] In some embodiments of the present disclosure, determining the latency of the to-be-tested extended reality device based on the differences between the first time-pose curves and the second time-pose curves corresponding to the first time-pose curves in time includes: determining a first statistical index based on the differences between the first time-pose curves and the second time-pose curves corresponding to the first time-pose curves in time, where the first statistical index includes at least one of the latency extreme value, latency average value, latency median, and latency standard deviation of the to-be-tested extended reality device.

[0023] In some embodiments of the present disclosure, determining the latency of the to-be-tested extended reality device based on the differences between the first time-pose curves and the second time-pose curves corresponding to the first time-pose curves in time further includes: determining a second statistical index according to at least one of the mean value and standard deviation of the first statistical index obtained through multiple measurements.

[0024] In some embodiments of the present disclosure, before fitting the plurality of motion segments of the sequence of visual poses and the plurality of motion segments of the sequence of motion poses respectively, the method further includes: performing zero-bias preprocessing on the sequence of visual poses and the sequence of motion poses respectively.

[0025] In some embodiments of the present disclosure, the measurement method further includes: performing time synchronization on the motion platform and the vision platform before and during the synchronous motion.

[0026] In some embodiments of the present disclosure, obtaining the visual pose sequence of the to-be-measured extended reality device based on the plurality of consecutive images includes: extracting the pixel coordinates of the feature points of the calibration board based on the plurality of consecutive images; converting the pixel coordinates into a preliminary measurement pose sequence in a first coordinate system with reference to the vision platform that captured the plurality of consecutive images according to the preset spatial geometric constraint relationship between the feature points; and converting the preliminary measurement pose sequence into the visual pose sequence in a second coordinate system with reference to the motion platform.

[0027] In some embodiments of the present disclosure, the measurement method further includes: performing time correction on the preliminary measurement pose sequence before converting the preliminary measurement pose sequence into the visual pose sequence.

[0028] In some embodiments of the present disclosure, the measurement method further includes: processing the plurality of consecutive images before extracting the pixel coordinates of the feature points of the calibration board based on the plurality of consecutive images, and the processing includes at least one of downsampling processing and Gaussian blur processing.

[0029] In some embodiments of the present disclosure, the calibration board is a virtual calibration board, and the virtual calibration board includes a visual marker calibration board in a visual reference library.

[0030] In some embodiments of the present disclosure, the visual pose sequence includes a plurality of first time-pose information arranged in chronological order, the first time-pose information includes first time information and a first pose angle corresponding to the first time information, and the first pose angle includes a first roll angle, a first yaw angle, and a first pitch angle; and the motion pose sequence includes a plurality of second time-pose information arranged in chronological order, the second time-pose information includes second time information and a second pose angle corresponding to the second time information, and the second pose angle includes a second roll angle, a second yaw angle, and a second pitch angle.

[0031] In a third aspect, embodiments of the present disclosure provide an electronic device, which includes: at least one processor; a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the measurement method described in any implementation manner of the second aspect.

[0032] Fourthly, embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to execute a measurement method described in any implementation manner of the second aspect when executed.

[0033] Fifthly, embodiments of the present disclosure provide a computer program product including a computer program, which can implement the measurement method described in any implementation manner of the second aspect when executed by a processor.

[0034] According to the delay measurement system, delay measurement method, electronic device, computer-readable storage medium, and computer program product provided by at least one embodiment of the present disclosure, a motion platform that moves synchronously with the to-be-tested extended reality device can simulate continuous complex three-dimensional motions of a human body. Therefore, it can comprehensively evaluate the delay performance of the extended reality device in different motion states, and can observe the dynamic changes of the extended reality device in continuous complex three-dimensional motions. By using a vision platform, multiple consecutive images of a target board displayed via the to-be-tested extended reality device can be obtained, and then a visual pose sequence of the to-be-tested extended reality device can be determined. By comparing the difference between the visual pose sequence of the to-be-tested extended reality device and the motion pose sequence of the motion platform, the MTP delay of the extended reality device in continuous complex three-dimensional motions of a human body can be accurately measured.

[0035] It should be understood that the content described in this part is not intended to identify the key features or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, objectives, and advantages related to the embodiments of the present disclosure will become more obvious by reading the detailed description of the non-limiting embodiments with reference to the following drawings. Among them:

[0037] Figure 1 Shows a schematic diagram of a delay measurement system according to an exemplary embodiment of the present disclosure;

[0038] Figure 2 Shows a schematic diagram of a to-be-tested extended reality device according to an exemplary embodiment of the present disclosure;

[0039] Figure 3 Shows a schematic diagram of a target board according to an exemplary embodiment of the present disclosure;

[0040] Figure 4 Shows a schematic diagram of one image among multiple consecutive images displayed by a to-be-tested extended reality device according to an exemplary embodiment of the present disclosure;

[0041] Figure 5Shows a schematic diagram of a processing platform according to an exemplary embodiment of the present disclosure;

[0042] Figure 6 Shows a schematic diagram of a processing platform according to an exemplary embodiment of the present disclosure;

[0043] Figure 7 Shows a schematic flowchart of segmenting and processing a visual pose sequence and a motion pose sequence to determine the latency of an extended reality device to be measured according to an exemplary embodiment of the present disclosure;

[0044] Figure 8 Shows a schematic flowchart of generating a first time-pose curve and a second time-pose curve according to an exemplary embodiment of the present disclosure;

[0045] Figure 9 Shows a schematic diagram of a statistical result including a first statistical index and a second statistical index according to an exemplary embodiment of the present disclosure;

[0046] Figure 10 Is a flowchart of a latency measurement method provided by an embodiment of the present disclosure;

[0047] Figure 11 Shows a schematic block diagram of an example electronic device that can be used to implement the embodiments of the present disclosure. Detailed implementation manners

[0048] To better understand the present disclosure, more detailed descriptions of various aspects of the present disclosure will be made with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present disclosure and do not limit the scope of the present disclosure in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0049] It should be noted that in this specification, the expressions such as first, second, etc. are only used to distinguish one feature from another feature and do not represent any limitation on the feature. Therefore, without departing from the teachings of the present disclosure, the first computing subunit discussed below may also be referred to as the second computing subunit, and the first time-pose curve may also be referred to as the second time-pose curve.

[0050] It should also be understood that expressions such as "including", "including having", "having", "containing" and / or "containing having" are open-ended rather than closed-ended expressions in this specification, which means that there are the stated features, elements and / or components, but do not exclude the existence of one or more other features, elements, components and / or their combinations. Additionally, "exemplarily" is used to refer to an example or illustration.

[0051] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood as having a meaning that is consistent with their meaning in the context of the relevant art and should not be interpreted in an idealized or overly formal sense unless expressly so defined in this disclosure.

[0052] It should be noted that, without conflict, the embodiments in this disclosure and the features in the embodiments may be combined with each other. Additionally, unless expressly defined or in contradiction with the context, the specific steps included in the methods described in this disclosure do not have to be limited to the recited order and may be executed in any order or executed in parallel. The following will detail this disclosure with reference to the accompanying drawings and in conjunction with embodiments.

[0053] Figure 1 A schematic diagram of a delay measurement system 1000 according to an exemplary embodiment of this disclosure is shown. Figure 2 A schematic diagram of an extended reality device 400 to be measured according to an exemplary embodiment of this disclosure is shown. Figure 3 A schematic diagram of a target board 410 according to an exemplary embodiment of this disclosure is shown. Figure 4 A schematic diagram of an image 420 among a plurality of consecutive images displayed by the extended reality device 400 to be measured according to an exemplary embodiment of this disclosure is shown.

[0054] As Figures 1 - 4 shown, an embodiment of this disclosure provides a delay measurement system 1000. The delay measurement system 1000 may include a motion platform 100, a vision platform 200, and a processing platform 300. The motion platform 100 is fixedly connected to the extended reality device 400 to be measured and drives the extended reality device 400 to move synchronously. The vision platform 200 is also fixedly connected to the extended reality device 400 to be measured and acquires a plurality of consecutive images of the target board 410 displayed via the extended reality device 400, such as Figure 4 shown an image 420 among a plurality of consecutive images. The processing platform 300 obtains a visual pose sequence of the extended reality device 400 to be measured based on the plurality of consecutive images, and determines the delay of the extended reality device 400 to be measured based on the visual pose sequence and the motion pose sequence of the motion platform 100.

[0055] According to the delay measurement system provided by at least one embodiment of the present disclosure, the motion platform that moves synchronously with the extended reality device to be measured can simulate the continuous complex three-dimensional motion of the human body. Therefore, it can comprehensively evaluate the delay performance of the extended reality device in different motion states, and can observe the dynamic changes of the extended reality device in the continuous complex three-dimensional motion. By using the vision platform, multiple consecutive images of the calibration board displayed via the extended reality device to be measured can be obtained, and then the visual pose sequence of the extended reality device to be measured can be determined. By comparing the difference between the visual pose sequence of the extended reality device to be measured and the motion pose sequence of the motion platform, the MTP delay of the extended reality device in the continuous complex three-dimensional motion of the human body can be accurately measured.

[0056] Specifically, in some embodiments of the present disclosure, the extended reality device 400 to be measured may include a virtual reality device, an augmented reality device, or a mixed reality device. It should be understood that the delay measurement system 1000 can also be applied to the application scenarios of delay measurement of other display devices, which is not limited here.

[0057] The motion platform 100 is fixedly connected to the extended reality device 400 to be measured, such as a rigid fixed connection. In addition, the motion platform 100 can drive the extended reality device 400 to move synchronously. The motion platform 100 can include any suitable structural components and can perform various translational and rotational motions in the world coordinate system. Taking the extended reality device 400 to be measured as an HMD as an example, the motion platform 100 fixedly connected to the HMD can simulate the horizontal movement, horizontal rotation, tilting motion, etc. of the human body.

[0058] The vision platform 200 can be used to simulate the scenario where the human eye receives the continuous images displayed by the extended reality device 400 to be measured. Optionally, the vision platform 200 can include at least one of a camera and a video camera. The camera can include an IR camera (Infrared Camera), an RGB camera (Red Green Blue Camera), or a black and white camera, etc. In addition, the vision platform 200 can also include other structures such as a photoelectric sensor. The present disclosure does not limit the specific internal structure settings of the vision platform 200.

[0059] Optionally, the calibration board 410 can be a virtual calibration board located inside the extended reality device 400 to be measured. The virtual calibration board can include a visual marker calibration board in the visual reference library. For example, QR code calibration boards such as ApriTag and ArUco. Using the virtual calibration board for delay measurement can simplify the structure of the delay measurement system and reduce the implementation difficulty of delay measurement.

[0060] The processing platform 300 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processing platform 300 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processing platform 300 obtains a visual pose sequence of the extended reality device 400 to be measured based on multiple consecutive images, and determines the latency of the extended reality device 400 to be measured based on the visual pose sequence and the motion pose sequence of the motion platform 100.

[0061] Optionally, the processing platform 300 may include various suitable communication buses and input / output interfaces for realizing the connection with the motion platform 100 and the visual platform 200.

[0062] Referring again to Figure 1 , in some embodiments of the present disclosure, the latency measurement system 1000 further includes a time platform 500, and the time platform 500 may include a clock source, where the clock source is used to synchronize the time of the motion platform 100 and the visual platform 200 before and during the synchronous motion.

[0063] Specifically, the clock source can uniformly provide time to the motion platform 100 and the visual platform 200 through the Precision Time Protocol (PTP). Using a clock source with high precision and high accuracy as the master clock, it sends timestamps to the motion platform 100 and the visual platform 200 as slave clocks. The slave clocks compare the timestamps with their local times and adjust the time according to the differences, so as to ensure the time synchronization of the motion pose sequence obtained based on the motion platform 100 and the visual pose sequence obtained based on the visual platform 200.

[0064] Optionally, in combination with Figure 1 and Figure 2 , the visual pose sequence may include a plurality of first time-pose information arranged in chronological order. The first time-pose information includes first time information and a first pose angle corresponding to the first time information, where the first pose angle includes a first roll angle, a first yaw angle, and a first pitch angle. For example, the form of the first time-pose information may include: timestamp_roll_pitch_yaw, where 000_10_20_30 may represent that the roll, pitch, and yaw angle postures at the starting moment are (10°, 20°, 30°) respectively. In a three-dimensional space right-handed Cartesian coordinate system, roll is the roll angle, around the z-axis; pitch is the pitch angle, around the x-axis; yaw is the yaw angle, around the y-axis.

[0065] The motion pose sequence may include a plurality of second time-pose information arranged in chronological order. The second time-pose information may include second time information and a second pose angle corresponding to the second time information, where the second pose angle includes a second roll angle, a second yaw angle, and a second pitch angle. Similarly, the form of the second time-pose information may also include: timestamp_roll_pitch_yaw.

[0066] In order to compare the motion pose changes in the real world with those in the virtual world of the to-be-tested extended reality device 400, it is necessary to unify these two motion pose changes in the same coordinate system. The motion changes in the virtual world can be determined in a first coordinate system (e.g., the camera coordinate system) with the visual platform 200 as a reference, and the motion changes in the real world can be determined in a second coordinate system (e.g., the world coordinate system) with the motion platform 100 as a reference. Therefore, it is necessary to calibrate the coordinate system conversion between the motion platform 100 and the visual platform 200.

[0067] Since the eye relief distances of different to-be-tested extended reality devices are different, it is necessary to adjust the position of the visual platform 200 before measurement to adapt to the to-be-tested extended reality device 400, which causes a change in the relative position between the visual platform 200 and the motion platform 100. The original conversion relationship is no longer applicable and needs to be recalibrated. To solve this problem, a feasible solution is to calibrate the external parameters of the camera or video camera in the visual platform 200 based on the virtual template displayed by the to-be-tested extended reality device captured by the visual platform 200.

[0068] Figure 5 The schematic diagram of the processing platform 300 according to an exemplary embodiment of the present disclosure is shown.

[0069] Specifically, referring to Figure 1 and Figure 5 , in some embodiments of the present disclosure, the processing platform 300 may include: a first conversion unit 320, a second conversion unit 330, and a calibration unit 350. The first conversion unit 320 may extract the pixel coordinates of the feature points of the template 410 (as shown in Figure 3 ) based on a plurality of consecutive images obtained by the visual platform 200. The second conversion unit 330 may convert the pixel coordinates into a preliminary measurement pose sequence in a first coordinate system with the visual platform 200 as a reference according to the preset spatial geometric constraint relationship between the feature points. The calibration unit 350 converts the preliminary measurement pose sequence into a visual pose sequence in a second coordinate system with the motion platform 100 as a reference.

[0070] Optionally, the processing platform 300 may further include an image processing unit 310. The image processing unit 310 may process a plurality of consecutive images before extracting the pixel coordinates of the feature points of the calibration board 410 based on the plurality of consecutive images. The processing includes at least one of downsampling processing and Gaussian blur processing. Since moiré patterns may cause inaccurate extraction of 2D feature points or even failure to extract them when photographing the screen of the to-be-tested extended reality device 400, it is necessary to process the obtained plurality of consecutive images. By performing at least one of downsampling processing and Gaussian blur processing, etc., the influence of moiré patterns on the above images can be reduced.

[0071] In addition, the processing platform 300 may further include a time correction unit 340. The time correction unit 340 may perform time correction on the initial measurement pose sequence before converting the initial measurement pose sequence into a visual pose sequence. Although the timestamps of the images of the calibration board taken above and the trajectory timestamps of the motion platform are under the same clock source, due to the existence of the MTP, there is a certain time delay in the process of taking images, so time correction is required.

[0072] It should be noted that, for clarity, Figure 5 the illustrated processing platform 300 shows both the image processing unit 310 and the time correction unit 340 that it includes in the optional case. It can be distinguished that the optional image processing unit 310 or time correction unit 340 is shown in a dashed box. In addition, in the case of including the image processing unit 310 or the time correction unit 340, the connection between the image processing unit 310 or the time correction unit 340 and other units is also shown by a dashed connection line.

[0073] Optionally, the fixed connection between the motion platform 100 and the to-be-tested extended reality device 400 can be considered as a rigid connection of a rigid body, and the fixed connection between the visual platform 200 and the to-be-tested extended reality device 400 can also be considered as a rigid connection of a rigid body. According to the principle of rotational invariance of a rigid body, time correction can be performed on the initial measurement pose sequence before converting the initial measurement pose sequence into a visual pose sequence. For example, after filtering the angular velocity of the pose, the time delay of the image-taking process caused by the existence of the MTP is obtained based on cross-correlation, and the relevant data is corrected and aligned in time.

[0074] Figure 6 A schematic diagram of a processing platform 300 according to an exemplary embodiment of the present disclosure is shown. Figure 7 A schematic flowchart of segmenting and processing a visual pose sequence and a motion pose sequence to determine the time delay of the to-be-tested extended reality device according to an exemplary embodiment of the present disclosure is shown. Figure 8 A schematic flowchart of generating a first time-pose curve and a second time-pose curve according to an exemplary embodiment of the present disclosure is shown.

[0075] As Figure 1 , Figures 6 - 8 shown, in some embodiments of the present disclosure, the processing platform 300 further includes a splitting unit 370, a fitting unit 380, and a delay unit 390. The splitting unit 370 splits the visual pose sequence and the motion pose sequence into multiple motion segments respectively according to the motion characteristic parameters of the synchronous motion of the motion platform 100 and the extended reality device 400 to be measured (such as Figure 2 shown). The fitting unit 380 fits the multiple motion segments of the visual pose sequence into multiple first time-pose curves respectively, and fits the multiple motion segments of the motion pose sequence into multiple second time-pose curves respectively. The delay unit 390 determines the delay of the extended reality device 400 to be measured based on the difference between the first time-pose curve and the second time-pose curve corresponding to the first time-pose curve in time.

[0076] Specifically, traditional MTP delay measurement can only obtain the delay at the motion start node or the motion end node, and the delay during the motion process cannot be measured. Or rather, traditional MTP measurement can be divided into manual calculation of delay and automated calculation of delay. Manual calculation of delay can calculate the delay of the extended reality device through a numerically estimated value by humans, with a large error and unable to objectively reflect the true delay of the extended display device. Automated calculation of delay measures the delay of the extended reality device through a measuring device. The key lies in capturing the changes in the motion state of the extended reality device and monitoring the changes in the images reflected on the screen of the extended reality device, and obtaining the delay of the extended reality device by comparing the delays between the two. However, automated calculation of delay is limited to measuring the delay generated under a single motion node, such as the delay at the motion start node or the motion end node. In actual use scenarios, the motion of the extended reality device and the response of the virtual reality scene in the extended reality device are a continuous process, and the delay generated during this process is a set of dynamic continuous data. Therefore, only measuring the delay generated under a single motion node cannot reflect the dynamic change of the delay of the extended reality device during continuous motion. Especially considering that human motion is mostly continuous three-dimensional composite motion, for an extended display device that supports a motion prediction algorithm, analyzing the delay change in continuous composite three-dimensional motion is a key indicator for evaluating its motion prediction algorithm.

[0077] According to the delay measurement system provided by at least one embodiment of the present disclosure, the motion platform that moves synchronously with the extended reality device to be measured can simulate continuous complex three-dimensional human motions. Therefore, it can comprehensively evaluate the delay performance of the extended reality device in different motion states, and can observe the dynamic changes of the extended reality device in continuous complex three-dimensional motions. By using the vision platform, multiple consecutive images of the target board displayed via the extended reality device to be measured can be obtained, and then the visual pose sequence of the extended reality device to be measured can be determined. By comparing the difference between the visual pose sequence of the extended reality device to be measured and the motion pose sequence of the motion platform, the MTP delay of the extended reality device in continuous complex three-dimensional human motions can be accurately measured.

[0078] To enhance the above effects and improve the accuracy and precision of MTP delay measurement, the visual pose sequence and the motion pose sequence can be segmented respectively. By using motion segment similarity analysis, the visual pose sequence and the motion pose sequence are respectively split into multiple motion segments; by analyzing each motion segment one by one, fitting line segments for each segment, and performing "gap" mean statistics on the first time-pose curve and the second time-pose curve after fitting, the MTP delay of the extended reality device in continuous complex three-dimensional human motions can be accurately measured.

[0079] Optionally, during the process of using motion segment similarity analysis, the motion characteristic parameters of the synchronous motion may include at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion. Therefore, appropriate synchronous motion characteristic parameters can be selected according to the type of the extended reality device to be measured and actual requirements, or according to the characteristics of the synchronous motion, so as to accurately measure the MTP delay of the extended reality device in continuous complex three-dimensional human motions. It should be noted that the present disclosure does not limit the specific content of the motion characteristic parameters of the synchronous motion.

[0080] In addition, during the process of using motion segment similarity analysis to split the visual pose sequence and the motion pose sequence into multiple motion segments respectively, the visual pose sequence and the motion pose sequence can be split into multiple motion segments according to the motion characteristic parameters of the synchronous motion of the motion platform 100 and the extended reality device 400 to be measured. For example, at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion is regarded as the volatility reference data, and a predetermined threshold is set. When at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion is greater than its corresponding predetermined threshold, it can be considered that the volatility of this part of the data is too large. Based on this, in the visual pose sequence or the motion pose sequence, the previous time-pose information is determined as the end point of the previous motion segment, and the next time-pose information is determined as the start point of the next motion segment.

[0081] Optionally, as Figure 6 shown, the processing platform 300 may further include a preprocessing unit 360. The preprocessing unit 360 may perform zero-bias preprocessing on the visual pose sequence and the motion pose sequence respectively before fitting multiple motion segments of the visual pose sequence and multiple motion segments of the motion pose sequence respectively (such as Figure 7 the "preprocessing" process shown). Considering error factors such as device assembly error and jitter error, performing zero-bias preprocessing on the visual pose sequence and the motion pose sequence respectively can improve the accuracy and precision of MTP delay measurement.

[0082] It should be noted that, for clarity, Figure 6 the shown processing platform 300 shows the preprocessing unit 360 it includes in the optional case. It can be distinguished that the optional preprocessing unit 360 is shown in a dashed box. In addition, the connection situation between the preprocessing unit 360 and other units is shown by a dashed connecting line in the case of including the preprocessing unit 360.

[0083] In the process of analyzing multiple motion segments one by one and performing line fitting segment by segment, the fitting can be based on a straight line or a quadratic curve. It should be noted that the present disclosure does not limit the way of line fitting. After that, the "gap" mean value statistics can be performed on the first time-pose curve and the second time-pose curve after fitting, and the MTP delay of the extended reality device in the continuous composite three-dimensional motion of the human body can be accurately measured.

[0084] Figure 8 shows the process of generating the first time-pose curve and the second time-pose curve according to an exemplary embodiment of the present disclosure. In Figure 8 the first figure of, the first motion trajectory represents the visual pose sequence, and the second motion trajectory represents the motion pose sequence. Therefore, Figure 8 the first figure of shows the relationship between the pose angles of the visual pose sequence and the motion pose sequence changing with time respectively. Using motion segment similarity analysis, the visual pose sequence and the motion pose sequence can be respectively split into multiple motion segments according to the motion characteristic parameters of the synchronous motion of the motion platform and the extended reality device to be measured. For example, Figure 8 the first figure of shows a motion segment of the visual pose sequence and a motion segment of the motion pose sequence that have a corresponding relationship in time, and both are shown by a dashed box A. Figure 8 The second figure and the third figure of show the process of performing linear fitting on the two motion segments within the dashed box A in the first figure of Figure 8 respectively, where Figure 8The first motion trajectory and the second motion trajectory in the second and third figures respectively represent a motion segment of the visual pose sequence, and a motion segment of the motion pose sequence corresponding in time to a motion segment of the visual pose sequence. Figure 8 The first motion trajectory after fitting and the second motion trajectory after fitting in the fourth figure of [] respectively represent Figure 8 The relationship of the change in the pose angle over time after fitting of a motion segment of the visual pose sequence in the second and third figures of [], and the relationship of the change in the pose angle over time after fitting of a motion segment of the motion pose sequence in the above figures. Or rather, Figure 8 The first motion trajectory after fitting and the second motion trajectory after fitting in the fourth figure of [] respectively represent the first time-pose curve and the second time-pose curve after fitting. By performing "gap" mean statistics on the first time-pose curve and the second time-pose curve, the Figure 8 delay curve shown in the fifth figure of [] can be obtained. The delay curve can accurately represent the MTP delay of the extended reality device during the entire motion process of the continuous complex three-dimensional motion of the human body.

[0085] In addition, in order to further improve the accuracy and precision of MTP delay measurement, expand the application field of the delay measurement system, and optimize the application scenario of the delay measurement system, multi-dimensional delay evaluation indicators can be used to characterize the results of MTP delay measurement.

[0086] Figure 9 shows a schematic diagram of the statistical results including the first statistical indicator and the second statistical indicator according to an exemplary embodiment of the present disclosure.

[0087] Optionally, referring to Figure 6 and Figure 9 , the delay unit 390 may further include: a first calculation subunit 391. The first calculation subunit 391 determines the first statistical indicator based on the difference between the first time-pose curve and the second time-pose curve corresponding to the first time-pose curve in time, where the first statistical indicator may include at least one of the delay extreme value, delay average value, delay median value, and delay standard deviation of the extended reality device 400 to be measured (such as Figure 2 shown).

[0088] In addition, the delay unit 390 may further include: a second calculation subunit 392. The second calculation subunit 392 may determine the second statistical indicator based on at least one of the mean value and the standard deviation of the first statistical indicator obtained from multiple measurements.

[0089] The delay evaluation index for characterizing the results of MTP delay measurement may include at least one of a first statistical index and a second statistical index. The first statistical index includes delay extreme values, delay average values, delay medians, delay standard deviations, etc., which can be used to measure the delay situation of the to-be-tested extended reality device during a single measurement process. The second statistical index is at least one of the mean value and the standard deviation of the first statistical index of multiple measurements, which can be used to measure the stability of the delay situation of the to-be-tested extended reality device macroscopically or overall. Figure 9 Fig. shows the statistical results of a to-be-tested extended reality device 400 performing 6 times of MTP delay measurements provided according to at least one embodiment of the present disclosure. The user can select a suitable delay evaluation index from the above multi-dimensional delay evaluation indexes to characterize the results of the MTP delay measurement based on the actual situation such as the type and application field of the extended reality device.

[0090] Therefore, in the delay measurement system provided according to at least one embodiment of the present disclosure, the motion platform that moves synchronously with the to-be-tested extended reality device can simulate the continuous composite three-dimensional motion of the human body. Therefore, it can comprehensively evaluate the delay performance of the extended reality device in different motion states, and can observe the dynamic change situation of the extended reality device in the continuous composite three-dimensional motion. The vision platform can be used to obtain multiple consecutive images of the target board displayed by the to-be-tested extended reality device, and then the vision pose sequence of the to-be-tested extended reality device can be determined. By comparing the difference between the vision pose sequence of the to-be-tested extended reality device and the motion pose sequence of the motion platform, the MTP delay of the extended reality device in the continuous composite three-dimensional motion of the human body can be accurately measured.

[0091] Figure 10 Fig. is a flowchart of a delay measurement method 2000 provided by an embodiment of the present disclosure. The delay measurement method 2000 may include the following steps:

[0092] S1: Obtain the motion pose sequence of the motion platform, where the motion platform is fixedly connected to the to-be-tested extended reality device and drives the to-be-tested extended reality device to move synchronously.

[0093] S2: Obtain multiple consecutive images of the target board displayed by the to-be-tested extended reality device.

[0094] S3: Obtain the vision pose sequence of the to-be-tested extended reality device based on the multiple consecutive images.

[0095] S4: Determine the delay of the to-be-tested extended reality device based on the vision pose sequence and the motion pose sequence.

[0096] The following will describe each step of the above delay measurement method 2000 in detail with reference to the drawings.

[0097] Step S1

[0098] Reference Figure 1 、 Figure 2 and Figure 10 In some embodiments of the present disclosure, the motion platform 100 is fixedly connected to the to-be-tested extended reality device 400, such as a rigid fixed connection. In addition, the motion platform 100 can drive the to-be-tested extended reality device 400 to move synchronously. The motion platform 100 can include any suitable structural components and can perform various translational and rotational motions in the world coordinate system. Taking the to-be-tested extended reality device 400 as an HMD as an example, the motion platform 100 fixedly connected to the HMD can simulate the horizontal movement, horizontal rotation, tilting motion, etc. of the human body.

[0099] Optionally, by using the vision platform 200 to obtain a plurality of consecutive images of the target board 410 (as shown in Figure 3 ), the time delay measurement method 2000 further includes: before and during the synchronous movement, synchronizing the time of the motion platform 100 and the vision platform 200.

[0100] For example, using a clock source to synchronize the time of the motion platform 100 and the vision platform 200. During the time synchronization process, the motion platform 100 and the vision platform 200 can be uniformly timed through the PTP protocol. Using a clock source with high precision and high accuracy as the master clock, sending timestamps to the motion platform 100 and the vision platform 200 as slave clocks. The slave clocks compare the timestamps with their own local times and adjust the time according to the differences, so as to ensure that the subsequent motion pose sequence obtained based on the motion platform 100 and the vision pose sequence obtained based on the vision platform 200 are time-synchronized.

[0101] Step S2

[0102] Reference Figures 1 - 4 、 Figure 10 In some embodiments of the present disclosure, the vision platform 200 can be used to obtain a plurality of consecutive images of the target board 410 displayed via the to-be-tested extended reality device 400, for example Figure 4 shows an image 420 among the plurality of consecutive images.

[0103] The vision platform 200 can be used to simulate the scenario where the human eye receives consecutive images displayed by the to-be-tested extended reality device 400. Optionally, the vision platform 200 can include at least one of a camera and a video camera. The camera can include an IR camera (Infrared Camera), an RGB camera (Red Green Blue Camera), or a black and white camera, etc. In addition, the vision platform 200 can also include other structures such as a photoelectric sensor. The present disclosure does not limit the specific internal structure settings of the vision platform 200.

[0104] Optionally, the calibration board 410 may be a virtual calibration board located within the extended reality device 400 to be measured. The virtual calibration board may include a visual marker calibration board in a visual reference library. For example, a two-dimensional code calibration board such as ApriTag, ArUco, etc. Using the virtual calibration board for latency measurement can simplify the structure of the latency measurement system and reduce the implementation difficulty of latency measurement.

[0105] Step S3

[0106] Reference Figure 1 、 Figure 3 and Figure 4 , in some embodiments of the present disclosure, obtaining the visual pose sequence of the extended reality device to be measured based on multiple consecutive images in step S3 may, for example, include: extracting the pixel coordinates of the feature points of the calibration board 410 based on the multiple consecutive images; converting the pixel coordinates into a preliminary pose sequence in a first coordinate system referenced to the visual platform 200 that captures the multiple consecutive images according to the preset spatial geometric constraint relationships between the feature points; and converting the preliminary pose sequence into a visual pose sequence in a second coordinate system referenced to the motion platform 100.

[0107] Specifically, the processing platform 300 may be used to obtain the visual pose sequence of the extended reality device to be measured based on multiple consecutive images. The processing platform 300 may be various general-purpose and / or dedicated processing components with processing and computing capabilities. Some examples of the processing platform 300 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc.

[0108] The processing platform 300 obtains the visual pose sequence of the extended reality device 400 to be measured based on multiple consecutive images, and determines the latency of the extended reality device 400 to be measured based on the visual pose sequence and the motion pose sequence of the motion platform 100. Optionally, the processing platform 300 may include various suitable communication buses and input / output interfaces for connecting to the motion platform 100 and the visual platform 200.

[0109] In addition, before extracting the pixel coordinates of the feature points of the calibration board 410 based on multiple consecutive images, the multiple consecutive images may be processed, and the processing may include at least one of downsampling processing and Gaussian blur processing. Since the appearance of moiré patterns will cause inaccurate extraction of two-dimensional feature points, or even inability to extract them, when photographing the screen of the extended reality device 400 to be measured, it is necessary to process the obtained multiple consecutive images. By at least one of downsampling processing and Gaussian blur processing, etc., the influence of moiré patterns on the above images can be reduced.

[0110] In addition, in order to compare the motion posture changes in the real world with those in the virtual world of the to-be-tested extended reality device 400, it is necessary to unify these two types of motion posture changes under the same coordinate system. The motion changes in the virtual world can be determined under the first coordinate system (e.g., the camera coordinate system) with the visual platform 200 as the reference, and the motion changes in the real world can be determined under the second coordinate system (e.g., the world coordinate system) with the motion platform 100 as the reference. Therefore, it is necessary to calibrate the coordinate system conversion of the motion platform 100 and the visual platform 200.

[0111] Since the eye relief distances of different to-be-tested extended reality devices are different, it is necessary to adjust the position of the visual platform 200 before measurement to adapt to the to-be-tested extended reality device 400, which further causes the relative positions of the visual platform 200 and the motion platform 100 to change, resulting in the inapplicability of the original conversion relationship and the need for recalibration.

[0112] Optionally, before converting the initial measurement pose sequence into the visual pose sequence, time correction can be performed on the initial measurement pose sequence. Although the timestamps of the images of the calibration board and the trajectory timestamps of the motion platform are under the same clock source, due to the existence of MTP, there is a certain time delay in the process of capturing images. Therefore, time correction is required.

[0113] For example, the fixed connection between the motion platform 100 and the to-be-tested extended reality device 400 can be considered as a rigid connection of a rigid body, and the fixed connection between the visual platform 200 and the to-be-tested extended reality device 400 can also be considered as a rigid connection of a rigid body. According to the principle of rotational invariance of a rigid body, time correction can be performed on the initial measurement pose sequence before converting the initial measurement pose sequence into the visual pose sequence. For example, after filtering the angular velocity of the pose, the time delay of the process of capturing images caused by the existence of MTP is obtained based on cross-correlation, and time correction and alignment are performed on the relevant data.

[0114] Step S4

[0115] Reference Figure 1 、 Figure 2 And Figures 7 - 8 , step S4 for determining the time delay of the to-be-tested extended reality device based on the visual pose sequence and the motion pose sequence may include, for example: splitting the visual pose sequence and the motion pose sequence into multiple motion segments respectively according to the synchronous motion characteristic parameters; fitting the multiple motion segments of the visual pose sequence into multiple first time-pose curves respectively, and fitting the multiple motion segments of the motion pose sequence into multiple second time-pose curves respectively; and determining the time delay of the to-be-tested extended reality device based on the differences between the first time-pose curves and the second time-pose curves corresponding to the first time-pose curves in time.

[0116] Optionally, the visual pose sequence may include a plurality of first time-pose information arranged in chronological order. The first time-pose information includes first time information and a first pose angle corresponding to the first time information. The first pose angle includes a first roll angle, a first yaw angle, and a first pitch angle. For example, the form of the first time-pose information may include: timestamp_roll_pitch_yaw. For instance, 000_10_20_30 may represent that the angular poses of roll, pitch, and yaw at the starting moment are (10°, 20°, 30°) respectively. In a right-handed Cartesian coordinate system in three-dimensional space, roll is the roll angle, around the z-axis; pitch is the pitch angle, around the x-axis; and yaw is the yaw angle, around the y-axis.

[0117] The motion pose sequence may include a plurality of second time-pose information arranged in chronological order. The second time-pose information may include second time information and a second pose angle corresponding to the second time information. The second pose angle includes a second roll angle, a second yaw angle, and a second pitch angle. Similarly, the form of the second time-pose information may also include: timestamp_roll_pitch_yaw.

[0118] Traditional MTP latency measurement can only obtain the latency at the start node or the end node of the motion, and it is impossible to measure the latency during the motion process. Or rather, traditional MTP measurement can be divided into manual calculation of latency and automated calculation of latency. Manual calculation of latency can calculate the latency of the extended reality device through artificially estimated values, with a large error and unable to objectively reflect the true latency of the extended display device. Automated calculation of latency measures the latency of the extended reality device through a measuring device. The key lies in capturing the changes in the motion state of the extended reality device and monitoring the changes in the images reflected on the screen of the extended reality device. By comparing the delays between the two, the latency of the extended reality device is obtained. However, automated calculation of latency is limited to measuring the latency generated under a single motion node, such as the latency at the start node or the end node of the motion. In actual usage scenarios, the motion of the extended reality device and the response of the virtual reality scene in the extended reality device are a continuous process, and the latency generated during this process is a set of dynamically continuous data. Therefore, only measuring the latency generated under a single motion node cannot reflect the dynamic change of the latency of the extended reality device during continuous motion. Especially considering that human motion is mostly continuous three-dimensional composite motion, for an extended display device that supports a motion prediction algorithm, analyzing the delay change in continuous composite three-dimensional motion is a key indicator for evaluating its motion prediction algorithm.

[0119] According to the delay measurement method provided by at least one embodiment of the present disclosure, a motion platform that moves synchronously with the extended reality device to be measured can simulate continuous complex three-dimensional human motion. Therefore, it can comprehensively evaluate the delay performance of the extended reality device in different motion states, and can observe the dynamic changes of the extended reality device in continuous complex three-dimensional motion. Based on the obtained multiple consecutive images of the calibration board displayed by the extended reality device to be measured, the visual pose sequence of the extended reality device to be measured can be determined. By comparing the difference between the visual pose sequence of the extended reality device to be measured and the motion pose sequence of the motion platform, the MTP delay of the extended reality device in continuous complex three-dimensional human motion can be accurately measured.

[0120] To enhance the above effects and improve the accuracy and precision of MTP delay measurement, the visual pose sequence and the motion pose sequence can be segmented respectively. Using motion segment similarity analysis, the visual pose sequence and the motion pose sequence are respectively split into multiple motion segments; by analyzing each motion segment one by one, performing line fitting for each segment, and performing "gap" mean statistics on the first time-pose curve and the second time-pose curve after fitting, the MTP delay of the extended reality device in continuous complex three-dimensional human motion can be accurately measured.

[0121] Optionally, during the process of using motion segment similarity analysis, the motion characteristic parameters of the synchronous motion may include at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion. Therefore, appropriate synchronous motion characteristic parameters can be selected according to the type of the extended reality device to be measured and actual requirements, or according to the characteristics of the synchronous motion, so as to accurately measure the MTP delay of the extended reality device in continuous complex three-dimensional human motion. It should be noted that the present disclosure does not limit the specific content of the motion characteristic parameters of the synchronous motion.

[0122] In addition, during the process of using motion segment similarity analysis to split the visual pose sequence and the motion pose sequence into multiple motion segments respectively, the visual pose sequence and the motion pose sequence can be split into multiple motion segments according to the motion characteristic parameters of the synchronous motion of the motion platform 100 and the extended reality device 400 to be measured. For example, at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion is regarded as the volatility reference data, and a predetermined threshold is set. When at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion is greater than its corresponding predetermined threshold, it can be considered that the volatility of this part of the data is too large. Based on this, in the visual pose sequence or the motion pose sequence, the previous time-pose information is identified as the end point of the previous motion segment, and the next time-pose information is identified as the start point of the next motion segment.

[0123] In addition, in some embodiments of the present disclosure, considering error factors such as device assembly error and jitter error, zero-bias preprocessing can be performed on the visual pose sequence and the motion pose sequence respectively before fitting multiple motion segments of the visual pose sequence and multiple motion segments of the motion pose sequence.

[0124] In the process of analyzing multiple motion segments one by one and performing line segment fitting segment by segment, fitting can be based on a straight line or a quadratic curve. It should be noted that the present disclosure does not limit the method of line segment fitting. After that, the "gap" mean value statistics can be performed on the first time-pose curve and the second time-pose curve after fitting, and the MTP delay of the extended reality device during the entire motion process of the continuous complex three-dimensional motion of the human body can be accurately measured.

[0125] In order to further improve the accuracy and precision of MTP delay measurement, expand the application field of the delay measurement system and optimize the application scenario of the delay measurement system, multi-dimensional delay evaluation indicators can be used to characterize the results of MTP delay measurement.

[0126] The delay evaluation indicators characterizing the results of MTP delay measurement can include at least one of the first statistical indicator and the second statistical indicator. The first statistical indicator includes delay extreme values, delay average values, delay medians, delay standard deviations, etc., which can be used to measure the delay situation of the extended reality device to be measured during a single measurement process. The second statistical indicator is at least one of the mean value and the standard deviation of the first statistical indicator for multiple measurements, which can be used to measure the stability of the delay situation of the extended reality device to be measured from a macroscopic or overall perspective. Figure 9 Fig. shows the statistical results of 6 MTP delay measurements performed on an extended reality device 400 to be measured according to at least one embodiment of the present disclosure. Users can select appropriate delay evaluation indicators from the above multi-dimensional delay evaluation indicators to characterize the results of MTP delay measurement based on the actual situation such as the type and application field of their extended reality devices.

[0127] Therefore, according to the delay measurement method provided by at least one embodiment of the present disclosure, the motion platform that moves synchronously with the extended reality device to be measured can simulate the continuous complex three-dimensional motion of the human body, can comprehensively evaluate the delay performance of the extended reality device in different motion states, and can observe the dynamic changes of the extended reality device in the continuous complex three-dimensional motion. Based on the obtained multiple consecutive images of the calibration board displayed by the extended reality device to be measured, the visual pose sequence of the extended reality device to be measured can be determined. By comparing the difference between the visual pose sequence of the extended reality device to be measured and the motion pose sequence of the motion platform, the MTP delay of the extended reality device in the continuous complex three-dimensional motion of the human body can be accurately measured.

[0128] Figure 11FIG. shows a schematic block diagram of an exemplary electronic device 600 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0129] As Figure 11 shown, the device 600 includes a computing unit 601 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0130] A plurality of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0131] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 executes the various methods and processes described above, such as the delay measurement method. For example, in some embodiments, the delay measurement method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the delay measurement method described above can be executed. Alternatively, in other embodiments, the computing unit 601 can be configured to execute the delay measurement method by any other suitable means (e.g., by means of firmware).

[0132] Various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), systems-on-a-chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0136] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0137] A computer system may include a client and a server. The client and the server are generally far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability existing in traditional physical hosts and virtual private server (VPS) services.

[0138] It should be understood that various forms of processes shown above can be used, and steps can be reordered, added, or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and no limitations are imposed herein.

[0139] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A delay measurement system, characterized in that: include: A motion platform, fixedly connected to the extended reality device to be tested, and driving the extended reality device to be tested to move synchronously; A visual platform, fixedly connected to the extended reality device to be tested, and acquiring a plurality of continuous images of a target plate displayed by the extended reality device to be tested; as well as The processing platform obtains a visual pose sequence of the extended reality device to be tested based on the multiple continuous images, and determines a delay of the extended reality device to be tested based on the visual pose sequence and the motion pose sequence of the motion platform. Wherein, the processing platform also includes: a splitting unit, which splits the visual posture sequence and the motion posture sequence into a plurality of motion segments according to motion characteristic parameters of the synchronous motion; a fitting unit, which fits the multiple motion segments of the visual pose sequence into multiple first time-pose curves, and fits the multiple motion segments of the motion pose sequence into multiple second time-pose curves; and The delay unit determines the delay of the extended reality device to be tested based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time.

2. The system according to claim 1, wherein: The motion characteristic parameter of the synchronous motion includes at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion.

3. The system according to claim 1, wherein: The delay unit also includes: The first calculation subunit determines a first statistical indicator based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time, wherein the first statistical indicator includes at least one of the extreme value of delay, the average value of delay, the median value of delay, and the standard deviation of delay of the extended reality device to be tested.

4. The system according to claim 3, wherein: The delay unit also includes: The second calculation subunit determines a second statistical indicator according to at least one of a mean value and a standard deviation of the first statistical indicator obtained through multiple measurements.

5. The system according to claim 1, wherein: The processing platform also includes: The preprocessing unit performs zero bias preprocessing on the visual pose sequence and the motion pose sequence respectively before fitting the multiple motion segments of the visual pose sequence and the multiple motion segments of the motion pose sequence respectively.

6. The system according to claim 1, wherein: The system further comprises: The time platform comprises a clock source, wherein the clock source is used to synchronize the time of the motion platform and the visual platform before and during the synchronous movement.

7. The system according to claim 1, wherein: The processing platform also includes: A first conversion unit extracts pixel coordinates of feature points of the target based on the plurality of continuous images; A second conversion unit, converting the pixel coordinates into an initial measured pose sequence in a first coordinate system with the visual platform as a reference, according to a preset spatial geometric constraint relationship between the feature points; and The calibration unit converts the initially measured posture sequence into the visual posture sequence in a second coordinate system with the motion platform as a reference.

8. The system according to claim 7, wherein: The processing platform also includes: A time correction unit performs time correction on the initially measured pose sequence before converting the initially measured pose sequence into the visual pose sequence.

9. The system according to claim 7, wherein: The processing platform also includes: The image processing unit processes the plurality of continuous images before extracting the pixel coordinates of the feature points of the target based on the plurality of continuous images, wherein the processing includes at least one of downsampling processing and Gaussian blur processing.

10. The system according to claim 1, wherein: The target plate is a virtual target plate, wherein the virtual target plate includes a visual marker calibration plate in a visual reference library.

11. The system according to claim 1, wherein: The visual posture sequence includes a plurality of first time-pose information arranged in time order, the first time-pose information includes first time information and a first posture angle corresponding to the first time information, wherein the first posture angle includes a first roll angle, a first yaw angle and a first pitch angle; and The motion posture sequence includes a plurality of second time-posture information arranged in chronological order, the second time-posture information including second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle and a second pitch angle.

12. A delay measurement method, characterized in that: include: Acquiring a motion posture sequence of a motion platform, wherein the motion platform is fixedly connected to the extended reality device to be tested, and drives the extended reality device to be tested to move synchronously; Acquire a plurality of continuous images of a target displayed by the extended reality device to be tested; Obtaining a visual pose sequence of the extended reality device to be tested based on the multiple continuous images; as well as Determine the delay of the extended reality device to be tested based on the visual pose sequence and the motion pose sequence, Wherein, determining the delay of the extended reality device to be tested based on the visual posture sequence and the motion posture sequence includes: According to the motion characteristic parameters of the synchronous motion, the visual posture sequence and the motion posture sequence are respectively split into a plurality of motion segments; Fitting a plurality of motion segments of the visual pose sequence into a plurality of first time-pose curves, and fitting a plurality of motion segments of the motion pose sequence into a plurality of second time-pose curves; and Based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time, the delay of the extended reality device to be tested is determined.

13. The method according to claim 12, wherein: The motion characteristic parameter of the synchronous motion includes at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion.

14. The method according to claim 12, wherein: Determining the delay of the measured augmented reality device based on a difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time includes: Based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time, a first statistical indicator is determined, wherein the first statistical indicator includes at least one of the extreme value of delay, the average value of delay, the median value of delay, and the standard deviation of delay of the extended reality device to be tested.

15. The method according to claim 14, wherein: Determining the delay of the measured extended reality device based on a difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time further includes: The second statistical indicator is determined according to at least one of a mean value and a standard deviation of the first statistical indicator obtained through multiple measurements.

16. The method according to claim 12, wherein: Before fitting the multiple motion segments of the visual pose sequence and the multiple motion segments of the motion pose sequence respectively, the method further includes: Zero bias preprocessing is performed on the visual pose sequence and the motion pose sequence respectively.

17. The method according to claim 12, wherein: The plurality of continuous images are acquired by using a visual platform, and the method further comprises: before and during the synchronous movement, performing time synchronization on the motion platform and the visual platform.

18. The method according to claim 12, wherein: Acquiring a visual pose sequence of the extended reality device to be tested based on the multiple continuous images includes: Extracting pixel coordinates of feature points of the target based on the plurality of continuous images; According to the preset spatial geometric constraint relationship between the feature points, the pixel coordinates are converted into an initial measured pose sequence in a first coordinate system with a visual platform for taking the multiple consecutive images as a reference; and The initially measured pose sequence is converted into the visual pose sequence in a second coordinate system with the motion platform as a reference.

19. The method according to claim 18, wherein: The method further includes: before converting the initially measured pose sequence into the visual pose sequence, performing time correction on the initially measured pose sequence.

20. The method according to claim 18, wherein: The method further includes: before extracting the pixel coordinates of the feature points of the target based on the multiple continuous images, processing the multiple continuous images, wherein the processing includes at least one of downsampling processing and Gaussian blur processing.

21. The method according to claim 12, wherein: The target plate is a virtual target plate, wherein the virtual target plate includes a visual marker calibration plate in a visual reference library.

22. The method according to claim 12, wherein: The visual posture sequence includes a plurality of first time-pose information arranged in time order, the first time-pose information includes first time information and a first posture angle corresponding to the first time information, wherein the first posture angle includes a first roll angle, a first yaw angle and a first pitch angle; and The motion posture sequence includes a plurality of second time-posture information arranged in chronological order, the second time-posture information including second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle and a second pitch angle.

23. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the delay measurement method according to any one of claims 12 to 22.

24. A computer-readable storage medium storing a computer program, characterized in that: The computer instructions are used to enable the computer to execute the delay measurement method according to any one of claims 12 to 22.

25. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the delay measurement method according to any one of claims 12 to 22 is implemented.

Images