Delay measurement method and system, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a delay measurement method, a delay measurement system, electronic equipment, a computer readable storage medium and a computer program product. The delay measurement system comprises a motion platform, a visual platform and a processing platform. The motion platform is fixedly connected with the to-be-tested augmented reality device and drives the to-be-tested augmented reality device to move synchronously. The visual platform is fixedly connected with the to-be-tested augmented reality device, and obtains a plurality of continuous images of the target displayed by the to-be-tested augmented reality device. And the processing platform obtains a visual pose sequence of the to-be-tested augmented reality device based on the plurality of continuous images, and determines the time delay of the to-be-tested augmented reality device based on the visual pose sequence and the motion pose sequence of the motion platform.

Description

Technical Field

[0001] The present disclosure relates to the technical field of extended reality devices, and in particular to a delay measurement method, a delay measurement system, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of science and technology, higher requirements are placed on the performance of extended reality devices such as virtual reality devices, augmented reality devices, and mixed reality devices.

[0003] Taking the head-mounted display (HMD) as an example, the extended reality device needs to respond to the user's head movement in real time. Although the user's head movement can be measured by a variety of sensors, the HMD cannot immediately display the corresponding image to the user due to the time constraints of transmission and calculation, which will cause image delay. This delay will cause the difference between visual motion perception and vestibular motion perception, which will not only reduce the sense of presence of the extended reality device, but also increase the burden on the brain and cause dizziness and nausea. MTP (motion-to-photon) delay can include the time from the start of the user's movement to the display of the corresponding image on the screen. It represents the delay time between the image seen by the user in the extended reality device such as HMD and the user's head movement, and can quantify the degree of match between visual observation and user head movement. Therefore, for the user, the smaller the MTP delay, the better the user's immersion; the larger the MTP delay, the stronger the user's dizziness.

[0004] Considering that human head movements are mostly continuous three-dimensional (three-axis) compound movements, for extended display devices that support motion prediction algorithms, analyzing the delay changes in continuous compound three-dimensional movements is a key indicator for evaluating their motion prediction algorithms.

[0005] Therefore, how to accurately measure the MTP delay of extended reality devices in the continuous complex three-dimensional movement of the human body has become one of the research hotspots in the field of extended reality device technology. Summary of the invention

[0006] The embodiments of the present disclosure propose a delay measurement method, a delay measurement system, an electronic device, a computer-readable storage medium, and a computer program product, which realize the accurate measurement of the MTP delay of an extended reality device during the continuous compound three-dimensional movement of a human body.

[0007] In a first aspect, an embodiment of the present disclosure proposes a delay measurement system, which includes: a motion platform, which is fixedly connected to the extended reality device to be tested and drives the extended reality device to be tested to move synchronously; a visual platform, which is fixedly connected to the extended reality device to be tested and obtains multiple continuous images of a target plate displayed by the extended reality device to be tested; and a processing platform, which obtains a visual pose sequence of the extended reality device to be tested based on the multiple continuous images, and determines the delay of the extended reality device to be tested based on the visual pose sequence and the motion pose sequence of the motion platform.

[0008] In some embodiments of the present disclosure, the processing platform also includes: a splitting unit, which splits the visual pose sequence and the motion pose sequence into multiple motion segments according to the motion feature parameters of the synchronous motion; a fitting unit, which fits the multiple motion segments of the visual pose sequence into multiple first time-pose curves, and fits the multiple motion segments of the motion pose sequence into multiple second time-pose curves; and a delay unit, which determines the delay of the extended reality device to be tested based on the difference between the first time-pose curve and the second time-pose curve corresponding to the first time-pose curve in time.

[0009] In some embodiments of the present disclosure, the motion characteristic parameter of the synchronous motion includes at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion.

[0010] In some embodiments of the present disclosure, the delay unit also includes: a first calculation subunit, which determines a first statistical indicator based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time, wherein the first statistical indicator includes at least one of the delay extreme value, delay average value, delay median value, and delay standard deviation of the extended reality device to be tested.

[0011] In some embodiments of the present disclosure, the delay unit further includes: a second calculation subunit, which determines a second statistical indicator according to at least one of a mean value and a standard deviation of the first statistical indicator obtained from multiple measurements.

[0012] In some embodiments of the present disclosure, the processing platform further includes: a preprocessing unit, which performs zero bias preprocessing on the visual pose sequence and the motion pose sequence before fitting the multiple motion segments of the visual pose sequence and the multiple motion segments of the motion pose sequence respectively.

[0013] In some embodiments of the present disclosure, the system further includes: a time platform including a clock source, wherein the clock source is used to perform time synchronization between the motion platform and the visual platform before and during the synchronous movement.

[0014] In some embodiments of the present disclosure, the processing platform also includes: a first conversion unit, which extracts the pixel coordinates of the feature points of the target plate based on the multiple continuous images; a second conversion unit, which converts the pixel coordinates into an initial measured pose sequence in a first coordinate system with the visual platform as a reference according to a preset spatial geometric constraint relationship between the feature points; and a calibration unit, which converts the initial measured pose sequence into the visual pose sequence in a second coordinate system with the motion platform as a reference.

[0015] In some embodiments of the present disclosure, the processing platform further includes: a time correction unit, which performs time correction on the initial measured pose sequence before converting the initial measured pose sequence into the visual pose sequence.

[0016] In some embodiments of the present disclosure, the processing platform also includes: an image processing unit, which processes the multiple continuous images before extracting the pixel coordinates of the feature points of the target plate based on the multiple continuous images, and the processing includes at least one of downsampling processing and Gaussian blur processing.

[0017] In some embodiments of the present disclosure, the target is a virtual target, wherein the virtual target includes a visual marker calibration plate in a visual reference library.

[0018] In some embodiments of the present disclosure, the visual posture sequence includes a plurality of first time-pose information arranged in chronological order, the first time-pose information including first time information and a first posture angle corresponding to the first time information, wherein the first posture angle includes a first roll angle, a first yaw angle, and a first pitch angle; and the motion posture sequence includes a plurality of second time-pose information arranged in chronological order, the second time-pose information including second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle, and a second pitch angle.

[0019] In the second aspect, an embodiment of the present disclosure proposes a delay measurement method, which includes: obtaining a motion posture sequence of a motion platform, wherein the motion platform is fixedly connected to an extended reality device to be tested, and drives the extended reality device to be tested to move synchronously; obtaining multiple continuous images of a target plate displayed by the extended reality device to be tested; obtaining a visual posture sequence of the extended reality device to be tested based on the multiple continuous images; and determining the delay of the extended reality device to be tested based on the visual posture sequence and the motion posture sequence.

[0020] In some embodiments of the present disclosure, determining the delay of the extended reality device to be tested based on the visual pose sequence and the motion pose sequence includes: splitting the visual pose sequence and the motion pose sequence into multiple motion segments according to the motion characteristic parameters of the synchronous motion; fitting the multiple motion segments of the visual pose sequence into multiple first time-pose curves, and fitting the multiple motion segments of the motion pose sequence into multiple second time-pose curves; and determining the delay of the extended reality device to be tested based on the difference between the first time-pose curve and the second time-pose curve corresponding to the first time-pose curve in time.

[0021] In some embodiments of the present disclosure, the motion characteristic parameter of the synchronous motion includes at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion.

[0022] In some embodiments of the present disclosure, determining the delay of the extended reality device to be tested based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time includes: determining a first statistical indicator based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time, wherein the first statistical indicator includes at least one of the delay extreme value, delay average value, delay median value, and delay standard deviation of the extended reality device to be tested.

[0023] In some embodiments of the present disclosure, determining the delay of the extended reality device to be tested based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time also includes: determining a second statistical indicator based on at least one of the mean and standard deviation of the first statistical indicator obtained from multiple measurements.

[0024] In some embodiments of the present disclosure, before fitting the multiple motion segments of the visual pose sequence and the multiple motion segments of the motion pose sequence respectively, the method further includes: performing zero bias preprocessing on the visual pose sequence and the motion pose sequence respectively.

[0025] In some embodiments of the present disclosure, the measurement method further comprises: before and during the synchronous movement, performing time synchronization on the motion platform and the visual platform.

[0026] In some embodiments of the present disclosure, obtaining the visual pose sequence of the extended reality device to be tested based on the multiple continuous images includes: extracting pixel coordinates of feature points of the target plate based on the multiple continuous images; converting the pixel coordinates into an initial measured pose sequence in a first coordinate system with a visual platform that takes the multiple continuous images as a reference according to a preset spatial geometric constraint relationship between the feature points; and converting the initial measured pose sequence into the visual pose sequence in a second coordinate system with the motion platform as a reference.

[0027] In some embodiments of the present disclosure, the measurement method further includes: before converting the initially measured pose sequence into the visual pose sequence, performing time correction on the initially measured pose sequence.

[0028] In some embodiments of the present disclosure, the measurement method further includes: before extracting the pixel coordinates of the feature points of the target plate based on the multiple continuous images, processing the multiple continuous images, wherein the processing includes at least one of downsampling processing and Gaussian blur processing.

[0029] In some embodiments of the present disclosure, the target is a virtual target, wherein the virtual target includes a visual marker calibration plate in a visual reference library.

[0030] In some embodiments of the present disclosure, the visual posture sequence includes a plurality of first time-pose information arranged in chronological order, the first time-pose information including first time information and a first posture angle corresponding to the first time information, wherein the first posture angle includes a first roll angle, a first yaw angle, and a first pitch angle; and the motion posture sequence includes a plurality of second time-pose information arranged in chronological order, the second time-pose information including second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle, and a second pitch angle.

[0031] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the measurement method described in any implementation method in the second aspect when executing.

[0032] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the measurement method described in any implementation manner in the second aspect when executed.

[0033] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the measurement method described in any implementation manner in the second aspect.

[0034] According to the delay measurement system, delay measurement method, electronic device, computer-readable storage medium, computer program product provided by at least one embodiment of the present disclosure, the motion platform that moves synchronously with the extended reality device to be tested can simulate the continuous compound three-dimensional motion of the human body, so that the delay performance of the extended reality device in different motion states can be comprehensively evaluated, and the dynamic changes of the extended reality device in the continuous compound three-dimensional motion can be observed. Using the visual platform, multiple continuous images of the target plate displayed by the extended reality device to be tested can be obtained, and then the visual posture sequence of the extended reality device to be tested can be determined. By comparing the difference between the visual posture sequence of the extended reality device to be tested and the motion posture sequence of the motion platform, the MTP delay of the extended reality device in the continuous compound three-dimensional motion of the human body can be accurately measured.

[0035] It should be understood that the content described in this section is not intended to identify the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, objectives and advantages of the disclosed embodiments will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings, in which: Figure 1 A schematic diagram of a delay measurement system according to an exemplary embodiment of the present disclosure is shown; Figure 2 A schematic diagram of an extended reality device to be tested according to an exemplary embodiment of the present disclosure is shown; Figure 3 A schematic diagram of a target according to an exemplary embodiment of the present disclosure is shown; Figure 4 A schematic diagram showing one image among a plurality of continuous images displayed by an extended reality device to be tested according to an exemplary embodiment of the present disclosure is shown; Figure 5 A schematic diagram of a processing platform according to an exemplary embodiment of the present disclosure is shown; Figure 6A schematic diagram of a processing platform according to an exemplary embodiment of the present disclosure is shown; Figure 7 A schematic diagram of a process for determining the delay of an extended reality device to be tested by segmenting a visual pose sequence and a motion pose sequence according to an exemplary embodiment of the present disclosure is shown; Figure 8 A schematic diagram of a process for generating a first time-pose curve and a second time-pose curve according to an exemplary embodiment of the present disclosure is shown; Fig. 9 A schematic diagram of statistical results including a first statistical indicator and a second statistical indicator according to an exemplary embodiment of the present disclosure is shown; Fig.10 A flow chart of a delay measurement method provided in an embodiment of the present disclosure; Fig.11 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0037] In order to better understand the present disclosure, various aspects of the present disclosure will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present disclosure and are not intended to limit the scope of the present disclosure in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0038] It should be noted that in this specification, the expressions first, second, etc. are only used to distinguish one feature from another feature, and do not represent any limitation on the feature. Therefore, without departing from the teaching of the present disclosure, the first calculation subunit discussed below may also be referred to as the second calculation subunit, and the first time-pose curve may also be referred to as the second time-pose curve.

[0039] It should also be understood that expressions such as "include", "comprising", "having", "including" and / or "comprising" are open rather than closed expressions in this specification, which indicate the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, the word "exemplarily" is intended to refer to an example or illustration.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used in the present disclosure have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs. It should also be understood that terms (such as those defined in commonly used dictionaries) should be understood to have the same meaning as they have in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense, unless explicitly defined in the present disclosure.

[0041] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other. In addition, unless explicitly limited or contradictory to the context, the specific steps included in the method described in the present disclosure are not necessarily limited to the order described, but can be performed in any order or in parallel. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] Figure 1 A schematic diagram of a delay measurement system 1000 according to an exemplary embodiment of the present disclosure is shown. Figure 2 A schematic diagram of an extended reality device 400 to be tested according to an exemplary embodiment of the present disclosure is shown. Figure 3 A schematic diagram of a target 410 according to an exemplary embodiment of the present disclosure is shown. Figure 4 A schematic diagram showing one image 420 among a plurality of continuous images displayed by an extended reality device 400 to be tested according to an exemplary embodiment of the present disclosure is shown.

[0043] like Figure 1-Figure 4 As shown, the embodiment of the present disclosure provides a delay measurement system 1000, which may include a motion platform 100, a visual platform 200, and a processing platform 300. The motion platform 100 is fixedly connected to the extended reality device 400 to be tested, and drives the extended reality device 400 to be tested to move synchronously. The visual platform 200 is also fixedly connected to the extended reality device 400 to be tested, and obtains multiple continuous images of a target plate 410 displayed by the extended reality device 400 to be tested, such as Figure 4 One image 420 among the multiple continuous images is shown. The processing platform 300 obtains the visual pose sequence of the XR device 400 to be tested based on the multiple continuous images, and determines the delay of the XR device 400 to be tested based on the visual pose sequence and the motion pose sequence of the motion platform 100.

[0044] According to the delay measurement system provided by at least one embodiment of the present disclosure, the motion platform that moves synchronously with the extended reality device to be tested can simulate the continuous compound three-dimensional motion of the human body, so that the delay performance of the extended reality device in different motion states can be comprehensively evaluated, and the dynamic changes of the extended reality device in the continuous compound three-dimensional motion can be observed. Using the visual platform, multiple continuous images of the target plate displayed by the extended reality device to be tested can be obtained, and then the visual posture sequence of the extended reality device to be tested can be determined. By comparing the difference between the visual posture sequence of the extended reality device to be tested and the motion posture sequence of the motion platform, the MTP delay of the extended reality device in the continuous compound three-dimensional motion of the human body can be accurately measured.

[0045] Specifically, in some embodiments of the present disclosure, the extended reality device 400 to be tested may include a virtual reality device, an augmented reality device, or a mixed reality device. It should be understood that the delay measurement system 1000 can also be applied to application scenarios of delay measurement of other display devices, which is not limited here.

[0046] The motion platform 100 is fixedly connected to the extended reality device 400 to be tested, for example, rigidly fixedly connected. In addition, the motion platform 100 can drive the extended reality device 400 to be tested to move synchronously. The motion platform 100 may include any suitable structural components and may perform various translational and rotational movements in the world coordinate system. Taking the extended reality device 400 to be tested as an HMD as an example, the motion platform 100 fixedly connected to the HMD can simulate the horizontal movement, horizontal rotation, tilting movement, etc. of the human body.

[0047] The visual platform 200 can be used to simulate the scene where the human eye receives the continuous images displayed by the extended reality device 400 to be tested. Optionally, the visual platform 200 may include at least one of a camera and a video camera, and the camera may include an IR camera (Infrared Camera), an RGB camera (RedGreenBlue Camera), or a black and white camera. In addition, the visual platform 200 may also include other structures such as photoelectric sensors, and the present disclosure does not limit the specific internal structure of the visual platform 200.

[0048] Optionally, the target plate 410 may be a virtual target plate located in the extended reality device 400 to be tested. The virtual target plate may include a visual marker calibration plate in the visual reference library. For example, a two-dimensional code calibration plate such as ApriTag and ArUco. Using a virtual target plate for delay measurement can simplify the structure of the delay measurement system and reduce the difficulty of implementing delay measurement.

[0049] The processing platform 300 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processing platform 300 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processing platform 300 obtains a visual pose sequence of the extended reality device 400 to be tested based on a plurality of continuous images, and determines the delay of the extended reality device 400 to be tested based on the visual pose sequence and the motion pose sequence of the motion platform 100.

[0050] Optionally, the processing platform 300 may include various suitable communication buses and input / output interfaces for achieving connection with the motion platform 100 and the visual platform 200 .

[0051] Reference again Figure 1 In some embodiments of the present disclosure, the delay measurement system 1000 also includes a time platform 500, and the time platform 500 may include a clock source, wherein the clock source is used to synchronize the motion platform 100 and the visual platform 200 before and during the synchronous movement.

[0052] Specifically, the clock source can provide unified time synchronization to the motion platform 100 and the visual platform 200 through the PTP (Precision Time Protocol) protocol. A clock source with high precision and accuracy is used as the master clock to send timestamps to the motion platform 100 and the visual platform 200 as slave clocks. The slave clock compares the timestamp with its own local time and adjusts the time according to the difference, thereby ensuring that the subsequent motion posture sequence obtained based on the motion platform 100 is synchronized with the visual posture sequence obtained based on the visual platform 200.

[0053] Optionally, combined Figure 1 and Figure 2 , the visual posture sequence may include a plurality of first time-pose information arranged in time order, the first time-pose information includes the first time information and the first posture angle corresponding to the first time information, wherein the first posture angle includes the first roll angle, the first yaw angle and the first pitch angle. For example, the first time-pose information may include: timestamp_roll_pitch_yaw, wherein 000_10_20_30 may represent the angle postures of roll, pitch and yaw at the starting time (10°, 20°, 30°), respectively. In the right-handed Cartesian coordinate system in three-dimensional space, roll is the roll angle, around the z-axis; pitch is the pitch angle, around the x-axis; yaw is the yaw angle, around the y-axis.

[0054] The motion posture sequence may include a plurality of second time-posture information arranged in time order, and the second time-posture information may include second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle, and a second pitch angle. Similarly, the second time-posture information may also include: timestamp_roll_pitch_yaw.

[0055] In order to compare the motion posture changes in the real world with the motion posture changes in the virtual world of the xed reality device 400 to be tested, it is necessary to unify the two motion posture changes in the same coordinate system. The motion changes in the virtual world can be determined in a first coordinate system (e.g., a camera coordinate system) with the visual platform 200 as a reference, and the motion changes in the real world can be determined in a second coordinate system (e.g., a world coordinate system) with the motion platform 100 as a reference, so it is necessary to calibrate the motion platform 100 and the visual platform 200 for coordinate system conversion.

[0056] Since the eye-adaptive distances of different extended reality devices to be tested are different, the position of the visual platform 200 needs to be adjusted before measurement to adapt to the extended reality device 400 to be tested, which causes the relative position of the visual platform 200 and the motion platform 100 to change, and the original conversion relationship is no longer applicable, and recalibration is required. In order to solve this problem, a feasible solution is to calibrate the external parameters of the camera or video camera in the visual platform 200 based on the visual platform 200 shooting the virtual standard displayed by the extended reality device to be tested.

[0057] Figure 5 A schematic diagram of a processing platform 300 according to an exemplary embodiment of the present disclosure is shown.

[0058] Specifically, refer to Figure 1 and Figure 5 In some embodiments of the present disclosure, the processing platform 300 may include: a first conversion unit 320, a second conversion unit 330 and a calibration unit 350. The first conversion unit 320 may extract a target plate 410 (such as Figure 3 The second conversion unit 330 can convert the pixel coordinates into a preliminary measured pose sequence in a first coordinate system with the visual platform 200 as a reference according to the spatial geometric constraint relationship between the preset feature points. The calibration unit 350 converts the preliminary measured pose sequence into a visual pose sequence in a second coordinate system with the motion platform 100 as a reference.

[0059] Optionally, the processing platform 300 may further include an image processing unit 310, which may process the plurality of continuous images before extracting the pixel coordinates of the feature points of the target plate 410 based on the plurality of continuous images, wherein the processing includes at least one of downsampling processing and Gaussian blur processing. Since the appearance of moiré patterns may lead to inaccurate or even impossible extraction of 2D feature points when photographing the screen of the extended reality device 400 to be tested, it is necessary to process the plurality of continuous images obtained, and the influence of moiré patterns on the above-mentioned images may be reduced by at least one of downsampling processing and Gaussian blur processing.

[0060] In addition, the processing platform 300 may also include a time correction unit 340, which can perform time correction on the initial measured pose sequence before converting the initial measured pose sequence into a visual pose sequence. Although the timestamp of the image of the photographed standard and the timestamp of the trajectory of the motion platform are under the same clock source, due to the existence of MTP, the process of photographing the image has a certain delay, so time correction is required.

[0061] It should be noted that, for the sake of clarity, Figure 5 The processing platform 300 shown shows the optional image processing unit 310 and time correction unit 340. It can be distinguished that the optional image processing unit 310 or time correction unit 340 is shown with a dotted line frame, and the connection between the image processing unit 310 or the time correction unit 340 and other units is shown with a dotted connecting line when the image processing unit 310 or the time correction unit 340 is included.

[0062] Optionally, the fixed connection between the motion platform 100 and the extended reality device 400 to be tested can be considered as a rigid connection of a rigid body, and the fixed connection between the visual platform 200 and the extended reality device 400 to be tested can also be considered as a rigid connection of a rigid body. Based on the principle of rigid body rotation invariance, the initial measured pose sequence can be time-corrected before converting the initial measured pose sequence into a visual pose sequence. For example, after filtering the angular velocity of the pose, the time delay of the image shooting process caused by the existence of MTP is obtained based on cross-correlation, and the relevant data is time-corrected and aligned.

[0063] Figure 6 A schematic diagram of a processing platform 300 according to an exemplary embodiment of the present disclosure is shown. Figure 7 A schematic diagram of a process for determining the delay of an extended reality device to be tested by segmenting a visual pose sequence and a motion pose sequence according to an exemplary embodiment of the present disclosure is shown. Figure 8 A schematic diagram of a process for generating a first time-pose curve and a second time-pose curve according to an exemplary embodiment of the present disclosure is shown.

[0064] like Figure 1 , Figure 6-Figure 8 As shown, in some embodiments of the present disclosure, the processing platform 300 further includes a splitting unit 370, a fitting unit 380 and a delay unit 390. The splitting unit 370 is configured to perform a time-delay operation according to the motion platform 100 and the extended reality device 400 to be tested (such as Figure 2 The motion characteristic parameters of the synchronous motion of the visual pose sequence and the motion pose sequence are respectively divided into multiple motion segments. The fitting unit 380 fits the multiple motion segments of the visual pose sequence into multiple first time-pose curves, and fits the multiple motion segments of the motion pose sequence into multiple second time-pose curves. The delay unit 390 determines the delay of the extended reality device 400 to be tested based on the difference between the first time-pose curve and the second time-pose curve corresponding to the first time-pose curve in time.

[0065] Specifically, the traditional MTP delay measurement can only obtain the delay of the start node or the end node of the movement, and the delay during the movement process cannot be measured. In other words, the traditional MTP measurement can be divided into manually calculated delay and automatically calculated delay. Manually calculated delay can calculate the delay of the extended reality device through the numerical value estimated by humans, with large errors, and cannot objectively reflect the real delay of the extended display device. Automatically calculated delay measures the delay of the extended reality device through the measuring device. The key is to capture the changes in the motion state of the extended reality device and monitor the changes in the image reflected on the screen of the extended reality device. By comparing the delay between the two, the delay of the extended reality device is obtained. However, the automatic calculation of delay is limited to measuring the delay generated at a single motion node, such as the delay at the start node or the end node of the movement. In actual use scenarios, the movement of the extended reality device and the response of the virtual reality scene in the extended reality device are both a continuous process, and the delay generated in this process is a set of dynamic and continuous data. Therefore, only measuring the delay generated at a single motion node cannot reflect the dynamic changes in the delay of the extended reality device in continuous motion. Especially considering that human body movements are mostly continuous three-dimensional compound movements, for extended display devices that support motion prediction algorithms, analyzing the delay changes in continuous compound three-dimensional movements is a key indicator for evaluating their motion prediction algorithms.

[0066] According to the delay measurement system provided by at least one embodiment of the present disclosure, the motion platform that moves synchronously with the extended reality device to be tested can simulate the continuous compound three-dimensional motion of the human body, so that the delay performance of the extended reality device in different motion states can be comprehensively evaluated, and the dynamic changes of the extended reality device in the continuous compound three-dimensional motion can be observed. Using the visual platform, multiple continuous images of the target plate displayed by the extended reality device to be tested can be obtained, and then the visual posture sequence of the extended reality device to be tested can be determined. By comparing the difference between the visual posture sequence of the extended reality device to be tested and the motion posture sequence of the motion platform, the MTP delay of the extended reality device in the continuous compound three-dimensional motion of the human body can be accurately measured.

[0067] In order to enhance the above effects and improve the precision and accuracy of MTP delay measurement, the visual posture sequence and the motion posture sequence can be segmented respectively, and the visual posture sequence and the motion posture sequence can be split into multiple motion segments respectively by using motion segment similarity analysis; by analyzing multiple motion segments segment by segment, performing line segment fitting segment by segment, and performing "gap" mean statistics on the fitted first time-posture curve and the second time-posture curve, the MTP delay of the extended reality device in the continuous compound three-dimensional motion of the human body can be accurately measured.

[0068] Optionally, in the process of using motion segment similarity analysis, the motion characteristic parameters of the synchronous motion may include at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion. Therefore, appropriate synchronous motion characteristic parameters can be selected according to the type of the extended reality device to be tested and the actual needs, or according to the characteristics of the synchronous motion, so as to accurately measure the MTP delay of the extended reality device in the continuous composite three-dimensional motion of the human body. It should be noted that the present disclosure does not limit the specific content of the motion characteristic parameters of the synchronous motion.

[0069] In addition, in the process of splitting the visual pose sequence and the motion pose sequence into multiple motion segments respectively by using motion segment similarity analysis, the visual pose sequence and the motion pose sequence can be split into multiple motion segments respectively according to the motion characteristic parameters of the synchronous motion of the motion platform 100 and the extended reality device 400 to be tested. For example, at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion is regarded as the volatility reference data, and a predetermined threshold is set. When at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion is greater than the corresponding predetermined threshold, it can be considered that the volatility of this part of the data is too large. Based on this, the previous time-pose information in the visual pose sequence or the motion pose sequence can be identified as the end point of the previous motion segment, and the next time-pose information can be identified as the starting point of the next motion segment.

[0070] Alternatively, if Figure 6 As shown, the processing platform 300 may further include a preprocessing unit 360, which may perform zero bias preprocessing (such as Figure 7 Taking into account the error factors such as equipment assembly error and jitter error, the visual pose sequence and motion pose sequence are preprocessed with zero bias to improve the precision and accuracy of MTP delay measurement.

[0071] It should be noted that, for the sake of clarity, Figure 6 The processing platform 300 shown also shows the optional pre-processing unit 360. It can be distinguished that the optional pre-processing unit 360 is shown with a dotted line frame, and the connection between the pre-processing unit 360 and other units is also shown with a dotted connecting line when the pre-processing unit 360 is included.

[0072] In the process of analyzing multiple motion segments segment by segment and fitting the line segments segment by segment, the fitting can be based on a straight line or a quadratic curve. It should be noted that the present disclosure does not limit the method of line segment fitting. Afterwards, the "gap" mean statistics can be performed on the first time-posture curve and the second time-posture curve after fitting, and the MTP delay of the extended reality device in the continuous composite three-dimensional movement of the human body can be accurately measured.

[0073] Figure 8 The process of generating the first time-pose curve and the second time-pose curve according to an exemplary embodiment of the present disclosure is shown. Figure 8 In the first figure, the first motion trajectory represents the visual pose sequence, and the second motion trajectory represents the motion pose sequence, so Figure 8 The first figure shows the relationship between the posture angle changes of the visual posture sequence and the motion posture sequence over time. Using the motion segment similarity analysis, the visual posture sequence and the motion posture sequence can be split into multiple motion segments according to the motion feature parameters of the synchronous motion of the motion platform and the extended reality device to be tested. For example, Figure 8 The first figure shows a motion segment of a visual pose sequence and a motion segment of a motion pose sequence that have a corresponding relationship in time, both of which are shown by a dotted box A. Figure 8 The second and third figures show the Figure 8 The process of linear fitting of the two motion segments in the dashed box A in the first figure, Figure 8The first motion trajectory and the second motion trajectory in the second figure and the third figure respectively represent a motion segment of the visual pose sequence and a motion segment of the motion pose sequence corresponding in time to a motion segment of the visual pose sequence. Figure 8 The first motion trajectory after fitting and the second motion trajectory after fitting in the fourth figure respectively represent Figure 8 The second and third figures show the relationship between the attitude angle of a motion segment of the visual pose sequence after fitting over time, and the attitude angle of a motion segment of the motion pose sequence in the above figure after fitting over time. In other words, Figure 8 The first motion trajectory after fitting and the second motion trajectory after fitting in the fourth figure of respectively represent the first time-position curve and the second time-position curve after fitting. By performing “gap” mean statistics on the first time-position curve and the second time-position curve, we can obtain Figure 8 The delay curve shown in the fifth figure can accurately characterize the MTP delay of the extended reality device during the entire movement process of the human body's continuous complex three-dimensional movement.

[0074] In addition, in order to further improve the precision and accuracy of MTP delay measurement, expand the application field of the delay measurement system and optimize the application scenarios of the delay measurement system, multi-dimensional delay evaluation indicators can be used to characterize the results of MTP delay measurement.

[0075] Fig. 9 A schematic diagram of statistical results including a first statistical indicator and a second statistical indicator according to an exemplary embodiment of the present disclosure is shown.

[0076] Optionally, refer to Figure 6 and Fig. 9 The delay unit 390 may further include: a first calculation subunit 391. The first calculation subunit 391 determines a first statistical indicator based on the difference between the first time-pose curve and the second time-pose curve corresponding to the first time-pose curve in time, wherein the first statistical indicator may include the measured extended reality device 400 (such as Figure 2 At least one of the delay extreme value, delay average value, delay median value, and delay standard deviation (as shown).

[0077] In addition, the delay unit 390 may further include: a second calculation subunit 392. The second calculation subunit 392 may determine the second statistical indicator according to at least one of a mean value and a standard deviation of the first statistical indicator obtained through multiple measurements.

[0078] The delay evaluation index characterizing the result of MTP delay measurement may include at least one of a first statistical index and a second statistical index, wherein the first statistical index includes the delay extreme value, the delay average value, the delay median value, the delay standard deviation, etc., which can be used to measure the delay of the extended reality device to be tested in a single measurement process; the second statistical index is at least one of the mean and standard deviation of the first statistical index of multiple measurements, which can be used to measure the stability of the delay of the extended reality device to be tested from a macro or overall perspective. Fig. 9 The figure shows the statistical results of six MTP delay measurements of an extended reality device 400 to be tested according to at least one embodiment of the present disclosure. Users can select appropriate delay evaluation indicators from the above multi-dimensional delay evaluation indicators to represent the results of MTP delay measurement based on the actual situation such as the type and application field of their extended reality device.

[0079] Therefore, according to the delay measurement system provided by at least one embodiment of the present disclosure, the motion platform that moves synchronously with the extended reality device to be tested can simulate the continuous compound three-dimensional motion of the human body, so that the delay performance of the extended reality device in different motion states can be comprehensively evaluated, and the dynamic changes of the extended reality device in the continuous compound three-dimensional motion can be observed. Using the visual platform, multiple continuous images of the target plate displayed by the extended reality device to be tested can be obtained, and then the visual posture sequence of the extended reality device to be tested can be determined. By comparing the difference between the visual posture sequence of the extended reality device to be tested and the motion posture sequence of the motion platform, the MTP delay of the extended reality device in the continuous compound three-dimensional motion of the human body can be accurately measured.

[0080] Fig.10 The flowchart of a delay measurement method 2000 provided in an embodiment of the present disclosure, wherein the delay measurement method 2000 may include the following steps: S1: Obtain a motion posture sequence of a motion platform, wherein the motion platform is fixedly connected to the extended reality device to be tested, and drives the extended reality device to be tested to move synchronously.

[0081] S2: Acquire multiple continuous images of a target displayed by the extended reality device to be tested.

[0082] S3: Obtain a visual pose sequence of the extended reality device to be tested based on multiple continuous images.

[0083] S4: Determine the delay of the extended reality device to be tested based on the visual pose sequence and the motion pose sequence.

[0084] The various steps of the delay measurement method 2000 will be described in detail below with reference to the accompanying drawings.

[0085] Step S1

[0086] refer to Figure 1 , Figure 2 and Fig.10 In some embodiments of the present disclosure, the motion platform 100 is fixedly connected to the extended reality device 400 to be tested, for example, rigidly fixedly connected. In addition, the motion platform 100 can drive the extended reality device 400 to be tested to move synchronously. The motion platform 100 may include any suitable structural components and may perform various translational and rotational movements in a world coordinate system. Taking the extended reality device 400 to be tested as an HMD as an example, the motion platform 100 fixedly connected to the HMD can simulate the horizontal movement, horizontal rotation, tilting movement, etc. of the human body.

[0087] Optionally, the visual platform 200 is used to obtain a target plate 410 (such as Figure 3 The delay measurement method 2000 further includes: before and during the synchronous movement, performing time synchronization on the motion platform 100 and the visual platform 200.

[0088] For example, the motion platform 100 and the visual platform 200 are synchronized by using a clock source. During the time synchronization process, the motion platform 100 and the visual platform 200 can be synchronized by using the PTP protocol. A clock source with high precision and high accuracy is used as the master clock to send a timestamp to the motion platform 100 and the visual platform 200 as slave clocks. The slave clock compares the timestamp with its own local time and adjusts the time according to the difference, thereby ensuring that the subsequent motion posture sequence obtained based on the motion platform 100 is synchronized with the visual posture sequence obtained based on the visual platform 200.

[0089] Step S2

[0090] refer to Figure 1-Figure 4 , Fig.10 In some embodiments of the present disclosure, the visual platform 200 may be used to obtain a plurality of continuous images of the target plate 410 displayed by the XR device 400 to be tested, for example Figure 4 One image 420 of a plurality of consecutive images is shown.

[0091] The visual platform 200 can be used to simulate the scene where the human eye receives the continuous images displayed by the extended reality device 400 to be tested. Optionally, the visual platform 200 may include at least one of a camera and a video camera, and the camera may include an IR camera (Infrared Camera), an RGB camera (RedGreenBlue Camera), or a black and white camera. In addition, the visual platform 200 may also include other structures such as photoelectric sensors, and the present disclosure does not limit the specific internal structure of the visual platform 200.

[0092] Optionally, the target plate 410 may be a virtual target plate located in the extended reality device 400 to be tested. The virtual target plate may include a visual marker calibration plate in the visual reference library. For example, a two-dimensional code calibration plate such as ApriTag and ArUco. Using a virtual target plate for delay measurement can simplify the structure of the delay measurement system and reduce the difficulty of implementing delay measurement.

[0093] Step S3

[0094] refer to Figure 1 , Figure 3 and Figure 4 In some embodiments of the present disclosure, step S3 of obtaining a visual pose sequence of the extended reality device to be tested based on multiple continuous images may, for example, include: extracting pixel coordinates of feature points of the target plate 410 based on multiple continuous images; converting the pixel coordinates into an initial measured pose sequence in a first coordinate system with a visual platform 200 for shooting the multiple continuous images as a reference according to a preset spatial geometric constraint relationship between the feature points; and converting the initial measured pose sequence into a visual pose sequence in a second coordinate system with a motion platform 100 as a reference.

[0095] Specifically, the processing platform 300 can be used to obtain a visual pose sequence of the extended reality device to be tested based on multiple continuous images. The processing platform 300 can be various general and / or special processing components with processing and computing capabilities. Some examples of the processing platform 300 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc.

[0096] The processing platform 300 obtains the visual pose sequence of the extended reality device 400 to be tested based on a plurality of continuous images, and determines the delay of the extended reality device 400 to be tested based on the visual pose sequence and the motion pose sequence of the motion platform 100. Optionally, the processing platform 300 may include various suitable communication buses and input / output interfaces for realizing connection with the motion platform 100 and the visual platform 200.

[0097] In addition, before extracting the pixel coordinates of the feature points of the target plate 410 based on the multiple continuous images, the multiple continuous images may be processed, and the processing may include at least one of downsampling processing and Gaussian blur processing. Since the appearance of moiré when shooting the screen of the extended reality device 400 to be tested will cause inaccurate extraction of 2D feature points, or even failure to extract them, it is necessary to process the multiple continuous images obtained, and the influence of moiré on the above images can be reduced by at least one of downsampling processing and Gaussian blur processing.

[0098] In addition, in order to compare the motion posture changes in the real world with the motion posture changes in the virtual world of the extended reality device 400 to be tested, it is necessary to unify the two motion posture changes in the same coordinate system. The motion changes in the virtual world can be determined in a first coordinate system (e.g., a camera coordinate system) with reference to the visual platform 200, and the motion changes in the real world can be determined in a second coordinate system (e.g., a world coordinate system) with reference to the motion platform 100, so it is necessary to calibrate the motion platform 100 and the visual platform 200 for coordinate system conversion.

[0099] Since the eye-adaptive distances of different extended reality devices to be tested are different, the position of the visual platform 200 needs to be adjusted to adapt to the extended reality device 400 to be tested before measurement, which further causes the relative positions of the visual platform 200 and the motion platform 100 to change, causing the original conversion relationship to no longer be applicable and requiring recalibration.

[0100] Optionally, before converting the initial measured pose sequence into a visual pose sequence, the initial measured pose sequence may be time-corrected. Although the timestamp of the image of the photographed standard and the timestamp of the trajectory of the motion platform are under the same clock source, due to the existence of MTP, the process of photographing the image has a certain delay, so time correction is required.

[0101] For example, the fixed connection between the motion platform 100 and the extended reality device 400 to be tested can be considered as a rigid connection of a rigid body, and the fixed connection between the visual platform 200 and the extended reality device 400 to be tested can also be considered as a rigid connection of a rigid body. Based on the principle of rigid body rotation invariance, the initial measured pose sequence can be time-corrected before converting the initial measured pose sequence into a visual pose sequence. For example, after filtering the angular velocity of the pose, the time delay of the image shooting process caused by the existence of MTP is obtained based on cross-correlation, and the relevant data is time-corrected and aligned.

[0102] Step S4

[0103] refer to Figure 1 , Figure 2 as well as Figure 7-Figure 8 Step S4 determines the delay of the extended reality device to be tested based on the visual pose sequence and the motion pose sequence, which may, for example, include: splitting the visual pose sequence and the motion pose sequence into multiple motion segments according to the motion characteristic parameters of the synchronous motion; fitting the multiple motion segments of the visual pose sequence into multiple first time-pose curves, and fitting the multiple motion segments of the motion pose sequence into multiple second time-pose curves; and determining the delay of the extended reality device to be tested based on the difference between the first time-pose curve and the second time-pose curve corresponding to the first time-pose curve in time.

[0104] Optionally, the visual posture sequence may include a plurality of first time-pose information arranged in time order, the first time-pose information including the first time information and the first posture angle corresponding to the first time information, wherein the first posture angle includes the first roll angle, the first yaw angle and the first pitch angle. For example, the first time-pose information may include: timestamp_roll_pitch_yaw, wherein 000_10_20_30 may represent that the angle postures of roll, pitch and yaw at the starting time are (10°, 20°, 30°) respectively, and in the right-handed Cartesian coordinate system of three-dimensional space, roll is the roll angle, around the z-axis; pitch is the pitch angle, around the x-axis; yaw is the yaw angle, around the y-axis.

[0105] The motion posture sequence may include a plurality of second time-posture information arranged in time order, and the second time-posture information may include second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle, and a second pitch angle. Similarly, the second time-posture information may also include: timestamp_roll_pitch_yaw.

[0106] Traditional MTP delay measurement can only obtain the delay of the start node or the end node of the movement, and cannot measure the delay during the movement. In other words, traditional MTP measurement can be divided into manually calculated delay and automatically calculated delay. Manually calculated delay can calculate the delay of the extended reality device through manually estimated numerical values, which has a large error and cannot objectively reflect the real delay of the extended display device. Automatically calculated delay uses a measuring device to measure the delay of the extended reality device. The key is to capture the changes in the motion state of the extended reality device and monitor the changes in the image reflected on the screen of the extended reality device. By comparing the delay between the two, the delay of the extended reality device can be obtained. However, the automatic calculation of delay is limited to measuring the delay generated at a single motion node, such as the delay at the start node or the end node of the movement. In actual use scenarios, the movement of the extended reality device and the response of the virtual reality scene in the extended reality device are a continuous process, and the delay generated in this process is a set of dynamic and continuous data. Therefore, only measuring the delay generated at a single motion node cannot reflect the dynamic changes in the delay of the extended reality device in continuous motion. Especially considering that human body movements are mostly continuous three-dimensional compound movements, for extended display devices that support motion prediction algorithms, analyzing the delay changes in continuous compound three-dimensional movements is a key indicator for evaluating their motion prediction algorithms.

[0107] According to the delay measurement method provided by at least one embodiment of the present disclosure, the motion platform that moves synchronously with the extended reality device to be tested can simulate the continuous compound three-dimensional motion of the human body, so that the delay performance of the extended reality device in different motion states can be comprehensively evaluated, and the dynamic changes of the extended reality device in the continuous compound three-dimensional motion can be observed. Based on the multiple continuous images of the target plate displayed by the extended reality device to be tested, the visual posture sequence of the extended reality device to be tested can be determined. By comparing the difference between the visual posture sequence of the extended reality device to be tested and the motion posture sequence of the motion platform, the MTP delay of the extended reality device in the continuous compound three-dimensional motion of the human body can be accurately measured.

[0108] In order to enhance the above effects and improve the precision and accuracy of MTP delay measurement, the visual posture sequence and the motion posture sequence can be segmented respectively, and the visual posture sequence and the motion posture sequence can be split into multiple motion segments respectively by using motion segment similarity analysis; by analyzing multiple motion segments segment by segment, performing line segment fitting segment by segment, and performing "gap" mean statistics on the fitted first time-posture curve and the second time-posture curve, the MTP delay of the extended reality device in the continuous compound three-dimensional motion of the human body can be accurately measured.

[0109] Optionally, in the process of using motion segment similarity analysis, the motion characteristic parameters of the synchronous motion may include at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion. Therefore, appropriate synchronous motion characteristic parameters can be selected according to the type of the extended reality device to be tested and the actual needs, or according to the characteristics of the synchronous motion, so as to accurately measure the MTP delay of the extended reality device in the continuous composite three-dimensional motion of the human body. It should be noted that the present disclosure does not limit the specific content of the motion characteristic parameters of the synchronous motion.

[0110] In addition, in the process of splitting the visual pose sequence and the motion pose sequence into multiple motion segments respectively by using motion segment similarity analysis, the visual pose sequence and the motion pose sequence can be split into multiple motion segments respectively according to the motion characteristic parameters of the synchronous motion of the motion platform 100 and the extended reality device 400 to be tested. For example, at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion is regarded as the volatility reference data, and a predetermined threshold is set. When at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion is greater than the corresponding predetermined threshold, it can be considered that the volatility of this part of the data is too large. Based on this, the previous time-pose information in the visual pose sequence or the motion pose sequence can be identified as the end point of the previous motion segment, and the next time-pose information can be identified as the starting point of the next motion segment.

[0111] In addition, in some embodiments of the present disclosure, taking into account error factors such as device assembly error and jitter error, the visual pose sequence and the motion pose sequence may be subjected to zero bias preprocessing before fitting multiple motion segments of the visual pose sequence and multiple motion segments of the motion pose sequence, respectively.

[0112] In the process of analyzing multiple motion segments segment by segment and fitting the line segments segment by segment, the fitting can be based on a straight line or a quadratic curve. It should be noted that the present disclosure does not limit the method of line segment fitting. Afterwards, the "gap" mean statistics can be performed on the first time-posture curve and the second time-posture curve after fitting, and the MTP delay of the extended reality device in the entire movement process of the continuous compound three-dimensional movement of the human body can be accurately measured.

[0113] In order to further improve the precision and accuracy of MTP delay measurement, expand the application field of the delay measurement system and optimize the application scenarios of the delay measurement system, multi-dimensional delay evaluation indicators can be used to characterize the results of MTP delay measurement.

[0114] The delay evaluation index characterizing the result of MTP delay measurement may include at least one of a first statistical index and a second statistical index, wherein the first statistical index includes the delay extreme value, the delay average value, the delay median value, the delay standard deviation, etc., which can be used to measure the delay of the extended reality device to be tested in a single measurement process; the second statistical index is at least one of the mean and standard deviation of the first statistical index of multiple measurements, which can be used to measure the stability of the delay of the extended reality device to be tested from a macro or overall perspective. Fig. 9 The figure shows the statistical results of six MTP delay measurements of an extended reality device 400 to be tested according to at least one embodiment of the present disclosure. Users can select appropriate delay evaluation indicators from the above multi-dimensional delay evaluation indicators to represent the results of MTP delay measurement based on the actual situation such as the type and application field of their extended reality device.

[0115] Therefore, according to the delay measurement method provided by at least one embodiment of the present disclosure, the motion platform that moves synchronously with the extended reality device to be tested can simulate the continuous compound three-dimensional motion of the human body, and can comprehensively evaluate the delay performance of the extended reality device in different motion states, and can observe the dynamic changes of the extended reality device in the continuous compound three-dimensional motion. Based on the multiple continuous images of the target plate displayed by the extended reality device to be tested, the visual posture sequence of the extended reality device to be tested can be determined. By comparing the difference between the visual posture sequence of the extended reality device to be tested and the motion posture sequence of the motion platform, the MTP delay of the extended reality device in the continuous compound three-dimensional motion of the human body can be accurately measured.

[0116] Fig.11A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0117] like Fig.11 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0118] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0119] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the delay measurement method. For example, in some embodiments, the delay measurement method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the delay measurement method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the delay measurement method in any other appropriate manner (e.g., by means of firmware).

[0120] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0122] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0124] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0125] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private server (VPS) services.

[0126] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0127] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A delay measurement system, characterized in that: include: A motion platform, fixedly connected to the extended reality device to be tested, and driving the extended reality device to be tested to move synchronously; A visual platform, fixedly connected to the extended reality device to be tested, and acquiring a plurality of continuous images of a target plate displayed by the extended reality device to be tested; as well as The processing platform obtains a visual pose sequence of the extended reality device to be tested based on the multiple continuous images, and determines a delay of the extended reality device to be tested based on the visual pose sequence and a motion pose sequence of the motion platform.

2. The system according to claim 1, wherein: The processing platform also includes: a splitting unit, which splits the visual posture sequence and the motion posture sequence into a plurality of motion segments according to motion characteristic parameters of the synchronous motion; a fitting unit, which fits the multiple motion segments of the visual pose sequence into multiple first time-pose curves, and fits the multiple motion segments of the motion pose sequence into multiple second time-pose curves; and The delay unit determines the delay of the extended reality device to be tested based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time.

3. The system according to claim 2, wherein: The motion characteristic parameter of the synchronous motion includes at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion.

4. The system according to claim 2, wherein: The delay unit also includes: The first calculation subunit determines a first statistical indicator based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time, wherein the first statistical indicator includes at least one of the extreme value of delay, the average value of delay, the median value of delay, and the standard deviation of delay of the extended reality device to be tested.

5. The system according to claim 4, wherein: The delay unit also includes: The second calculation subunit determines a second statistical indicator according to at least one of a mean value and a standard deviation of the first statistical indicator obtained through multiple measurements.

6. The system according to claim 2, wherein: The processing platform also includes: The preprocessing unit performs zero bias preprocessing on the visual pose sequence and the motion pose sequence respectively before fitting the multiple motion segments of the visual pose sequence and the multiple motion segments of the motion pose sequence respectively.

7. The system according to claim 1, wherein: The system further comprises: The time platform comprises a clock source, wherein the clock source is used to synchronize the time of the motion platform and the visual platform before and during the synchronous movement.

8. The system according to claim 1, wherein: The processing platform also includes: A first conversion unit extracts pixel coordinates of feature points of the target based on the plurality of continuous images; A second conversion unit, converting the pixel coordinates into an initial measured pose sequence in a first coordinate system with the visual platform as a reference, according to a preset spatial geometric constraint relationship between the feature points; and The calibration unit converts the initially measured posture sequence into the visual posture sequence in a second coordinate system with the motion platform as a reference.

9. The system according to claim 8, wherein: The processing platform also includes: A time correction unit performs time correction on the initially measured pose sequence before converting the initially measured pose sequence into the visual pose sequence.

10. The system according to claim 8, wherein: The processing platform also includes: The image processing unit processes the plurality of continuous images before extracting the pixel coordinates of the feature points of the target based on the plurality of continuous images, wherein the processing includes at least one of downsampling processing and Gaussian blur processing.

11. The system according to claim 1, wherein: The target plate is a virtual target plate, wherein the virtual target plate includes a visual marker calibration plate in a visual reference library.

12. The system of claim 1, wherein: The visual posture sequence includes a plurality of first time-pose information arranged in time order, the first time-pose information includes first time information and a first posture angle corresponding to the first time information, wherein the first posture angle includes a first roll angle, a first yaw angle and a first pitch angle; and The motion posture sequence includes a plurality of second time-posture information arranged in chronological order, the second time-posture information including second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle and a second pitch angle.

13. A delay measurement method, characterized in that: include: Acquiring a motion posture sequence of a motion platform, wherein the motion platform is fixedly connected to the extended reality device to be tested, and drives the extended reality device to be tested to move synchronously; Acquire a plurality of continuous images of a target displayed by the extended reality device to be tested; Obtaining a visual pose sequence of the extended reality device to be tested based on the multiple continuous images; as well as The delay of the extended reality device to be tested is determined based on the visual pose sequence and the motion pose sequence.

14. The method according to claim 13, wherein: Determining the delay of the extended reality device to be tested based on the visual pose sequence and the motion pose sequence includes: According to the motion characteristic parameters of the synchronous motion, the visual posture sequence and the motion posture sequence are respectively split into a plurality of motion segments; Fitting a plurality of motion segments of the visual pose sequence into a plurality of first time-pose curves, and fitting a plurality of motion segments of the motion pose sequence into a plurality of second time-pose curves; and Based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time, the delay of the extended reality device to be tested is determined.

15. The method according to claim 14, wherein: The motion characteristic parameter of the synchronous motion includes at least one of the speed of the synchronous motion, the acceleration of the synchronous motion, and the derivative of the acceleration of the synchronous motion.

16. The method according to claim 14, wherein: Determining the delay of the measured augmented reality device based on a difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time includes: Based on the difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time, a first statistical indicator is determined, wherein the first statistical indicator includes at least one of the extreme value of delay, the average value of delay, the median value of delay, and the standard deviation of delay of the extended reality device to be tested.

17. The method according to claim 16, wherein: Determining the delay of the measured extended reality device based on a difference between the first time-pose curve and a second time-pose curve corresponding to the first time-pose curve in time further includes: The second statistical indicator is determined according to at least one of a mean value and a standard deviation of the first statistical indicator obtained through multiple measurements.

18. The method according to claim 14, wherein: Before fitting the multiple motion segments of the visual pose sequence and the multiple motion segments of the motion pose sequence respectively, the method further includes: Zero bias preprocessing is performed on the visual pose sequence and the motion pose sequence respectively.

19. The method according to claim 13, wherein: The plurality of continuous images are acquired by using a visual platform, and the method further comprises: before and during the synchronous movement, performing time synchronization on the motion platform and the visual platform.

20. The method according to claim 13, wherein: Acquiring a visual pose sequence of the extended reality device to be tested based on the multiple continuous images includes: Extracting pixel coordinates of feature points of the target based on the plurality of continuous images; According to the preset spatial geometric constraint relationship between the feature points, the pixel coordinates are converted into an initial measured pose sequence in a first coordinate system with a visual platform for taking the multiple consecutive images as a reference; and The initially measured pose sequence is converted into the visual pose sequence in a second coordinate system with the motion platform as a reference.

21. The method according to claim 20, wherein: The method further includes: before converting the initially measured pose sequence into the visual pose sequence, performing time correction on the initially measured pose sequence.

22. The method according to claim 20, wherein: The method further includes: before extracting the pixel coordinates of the feature points of the target based on the multiple continuous images, processing the multiple continuous images, wherein the processing includes at least one of downsampling processing and Gaussian blur processing.

23. The method according to claim 13, wherein: The target plate is a virtual target plate, wherein the virtual target plate includes a visual marker calibration plate in a visual reference library.

24. The method according to claim 13, wherein: The visual posture sequence includes a plurality of first time-pose information arranged in time order, the first time-pose information includes first time information and a first posture angle corresponding to the first time information, wherein the first posture angle includes a first roll angle, a first yaw angle and a first pitch angle; and The motion posture sequence includes a plurality of second time-posture information arranged in chronological order, the second time-posture information including second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle and a second pitch angle.

25. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the delay measurement method according to any one of claims 13 to 24.

26. A computer-readable storage medium storing a computer program, characterized in that: The computer instructions are used to enable the computer to execute the delay measurement method according to any one of claims 13 to 24.

27. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the delay measurement method according to any one of claims 13 to 24 is implemented.

Images