Delay compensation method and device for hand key point position, equipment and medium

Abstract

The present disclosure provides a method and device for hand key point position delay compensation, equipment and medium. The specific implementation scheme is: based on the obtained hand image containing the hand at the current exposure time and the attitude of the terminal device under the reference coordinate system, the spatial position of each hand key point under the reference coordinate system at the current exposure time is determined; based on the spatial position of the hand key point under the reference coordinate system at the current exposure time, the post-filtering state of the hand key point at the current exposure time is determined by using a preset filtering algorithm; based on the target display time, the current exposure time and the post-filtering state of the hand key point at the current exposure time, the post-filtering position of the hand key point at the current exposure time is compensated for delay to obtain the target position of the hand key point under the reference coordinate system at the target display time; wherein the target display time is later than the current exposure time; the display system is configured on the terminal device; the target position is output to the display system.

Description

Technical Field

[0001] This disclosure relates to the field of gesture recognition technology, and in particular to a method, apparatus, device, and medium for delay compensation of key hand position. Background Technology

[0002] With the rapid development of science and technology, gesture recognition has been applied to various fields, such as smart wearable products, where gesture recognition is used to complete corresponding controls. Summary of the Invention

[0003] In view of this, the purpose of this disclosure is to provide a method, device, equipment and medium for delay compensation of hand key point positions, which performs delay compensation of hand key point positions based on hand movements, effectively improves the real-time performance and tracking of hand key point positions, and improves visual perception and interactive experience.

[0004] In a first aspect, embodiments of this disclosure provide a delay compensation method for the position of hand key points, comprising: determining the spatial position of each hand key point in the reference coordinate system at the current exposure time based on an image of the hand containing the hand at the current exposure time and the posture of the terminal device in the reference coordinate system; wherein the hand image is acquired by a camera configured on the terminal device; determining the filtered state of the hand key points at the current exposure time using a preset filtering algorithm based on the spatial position of the hand key points in the reference coordinate system at the current exposure time; wherein the filtered state includes filtered position, filtered velocity, and filtered acceleration; performing delay compensation on the filtered position of the hand key points at the current exposure time based on the target display time, the current exposure time, and the filtered state of the hand key points at the current exposure time to obtain the target position of the hand key points in the reference coordinate system at the target display time; wherein the target display time is later than the current exposure time; the target display time represents the planned screen refresh time of the display system; the display system is configured on the terminal device; and outputting the target position to the display system.

[0005] Secondly, embodiments of this disclosure provide a delay compensation device for hand key point positions, comprising: a first processing module, configured to determine the spatial position of each hand key point in the reference coordinate system at the current exposure time based on an image of the hand containing the hand at the current exposure time and the posture of the terminal device in the reference coordinate system; wherein the hand image is acquired by a camera configured on the terminal device; a second processing module, configured to determine the filtered state of the hand key points at the current exposure time using a preset filtering algorithm based on the spatial position of the hand key points in the reference coordinate system at the current exposure time; wherein the filtered state includes filtered position, filtered velocity, and filtered acceleration; a third processing module, configured to perform delay compensation on the filtered position of the hand key points at the current exposure time based on the target display time, the current exposure time, and the filtered state of the hand key points at the current exposure time, to obtain the target position of the hand key points in the reference coordinate system at the target display time; wherein the target display time is later than the current exposure time; the target display time represents the planned screen refresh time of the display system; the display system is configured on the terminal device; and an output module, configured to output the target position to the display system.

[0006] Thirdly, embodiments of this disclosure provide a computer-readable storage medium storing a computer program for executing the delay compensation method for key hand position provided in any of the above embodiments of this disclosure.

[0007] Fourthly, embodiments of this disclosure provide an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the delay compensation method for hand key point positions provided in any of the above embodiments of this disclosure.

[0008] Fifthly, embodiments of this disclosure provide a head-mounted display device, the head-mounted display device comprising:

[0009] Processor; memory for storing processor-executable instructions; processor for reading executable instructions from memory and executing the instructions to implement the delay compensation method for hand key point positions provided in any of the above embodiments of this disclosure.

[0010] Sixthly, embodiments of this disclosure provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the delay compensation method for key hand position provided in any of the above embodiments of this disclosure.

[0011] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0012] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 This is an exemplary system architecture that can be applied to embodiments of the delay compensation method or apparatus for hand key point positions disclosed herein;

[0014] Figure 2 This is a flowchart illustrating a delay compensation method for key hand positions provided in an exemplary embodiment of this disclosure.

[0015] Figure 3 This is a flowchart illustrating step 40 provided in an exemplary embodiment of this disclosure;

[0016] Figure 4 This is a flowchart illustrating step 30 provided in an exemplary embodiment of this disclosure;

[0017] Figure 5 This is a flowchart illustrating step 20 provided in an exemplary embodiment of this disclosure;

[0018] Figure 6 This is a schematic diagram of the time node for delay compensation provided in an exemplary embodiment of this disclosure;

[0019] Figure 7 This is a schematic diagram of the structure of a delay compensation device for key hand positions provided in an exemplary embodiment of the present disclosure;

[0020] Figure 8 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0021] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0022] In the description of this disclosure, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the purpose of simplifying the description of this disclosure and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this disclosure.

[0023] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0024] In the description of this disclosure, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linkage" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure based on the specific circumstances.

[0025] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0026] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0027] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0028] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0029] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0030] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0031] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0032] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0033] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0034] This disclosure can be applied to electronic devices such as head-mounted displays, terminal devices, and servers, and can operate in conjunction with numerous other general-purpose or special-purpose computing system environments or configurations. These electronic devices can be described in the general context of computer-executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0035] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0036] Figure 1 This is an exemplary system architecture applicable to embodiments of the delay compensation method or apparatus for hand key point positioning disclosed herein. The system architecture may include a head-mounted display device 1, a network 2, and a server 3. The network 2 can provide a communication link between the head-mounted display device 1 and the server 3. The network 2 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0037] The aforementioned head-mounted display device 1 can be an electronic device with image display capabilities, including but not limited to AR (Augmented Reality) smart glasses, VR (Virtual Reality) smart glasses, etc. The head-mounted display device 1 can be an all-in-one device, integrating the functionality of a computing unit. This computing unit provides processing power to the head-mounted display device, and various client applications installed on the head-mounted display device are actually installed within the computing unit. Alternatively, the head-mounted display device can also be a separate device, comprising a head-mounted display and a computing unit separately positioned from the head-mounted display. The display screen of the head-mounted display is used to display the images processed by the computing unit. The computing unit in the separate head-mounted display device can be a terminal device, such as a mobile phone, controller, adapter, computer, tablet, or in-vehicle infotainment system. Here, for ease of illustration, when the execution subject is a head-mounted display device, this document will subsequently use an all-in-one head-mounted display device as the action execution subject to describe the solution disclosed herein, and will use head-mounted display device 1 to represent it in the figures. Those skilled in the art should understand that, for split-type head-mounted display devices, the execution entity is the computing unit.

[0038] The head-mounted display device 1 can provide various services. For example, it can acquire a hand image containing the hand based on a camera configured on the head-mounted display device 1; based on the hand image containing the hand at the current exposure time and the posture of the head-mounted display device in the reference coordinate system, it can determine the spatial position of each hand key point in the reference coordinate system at the current exposure time; based on the spatial position of the hand key points in the reference coordinate system at the current exposure time, it can use a preset filtering algorithm to determine the filtered state of the hand key points at the current exposure time; wherein, the filtered state includes the filtered position, filtered velocity, and filtered acceleration; then, based on the target display time, the current exposure time, and the filtered state of the hand key points at the current exposure time, it can perform delay compensation on the filtered position of the hand key points at the current exposure time to obtain the target position of the hand key points in the reference coordinate system at the target display time; wherein, the target display time is later than the current exposure time; the target display time represents the planned screen refresh time of the display system; the display system is configured on the head-mounted display device; and the target position of the hand key points in the reference coordinate system is output to the display system. This allows the display system to display virtual content related to the hand key points at the target location when the target display time arrives. The delay compensation method for hand key point location provided in this embodiment is typically executed by the head-mounted display device 1, and correspondingly, the delay compensation device for hand key point location is typically provided in the head-mounted display device 1.

[0039] It should also be noted that the delay compensation method for hand key point positions provided in this disclosure can also be applied to server 3, which can be a backend page server. When the method of this disclosure is applied to server 3, after receiving a hand image containing a hand from the head-mounted display device, server 3 can determine the target position of the corresponding delay-compensated hand key points based on the hand image, and output the target position to the head-mounted display device for display. Accordingly, a delay compensation device for hand key point positions can also be set in server 3.

[0040] Besides servers and head-mounted displays, the latency compensation method for hand keypoint positioning can also be applied to other terminal devices (not shown in the figure). These other terminal devices can be mobile phones, tablets, computers, etc.

[0041] Exemplary methods

[0042] Figure 2 This is a schematic flowchart illustrating a delay compensation method for key hand position measurements provided in an exemplary embodiment of this disclosure. Embodiments of this disclosure can be applied to electronic devices, such as… Figure 2 As shown, the method includes the following steps:

[0043] Step 20: Based on the hand image containing the hand at the current exposure time and the posture of the terminal device in the reference coordinate system, determine the spatial position of each key point of the hand in the reference coordinate system at the current exposure time.

[0044] In some optional embodiments of this disclosure, the hand image is acquired by a camera configured on the terminal device.

[0045] The execution entity for the delay compensation method for key hand points (which can be a terminal device, for example, a...) Figure 1 The head-mounted display device (or server shown) acquires a hand image containing the hand at the current exposure moment. Based on this image and the device's pose in the reference coordinate system, it determines the spatial position of each key point of the hand at the current exposure moment. The terminal device can be a device equipped with a camera for capturing hand images, such as a head-mounted display device, mobile phone, or tablet. The current exposure moment refers to the exposure moment of the latest acquired frame of the hand image. The reference coordinate system is a relatively stable coordinate system, relatively stationary compared to the real world, and can be set according to actual needs. For example, it could be the camera coordinate system when the camera captures the first frame, the world coordinate system, or other relative three-dimensional coordinate systems; the specific choice is not limited. The camera coordinate system when the camera captures the first frame can help confirm the directions of the three axes of the reference coordinate system.

[0046] In some optional embodiments of this disclosure, the specific points included in the key points of the hand can be set according to actual needs. For example, the key points of the hand may include at least one of 21 key points representing the fingertip, knuckle, base of finger joint, thumb carpometacarpal joint, and wrist.

[0047] In some optional embodiments of this disclosure, hand key points can be extracted from hand images in any feasible manner, such as based on any feasible hand key point extraction model obtained through pre-training, which can be set according to actual needs.

[0048] Step 30: Based on the spatial position of the key hand points in the reference coordinate system at the current exposure time, a preset filtering algorithm is used to determine the filtered state of the key hand points at the current exposure time.

[0049] In some optional embodiments of this disclosure, the filtered state includes the filtered position, the filtered velocity, and the filtered acceleration.

[0050] In some optional embodiments of this disclosure, the preset filtering algorithm can be set according to actual needs. For example, the preset filtering algorithm may include at least one of a position filter, a velocity filter, and an acceleration filter. The position filter, velocity filter, and acceleration filter may be filters of the same type or different types. Filters may include, for example, a 1 Euro filter, a Kalman filter, etc.

[0051] Step 40: Based on the target display time, the current exposure time, and the filtered state of the hand key points at the current exposure time, perform delay compensation on the filtered position of the hand key points at the current exposure time to obtain the target position of the hand key points in the reference coordinate system at the target display time.

[0052] In some optional embodiments of this disclosure, the target display time is later than the current exposure time; the target display time represents the planned screen refresh time of the display system; the display system is configured on a terminal device.

[0053] In some optional embodiments of this disclosure, the target display time can be characterized by an absolute time. For example, the target display time can be a timestamp, which can be used to determine the specific year, month, day, hour, minute, and second. Based on the target display time and the current exposure time, the required compensation duration for delay compensation can be determined.

[0054] In some optional embodiments of this disclosure, the target display time can also be characterized by a relative time. For example, the message received from the display system may contain a time interval relative to the current time (e.g., denoted as t1) or the time when the display system sends the message to the target (e.g., denoted as t2), such as 100 milliseconds (ms). Based on the current time t1 or the sending time t2, the target display time can be determined, and the required compensation duration for delay compensation can be calculated. Taking the current time as an example, adding 100 milliseconds to the current time t1, and then adding the time interval between the current time and the current exposure time, yields the compensation duration.

[0055] In some optional embodiments of this disclosure, delay compensation for the filtered position of key hand points at the current exposure time can be achieved based on the principle of hand acceleration. For example, based on the filtered acceleration, filtered velocity, and compensation duration, the displacement change of the key hand points can be estimated, and then delay compensation can be performed on the filtered position based on the displacement change to obtain the target position of the key hand points at the target display time.

[0056] Step 50: Output the target location to the display system.

[0057] In some alternative embodiments of this disclosure, the display system may be a display system for a head-mounted display device.

[0058] After obtaining the target position of the hand key point in the reference coordinate system at the target display time, the execution subject of the delay compensation method for the hand key point position outputs the target position of each hand key point in the reference coordinate system to the display system. The display system has a certain control and processing capability. After receiving the target position, it can control the rendering to be completed and displayed when the target display time arrives.

[0059] The time delay compensation method for hand key point position provided in this embodiment of the present disclosure obtains the accurate and effective filtered state of the hand key point at the current exposure time by using a preset filtering algorithm based on the spatial position of the hand key point in the reference coordinate system at the current exposure time. Then, based on the target display time, the current exposure time, and the filtered state of the hand key point at the current exposure time, time delay compensation is performed on the filtered position of the hand key point at the current exposure time to obtain the target position of the hand key point in the reference coordinate system at the target display time. This makes the position of the hand key point displayed at the target display time more real-time, better able to follow hand movements, effectively improve real-time performance and tracking performance, and thus improve visual perception and interactive experience.

[0060] Figure 3 This is a flowchart illustrating step 40 provided in an exemplary embodiment of this disclosure.

[0061] In some optional embodiments of this disclosure, step 40, based on the target display time, the current exposure time, and the filtered state of the hand key points at the current exposure time, performs delay compensation on the filtered position of the hand key points at the current exposure time to obtain the target position of the hand key points in the reference coordinate system at the target display time, including:

[0062] Step 410: Based on the filtered acceleration and filtered velocity at the current exposure time, determine the displacement change of the key hand points from the current exposure time to the target display time.

[0063] In some optional embodiments of this disclosure, the filtered acceleration at the current exposure time can be expressed as: The filtered velocity can be expressed as Let w represent the reference coordinate system and t represent the current exposure time. Then, the displacement change Δx of the key point of the hand from the current exposure time to the target display time can be expressed as follows:

[0064]

[0065] in, Indicates the time when the target is displayed.

[0066] Step 420: Based on the filtered position and displacement change at the current exposure time, determine the target position of the key hand points in the reference coordinate system at the target display time.

[0067] In some optional embodiments of this disclosure, the filtered position of the current exposure time can be represented as: The target displays the position of the hand key points in the reference coordinate system at the moment of target display. It can be represented as follows:

[0068]

[0069] The meanings of each symbol are explained above.

[0070] This embodiment estimates the displacement change of the hand key point from the current exposure time to the target display time by using filtered acceleration and filtered velocity. This is then used to compensate for the delay in the filtered position, thereby obtaining the target position of the hand key point at the target display time. This enables the real-time display of the hand key point at the target display time, further improving real-time performance and tracking accuracy.

[0071] Figure 4 This is a flowchart illustrating step 30 provided in an exemplary embodiment of this disclosure.

[0072] In some optional embodiments of this disclosure, step 30, which determines the filtered state of the hand key points at the current exposure time based on the spatial position of the hand key points in the reference coordinate system at the current exposure time using a preset filtering algorithm, includes:

[0073] Step 310: Based on the position filter coefficients of the previous exposure time and the spatial position of the hand key points in the reference coordinate system at the current exposure time, the filtered position of the hand key points at the current exposure time is determined by using a position filter.

[0074] In some optional embodiments of this disclosure, the position filter coefficients at the previous exposure time are position filter coefficients obtained by performing position filtering at the previous exposure time. For example, the position filter coefficients at the previous exposure time can be expressed as... t represents the current exposure time, t-1 represents the previous exposure time, and x represents the position. The previous exposure time is the exposure time of the hand image in the frame adjacent to the current exposure time.

[0075] In some optional embodiments of this disclosure, the filtered position of the hand key points at the current exposure time. It can be represented as follows:

[0076]

[0077] in, f represents the spatial position of the hand keypoint in the reference coordinate system at the current exposure moment. x (,) represents a position filter.

[0078] Step 320: Based on the filtered position of the hand key point at the current exposure time, the velocity filter coefficient at the previous exposure time, and the filtered position of the hand key point at the previous exposure time, the velocity filter is used to determine the filtered velocity of the hand key point at the current exposure time.

[0079] In some optional embodiments of this disclosure, the velocity filter coefficients at the previous exposure time are the velocity filter coefficients obtained when velocity filtering is performed at the previous exposure time. For example, the velocity filter coefficients at the previous exposure time can be expressed as... v represents speed.

[0080] In some optional embodiments of this disclosure, the filtered velocity of the hand key points at the current exposure time It can be represented as follows:

[0081]

[0082] in, This indicates the filtered position of key hand points at the previous exposure time. This indicates the filtered position of the key points of the hand at the current exposure time, f. v (,) represents a velocity filter.

[0083] Step 330: Based on the filtered velocity of the hand key points at the current exposure time, the acceleration filter coefficients at the previous exposure time, and the filtered velocity of the hand key points at the previous exposure time, the filtered acceleration of the hand key points at the current exposure time is determined using an acceleration filter.

[0084] In some optional embodiments of this disclosure, the acceleration filter coefficients at the previous exposure time are the acceleration filter coefficients obtained when performing acceleration filtering at the previous exposure time. For example, the acceleration filter coefficients at the previous exposure time can be expressed as... 'a' represents acceleration.

[0085] In some optional embodiments of this disclosure, the filtered acceleration of key hand points at the current exposure time. It can be represented as follows:

[0086]

[0087] in, This indicates the filtered velocity of key hand points at the previous exposure time. f represents the filtered velocity of the key hand points at the current exposure moment. a (,) represents an acceleration filter.

[0088] In some optional embodiments of this disclosure, the position filter, velocity filter, and acceleration filter may be a 1 Euro filter, a Kalman filter, or other feasible filters as needed, without any specific limitation.

[0089] In one optional example, a one-euro filter is used as a position filter.

[0090]

[0091] Among them, the position filter coefficients in the one-euro filter It can be called an adaptive smoothing factor. The adaptive smoothing factor can be determined by the influencing parameters (including the velocity coefficient, minimum cutoff frequency, and constant cutoff frequency). The determination method of each influencing parameter can be found in the relevant technology. The specific principle will not be elaborated here.

[0092] This embodiment determines the filtered state of the key hand points at the current exposure time by using a position filter, a velocity filter, and an acceleration filter. This allows for a more accurate determination of the key hand points at the current exposure time, which in turn ensures the accuracy of the target position of the key hand points at the time of the target display after compensation when used for time delay compensation.

[0093] In some optional embodiments of this disclosure, step 310, which uses a position filter to determine the filtered position of the hand key point at the current exposure time based on the position filter coefficients of the previous exposure time and the spatial position of the hand key point in the reference coordinate system at the current exposure time, includes: using a position filter to determine the filtered position and position filter coefficients of the hand key point at the current exposure time based on the position filter coefficients of the previous exposure time and the spatial position of the hand key point in the reference coordinate system at the current exposure time.

[0094] In some optional embodiments of this disclosure, the position filter coefficients also need to be continuously updated with position filtering, which can be expressed as follows:

[0095]

[0096] in, This represents the position filter coefficient updated at the current exposure time; other symbols are described above.

[0097] In some optional embodiments of this disclosure, step 320, which involves determining the filtered velocity of the hand keypoint at the current exposure time based on the filtered position of the hand keypoint at the current exposure time, the velocity filter coefficient at the previous exposure time, and the filtered position of the hand keypoint at the previous exposure time, using a velocity filter, includes: determining the filtered velocity and velocity filter coefficient of the hand keypoint at the current exposure time based on the filtered position of the hand keypoint at the current exposure time, the velocity filter coefficient at the previous exposure time, and the filtered position of the hand keypoint at the previous exposure time.

[0098] In some optional embodiments of this disclosure, similar to the position filter coefficients, the velocity filter coefficients also need to be updated as the velocity is filtered, as can be expressed as follows:

[0099]

[0100] in, This represents the updated velocity filter coefficients at the current exposure time; other symbols are described above.

[0101] In some optional embodiments of this disclosure, step 330, which involves determining the filtered acceleration of the hand keypoint at the current exposure time based on the filtered velocity of the hand keypoint at the current exposure time, the acceleration filter coefficient at the previous exposure time, and the filtered velocity of the hand keypoint at the previous exposure time, using an acceleration filter, includes: determining the filtered acceleration and acceleration filter coefficient of the hand keypoint at the current exposure time based on the filtered velocity of the hand keypoint at the current exposure time, the acceleration filter coefficient at the previous exposure time, and the filtered velocity of the hand keypoint at the previous exposure time.

[0102] In some optional embodiments of this disclosure, similar to the position filter coefficients, the acceleration filter coefficients also need to be updated as acceleration is filtered, as can be represented as follows:

[0103]

[0104] in, This represents the acceleration filter coefficient updated at the current exposure time; other symbols are described above.

[0105] This embodiment improves the accuracy of the filtered state by updating the position filter coefficients, velocity filter coefficients, and acceleration filter coefficients in real time during filtering.

[0106] Figure 5 This is a flowchart illustrating step 20 provided in an exemplary embodiment of this disclosure.

[0107] In some optional embodiments of this disclosure, step 20, which determines the spatial position of each key point of the hand in the reference coordinate system at the current exposure time based on the obtained hand image containing the hand at the current exposure time and the pose of the terminal device in the reference coordinate system, includes:

[0108] Step 210: Use the hand key point extraction model to process the hand image to obtain the camera coordinate system position of the hand key points at the current exposure time.

[0109] In some optional embodiments of this disclosure, the camera coordinate system position includes at least one of a two-dimensional position and a three-dimensional position.

[0110] In some optional embodiments of this disclosure, the hand keypoint extraction model can be any implementable model, and there is no specific limitation. The hand keypoint extraction model needs to be pre-trained. For example, the hand keypoint extraction model can be obtained by training a hand keypoint extraction network based on annotated images including hand keypoints as training images.

[0111] In some optional embodiments of this disclosure, the camera coordinate system position can be directly obtained by the hand key point extraction model, that is, the output result of the hand key point extraction model is directly the camera coordinate system position of the hand key points.

[0112] In some optional embodiments of this disclosure, the hand keypoint extraction model can also extract the image coordinate system position, and then obtain the camera coordinate system position by transforming the image coordinate system position to the camera coordinate system. The specific transformation can be set according to actual needs. The transformation from the image coordinate system to the camera coordinate system can be implemented based on the camera's intrinsic parameters.

[0113] Step 220: Based on the camera coordinate system position of the hand key points at the current exposure time and the attitude of the terminal device in the reference coordinate system, determine the spatial position of the hand key points in the reference coordinate system at the current exposure time.

[0114] In some optional embodiments of this disclosure, the attitude of the terminal device in the reference coordinate system can be used to determine the transformation relationship from the terminal device coordinate system to the reference coordinate system, which facilitates determining the spatial position of the hand key points in the reference coordinate system at the current exposure time. There is also a certain transformation relationship between the camera coordinate system and the terminal device coordinate system. This transformation relationship can be obtained through pre-calibration and can be used to transform the position of the camera coordinate system to the terminal device coordinate system, and then from the terminal device coordinate system to the reference coordinate system.

[0115] This embodiment determines the spatial position of key hand points in the reference coordinate system at the current exposure time for subsequent filtering. This reduces the instability caused by device shake, improves the stability of the filtering state, and further enhances the accuracy of the compensation results.

[0116] In some optional embodiments of this disclosure, the hand image including the hand comprises a left-eye hand image and a right-eye hand image. The left-eye hand image and the right-eye hand image have parallax.

[0117] Step 210 uses the hand keypoint extraction model to process the hand image and obtain the camera coordinate system position of the hand keypoints at the current exposure time, including:

[0118] Step 2110: Use a two-dimensional key point detection model to process the left eye hand image to obtain the left eye two-dimensional position of the key points of the hand at the current exposure time.

[0119] In some optional embodiments of this disclosure, the two-dimensional keypoint detection model can employ any implementable two-dimensional hand keypoint detection network. The detection network can be trained using labeled images containing two-dimensional location labels including hand keypoints (e.g., 21 hand keypoints) as training images to obtain the two-dimensional keypoint detection model.

[0120] Step 2120: Use a two-dimensional key point detection model to process the right eye hand image to obtain the right eye two-dimensional position of the hand key points at the current exposure time.

[0121] Step 2130: Use at least one of the left-eye 2D position and the right-eye 2D position of the hand key point to restore the depth information and obtain the 3D position of the hand key point in the camera coordinate system. This position is used as the camera coordinate system position of the hand key point at the current exposure time. The camera coordinate system includes at least one of the left-eye camera coordinate system and the right-eye camera coordinate system.

[0122] In some optional embodiments of this disclosure, depth information can be reconstructed using the left or right 2D position of the hand keypoint to obtain the 3D position of the hand keypoint in the camera coordinate system, which serves as the camera coordinate system position of the hand keypoint at the current exposure time. For example, a dimensionality-enhancing network model can be used for depth information reconstruction. The dimensionality-enhancing network model can be trained together with a 2D keypoint detection model. By processing the left or right 2D position using the dimensionality-enhancing network model, the 3D position of the hand keypoint in the camera coordinate system can be obtained.

[0123] In some optional embodiments of this disclosure, depth information can also be restored by combining the left-eye and right-eye 2D positions of the hand keypoints to obtain the 3D position of the hand keypoints in the camera coordinate system. For example, depth information can be restored based on the disparity of the left-eye and right-eye 2D positions, combined with the intrinsic and extrinsic parameters of the left and right cameras. Specifically, the process of depth information restoration based on binocular disparity includes: determining a disparity map using a stereo matching algorithm based on the left-eye and right-eye hand images; and solving for a depth map based on the disparity map to restore the depth information.

[0124] In some optional embodiments of this disclosure, depth information can be reconstructed using the left-eye 2D position of the hand keypoints to obtain the left-eye 3D position, and the right-eye 2D position of the hand keypoints can be reconstructed to obtain the right-eye 3D position. Then, based on the left-eye 3D position, left-eye camera parameters, right-eye 3D position, and right-eye camera parameters, the 3D position of the hand keypoints in the camera coordinate system is determined. Specifically, the left-eye 3D position can be transformed to the right-eye camera coordinate system, or the right-eye 3D position can be transformed to the left-eye camera coordinate system, achieving a unification of the two coordinate systems and obtaining the left-eye and right-eye 3D positions in the same camera coordinate system. Then, by setting confidence weights for the left and right eyes respectively, the left-eye and right-eye 3D positions in the same camera coordinate system are weighted to obtain the corrected 3D position of the hand keypoints. By weighting the left and right eyes, effective correction of the 3D position can be achieved, further improving the accuracy of the 3D position.

[0125] This embodiment can reconstruct the 3D position of the hand key points in the camera coordinate system by using at least one of the left-eye and right-eye 2D positions of the hand key points. Therefore, as long as at least one of the left and right eyes detects the hand image, the 3D position of the hand key points can be obtained, improving versatility.

[0126] In some optional embodiments of this disclosure, step 220, which determines the spatial position of the hand keypoint in the reference coordinate system at the current exposure time based on the camera coordinate system position of the hand keypoint and the attitude of the terminal device in the reference coordinate system, includes:

[0127] Step 2210: Based on the data collected by the inertial measurement unit configured on the terminal device, determine the attitude of the terminal device in the reference coordinate system at the current exposure time.

[0128] In some optional embodiments of this disclosure, the data collected by the Inertial Measurement Unit (IMU) may include angular velocity and acceleration. The attitude of the terminal device in the reference coordinate system at the current exposure moment can be determined by summing the change in angle of the terminal device from the previous exposure moment to the current exposure moment with the attitude of the terminal device in the reference coordinate system at the previous exposure moment. The change in angle can be obtained by integrating the angular velocity collected by the IMU from the previous exposure moment to the current exposure moment; the specific principle will not be elaborated further.

[0129] Step 2220: Based on the attitude of the terminal device in the reference coordinate system at the current exposure time, determine the first transformation matrix from the terminal device coordinate system to the reference coordinate system.

[0130] In some optional embodiments of this disclosure, the attitude of the terminal device in the reference coordinate system at the current exposure time may include yaw, pitch, and roll. Since the relative position of the hand keypoints to the terminal device can also be used to determine the target position of the hand keypoints at the target display time, the first transformation matrix can be determined based on the attitude of the terminal device in the reference coordinate system at the current exposure time.

[0131] In some optional embodiments of this disclosure, the position of the terminal device in the reference coordinate system can also be determined based on the data collected by the IMU. Based on the attitude and position of the terminal device in the reference coordinate system at the current exposure time, a first transformation matrix from the terminal device coordinate system to the reference coordinate system can be determined. The specific transformation matrix can be set according to actual needs.

[0132] In some optional embodiments of this disclosure, the terminal device is a head-mounted display device, in which case the coordinate system of the terminal device is a head coordinate system. The head coordinate system can be, for example, with the center of the user's head (or the center of the headband of the head-mounted display device's glasses) as the origin, the direction of the user's gaze straight ahead as the Z-axis, the direction from the left eye to the right eye as the X-axis, and the direction from the center of the head to the neck as the Y-axis. After the user wears the head-mounted display device, the head coordinate system can be considered rigidly connected to the camera coordinate system, and the transformation from the camera coordinate system to the head coordinate system can be completed through translation.

[0133] Step 2230: Based on the first transformation matrix and the second transformation matrix from the camera coordinate system to the terminal device coordinate system, the camera coordinate system position of the hand key point at the current exposure time is transformed to the reference coordinate system to obtain the spatial position of the hand key point in the reference coordinate system at the current exposure time.

[0134] In some optional embodiments of this disclosure, the first transformation matrix may represent The second transformation matrix can be expressed as: The spatial position of the key points of the hand in the reference coordinate system at the current exposure time. It can be represented as follows:

[0135]

[0136] in, The camera coordinate system position of the key point of the hand at the current exposure time is represented by h, which represents the terminal device coordinate system, and c represents the camera coordinate system.

[0137] This embodiment realizes the transformation from the camera coordinate system position to the reference coordinate system through the first transformation matrix and the second transformation matrix. When performing smoothing filtering in the subsequent process, it can cover more instabilities and further improve the accuracy of the filtering results.

[0138] In some optional embodiments of this disclosure, the method further includes:

[0139] Step 610: Obtain the left and right eye images captured by the camera at the current exposure time.

[0140] In some optional embodiments of this disclosure, the camera is a binocular camera, and the left and right eye images have parallax.

[0141] Step 620: Correct the left eye image based on the left eye camera parameters to obtain the corrected left eye image.

[0142] In some optional embodiments of this disclosure, the left-eye camera parameters may include intrinsic and extrinsic parameters. Correction of the left-eye image based on the left-eye camera parameters mainly corrects certain pixels in the left-eye image, such as distortion in the four corner regions of the left-eye image; the specific principles are not elaborated further.

[0143] Step 630: Correct the right eye image based on the right eye camera parameters to obtain the corrected right eye image.

[0144] The method for correcting the right eye image is similar to that for the left eye, and will not be repeated here.

[0145] Step 640: Use the hand recognition model to perform hand recognition on the corrected images of the left and right eyes respectively.

[0146] In some optional embodiments of this disclosure, the hand recognition model can be any implementable model, which can be set according to actual needs. For example, the hand recognition model can be a deep learning-based object detection model.

[0147] Step 650: If a hand is identified from at least one of the left-eye corrected image and the right-eye corrected image, determine that the left-eye corrected image and the right-eye corrected image are the left-eye hand image and the right-eye hand image, respectively.

[0148] In some optional embodiments of this disclosure, if a hand is identified from the corrected image of the left eye, the corrected image of the left eye is determined to be the left eye hand image; if a hand is identified from the corrected image of the right eye, the corrected image of the right eye is determined to be the right eye hand image. Alternatively, if a hand is identified from one of the corrected images of the two eyes, the corrected images of both eyes can be used as the left eye hand image and the right eye hand image, respectively.

[0149] In some optional embodiments of this disclosure, when a hand is identified from at least one of the left-eye corrected image and the right-eye corrected image, the region containing the hand in the left-eye corrected image can be used as the left-eye hand image, and the region containing the hand in the right-eye corrected image can be used as the right-eye hand image. For example, a hand recognition model can be used to recognize the hand in the left-eye corrected image to obtain a bounding box of the hand region. Based on this bounding box, the image region corresponding to the bounding box can be extracted from the left-eye corrected image as the left-eye hand image, which facilitates the subsequent extraction of hand key points. After extracting the hand key points, the hand key points are restored to the original corrected image to obtain the position of each hand key point in the corresponding camera coordinate system.

[0150] This embodiment identifies the left and right hand images when the hand is recognized, and performs subsequent processing accordingly. This avoids invalid processing of images that do not contain the hand, thus reducing resource waste.

[0151] In some optional embodiments of this disclosure, step 40, which involves delay compensation of the filtered position of the hand keypoint at the current exposure time based on the target display time, the current exposure time, and the filtered state of the hand keypoint at the current exposure time, includes:

[0152] Determine whether the difference between the target display time and the current exposure time is greater than a preset time threshold; if the difference between the target display time and the current exposure time is less than or equal to the preset time threshold, then perform time delay compensation on the filtered position of the key hand points at the current exposure time.

[0153] In some optional embodiments of this disclosure, the preset time threshold can be set according to actual needs. For example, the preset time threshold can be determined based on the time interval between two frames of images. For example, if the time interval between two frames of images is 33ms, the preset time threshold can be set to about 130ms for 4 frames. The specific time threshold is not limited.

[0154] In some optional embodiments of this disclosure, the delay compensation is limited based on a preset time threshold to avoid the compensation duration being too long, which could easily lead to a large error in the target position at the compensated target display time. When the difference between the target display time and the current exposure time is greater than the preset time threshold, it indicates that the compensation duration is too long, and the effect of delay compensation may not be good. In this case, the position at the current exposure time is not used for delay compensation. If the difference between the target display time and the current exposure time is less than the preset time threshold, it indicates that delay compensation can be performed. In this case, the filtered position of the hand key points at the current exposure time is used for delay compensation.

[0155] In one optional example, Figure 6This is a schematic diagram of the time points for delay compensation provided in an exemplary embodiment of this disclosure. The execution subject is the entity that executes the method of this disclosure, and the frequency at which the execution subject receives images is limited by the sampling frequency of the camera. For example, if the sampling frequency of the camera is 30Hz, then the frequency at which the execution subject receives images is also 30Hz. The execution subject will consume some time to process the image, for example, this time will not exceed 33ms. After the execution subject calculates the delay compensation, it transmits it to the display system for display. From the moment the execution subject receives the image a1 to the completion of display c1, it usually takes about 100ms. For time c2, if no hand is detected at time b2, c2 can use the filtered position of the hand image corresponding to b1 for delay compensation. If no hand is detected at time c3, the filtered position of the hand image of b1 can still be used for delay compensation. However, the time interval between the display time (target display time) and the current exposure time cannot exceed a preset time threshold. If the preset time threshold is exceeded, a tracking failure prompt message can be output. If the subject cannot detect a hand while processing the image, and the gesture output is interrupted for more than a certain time (e.g., 30ms), the subject will stop processing and resume image processing only after receiving the next frame. The preset time threshold is set based on various time-consuming factors, such as the subject processing the image and the maximum drawing limit of the display system, which may result in the time elapsed from when the subject receives the image to when the image is displayed on the screen.

[0156] This embodiment determines whether the difference between the target display time and the current exposure time is greater than a preset time threshold. Delay compensation is only performed when the difference between the target display time and the current exposure time is less than or equal to the preset time threshold, so as to ensure the accuracy and effectiveness of the delay compensation result.

[0157] In some alternative embodiments of this disclosure, the terminal device may be a head-mounted display device.

[0158] In some optional embodiments of this disclosure, the execution entity can be a mobile phone, handheld device, adapter, computer, tablet, vehicle-mounted system, etc. The head-mounted display device can be equipped with a binocular camera and a display system. The binocular camera acquires left and right eye images and sends them to the execution entity. The execution entity processes the images, performs delay compensation, obtains the target position of the hand key points in the reference coordinate system at the target display moment, and transmits the target position to the display system of the head-mounted display device. The display system renders and displays the image on the screen.

[0159] The embodiments or optional examples disclosed herein can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit it.

[0160] Any of the delay compensation methods for hand key point positions provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the delay compensation methods for hand key point positions provided in this disclosure can be executed by a processor, such as by a processor executing any of the delay compensation methods for hand key point positions mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated upon below.

[0161] Exemplary device

[0162] Figure 7 This is a schematic diagram of a delay compensation device for key hand position data provided in an exemplary embodiment of this disclosure. The device in this embodiment can be used to implement corresponding method embodiments of this disclosure, such as... Figure 7 The device shown includes: a first processing module 710, a second processing module 720, a third processing module 730, and an output module 740.

[0163] The first processing module 710 is used to determine the spatial position of each key point of the hand in the reference coordinate system at the current exposure time, based on the hand image containing the hand at the current exposure time and the pose of the terminal device in the reference coordinate system. The hand image is acquired by a camera configured on the terminal device.

[0164] The second processing module 720 is used to determine the filtered state of the hand key points at the current exposure time based on the spatial position of the hand key points in the reference coordinate system using a preset filtering algorithm. The filtered state includes the filtered position, filtered velocity, and filtered acceleration.

[0165] The third processing module 730 is used to perform delay compensation on the filtered position of the hand keypoint at the current exposure time based on the target display time, the current exposure time, and the filtered state of the hand keypoint at the current exposure time, to obtain the target position of the hand keypoint in the reference coordinate system at the target display time. The target display time is later than the current exposure time. The target display time represents the planned screen refresh time of the display system. The display system is configured on the terminal device.

[0166] Output module 740 is used to output the target position to the display system.

[0167] In some optional embodiments of this disclosure, the third processing module 730 is specifically used for:

[0168] Based on the filtered acceleration and velocity at the current exposure time, determine the displacement change of the hand key points from the current exposure time to the target display time. Based on the filtered position and displacement change at the current exposure time, determine the target position of the hand key points in the reference coordinate system at the target display time.

[0169] In some optional embodiments of this disclosure, the second processing module 720 is specifically used for:

[0170] Based on the position filter coefficients of the previous exposure and the spatial position of the hand keypoint in the reference coordinate system at the current exposure, a position filter is used to determine the filtered position of the hand keypoint at the current exposure. Based on the filtered position of the hand keypoint at the current exposure, the velocity filter coefficients of the previous exposure, and the filtered position of the hand keypoint at the previous exposure, a velocity filter is used to determine the filtered velocity of the hand keypoint at the current exposure. Based on the filtered velocity of the hand keypoint at the current exposure, the acceleration filter coefficients of the previous exposure, and the filtered velocity of the hand keypoint at the previous exposure, an acceleration filter is used to determine the filtered acceleration of the hand keypoint at the current exposure.

[0171] In some optional embodiments of this disclosure, the second processing module 720 is specifically used for:

[0172] Based on the position filter coefficients of the previous exposure and the spatial position of the hand keypoint in the reference coordinate system at the current exposure, a position filter is used to determine the filtered position and position filter coefficients of the hand keypoint at the current exposure. Based on the filtered position of the hand keypoint at the current exposure, the velocity filter coefficients of the previous exposure, and the filtered position of the hand keypoint at the previous exposure, a velocity filter is used to determine the filtered velocity and velocity filter coefficients of the hand keypoint at the current exposure. Based on the filtered velocity of the hand keypoint at the current exposure, the acceleration filter coefficients of the previous exposure, and the filtered velocity of the hand keypoint at the previous exposure, an acceleration filter is used to determine the filtered acceleration and acceleration filter coefficients of the hand keypoint at the current exposure.

[0173] In some optional embodiments of this disclosure, the first processing module 710 is specifically used for:

[0174] The hand keypoint extraction model is used to process the hand image to obtain the camera coordinate system position of the hand keypoints at the current exposure time. The camera coordinate system position includes at least one of two-dimensional and three-dimensional positions. Based on the camera coordinate system position of the hand keypoints at the current exposure time and the pose of the terminal device in the reference coordinate system, the spatial position of the hand keypoints in the reference coordinate system at the current exposure time is determined.

[0175] In some optional embodiments of this disclosure, the hand image including the hand comprises a left-eye hand image and a right-eye hand image. The left-eye hand image and the right-eye hand image have parallax. The first processing module 710 is specifically used for:

[0176] The left-eye hand image is processed using a 2D keypoint detection model to obtain the left-eye 2D position of the hand keypoints at the current exposure time. The right-eye hand image is also processed using the same model to obtain the right-eye 2D position of the hand keypoints at the current exposure time. Depth information is then reconstructed using at least one of the left-eye and right-eye 2D positions of the hand keypoints to obtain the 3D position of the hand keypoints in the camera coordinate system. This 3D position is used as the camera coordinate system position of the hand keypoints at the current exposure time, and the camera coordinate system includes at least one of the left-eye and right-eye camera coordinate systems.

[0177] In some optional embodiments of this disclosure, the first processing module 710 is specifically used for:

[0178] Based on data collected by the inertial measurement unit configured on the terminal device, the attitude of the terminal device in the reference coordinate system at the current exposure time is determined. Based on the attitude of the terminal device in the reference coordinate system at the current exposure time, a first transformation matrix from the terminal device coordinate system to the reference coordinate system is determined. Based on the first transformation matrix and a second transformation matrix from the camera coordinate system to the terminal device coordinate system, the camera coordinate system position of the hand keypoint at the current exposure time is transformed to the reference coordinate system, obtaining the spatial position of the hand keypoint in the reference coordinate system at the current exposure time.

[0179] In some optional embodiments of this disclosure, the first processing module 710 is further configured to:

[0180] Acquire the left and right eye images captured by the camera at the current exposure time. The camera is a binocular camera, and there is parallax between the left and right eye images. Correct the left eye image based on the left eye camera parameters to obtain the corrected left eye image. Correct the right eye image based on the right eye camera parameters to obtain the corrected right eye image. Use a hand recognition model to perform hand recognition on both the corrected left and right eye images. If a hand is recognized from at least one of the corrected left and right eye images, determine the corrected left and right eye images as the left and right eye hand images, respectively.

[0181] In some optional embodiments of this disclosure, the third processing module 730 is specifically used for:

[0182] Determine if the difference between the target display time and the current exposure time is greater than a preset time threshold. If the difference is less than or equal to the preset time threshold, perform time-delay compensation on the filtered position of the hand key points at the current exposure time.

[0183] In some optional embodiments of this disclosure, the terminal device is a head-mounted display device.

[0184] It should be noted that the specific implementation of the delay compensation device for hand key point position in this disclosure is similar to the specific implementation of the delay compensation method for hand key point position in this disclosure. For details, please refer to the delay compensation method for hand key point position section. To reduce redundancy, it will not be described again here.

[0185] Exemplary electronic devices

[0186] This disclosure also provides an electronic device, including a processor and a memory.

[0187] Memory is used to store processor-executable instructions.

[0188] A processor is configured to read executable instructions from memory and execute the instructions to implement the delay compensation method for key hand position as described in any of the above embodiments of the present disclosure.

[0189] Exemplary head-mounted display device

[0190] This disclosure also provides a head-mounted display device, including a processor and a memory.

[0191] Memory is used to store processor-executable instructions.

[0192] A processor is configured to read executable instructions from memory and execute the instructions to implement the delay compensation method for key hand position as described in any of the above embodiments of the present disclosure.

[0193] Figure 8 This is a schematic diagram of the structure of one application embodiment of the electronic device disclosed herein. The structure of a head-mounted display device can be found in the structure of this electronic device. In this embodiment, the electronic device includes one or more processors 100 and a memory 200.

[0194] The processor 100 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0195] The memory 200 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 100 may execute the program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0196] In one example, the electronic device may also include an input device 300 and an output device 400, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0197] For example, the input device 300 may be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0198] In addition, the input device 300 may also include, for example, a keyboard, a mouse, etc.

[0199] The output device 400 can output various information to the outside, including determined distance information, direction information, etc. The output device 400 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0200] Of course, for the sake of simplicity, Figure 8 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0201] Exemplary computer program products and computer-readable storage media

[0202] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0203] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0204] Furthermore, embodiments of this disclosure may also be computer-readable storage media having a computer program stored thereon for performing steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0205] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0206] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0207] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0208] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0209] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0210] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0211] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0212] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A delay compensation method for key hand position data, comprising: Based on the hand image containing the hand at the current exposure time and the posture of the terminal device in the reference coordinate system, the spatial position of each key point of the hand in the reference coordinate system at the current exposure time is determined; wherein, the hand image is acquired by a camera configured on the terminal device; Based on the spatial position of the key hand points in the reference coordinate system at the current exposure time, a preset filtering algorithm is used to determine the filtered state of the key hand points at the current exposure time; wherein, the filtered state includes the filtered position, the filtered velocity, and the filtered acceleration; Based on the target display time, the current exposure time, and the filtered state of the hand key points at the current exposure time, delay compensation is applied to the filtered position of the hand key points at the current exposure time to obtain the target position of the hand key points at the target display time in the reference coordinate system; wherein, the target display time is later than the current exposure time; the target display time represents the planned screen refresh time of the display system; the display system is configured on the terminal device; The target position is output to the display system; The step of performing delay compensation on the filtered position of the hand key points at the current exposure time based on the target display time, the current exposure time, and the filtered state of the hand key points at the current exposure time to obtain the target position of the hand key points in the reference coordinate system at the target display time includes: Based on the filtered acceleration and filtered velocity at the current exposure time, determine the displacement change of the key hand points from the current exposure time to the target display time; Based on the filtered position and displacement change at the current exposure time, the target position of the hand key point at the target display time is determined in the reference coordinate system.

2. The method according to claim 1, wherein, The process of determining the filtered state of the hand key points at the current exposure time using a preset filtering algorithm, based on the spatial position of the hand key points in the reference coordinate system at the current exposure time, includes: Based on the position filter coefficients of the previous exposure time and the spatial position of the key hand points in the reference coordinate system at the current exposure time, the filtered position of the key hand points at the current exposure time is determined by using a position filter. Based on the filtered position of the key hand points at the current exposure time, the velocity filter coefficients at the previous exposure time, and the filtered position of the key hand points at the previous exposure time, the velocity filter is used to determine the filtered velocity of the key hand points at the current exposure time. Based on the filtered velocity of the key hand points at the current exposure time, the acceleration filter coefficients at the previous exposure time, and the filtered velocity of the key hand points at the previous exposure time, an acceleration filter is used to determine the filtered acceleration of the key hand points at the current exposure time.

3. The method according to claim 2, wherein, The process of determining the filtered position of the hand key points at the current exposure time using a position filter, based on the position filter coefficients of the previous exposure time and the spatial position of the hand key points in the reference coordinate system at the current exposure time, includes: Based on the position filter coefficients of the previous exposure time and the spatial position of the hand key points in the reference coordinate system at the current exposure time, the filtered position and position filter coefficients of the hand key points at the current exposure time are determined by using a position filter. The step of determining the filtered velocity of the hand keypoint at the current exposure time using a velocity filter, based on the filtered position of the hand keypoint at the current exposure time, the velocity filter coefficient at the previous exposure time, and the filtered position of the hand keypoint at the previous exposure time, includes: Based on the filtered position of the key hand points at the current exposure time, the velocity filter coefficients at the previous exposure time, and the filtered position of the key hand points at the previous exposure time, a velocity filter is used to determine the filtered velocity and velocity filter coefficients of the key hand points at the current exposure time. The process of determining the filtered acceleration of the hand keypoint at the current exposure time using an acceleration filter, based on the filtered velocity of the hand keypoint at the current exposure time, the acceleration filter coefficients at the previous exposure time, and the filtered velocity of the hand keypoint at the previous exposure time, includes: Based on the filtered velocity of the key hand points at the current exposure time, the acceleration filter coefficients at the previous exposure time, and the filtered velocity of the key hand points at the previous exposure time, an acceleration filter is used to determine the filtered acceleration and acceleration filter coefficients of the key hand points at the current exposure time.

4. The method according to claim 1, wherein, The step of determining the spatial position of each key hand point in the reference coordinate system at the current exposure time, based on the obtained hand image including the hand and the pose of the terminal device in the reference coordinate system, includes: The hand image is processed using a hand key point extraction model to obtain the camera coordinate system position of the hand key points at the current exposure time. The camera coordinate system position includes at least one of two-dimensional position and three-dimensional position. Based on the camera coordinate system position of the key hand points at the current exposure time and the attitude of the terminal device in the reference coordinate system, the spatial position of the key hand points in the reference coordinate system at the current exposure time is determined.

5. The method according to claim 4, wherein, The hand image containing the hand includes a left-eye hand image and a right-eye hand image; wherein the left-eye hand image and the right-eye hand image have parallax. The step of processing the hand image using a hand keypoint extraction model to obtain the camera coordinate system position of the hand keypoints at the current exposure time includes: The left-eye hand image is processed using a two-dimensional key point detection model to obtain the left-eye two-dimensional position of the key points of the hand at the current exposure time; The two-dimensional key point detection model is used to process the right eye hand image to obtain the right eye two-dimensional position of the key points of the hand at the current exposure time; Depth information is restored using at least one of the left-eye 2D position and the right-eye 2D position of the hand key point to obtain the 3D position of the hand key point in the camera coordinate system, which is used as the camera coordinate system position of the hand key point at the current exposure time. The camera coordinate system includes at least one of the left-eye camera coordinate system and the right-eye camera coordinate system.

6. The method according to claim 4, wherein, Determining the spatial position of the hand key points in the reference coordinate system at the current exposure time, based on the camera coordinate system position of the hand key points and the posture of the terminal device in the reference coordinate system, includes: Based on the data collected by the inertial measurement unit configured on the terminal device, the attitude of the terminal device in the reference coordinate system at the current exposure time is determined; Based on the posture, determine the first transformation matrix from the terminal device coordinate system to the reference coordinate system; Based on the first transformation matrix and the second transformation matrix from the camera coordinate system to the terminal device coordinate system, the camera coordinate system position of the key hand point at the current exposure time is transformed to the reference coordinate system, thereby obtaining the spatial position of the key hand point in the reference coordinate system at the current exposure time.

7. The method according to claim 5, wherein, Also includes: The left and right eye images captured by the camera at the current exposure time are obtained; wherein the camera is a binocular camera, and the left and right eye images have parallax. The left-eye image is corrected based on the parameters of the left-eye camera to obtain the corrected left-eye image; The right eye image is corrected based on the parameters of the right eye camera to obtain the corrected right eye image; Hand recognition models were used to perform hand recognition on the corrected left eye image and the corrected right eye image, respectively. If a hand is identified from at least one of the corrected left-eye image and the corrected right-eye image, the corrected left-eye image and the corrected right-eye image are determined to be the left-eye hand image and the right-eye hand image, respectively.

8. The method according to any one of claims 1-7, wherein, The method of performing delay compensation on the filtered position of the hand key points at the current exposure time based on the target display time, the current exposure time, and the filtered state of the hand key points at the current exposure time includes: Determine whether the difference between the target display time and the current exposure time is greater than a preset time threshold; If the difference between the target display time and the current exposure time is less than or equal to the preset time threshold, then delay compensation is performed on the filtered position of the key hand point at the current exposure time.

9. The method according to any one of claims 1-7, wherein, The terminal device is a head-mounted display device.

10. A delay compensation device for key hand position measurement, comprising: The first processing module is used to determine the spatial position of each key point of the hand in the reference coordinate system at the current exposure time based on the hand image containing the hand at the current exposure time and the posture of the terminal device in the reference coordinate system; wherein, the hand image is acquired by a camera configured on the terminal device; The second processing module is used to determine the filtered state of the hand key points at the current exposure time based on the spatial position of the hand key points in the reference coordinate system at the current exposure time using a preset filtering algorithm; wherein, the filtered state includes the filtered position, the filtered velocity, and the filtered acceleration; The third processing module is used to perform delay compensation on the filtered position of the hand key point at the current exposure time based on the target display time, the current exposure time, and the filtered state of the hand key point at the current exposure time, to obtain the target position of the hand key point at the target display time in the reference coordinate system; wherein, the target display time is later than the current exposure time; the target display time represents the planned screen refresh time of the display system; the display system is configured on the terminal device; The output module is used to output the target position to the display system; The third processing module is specifically used for: Based on the filtered acceleration and filtered velocity at the current exposure time, determine the displacement change of the key hand points from the current exposure time to the target display time; Based on the filtered position and displacement change at the current exposure time, the target position of the hand key point at the target display time is determined in the reference coordinate system.

11. A computer-readable storage medium storing a computer program for performing the delay compensation method for key hand position as described in any one of claims 1-9.

12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the delay compensation method for key hand position as described in any one of claims 1-9.

13. A head-mounted display device, the head-mounted display device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the delay compensation method for key hand position as described in any one of claims 1-9.

14. A computer program product comprising computer program instructions, which, when executed by a processor, cause the processor to perform a delay compensation method for key hand position according to any one of claims 1-9.

Images