Gesture recognition method, system and device based on VR cloud rendering and medium
By dynamically adjusting the difference data transmission interval and compressing byte length, the problem of low efficiency and accuracy of gesture recognition data transmission in cloud rendering environment of VR devices is solved, and efficient and real-time gesture data transmission is achieved, which improves VR The interactive experience of the application and the system scalability.
Patent Information
- Application Number
- CN202411176889.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing VR devices face poor data transmission efficiency and accuracy in gesture recognition, especially in cloud rendering environments, which are difficult to achieve efficient and real-time gesture data transmission.
Receive gesture data through the local side, dynamically adjust the transmission interval of the difference data and compressed byte length according to network congestion, and perform corresponding decoding and restoration on the cloud. Dynamic scaling factors and compression algorithms are used to process gesture data to improve transmission Efficiency and accuracy.
Significantly improve gesture recognition accuracy and user interaction fluency in VR applications, enhance immersion, reduce development and deployment costs, improve system maintainability and scalability, and ensure optimal maintenance in different network environments. Data transmission efficiency and user experience.
Smart Images

Figure CN119068547A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of data recognition technology, and in particular to a gesture recognition method, system, device and medium based on VR cloud rendering. Background Art
[0002] Currently, virtual reality (VR) technology provides users with an immersive experience by creating a computer-generated simulated environment. However, current wearable VR devices are limited by hardware performance and often face the bottleneck of insufficient computing power. Cloud rendering technology uses the powerful server resources in the cloud to perform rendering tasks, effectively alleviating the pressure on local hardware. However, in terms of gesture recognition, how to efficiently and in real time transmit local gesture data to the cloud and ensure that the data is seamlessly integrated into the cloud scene logic has become a technical challenge that needs to be solved. Existing cloud rendering solutions such as NVIDIA's CloudXR focus on cloud processing of rendering tasks, but do not directly cover a complete solution for remote gesture recognition.
[0003] It can be seen that there is an urgent need for a gesture recognition method based on VR cloud rendering that can improve data transmission efficiency and accuracy. Summary of the invention
[0004] In view of this, the embodiments of the present disclosure provide a gesture recognition method, system, device and medium based on VR cloud rendering, which at least partially solve the problems of poor data transmission efficiency and accuracy in the prior art.
[0005] In a first aspect, an embodiment of the present disclosure provides a gesture recognition method based on VR cloud rendering, including:
[0006] Step 1: The local end receives real-time gesture data from a gesture capture device, wherein the real-time gesture data includes the position and rotation angle of the root node of the hand;
[0007] Step 2: dynamically adjust the difference data sending interval and compressed byte length according to network congestion, and send it to the cloud;
[0008] Step 3, obtaining real-time gesture data within a preset period and calculating a dynamic scaling factor;
[0009] Step 4, compressing the real-time gesture data according to the dynamic scaling factor and the compression byte length;
[0010] Step 5: Send the compressed real-time gesture data to the cloud according to the preset transmission architecture;
[0011] Step 6: The cloud receives the compressed real-time gesture data, restores the gesture data according to the dynamic scaling factor and the compressed byte length, and broadcasts it to the target module.
[0012] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:
[0013] Get the current network congestion status and compare it with the preset threshold;
[0014] If the current network congestion state is greater than the preset threshold, it is determined to be a high congestion state, and the difference data sending interval and the compressed byte length are adjusted to the preset minimum value;
[0015] If the current network congestion state is less than or equal to the preset threshold, it is determined to be a low congestion state, and the difference data sending interval and the compressed byte length are adjusted to the preset maximum values;
[0016] The adjusted difference data sending interval and compressed byte length are sent to the cloud.
[0017] According to a specific implementation of the embodiment of the present disclosure, step 3 specifically includes:
[0018] Step 3.1, traverse the gesture data in the target period and calculate the difference between the position and rotation angle in adjacent frames;
[0019] Step 3.2, select the maximum value z among the differences;
[0020] Step 3.3, calculate the integer value range [-a, a] that can be expressed by the current byte length according to the compressed byte length;
[0021] Step 3.4, calculate the dynamic scaling factor, wherein the expression of the dynamic scaling factor is
[0022] DynamicScaleFactor=|a| / (z*T)
[0023] Where T is the redundant constant factor.
[0024] According to a specific implementation of the embodiment of the present disclosure, step 4 specifically includes:
[0025] Step 4.1, convert the root node of the real-time gesture data from the world coordinates to the observation coordinates and reassign it;
[0026] Step 4.2, scaling the coordinate components in the re-assigned real-time gesture data according to the dynamic scaling factor and rounding them to integers;
[0027] Step 4.3, calling a preset encoding function to encode each coordinate component according to the compressed byte length;
[0028] Step 4.4, store the encoded byte array into a new array.
[0029] According to a specific implementation of the embodiment of the present disclosure, step 4.3 specifically includes:
[0030] Step 4.3.1, check whether the byte length of the coordinate component is 2 or 3, if not, throw an exception;
[0031] Step 4.3.2, allocate the byte array bytes according to the byte length;
[0032] Step 4.3.3, for the 2-byte coordinate component, convert the integer to a short integer type, and store the high 8 bits and low 8 bits of the short integer value into bytes[0] and bytes[1] respectively;
[0033] Step 4.3.4, for the 3-byte coordinate components, operate on the lower 24 bits of the integer and store them in bytes[0], bytes[1], and bytes[2].
[0034] According to a specific implementation of the embodiment of the present disclosure, step 6 specifically includes:
[0035] Step 6.1, the cloud calls a preset decoding function to decode each byte array in the compressed real-time gesture data;
[0036] Step 6.2, dividing the decoded integer by the dynamic scaling factor to obtain the coordinate component;
[0037] Step 6.3, create and return new gesture data;
[0038] In step 6.4, the new gesture data is converted from the observation coordinates back to the world coordinates and reassigned, and then broadcast to the XRHandSkeletonDriver that drives the skeletal animation and the XRHandMeshController that drives the mesh rendering, thereby updating the gesture actions in the cloud virtual environment.
[0039] According to a specific implementation of the embodiment of the present disclosure, step 6.1 specifically includes:
[0040] Step 6.1.1, check if the byte length is 2 or 3, if not, throw an exception;
[0041] Step 6.1.2, decode the byte array according to the byte length;
[0042] Step 6.1.3, for a 2-byte byte array, shift bytes[0] left by 8 bits, and then perform a bitwise OR operation with bytes[1] to form a 16-bit integer;
[0043] Step 6.1.4, for the 3-byte byte array, assemble it into a 24-bit value and sign-extend it to 32 bits, shift bytes[0] left by 16 bits, shift bytes[1] left by 8 bits, and then perform a bitwise OR operation with bytes[2] to assemble it into a 24-bit value.
[0044] In a second aspect, the embodiment of the present disclosure provides a gesture recognition system based on VR cloud rendering, including:
[0045] A receiving module, used for receiving real-time gesture data from a gesture capture device at a local end, wherein the real-time gesture data includes the position and rotation angle of the root node of the hand;
[0046] An adjustment module is used to dynamically adjust the difference data sending interval and compressed byte length according to network congestion conditions, and send it to the cloud;
[0047] A calculation module, used to obtain real-time gesture data within a preset period and calculate a dynamic scaling factor;
[0048] A compression module, for compressing real-time gesture data according to a dynamic scaling factor and a compression byte length;
[0049] A transmission module, used to send compressed real-time gesture data to the cloud according to a preset transmission architecture;
[0050] The decompression module is used to receive the compressed real-time gesture data in the cloud, restore the gesture data according to the dynamic scaling factor and the compressed byte length, and broadcast it to the target module.
[0051] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:
[0052] at least one processor; and,
[0053] a memory communicatively connected to the at least one processor; wherein,
[0054] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the gesture recognition method based on VR cloud rendering in the aforementioned first aspect or any implementation of the first aspect.
[0055] In a fourth aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the gesture recognition method based on VR cloud rendering in the aforementioned first aspect or any implementation of the first aspect.
[0056] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the gesture recognition method based on VR cloud rendering in the aforementioned first aspect or any implementation of the first aspect.
[0057] The gesture recognition scheme based on VR cloud rendering in the embodiment of the present disclosure includes: step 1, the local end receives real-time gesture data from the gesture capture device, wherein the real-time gesture data includes the position and rotation angle of the root node of the hand; step 2, dynamically adjusts the difference data sending interval and the compressed byte length according to the network congestion situation, and sends it to the cloud; step 3, obtains the real-time gesture data within a preset period, and calculates the dynamic scaling factor; step 4, compresses the real-time gesture data according to the dynamic scaling factor and the compressed byte length; step 5, sends the compressed real-time gesture data to the cloud according to the preset transmission architecture; step 6, the cloud receives the compressed real-time gesture data, restores the gesture data according to the dynamic scaling factor and the compressed byte length, and broadcasts it to the target module.
[0058] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, the gesture recognition accuracy and user interaction fluency in VR applications are significantly improved through precise coordinate space conversion and efficient data transmission architecture, thereby enhancing the sense of immersion; the development and deployment costs are reduced, and the maintainability and scalability of the system are improved; the data compression algorithm with dynamic factor adjustment can dynamically adjust the compression performance and data accuracy according to network conditions, ensuring the best data transmission efficiency and user experience in different network environments; the upper-level application does not need to care about the specific source and processing process of the gesture data, thereby reducing the difficulty and complexity of development. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0060] Figure 1 A flowchart of a gesture recognition method based on VR cloud rendering provided by an embodiment of the present disclosure;
[0061] Figure 2 A schematic diagram of a transmission process of gesture data between a local terminal and a cloud provided in an embodiment of the present disclosure;
[0062] Figure 3A schematic diagram of a gesture recognition effect in a cloud rendering mode provided by an embodiment of the present disclosure;
[0063] Figure 4 A schematic diagram for comparing the effects of various compression strategies on data provided by an embodiment of the present disclosure;
[0064] Figure 5 A structural diagram of a gesture recognition system based on VR cloud rendering provided in an embodiment of the present disclosure;
[0065] Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0066] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0067] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.
[0068] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0069] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0070] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.
[0071] The embodiment of the present disclosure provides a gesture recognition method based on VR cloud rendering, which can be applied to a gesture recognition process in a virtual reality scene.
[0072] See also Figure 1 , is a flow chart of a gesture recognition method based on VR cloud rendering provided by an embodiment of the present disclosure. Figure 1 As shown, the method mainly comprises the following steps:
[0073] Step 1: The local end receives real-time gesture data from a gesture capture device, wherein the real-time gesture data includes the position and rotation angle of the root node of the hand;
[0074] The local end continuously collects hand posture data and dynamically adjusts the data sending strategy and compression ratio according to the network status.
[0075] Algorithm implementation instructions:
[0076] A. Loop continuously to check whether the current time meets the data or parameter update time interval.
[0077] B. If the parameter update time interval is met, call the updateParameters function to update N and P, calculate the DynamicScaleFactor, and then send the parameter update information.
[0078] C. If the data update time interval is met, select to send the baseline data or difference data according to the comparison result of curN and N, and update the relevant timestamp and data pointer.
[0079] In specific implementation, the present disclosure adopts the OpenXR standard as the basis for acquiring gesture data, and uses the XR Hands plug-in to capture the gesture data of VR devices in real time in the Unity3D development environment. OpenXR provides a cross-platform API to ensure the consistency and accuracy of gesture data, laying a solid foundation for subsequent data processing and transmission.
[0080] In the Unity3D development environment, use the XR Hands plug-in as the basic tool for acquiring gesture data. The XRHands plug-in provides a cross-platform API that supports OpenXR standard VR headsets to access hand tracking data. The specific steps are as follows:
[0081] Import the XR Hands plugin through the Package Manager in Unity3D and make sure it is installed and configured correctly.
[0082] Initialize the XR Hands subsystem (XRHandSubsystem), which is responsible for communicating with the VR hardware to obtain the user's hand position and posture data in real time.
[0083] Design the PlayerHandMgr management class to process and synchronize gesture data between the local end and the cloud. PlayerHandMgr listens to events of the XRHandTrackingEvents class to capture notifications of hand tracking status and data updates. When changes in the user's hand movements are detected, the corresponding processing function is triggered.
[0084] At the same time, a corresponding adaptation layer can also be designed. The cloud management class is responsible for receiving gesture data from the local end, decompressing and converting coordinates, and then submitting the processed data to the adaptation layer. The adaptation layer simulates the XR Hands data processing class to send data messages to the upper-level application for use. The introduction of the adaptation layer allows the upper-level application to process gesture data transmitted from the cloud like local data without having to worry about the specific source and processing details of the data. This greatly increases the ease of use of this solution.
[0085] Step 2: dynamically adjust the difference data sending interval and compressed byte length according to network congestion, and send it to the cloud;
[0086] Based on the above embodiment, step 2 specifically includes:
[0087] Get the current network congestion status and compare it with the preset threshold;
[0088] If the current network congestion state is greater than the preset threshold, it is determined to be a high congestion state, and the difference data sending interval and the compressed byte length are adjusted to the preset minimum value;
[0089] If the current network congestion state is less than or equal to the preset threshold, it is determined to be a low congestion state, and the difference data sending interval and the compressed byte length are adjusted to the preset maximum values;
[0090] The adjusted difference data sending interval and compressed byte length are sent to the cloud.
[0091] In the specific implementation, we consider that when people make gestures, no matter how fast they move, their range is limited. Therefore, the position and angle difference between the gesture node in the current frame and the same node in the previous frame will not be particularly large, which means we do not need to use a full float value to represent this difference.
[0092] After sending the baseline information containing the original data, the difference information is sent continuously (0<=N<=12) N times. After the cloud receives the baseline information, it will use it as a reference, and the subsequent difference information will be used to calculate the cumulative offset based on the baseline information. To further compress the data, we use a byte array of length P (2<=P<=4) to represent these differences, and use a mapping compression algorithm to convert float type differences to byte arrays. In addition, the local end maintains a congestionLevel variable to indicate the current network congestion status. At regular intervals (e.g., 10 seconds), the P and N values are dynamically adjusted through the congestionLevel variable and the recent average jitter range of the gesture data. When the network congestion is low, N=0 is selected, that is, no compression. When the congestion is medium, a medium N and byte compression (P=2 or 3) are selected; when the congestion is high, a larger N and a smaller byte compression (P=2) are selected. After P is determined, the average jitter values of the position and rotation angle of the gesture node during this period are calculated based on the gesture data of the recent period, and then the dynamic scaling factor DynamicScaleFactor is calculated based on these data and P. After the parameters are adjusted, the parameter adjustment information is sent to the cloud to keep the parameters consistent. The local end first maps the compressed data according to P and DynamicScaleFactor, and then further compresses it with ZSDT, and then sends the data to the cloud. After the cloud receives the data, it decompresses it according to the parameters.
[0093] For example, the compression algorithm pseudo code is as follows:
[0094] 1. System parameter definition
[0095] DATA_TIMER_INTERVAL: gesture data sending interval, in seconds.
[0096] NP_DATA_TIMER_INTERVAL: The interval for sending P value and scaling factor data, in seconds.
[0097] congestionLevel: Current network congestion status.
[0098] DynamicScaleFactor: Dynamic scaling factor.
[0099] N: difference data sending interval count.
[0100] curN: The number of times the difference data has been sent.
[0101] P: Byte length after compression.
[0102] gestureData: A list of gesture data from the recent period.
[0103] curGestureDataData: gesture data of the current frame.
[0104] lastGestureDataData: gesture data of the previous frame.
[0105] 2. Detailed description of the algorithm
[0106] 2.1 Update Parameters
[0107] Function: Dynamically adjust the difference data sending interval and compressed byte length according to the current network congestion status.
[0108] Input: Network congestion level.
[0109] Output: difference data sending interval N, compressed byte length P.
[0110] step:
[0111] A. Check the comparison result of congestionLevel with the preset thresholds (LOW_THRESHOLD, MEDIUM_THRESHOLD).
[0112] B. Set the values of N and P according to the comparison results. For example, in a low congestion state, N is set to the maximum value MAX_N and P is set to 4 (no compression); in a high congestion state, N is set to the minimum value MIN_N and P is set to 2 (high compression).
[0113] Step 3, obtaining real-time gesture data within a preset period and calculating a dynamic scaling factor;
[0114] Based on the above embodiment, step 3 specifically includes:
[0115] Step 3.1, traverse the gesture data in the target period and calculate the difference between the position and rotation angle in adjacent frames;
[0116] Step 3.2, select the maximum value z among the differences;
[0117] Step 3.3, calculate the integer value range [-a, a] that can be expressed by the current byte length according to the compressed byte length;
[0118] Step 3.4, calculate the dynamic scaling factor, wherein the expression of the dynamic scaling factor is
[0119] DynamicScaleFactor=|a| / (z*T)
[0120] Where T is the redundant constant factor.
[0121] During specific implementation, the dynamic scaling factor may be calculated based on gesture data of a recent period of time and the current compressed byte length.
[0122] Input: gesture data list gestureData, compressed byte length P.
[0123] Output: dynamic scaling factor DynamicScaleFactor.
[0124] The steps are as follows:
[0125] A. Traverse gestureData and calculate the difference in position and rotation angle in adjacent frames.
[0126] B. Find the maximum value z among these differences.
[0127] C. Calculate the integer value range [-a,a] that can be expressed by the current byte length based on P.
[0128] D. Calculate the dynamic scaling factor DynamicScaleFactor = |a| / (z*T), where T is a redundant constant factor.
[0129] Step 4, compressing the real-time gesture data according to the dynamic scaling factor and the compression byte length;
[0130] Based on the above embodiment, step 4 specifically includes:
[0131] Step 4.1, convert the root node of the real-time gesture data from the world coordinates to the observation coordinates and reassign it;
[0132] Step 4.2, scaling the coordinate components in the re-assigned real-time gesture data according to the dynamic scaling factor and rounding them to integers;
[0133] Step 4.3, calling a preset encoding function to encode each coordinate component according to the compressed byte length;
[0134] Step 4.4, store the encoded byte array into a new array.
[0135] Furthermore, the step 4.3 specifically includes:
[0136] Step 4.3.1, check whether the byte length of the coordinate component is 2 or 3, if not, throw an exception;
[0137] Step 4.3.2, allocate the byte array bytes according to the byte length;
[0138] Step 4.3.3, for the 2-byte coordinate component, convert the integer to a short integer type, and store the high 8 bits and low 8 bits of the short integer value into bytes[0] and bytes[1] respectively;
[0139] Step 4.3.4, for the 3-byte coordinate components, operate on the lower 24 bits of the integer and store them in bytes[0], bytes[1], and bytes[2].
[0140] During specific implementation, the root node of the gesture data is converted from the world coordinates to the observation coordinates and reassigned.
[0141] Input: XRHand: gesture data of the current frame, mainCameraTransform: transformation information of the main camera;
[0142] Output: gesture data after coordinate conversion;
[0143] The steps are as follows:
[0144] A. Calculate the transformation matrix:
[0145] If transformMatrix is not initialized or the camera position has changed, calculate the transformation matrix from world coordinates to view coordinates.
[0146] transformMatrix=mainCameraTransform.worldToLocalMatrix
[0147] B. Extract root node data:
[0148] Get the position and rotation of the root node (rootPose) from the XRHand data;
[0149] C. Convert coordinates:
[0150] Convert the root node's position from world coordinates to eye coordinates:
[0151] localPosition=transformMatrix.MultiplyPoint(rootPose.position)
[0152] Update the position of the root node in the XRHand data to localPosition;
[0153] D. Reassign to XRHand:
[0154] Reassign the root node of the XRHand data using the updated position and rotation information.
[0155] Input the re-assigned real-time gesture data Vector3 object vector, dynamic scaling factor dynamicScaleFactor, byte length byteLength, and output the array of byte arrays encodedVector. The specific steps are as follows:
[0156] A. Scale the x, y, and z components of the Vector and round them to integers.
[0157] B. Call the EncodeInt function to encode each component.
[0158] C. Store the encoded byte array into a new array and return it.
[0159] Among them, the specific process of EncodeInt function encoding is as follows:
[0160] Input: integer value, byte length byteLength (must be 2 or 3).
[0161] Output: byte array bytes.
[0162] step:
[0163] A. Check if byteLength is 2 or 3, and throw an exception if it is not.
[0164] B. Allocate the byte array bytes according to byteLength.
[0165] C. For 2-byte encoding:
[0166] Convert value to short type.
[0167] Store the high 8 bits and low 8 bits of the short value into bytes[0] and bytes[1] respectively.
[0168] D. For 3-byte encoding:
[0169] Directly operate on the lower 24 bits of the integer and store them in bytes[0], bytes[1], and bytes[2].
[0170] Step 5: Send the compressed real-time gesture data to the cloud according to the preset transmission architecture;
[0171] In specific implementation, the transmission process of gesture data between the local terminal and the cloud is as follows: Figure 2 As shown in the figure, the local server integrated design adopts the local server integrated data transmission architecture based on RPC (remote procedure call), and the network communication function is directly integrated into the local VR application process, which avoids the additional server process overhead, simplifies the system architecture, and improves the data transmission efficiency.
[0172] Low-latency synchronization framework: By optimizing network configuration and data packet processing, it ensures that gesture data can be transmitted to the cloud in real time and accurately, and immediately fed back to the virtual environment, improving the smoothness of user interaction.
[0173] Step 6: The cloud receives the compressed real-time gesture data, restores the gesture data according to the dynamic scaling factor and the compressed byte length, and broadcasts it to the target module.
[0174] Based on the above implementation, step 6 specifically includes:
[0175] Step 6.1, the cloud calls a preset decoding function to decode each byte array in the compressed real-time gesture data;
[0176] Step 6.2, dividing the decoded integer by the dynamic scaling factor to obtain the coordinate component;
[0177] Step 6.3, create and return new gesture data;
[0178] In step 6.4, the new gesture data is converted from the observation coordinates back to the world coordinates and reassigned, and then broadcast to the XRHandSkeletonDriver that drives the skeletal animation and the XRHandMeshController that drives the mesh rendering, thereby updating the gesture actions in the cloud virtual environment.
[0179] Furthermore, the step 6.1 specifically includes:
[0180] Step 6.1.1, check if the byte length is 2 or 3, if not, throw an exception;
[0181] Step 6.1.2, decode the byte array according to the byte length;
[0182] Step 6.1.3, for a 2-byte byte array, shift bytes[0] left by 8 bits, and then perform a bitwise OR operation with bytes[1] to form a 16-bit integer;
[0183] Step 6.1.4, for the 3-byte byte array, assemble it into a 24-bit value and sign-extend it to 32 bits, shift bytes[0] left by 16 bits, shift bytes[1] left by 8 bits, and then perform a bitwise OR operation with bytes[2] to assemble it into a 24-bit value.
[0184] The cloud receives data from the local end and processes it accordingly based on the data type.
[0185] Algorithm implementation instructions:
[0186] A. Keep looping to get the latest network news.
[0187] B. Execute corresponding processing logic according to the message type (baseline information, difference information, parameter update information):
[0188] C. For baseline information, use the updated data directly.
[0189] D. For the difference information, use the current P and DynamicScaleFactor to calculate the original value, and update the gesture information of the current frame in combination with the previous frame data.
[0190] E. For parameter update information, update P and DynamicScaleFactor.
[0191] In the specific implementation, the input byte array is encodedVector, the dynamic scaling factor dynamicScaleFactor, and the byte length byteLength.
[0192] Output: Vector3 object.
[0193] The steps are as follows:
[0194] A. Call the DecodeInt function to decode each byte array in encodedVector.
[0195] B. Divide the decoded integer value by dynamicScaleFactor to get the x, y, and z components.
[0196] C. Create and return a new Vector3 object.
[0197] The DecodeInt function is used to decode the byte array back to integer data. The specific decoding process is as follows:
[0198] Byte array bytes, byte length byteLength (must be 2 or 3).
[0199] Output: integer value.
[0200] step:
[0201] A. Check if byteLength is 2 or 3, and throw an exception if not.
[0202] B. Decode the byte array according to byteLength.
[0203] C. For 2-byte decoding:
[0204] Shift bytes[0] left by 8 bits, and then perform a bitwise OR operation with bytes[1] to form a 16-bit integer.
[0205] D. For 3-byte decoding: pack into 24-bit value and sign-extend to 32 bits.
[0206] Shift bytes[0] left by 16 bits, shift bytes[1] left by 8 bits, and then perform a bitwise OR operation with bytes[2] to form a 24-bit value. At this point, the lower 24 bits of value contain the original data, but the upper 8 bits (i.e. the sign-extended part) are undefined (usually 0).
[0207] Check the highest bit (i.e., the sign bit) of bytes[0]. If the highest bit is 1 (i.e., (bytes[0] & 0x80) != 0), this indicates that this is a negative number. In order to extend value from 24 bits to 32 bits and maintain its sign, we need to set the highest 8 bits of value to 1. This is achieved by value|=~0xFFFFFF;, that is, first perform a bitwise OR operation on ~0xFFFFFF (i.e., 0xFF000000) and value, thereby setting the highest 8 bits of value to 1 and keeping the lower 24 bits unchanged.
[0208] The cloud converts the received gesture data from the observation coordinates back to the world coordinates and broadcasts the update.
[0209] Input: remoteXRHand: decompressed gesture data, mainCameraTransform: main camera transformation information in the cloud
[0210] Output: Gesture data after coordinate conversion
[0211] step:
[0212] A. Calculate the inverse transformation matrix:
[0213] If inverseTransformMatrix is not initialized or the camera position has changed, computes the inverse transformation matrix from view coordinates to world coordinates.
[0214] inverseTransformMatrix=mainCameraTransform.localToWorldMatrix
[0215] B. Convert coordinates:
[0216] Convert the root node position of the gesture data from the observation coordinates back to the world coordinates:
[0217] worldPosition=inverseTransformMatrix.MultiplyPoint(localPosition)
[0218] C. Update and broadcast gesture data:
[0219] Update the gesture data with the updated position and rotation information, and broadcast the update via RemoteXRHandTrackingEvents. Figure 3 shown.
[0220] The gesture recognition method based on VR cloud rendering provided in this embodiment significantly improves the gesture recognition accuracy and user interaction fluency in VR applications and enhances the sense of immersion through precise coordinate space conversion and efficient data transmission architecture; reduces development and deployment costs and improves the maintainability and scalability of the system; the data compression algorithm with dynamic factor adjustment can dynamically adjust the compression performance and data accuracy according to network conditions, ensuring the best data transmission efficiency and user experience in different network environments; the upper-layer application does not need to care about the specific source and processing process of gesture data, reducing the difficulty and complexity of development.
[0221] The compression algorithm is described in detail below in conjunction with specific embodiments.
[0222] The local end receives real-time gesture data from the gesture capture device, including the position and rotation angle of the gesture node.
[0223] According to the network congestion situation, assuming that the current network congestion is moderate, set P=2, N=12, and send the parameters to the cloud.
[0224] Get the most recent gesture data and calculate the dynamic scaling factor DynamicScaleFactor=5461.
[0225] The gesture data is compressed using DynamicScaleFactor and P=2.
[0226] Send compressed gesture data to the cloud.
[0227] The cloud receives the compressed data, restores the gesture data according to P and DynamicScaleFactor, and updates the gesture information.
[0228] The effect of data compression under each strategy is compared. Figure 4 shown.
[0229] Corresponding to the above method embodiment, see Figure 5 The embodiment of the present disclosure further provides a gesture recognition system 50 based on VR cloud rendering, including:
[0230] The receiving module 501 is used for receiving the real-time gesture data from the gesture capture device at the local end, wherein the real-time gesture data includes the position and rotation angle of the root node of the hand;
[0231] An adjustment module 502 is used to dynamically adjust the difference data transmission interval and the compressed byte length according to the network congestion situation, and send it to the cloud;
[0232] A calculation module 503 is used to obtain real-time gesture data within a preset period and calculate a dynamic scaling factor;
[0233] A compression module 504, for compressing the real-time gesture data according to the dynamic scaling factor and the compression byte length;
[0234] The transmission module 505 is used to send the compressed real-time gesture data to the cloud according to a preset transmission architecture;
[0235] The decompression module 506 is used to receive the compressed real-time gesture data in the cloud, restore the gesture data according to the dynamic scaling factor and the compressed byte length, and broadcast it to the target module.
[0236] Figure 5 The system shown can correspondingly execute the contents of the above method embodiment. For the parts not described in detail in this embodiment, refer to the contents recorded in the above method embodiment and will not be repeated here.
[0237] See also Figure 6 The embodiment of the present disclosure further provides an electronic device 60, which includes: at least one processor and a memory in communication with the at least one processor. The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the gesture recognition method based on VR cloud rendering in the aforementioned method embodiment.
[0238] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the gesture recognition method based on VR cloud rendering in the aforementioned method embodiment.
[0239] The embodiments of the present disclosure also provide a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the gesture recognition method based on VR cloud rendering in the aforementioned method embodiment.
[0240] Reference below Figure 6, which shows a schematic diagram of the structure of an electronic device 60 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0241] like Figure 6 As shown, the electronic device 60 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 60 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0242] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 60 to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device 60 with various devices is shown in the figure, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.
[0243] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0244] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0245] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0246] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can execute the relevant steps of the method embodiment.
[0247] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can perform the relevant steps of the method embodiment.
[0248] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0249] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0250] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware.
[0251] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.
[0252] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A gesture recognition method based on VR cloud rendering, It is characterized in that include: Step 1: The local end receives real-time gesture data from a gesture capture device, wherein the real-time gesture data includes the position and rotation angle of the root node of the hand; Step 2: dynamically adjust the difference data sending interval and compressed byte length according to network congestion, and send it to the cloud; Step 3, obtaining real-time gesture data within a preset period and calculating a dynamic scaling factor; Step 4, compressing the real-time gesture data according to the dynamic scaling factor and the compression byte length; Step 5: Send the compressed real-time gesture data to the cloud according to the preset transmission architecture; Step 6: The cloud receives the compressed real-time gesture data, restores the gesture data according to the dynamic scaling factor and the compressed byte length, and broadcasts it to the target module.
2. The method according to claim 1, Features , the step 2 specifically includes: Get the current network congestion status and compare it with the preset threshold; If the current network congestion state is greater than the preset threshold, it is determined to be a high congestion state, and the difference data sending interval and the compressed byte length are adjusted to the preset minimum values; If the current network congestion state is less than or equal to the preset threshold, it is determined to be a low congestion state, and the difference data sending interval and the compressed byte length are adjusted to the preset maximum values; The adjusted difference data sending interval and compressed byte length are sent to the cloud.
3. The method according to claim 2, Features , the step 3 specifically includes: Step 3.1, traverse the gesture data in the target period and calculate the difference between the position and rotation angle in adjacent frames; Step 3.2, select the maximum value z among the differences; Step 3.3, calculate the integer value range [-a, a] that can be expressed by the current byte length according to the compressed byte length; Step 3.4, calculate the dynamic scaling factor, wherein the expression of the dynamic scaling factor is DynamicScaleFactor=|a| / (z*T) Where T is the redundant constant factor.
4. The method according to claim 3, Features , the step 4 specifically includes: Step 4.1, convert the root node of the real-time gesture data from the world coordinates to the observation coordinates and reassign it; Step 4.2, scaling the coordinate components in the re-assigned real-time gesture data according to the dynamic scaling factor and rounding them to integers; Step 4.3, calling a preset encoding function to encode each coordinate component according to the compressed byte length; Step 4.4, store the encoded byte array into a new array.
5. The method according to claim 4, Features , the step 4.3 specifically includes: Step 4.3.1, check whether the byte length of the coordinate component is 2 or 3, if not, throw an exception; Step 4.3.2, allocate the byte array bytes according to the byte length; Step 4.3.3, for the 2-byte coordinate component, convert the integer to a short integer type, and store the high 8 bits and low 8 bits of the short integer value into bytes[0] and bytes[1] respectively; Step 4.3.4, for the 3-byte coordinate components, operate on the lower 24 bits of the integer and store them in bytes[0], bytes[1], and bytes[2].
6. The method according to claim 5, Features , the step 6 specifically includes: Step 6.1, the cloud calls a preset decoding function to decode each byte array in the compressed real-time gesture data; Step 6.2, dividing the decoded integer by the dynamic scaling factor to obtain the coordinate component; Step 6.3, create and return new gesture data; In step 6.4, the new gesture data is converted from the observation coordinates back to the world coordinates and reassigned, and then broadcast to the XRHandSkeletonDriver that drives the skeletal animation and the XRHandMeshController that drives the mesh rendering, thereby updating the gesture actions in the cloud virtual environment.
7. The method according to claim 6, Features , the step 6.1 specifically includes: Step 6.1.1, check if the byte length is 2 or 3, if not, throw an exception; Step 6.1.2, decode the byte array according to the byte length; Step 6.1.3, for a 2-byte byte array, shift bytes[0] left by 8 bits, and then perform a bitwise OR operation with bytes[1] to form a 16-bit integer; Step 6.1.4, for the 3-byte byte array, assemble it into a 24-bit value and sign-extend it to 32 bits, shift bytes[0] left by 16 bits, shift bytes[1] left by 8 bits, and then perform a bitwise OR operation with bytes[2] to assemble it into a 24-bit value.
8. A gesture recognition system based on VR cloud rendering, It is characterized in that include: A receiving module, used for receiving real-time gesture data from a gesture capture device at a local end, wherein the real-time gesture data includes the position and rotation angle of the root node of the hand; An adjustment module is used to dynamically adjust the difference data sending interval and compressed byte length according to network congestion conditions, and send it to the cloud; A calculation module, used to obtain real-time gesture data within a preset period and calculate a dynamic scaling factor; A compression module, for compressing real-time gesture data according to a dynamic scaling factor and a compression byte length; A transmission module, used to send compressed real-time gesture data to the cloud according to a preset transmission architecture; The decompression module is used to receive the compressed real-time gesture data in the cloud, restore the gesture data according to the dynamic scaling factor and the compressed byte length, and broadcast it to the target module.
9. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the gesture recognition method based on VR cloud rendering as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the gesture recognition method based on VR cloud rendering according to any one of claims 1 to 7.
Citation Information
Patent Citations
Information processing method, device and equipment and readable storage medium
CN114842497A
Data transmission method and device, equipment and storage medium
CN116708301A
Cloud rendering method, cloud rendering device, medium and equipment
CN117956177A
