Cloud game virtual reality gesture interaction method and device based on adaptive multi-modal fusion and storage medium

Abstract

The invention provides a cloud game virtual reality gesture interaction method and device based on self-adaptive multi-modal fusion and a storage medium. The method comprises the following steps that S1, hand skeleton point data of a user is collected, three-dimensional coordinates of key skeleton points are obtained, and skeleton point feature vectors are generated; s2, collecting hand depth map data, extracting depth gradient features of a hand region, and generating a depth gradient feature vector; s3, collecting time sequence data of hand movement, calculating speed and acceleration characteristics of key skeleton points, and generating a time sequence characteristic vector; s4, performing weighted fusion on the skeleton point features, the depth map features and the time sequence features to generate a fusion feature vector; s5, inferring a user interaction intention and executing corresponding virtual reality operation in combination with the fusion feature vector and the scene information; through weighted fusion of multi-modal features, the method significantly improves the accuracy and anti-interference capability of gesture recognition, and is suitable for cloud games and virtual reality interaction scenes.

Description

Technical Field

[0001] The present invention relates to the field of cloud gaming virtual reality interaction technology, and in particular to a cloud gaming virtual reality gesture interaction method, device and storage medium based on adaptive multimodal fusion. Background Art

[0002] In cloud gaming virtual reality interaction scenarios, gesture recognition technology is crucial. Existing technologies usually perform gesture recognition based on a single modality, but there are the following problems in practical applications:

[0003] Limitations of a single modality: Skeletal point features are sensitive to noise and can easily lead to misjudgment due to detection errors;

[0004] Depth map features are easily distorted in complex backgrounds; time series features have limited ability to capture dynamic gestures and are difficult to process static gestures alone.

[0005] Insufficient adaptability to individual differences: Users’ hand size, finger length, and operating habits vary greatly, and the existing system cannot flexibly adapt, resulting in a decrease in recognition accuracy.

[0006] Multimodal fusion problem: The fusion of multimodal features is often a simple superposition, lacking a dynamic adjustment mechanism, resulting in insufficient robustness of the fused features.

[0007] Therefore, there is an urgent need for a cloud gaming virtual reality gesture interaction method that can adaptively fuse multimodal features to solve the above technical problems. Summary of the invention

[0008] The present invention provides a cloud game virtual reality gesture interaction method, device and storage medium based on adaptive multimodal fusion, which are used to solve the technical problems mentioned in the background technology.

[0009] To solve the above problems, the present invention provides the following technical solution: a cloud game virtual reality gesture interaction method based on adaptive multimodal fusion, comprising the following steps:

[0010] S1, collect the user's hand bone point data, obtain the three-dimensional coordinates of the key bone points of the hand, and generate the bone point feature vector;

[0011] S2, collecting the user's hand depth map data, extracting the depth gradient features of the hand area, and generating a depth gradient feature vector;

[0012] S3, collects the time series data of the user's hand movement, calculates the velocity and acceleration characteristics of key bone points, and generates a time series feature vector;

[0013] S4, weighted fusion of the skeleton point feature vector, the depth map feature vector and the time series feature vector to generate a fused feature vector;

[0014] S5, based on the fused feature vector and combined with the current scene information, infers the user's interaction intention and performs corresponding virtual reality interaction operations.

[0015] Further, step S1 includes:

[0016] S101, using a depth camera or sensor to capture the three-dimensional coordinates (x, y, z) of 21 key bone points of the user's hand, where the 21 key bone points include the center of the palm and the bases, first joints, second joints, and fingertips of five fingers;

[0017] S102, recording the skeleton point data at a sampling frequency of 30 frames / second, and obtaining the three-dimensional coordinates of the skeleton points in each frame;

[0018] S103, by calculating the Euclidean distance between key skeleton points, a skeleton point distance matrix D is generated, and the calculation formula is:

[0019] ;

[0020] Where, 1 ≤ i, j ≤ 21;

[0021] S104, extracting the skeleton point feature vector F_b from the upper triangular part of the distance matrix to form a geometric feature representation of the skeleton point.

[0022] Further, step S2 includes:

[0023] S201, using a depth sensor to obtain a depth image of the user's hand area, and generate depth value data in a matrix form, wherein the depth value of each pixel point represents the distance from the point to the camera;

[0024] S202, for each pixel point (i, j) in the depth map matrix, calculate its depth gradient value G(i, j), the calculation formula is:

[0025] ;

[0026] S203, dividing the depth gradient calculation result into grid areas, calculating the average gradient value of each grid, and generating a depth feature vector F_d.

[0027] Further, step S3 includes:

[0028] S301, recording the three-dimensional coordinate changes of the key skeleton points of the hand in consecutive frames;

[0029] S302, based on the coordinate changes in the continuous frames, the velocity and acceleration of the skeleton point are calculated using the difference method, and the calculation formulas of the velocity and acceleration are respectively:

[0030] ; a(t) = |v(t+1) - v(t)| / Δt;

[0031] S303, concatenating the calculated velocity and acceleration features to form a time series feature vector F_t.

[0032] Further, step S4 includes:

[0033] S401, obtaining a skeleton point feature vector F_b, a depth map feature vector F_d and a time series feature vector F_t, which respectively represent the geometric features, depth features and time variation features of the hand;

[0034] S402, set initial weights W_b, W_d and W_t for the skeleton point features, depth map features and time series features, and dynamically adjust the weights according to the confidence of real-time data. The weight calculation formula is:

[0035] W_b = C_b / (C_b + C_d + C_t)

[0036] W_d = C_d / (C_b + C_d + C_t)

[0037] W_t = C_t / (C_b + C_d + C_t)

[0038] Among them, C_b, C_d, and C_t are the confidences of the skeleton point features, depth map features, and time series features respectively;

[0039] S403, normalize the feature vector, the normalization formula is:

[0040] F'_b = (F_b - min(F_b)) / (max(F_b) - min(F_b))

[0041] F'_d = (F_d - min(F_d)) / (max(F_d) - min(F_d))

[0042] F'_t = (F_t - min(F_t)) / (max(F_t) - min(F_t))

[0043] S404, weighted fusion is performed on the normalized feature vectors according to the adjusted weights to generate a fused feature vector F_f. The fusion formula is:

[0044] F_f = W_b * F'_b + W_d * F'_d + W_t * F'_t.

[0045] Further, step S5 includes:

[0046] S501, constructing a gesture feature template library, where each gesture category corresponds to a feature template T and an interaction intention;

[0047] S502, by calculating the similarity between the fusion feature vector F_f and the template feature vector T, the matching degree between the current gesture and the template is determined. The similarity calculation formula is:

[0048] Similarity = (F_f · T) / (||F_f|| * ||T||)

[0049] S503: If the similarity is greater than a preset threshold, determining the interaction intention corresponding to the current gesture, and performing virtual reality interaction operations according to the intention, the operations including grabbing virtual objects, opening virtual menus, and moving virtual objects.

[0050] Furthermore,

[0051] When the user gesture is detected as "making a fist", the interaction intention is inferred to be "grabbing the virtual object" and the following operations are performed:

[0052] Calculate the user's hand position, select the nearest virtual object, and perform a grabbing action so that the virtual object moves synchronously with the user's palm;

[0053] When the user gesture is detected as "open palm", the interaction intention is inferred to be "open menu" and the following operations are performed:

[0054] A virtual menu interface pops up, waiting for the user to select a menu item through subsequent gestures;

[0055] When the user gesture is detected as "waving left", the interaction intention is inferred to be "moving the object to the left" and the following operations are performed:

[0056] Select the current virtual object and move the object to the left along the X axis by a predetermined distance.

[0057] The present invention also discloses a cloud gaming virtual reality gesture interaction device based on adaptive multimodal fusion, which is characterized by comprising:

[0058] The skeleton point data collection module is used to collect the skeleton point data of the user's hand, obtain the three-dimensional coordinates of the key skeleton points of the hand, and generate the skeleton point feature vector;

[0059] A depth map data acquisition module is used to collect the depth map data of the user's hand, extract the depth gradient features of the hand area, and generate a depth map feature vector;

[0060] The time series data acquisition module is used to collect the time series data of the user's hand movement, calculate the velocity and acceleration characteristics of key bone points, and generate a time series feature vector;

[0061] A feature fusion processing module is used to perform weighted fusion on the skeleton point feature vector, the depth map feature vector and the time series feature vector to generate a fused feature vector;

[0062] The scene interaction execution module is used to infer the user's interaction intention based on the fused feature vector and the current scene information, and perform corresponding virtual reality interaction operations.

[0063] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] The present invention significantly improves the accuracy and anti-interference ability of gesture recognition through weighted fusion of skeleton points, depth map and time series features. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 It is a flowchart of the steps of the cloud game virtual reality gesture interaction method based on adaptive multimodal fusion of the present invention;

[0068] Figure 2 It is a structural schematic diagram of a cloud gaming virtual reality gesture interaction device based on adaptive multimodal fusion of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention. It is understood that the drawings are only provided for reference and explanation, and are not used to limit the present invention.

[0070] See also Figure 1 ,

[0071] The present invention focuses on the collection, processing, calculation and fusion of multimodal gesture data, describes the implementation process of each step in detail, and displays the gesture recognition results outputted in the end. The following is a specific technical solution expansion and data calculation process.

[0072] Step S1, collecting skeleton point data, obtaining the three-dimensional coordinates of key skeleton points of the hand, and using a depth camera or sensor to capture the skeleton point data of the user's hand;

[0073] S101, collecting skeleton point data: including the three-dimensional coordinates (x, y, z) of 21 key skeleton points, the 21 key skeleton points are the center of the palm and the bases, first joints, second joints and fingertips of the five fingers.

[0074] For example, the data is: bone point 1 (X1, Y1, Z1), bone point 2 (X2, Y2, Z2)... bone point 21 (X21, Y21, Z21)

[0075] Sampling frequency: 30 frames / second

[0076] S102, bone point feature calculation:

[0077] 1. Calculate the Euclidean distance matrix between skeleton points:

[0078] The Euclidean distance formula between key bone points is:

[0079] ;

[0080] The generated distance matrix D (21×21) is a symmetric matrix (because D(i, j) = D(j, i)), and the values ​​on the diagonal are 0:

[0081] D = [

[0082] [0, D(1,2), D(1,3), ..., D(1,21)],

[0083] [D(2,1), 0, D(2,3), ..., D(2,21)],

[0084] [D(3,1), D(3,2), 0, ..., D(3,21)], ...

[0086] [D(21,1), D(21,2),D(21,3), ..., 0] ]

[0088] Assume that the coordinates of skeleton points 1, 2, and 3 are (1, 2, 3), (4, 5, 6), and (7, 8, 9) respectively:

[0089]

[0090]

[0091]

[0092] 2. Extract bone point feature vector:

[0093] Flatten the upper triangular part of the distance matrix to obtain the skeleton point feature vector F_b:

[0094] F_b = [D(1,2), D(1,3), ..., D(20,21)]

[0095] In this embodiment, the purpose of calculating the Euclidean distance matrix between skeleton points is:

[0096] Characterizing hand geometry: The Euclidean distance between skeleton points can fully capture the geometry of the palm and fingers, especially the relative position and curvature of the fingers.

[0097] Enhance the ability to distinguish static gestures: Different static gestures (such as "OK", "heart" or "fist") will cause the distance between skeleton points to change significantly. The Euclidean distance matrix can be used as an important feature of static gestures.

[0098] Provides a basis for symmetry analysis: By calculating the symmetry of the distance between the left and right hand bone points, it helps distinguish the movements of the left and right hands.

[0099] Step S2: Collect the user's hand depth map data and extract the depth gradient features of the hand area.

[0100] S201, using a depth sensor (such as a TOF camera or a structured light camera) to obtain a depth image of the hand area and generate depth value data in matrix form:

[0101] The resolution of the depth map is m × n (the number of rows is m and the number of columns is n). Each element D(i, j) of the matrix D represents the depth value of the pixel in the i-th row and j-th column.

[0102] The depth map matrix is:

[0103] D = [

[0104] [D(1,1), D(1,2), D(1,3), ..., D(1,n)],

[0105] [D(2,1), D(2,2), D(2,3), ..., D(2,n)], ...

[0107] [D(m,1), D(m,2), D(m,3), ..., D(m,n)] ]

[0109] The depth map matrix is ​​a two-dimensional array that represents the distance from each pixel in the hand area to the camera (in millimeters). Each element corresponds to the depth value of a pixel, and the specific matrix size depends on the resolution of the depth sensor (such as 640×480 or 1280×720, etc.).

[0110] Assuming the resolution of the depth map is 4×4, some depth values ​​are as follows:

[0111] D = [

[0112] [1500, 1503, 1510, 1515],

[0113] [1498, 1502, 1508, 1513],

[0114] [1495, 1500, 1505, 1510],

[0115] [1490, 1497, 1503, 1508] ]

[0117] in:

[0118] D(1,1) = 1500 means the depth value of the pixel in row 1 and column 1 is 1500 mm;

[0119] D(2,3) = 1508 means that the depth value of the pixel in the 2nd row and 3rd column is 1508 mm.

[0120] In actual scenarios, the resolution of the depth map is usually higher (such as 640×480), and the actual size of the matrix will contain more pixels and depth values.

[0121] S202, depth map feature calculation:

[0122] 1. Depth gradient calculation:

[0123] For each pixel (i, j) in the depth map matrix, calculate its gradient value:

[0124] ; For example: Assume that the pixel points D(i, j)=1500, D(i+1, j)=1503, D(i, j+1)=1502,

[0125]

[0126] 2. Generate depth gradient feature vector:

[0127] Divide the depth gradient calculation results into grid areas (such as dividing the depth map into 4×4 grids), calculate the average gradient value of each grid, and generate the depth feature vector F_d:

[0128] F_d = [G region 1, G region 2, ..., G region 16]

[0129] In this embodiment, the collection of depth map data is mainly used to solve the following problems:

[0130] Overcoming complex background interference: In actual application scenarios, the background where the hand is located may be complex (such as dynamic objects in the virtual scene or environmental noise around the user). The depth map records the distance information between the hand and the camera, which can effectively remove background interference and retain only the depth features of the hand.

[0131] Capturing hand geometry: Depth data can reflect changes in the 3D shape of the hand surface (such as bending, opening, or clenching fingers), thereby supplementing the hand surface information that skeleton points cannot directly provide.

[0132] Enhance the recognition ability of static gestures: In static gestures (such as open palm, clenched fist, "OK" gesture, etc.), the depth gradient characteristics of the hand are important for identification. Depth map data can more accurately describe these static features.

[0133] Step S3, collecting time series data of the user's hand movement and calculating the velocity and acceleration characteristics of key bone points;

[0134] S301, data collection:

[0135] Record the 3D coordinate changes of the key bone points of the hand in consecutive frames, for example:

[0136] Time t1: (x1, y1, z1), time t2: (x2, y2, z2), time t3: (x3, y3, z3), ...

[0137] S302, time series feature calculation:

[0138] 1. Calculate velocity and acceleration using the differential method:

[0139] ;

[0140] Acceleration calculation formula:

[0141] a(t) = |v(t+1) - v(t)| / Δt

[0142] For example, assuming that the positions of the captured bone point 1 in three consecutive frames are (10, 20, 30), (12, 22, 32), and (15, 25, 35), then:

[0143]

[0144] a(t1) = |5.19 - 3.46| / 1 = 1.73

[0145] 2. Generate time series feature vector

[0146] Concatenate the velocity and acceleration features to generate the feature vector F_t:

[0147] F_t = [v(t1), v(t2), ..., a(t1), a(t2), ...]

[0148] In this embodiment, the collection of time series data is used to solve the following problems:

[0149] Capturing the temporal variation characteristics of dynamic gestures: Dynamic gestures (such as waving, clicking, grabbing, etc.) involve the continuous trajectory of hand bone points over time. Time series data records these changes and is the key to dynamic gesture recognition.

[0150] Distinguishing static and dynamic features: Time series data can be used to distinguish static and dynamic gestures by calculating velocity and acceleration. For example, the velocity and acceleration of static gestures are close to zero, while dynamic gestures have obvious time-varying features.

[0151] Predicting gesture trends: By analyzing the changing patterns of time series data, it is possible to predict the trends of user hand movements (such as the completion of a grasping action, changes in the angle of palm rotation).

[0152] Step S4, weighted fusion of the skeleton point features, the depth map features and the time series features to generate a fused feature vector;

[0153] S401, obtaining multimodal feature data:

[0154] Skeleton point feature vector F_b:

[0155] Extracted from the Euclidean distance matrix of the skeleton points, including the features of the hand geometry. Assumptions:

[0156] F_b = [0.1, 0.2, 0.3, ...], with a feature dimension of m.

[0157] Depth map feature vector F_d:

[0158] Extract the depth features of the hand area from the depth gradient calculation results. Assumptions:

[0159] F_d = [0.4, 0.3, 0.2, ...], with feature dimension n.

[0160] Time series feature vector F_t:

[0161] Extracted from the speed and acceleration of hand movements, including the time-varying features of dynamic gestures. Assumptions:

[0162] F_t = [0.2, 0.4, 0.4, ...], with feature dimension k.

[0163] S402, feature normalization processing

[0164] In order to eliminate the dimensional differences of different modal features, each modal feature vector is normalized:

[0165] F'_b = (F_b - min(F_b)) / (max(F_b) - min(F_b))

[0166] F'_d = (F_d - min(F_d)) / (max(F_d) - min(F_d))

[0167] F'_t = (F_t - min(F_t)) / (max(F_t) - min(F_t))

[0168] Example normalized result:

[0169] F'_b = [0.33, 0.66, 1.0, ...]

[0170] F'_d = [1.0, 0.75, 0.5, ...]

[0171] F'_t = [0.5, 1.0, 1.0, ...]

[0172] S403, dynamic weight calculation

[0173] Dynamically adjust the weight of each modal feature according to the real-time confidence of multimodal data. Assumptions:

[0174] Confidence calculation:

[0175] The confidence level of each mode (C_b, C_d, C_t) can be calculated based on the noise level of the modal characteristics, data consistency, etc. For example:

[0176] C_b = 0.9, C_d = 0.8, C_t = 0.7

[0177] Weight calculation:

[0178] The weight of each modality is calculated using the following formula:

[0179] W_b = C_b / (C_b + C_d + C_t)

[0180] W_d = C_d / (C_b + C_d + C_t)

[0181] W_t = C_t / (C_b + C_d + C_t)

[0182] Example:

[0183] W_b = 0.9 / (0.9 + 0.8 + 0.7) ≈ 0.36

[0184] W_d = 0.8 / (0.9 + 0.8 + 0.7) ≈ 0.32

[0185] W_t = 0.7 / (0.9 + 0.8 + 0.7) ≈ 0.28

[0186] S404, fusion feature vector generation

[0187] The normalized multimodal features are weighted fused according to the weights to obtain the final fused feature vector F_f:

[0188] F_f = W_b * F'_b + W_d * F'_d + W_t * F'_t

[0189] Example calculation:

[0190] F_f = 0.36*[0.33, 0.66, 1.0] + 0.32*[1.0, 0.75, 0.5] + 0.28*[0.5,1.0, 1.0]

[0191] ≈ [0.12, 0.24, 0.36] + [0.32, 0.24, 0.16] + [0.14, 0.28, 0.28]

[0192] ≈ [0.58, 0.76, 0.80]

[0193] The fused feature vector F_f = [0.58, 0.76, 0.80] is the input for subsequent interaction intent inference.

[0194] Step S5, inferring the user's interaction intention based on the fused feature vector and the current scene information, and performing corresponding virtual reality interaction operations.

[0195] include:

[0196] S501, Construction of feature template library

[0197] A set of gesture templates are predefined, and each gesture template has a corresponding feature vector T and interaction intention, as shown in Table 1 below.

[0198] Table 1:

[0199] Gesture categories Feature vector (template) intention make a fist T1 = [0.6, 0.8, 0.9] "Grab virtual objects" Open palm T2 = [0.4, 0.7, 0.5] "Open Menu" Wave left T3 = [0.7, 0.5, 0.4] "Move object to the left"

[0200] S502, interaction intention matching

[0201] By calculating the similarity between the fused feature vector F_f and the template feature vector T, it is determined which template the current gesture is closest to.

[0202] Similarity calculation formula (cosine similarity):

[0203] Similarity = (F_f · T) / (||F_f|| * ||T||)

[0204] in:

[0205] F_f · T represents the vector dot product;

[0206] ||F_f|| and ||T|| represent the magnitude of the vector respectively.

[0207] Example calculation:

[0208] Assume F_f = [0.58, 0.76, 0.80], T1 = [0.6, 0.8, 0.9]:

[0209] F_f · T1 = (0.58*0.6 + 0.76*0.8 + 0.80*0.9) = 0.348 + 0.608 + 0.72 =1.676

[0210]

[0211]

[0212] Similarity = 1.676 / (1.16 * 1.32) ≈ 0.96

[0213] Matching results:

[0214] If the similarity > threshold (such as 0.8), the current gesture is determined to belong to template T1, and the corresponding intention is "grasping a virtual object".

[0215] S503, executing corresponding virtual reality interaction operations

[0216] Once the user's interaction intention is inferred, the system will perform the corresponding virtual reality interaction operations.

[0217] Operation mapping table:

[0218] Based on gesture categories and intentions, a mapping table of interactive operations is defined, as shown in Table 2 below.

[0219] Table 2:

[0220] Gesture categories intention Operation Description make a fist "Grab virtual objects" Select the virtual object pointed by the user's finger in the scene and grab it Open palm "Open Menu" Pop-up virtual menu interface Wave left "Move object to the left" Move the selected object 10 units to the left along the X axis

[0221] Operation execution mechanism:

[0222] Grasping virtual objects:

[0223] The user's gesture is detected as "making a fist" and the intention is inferred as "grasping a virtual object".

[0224] The system calculates the user's hand position and selects the nearest virtual object.

[0225] Perform a grabbing motion to lock the virtual object to move in sync with the user's hand.

[0226] Menu operation:

[0227] The user gesture is detected as "open palm" and the inferred intent is "open menu".

[0228] A virtual menu pops up, waiting for the user's subsequent operation (such as selecting a menu item through dynamic gestures).

[0229] Object movement:

[0230] The user gesture is detected as "waving left", and the intention is inferred as "moving the object to the left".

[0231] Performs a translation operation on the object along the X axis.

[0232] The present invention achieves a unified representation of multimodal data by integrating the generation of feature vectors; effectively infers user intentions through template matching and similarity calculation; and finally completes virtual reality interactive operations through mapping tables and real-time operation mechanisms. This process not only improves the accuracy of gesture recognition, but also enhances the naturalness and fluency of virtual reality interaction.

[0233] like Figure 2 As shown, the embodiment of the present application also provides a cloud gaming virtual reality gesture interaction device based on adaptive multimodal fusion, which is characterized by comprising:

[0234] The skeleton point data collection module 11 is used to collect the skeleton point data of the user's hand, obtain the three-dimensional coordinates of the key skeleton points of the hand, and generate the skeleton point feature vector;

[0235] The depth map data acquisition module 12 is used to collect the depth map data of the user's hand, extract the depth gradient features of the hand area, and generate a depth map feature vector;

[0236] The time series data acquisition module 13 is used to collect the time series data of the user's hand movement, calculate the speed and acceleration characteristics of the key bone points, and generate a time series feature vector;

[0237] A feature fusion processing module 20 is used to perform weighted fusion on the skeleton point feature vector, the depth map feature vector and the time series feature vector to generate a fused feature vector;

[0238] The scene interaction execution module 30 is used to infer the user's interaction intention based on the fused feature vector and the current scene information, and execute the corresponding virtual reality interaction operation.

[0239] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned cloud computer resource dynamic allocation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0240] In the specification and claims of the present application, the words "include / comprise" and the words "have / include" and their variations are used to specify the existence of stated features, values, steps or components, but do not exclude the existence or addition of one or more other features, values, steps, components or combinations thereof.

[0241] Some features of the present invention are described in different embodiments for clarity of description, however, these features may also be described in combination in a single embodiment. Conversely, some features of the present invention are described in only a single embodiment for brevity, however, these features may also be described in different embodiments alone or in any suitable combination.

[0242] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud game virtual reality gesture interaction method based on adaptive multimodal fusion, characterized in that: The following steps are involved: S1, collect the user's hand bone point data, obtain the three-dimensional coordinates of the key bone points of the hand, and generate the bone point feature vector; S2, collecting the user's hand depth map data, extracting the depth gradient features of the hand area, and generating a depth gradient feature vector; S3, collects the time series data of the user's hand movement, calculates the velocity and acceleration characteristics of key bone points, and generates a time series feature vector; S4, weighted fusion of the skeleton point feature vector, the depth map feature vector and the time series feature vector to generate a fused feature vector; S5, based on the fused feature vector and combined with the current scene information, infers the user's interaction intention and performs corresponding virtual reality interaction operations.

2. The cloud game virtual reality gesture interaction method based on adaptive multimodal fusion according to claim 1 is characterized in that: Step S1 includes: S101, using a depth camera or sensor to capture the three-dimensional coordinates (x, y, z) of 21 key bone points of the user's hand, where the 21 key bone points include the center of the palm and the bases, first joints, second joints, and fingertips of five fingers; S102, recording the skeleton point data at a sampling frequency of 30 frames / second, and obtaining the three-dimensional coordinates of the skeleton points in each frame; S103, by calculating the Euclidean distance between key skeleton points, a skeleton point distance matrix D is generated, and the calculation formula is: ； Where, 1 ≤ i, j ≤ 21; S104, extracting the skeleton point feature vector F_b from the upper triangular part of the distance matrix to form a geometric feature representation of the skeleton point.

3. The cloud game virtual reality gesture interaction method based on adaptive multimodal fusion according to claim 2 is characterized in that: Step S2 includes: S201, using a depth sensor to obtain a depth image of the user's hand area, and generate depth value data in a matrix form, wherein the depth value of each pixel point represents the distance from the point to the camera; S202, for each pixel point (i, j) in the depth map matrix, calculate its depth gradient value G(i, j), the calculation formula is: ； S203, dividing the depth gradient calculation result into grid areas, calculating the average gradient value of each grid, and generating a depth feature vector F_d.

4. The cloud game virtual reality gesture interaction method based on adaptive multimodal fusion according to claim 3 is characterized in that: Step S3 includes: S301, recording the three-dimensional coordinate changes of the key skeleton points of the hand in consecutive frames; S302, based on the coordinate changes in the continuous frames, the velocity and acceleration of the skeleton point are calculated using the difference method, and the calculation formulas of the velocity and acceleration are respectively: ； a(t) = |v(t+1) - v(t)| / Δt; S303, concatenating the calculated velocity and acceleration features to form a time series feature vector F_t.

5. The cloud game virtual reality gesture interaction method based on adaptive multimodal fusion according to claim 4 is characterized in that: Step S4 includes: S401, obtaining a skeleton point feature vector F_b, a depth map feature vector F_d and a time series feature vector F_t, which respectively represent the geometric features, depth features and time variation features of the hand; S402, set initial weights W_b, W_d and W_t for the skeleton point features, depth map features and time series features, and dynamically adjust the weights according to the confidence of real-time data. The weight calculation formula is: W_b = C_b / (C_b + C_d + C_t) W_d = C_d / (C_b + C_d + C_t) W_t = C_t / (C_b + C_d + C_t) Among them, C_b, C_d, and C_t are the confidences of the skeleton point features, depth map features, and time series features respectively; S403, normalize the feature vector, the normalization formula is: F'_b = (F_b - min(F_b)) / (max(F_b) - min(F_b)) F'_d = (F_d - min(F_d)) / (max(F_d) - min(F_d)) F'_t = (F_t - min(F_t)) / (max(F_t) - min(F_t)) S404, weighted fusion is performed on the normalized feature vectors according to the adjusted weights to generate a fused feature vector F_f. The fusion formula is: F_f = W_b * F'_b + W_d * F'_d + W_t * F'_t .

6. The cloud game virtual reality gesture interaction method based on adaptive multimodal fusion according to claim 5 is characterized in that: Step S5 includes: S501, constructing a gesture feature template library, where each gesture category corresponds to a feature template T and an interaction intention; S502, by calculating the similarity between the fusion feature vector F_f and the template feature vector T, the matching degree between the current gesture and the template is determined. The similarity calculation formula is: Similarity = (F_f · T) / (||F_f|| * ||T||) S503: If the similarity is greater than a preset threshold, determining the interaction intention corresponding to the current gesture, and performing virtual reality interaction operations according to the intention, the operations including grabbing virtual objects, opening virtual menus, and moving virtual objects.

7. The cloud game virtual reality gesture interaction method based on adaptive multimodal fusion according to any one of claims 1 to 6, characterized in that: When the user gesture is detected as "making a fist", the interaction intention is inferred to be "grabbing the virtual object" and the following operations are performed: Calculate the user's hand position, select the nearest virtual object, and perform a grabbing action so that the virtual object moves synchronously with the user's palm; When the user gesture is detected as "open palm", the interaction intention is inferred to be "open menu" and the following operations are performed: A virtual menu interface pops up, waiting for the user to select a menu item through subsequent gestures; When the user gesture is detected as "waving left", the interaction intention is inferred to be "moving the object to the left" and the following operations are performed: Select the current virtual object and move the object to the left along the X axis by a predetermined distance.

8. A cloud gaming virtual reality gesture interaction device based on adaptive multimodal fusion, characterized in that: include: The skeleton point data collection module is used to collect the skeleton point data of the user's hand, obtain the three-dimensional coordinates of the key skeleton points of the hand, and generate the skeleton point feature vector; A depth map data acquisition module is used to collect the depth map data of the user's hand, extract the depth gradient features of the hand area, and generate a depth map feature vector; The time series data acquisition module is used to collect the time series data of the user's hand movement, calculate the velocity and acceleration characteristics of key bone points, and generate a time series feature vector; A feature fusion processing module is used to perform weighted fusion on the skeleton point feature vector, the depth map feature vector and the time series feature vector to generate a fused feature vector; The scene interaction execution module is used to infer the user's interaction intention based on the fused feature vector and the current scene information, and perform corresponding virtual reality interaction operations.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images