Game sense operation control system based on gesture interaction

Abstract

The application relates to the technical field of program control, and discloses a game motion sensing operation control system based on gesture interaction. The system comprises a depth camera, a millimeter wave radar, a multi-modal fusion processor and a game host. The multi-modal fusion processor projects radar point clouds and hand depth maps into a three-dimensional voxel grid in a space under a camera coordinate system. The area variation coefficient of a hand foreground mask is calculated in real time, and when the area variation coefficient is lower than a shielding judgment threshold, it is judged that a shielding event occurs. In the shielding state, a connected domain with concentrated reflection intensity in the three-dimensional voxel is extracted, the centroid coordinates and the velocity vector of the connected domain are calculated, the direction component is mapped into a game perspective rotation control signal, and the modulus length component is mapped into a displacement step control signal. The application solves the problem that the control signal is interrupted when the optical gesture control is shielded, and the continuous output of the game control instruction is maintained by using the radar signal.

Description

Technical Field

[0001] This invention relates to the field of program control technology and discloses a game motion-sensing operation control system based on gesture interaction. Background Technology

[0002] Current gesture-based game motion-sensing operating systems mostly rely on external optical sensors to capture hand contours. The conventional approach involves a depth camera acquiring a depth map of the hand in the interaction space. A processor then segments the background and hand regions using a preset depth threshold, extracting the 2D contour of the hand's foreground mask. This contour is then used for contour fitting or depth feature extraction to reconstruct the hand's 3D spatial position and joint pose. The system calculates displacement vectors based on changes in the hand's 3D position between consecutive frames, mapping these vectors to movement or viewpoint rotation control commands for the virtual character in the game scene, thus completing the interactive loop from gesture actions to game controls.

[0003] The aforementioned conventional gesture control scheme based on optical sensors has a core flaw: when the player's hand is self-occluded or obscured by other objects such as the torso during game interaction, the depth camera cannot acquire complete optical reflection signals from the hand, causing the segmented foreground mask area of ​​the hand to shrink drastically or even disappear. Under this physical occlusion, because the optical sensor loses visibility of the hand's shape, the processor cannot continue to extract effective hand displacement vectors, resulting in interruptions or jumps in the control signals output to the game console, causing the virtual character's movements to stagnate during the occlusion period. Summary of the Invention

[0004] The purpose of this invention is to provide a game motion-sensing operation control system based on gesture interaction, which can effectively solve the problems in the background art mentioned above.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The gesture-based game motion control system includes: a depth camera, millimeter-wave radar, a multimodal fusion processor, and a game console; A depth camera captures hand depth images in the interactive space, while a millimeter-wave radar captures radar point cloud data of the interactive space. The multimodal fusion processor uses a pre-calibrated camera extrinsic matrix to project the radar point cloud from the radar coordinate system to the camera coordinate system where the depth map is located. It then constructs a spatial three-dimensional voxel mesh in the camera coordinate system and fills the hand foreground mask of the hand depth map and the reflection intensity value of the radar point cloud into the corresponding three-dimensional voxels. The multimodal fusion processor calculates the area variation coefficient of the hand foreground mask in the current frame in real time. When the area variation coefficient is lower than the preset occlusion determination threshold, an occlusion event is determined to have occurred. In the occlusion state, the multimodal fusion processor extracts the connected domains where the reflection intensity is concentrated in the three-dimensional voxel, calculates the centroid coordinates and velocity vector of the connected domain, maps the direction component of the velocity vector to the rotation control signal of the game view, maps the magnitude component of the velocity vector to the displacement step control signal of the virtual character in the game scene, and sends the rotation control signal and displacement step control signal to the game host. When the area variation coefficient recovers to above the occlusion determination threshold, the multimodal fusion processor switches to the hand joint extraction path based on the hand depth map to generate control signals and send them to the game console.

[0006] Preferably, the process of the multimodal fusion processor constructing a spatial three-dimensional voxel grid and filling data includes: taking the origin of the camera coordinate system as the center, dividing the three-dimensional voxel grid according to a fixed physical size, for each frame of hand depth map captured by the depth camera, using a depth threshold to remove background noise to generate a hand foreground mask, and projecting the pixels with valid depth values ​​in the hand foreground mask to the corresponding three-dimensional voxel grid positions and marking them as visual occupancy states. For radar point cloud data collected by millimeter-wave radar, the reflection intensity value carried by each radar point after projection transformation is accumulated to the corresponding three-dimensional voxel grid position. When the same three-dimensional voxel grid has both a visual occupancy marker and a reflection intensity value, the three-dimensional voxel grid is recorded as a multimodal overlapping voxel. The multimodal fusion processor counts the total number of multimodal overlapping voxels in each frame. If the total number of multimodal overlapping voxels is lower than a preset registration verification threshold, the online recalibration process of the camera extrinsic matrix is ​​triggered.

[0007] Preferably, the multimodal fusion processor calculates the area variation coefficient of the hand foreground mask by dividing the hand foreground mask of the current frame into multiple non-overlapping local regions on the image plane, calculating the pixel area of ​​the hand foreground mask in each local region, and calculating the ratio of the variance to the mean of the pixel areas of all local regions as the area variation coefficient. Before determining that an occlusion event has occurred, the multimodal fusion processor also extracts the bounding rectangle of the hand foreground mask on the image plane, calculates the displacement distance of the center point coordinates of the bounding rectangle relative to the center point coordinates of the bounding rectangle in the previous frame, and only when the displacement distance is less than the preset boundary walk threshold and the area variation coefficient is lower than the occlusion determination threshold, does it finally confirm that an occlusion event has occurred and trigger the radar connected domain extraction process.

[0008] Preferably, the process of the multimodal fusion processor extracting the connected components with concentrated reflection intensity in the three-dimensional voxels includes: after determining that an occlusion event has occurred, locking the three-dimensional voxel grid region corresponding to the hand foreground mask in the previous frame as the search starting point, using a preset lower limit of reflection intensity as the seed point screening condition, and using a three-dimensional region growing algorithm to expand in the three-dimensional voxel grid, and grouping the three-dimensional voxel grids with reflection intensity values ​​greater than the lower limit of reflection intensity and spatially adjacent into the same connected component; During the expansion process of the region growing algorithm, if the reflection intensity value of the current three-dimensional voxel mesh to be expanded is lower than the lower limit of reflection intensity, the reflection intensity gradient between the three-dimensional voxel mesh to be expanded and the seed point is calculated. When the reflection intensity gradient is less than the preset edge smoothing threshold, the three-dimensional voxel mesh to be expanded is incorporated into the connected region as a transition edge voxel.

[0009] Preferably, the process by which the multimodal fusion processor calculates the centroid coordinates and velocity vectors of the connected domain includes: weighting the spatial coordinates of all three-dimensional voxel meshes within the connected domain according to their corresponding reflection intensity values ​​to obtain the weighted centroid coordinates of the current frame; The multimodal fusion processor has a built-in Kalman filter. It uses the weighted centroid coordinates of the current frame as the observation input value of the Kalman filter, and calculates the smoothed centroid coordinates of the current frame by combining the predicted centroid coordinates and prediction error covariance of the Kalman filter output of the previous frame. The multimodal fusion processor extracts a smooth centroid coordinate sequence from multiple consecutive frames and calculates the smooth velocity vector of the current frame by using the ratio of the difference in smooth centroid coordinates between adjacent frames to the inter-frame time interval.

[0010] Preferably, the method by which the multimodal fusion processor maps the velocity vector into a rotation control signal and a displacement step control signal includes: inputting the direction component of the smooth velocity vector into a preset dead-zone nonlinear mapping function; setting the rotation control signal output to zero when the magnitude of the direction component is within the dead-zone range; and amplifying the rotation control signal according to a nonlinear scaling factor when the magnitude of the direction component exceeds the dead-zone range. When the multimodal fusion processor detects that the area variation coefficient has recovered to above the occlusion determination threshold, it records the coordinates of the first hand joint point extracted by the depth camera at the switching moment, calculates the spatial deviation vector between the coordinates of the first hand joint point and the smooth centroid coordinates of the last frame before the switching moment, and superimposes the spatial deviation vector into the control signal generated by the hand joint point extraction path in subsequent consecutive frames according to the exponential decay law until the magnitude of the spatial deviation vector decays to zero.

[0011] Preferably, the method of dividing the three-dimensional voxel mesh according to a fixed physical size is as follows: the multimodal fusion processor extracts the minimum depth distance of the effective depth value in the hand depth map in real time, and queries a preset distance resolution mapping table according to the minimum depth distance to obtain the target physical size corresponding to the current minimum depth distance; The multimodal fusion processor dynamically divides the current frame into a three-dimensional voxel mesh based on the target physical size, where the depth distance recorded in the distance resolution mapping table is positively correlated with the target physical size; When the minimum depth distance decreases, the target physical size of the 3D voxel mesh decreases accordingly to improve spatial resolution; when the minimum depth distance increases, the target physical size of the 3D voxel mesh increases accordingly to expand the spatial coverage.

[0012] Preferably, the occlusion determination threshold is determined as follows: the multimodal fusion processor maintains a sliding window queue of fixed length, and in the initial running phase when no occlusion event occurs, the area variation coefficients calculated from multiple consecutive frames are sequentially stored into the sliding window queue. When the sliding window queue is full, calculate the arithmetic mean and standard deviation of all area variation coefficients in the sliding window queue, and use the arithmetic mean minus the standard deviation by a preset multiple as the updated occlusion judgment threshold. During subsequent operation, the sliding window queue continuously updates the area variation coefficient according to the first-in-first-out principle, and the multimodal fusion processor recalculates and updates the occlusion determination threshold every preset update cycle.

[0013] Preferably, before calculating the weighted centroid coordinates of the current frame, the multimodal fusion processor also performs a spatial constraint filtering process: the multimodal fusion processor extracts the centroid positions of the hand foreground mask of a preset number of consecutive frames before the occlusion event occurs, and uses a linear extrapolation algorithm to predict the predicted three-dimensional region of the hand in the current frame under the occlusion state. The multimodal fusion processor uses the predicted 3D region of the hand as a spatial mask to remove the 3D voxel mesh located outside the predicted 3D region of the hand in the connected domain, and retain only the remaining 3D voxel mesh located inside the predicted 3D region of the hand. The multimodal fusion processor recalculates the weighted average using the spatial coordinates of the remaining three-dimensional voxel mesh and the corresponding reflection intensity values ​​to obtain the constrained weighted centroid coordinates.

[0014] Preferably, the process by which the multimodal fusion processor maps the magnitude component of the velocity vector to the displacement step control signal of the virtual character in the game scene includes: the multimodal fusion processor obtaining the motion state identifier of the virtual character currently rendered by the game host through the inter-process communication interface, the motion state identifier including the stationary state, the running state, and the jumping state; The multimodal fusion processor retrieves the corresponding target mapping coefficient from the preset mapping coefficient set according to the motion state identifier, multiplies the magnitude component of the velocity vector by the target mapping coefficient to generate a displacement step control signal, wherein the first mapping coefficient corresponding to the stationary state is greater than the second mapping coefficient corresponding to the running state, and the second mapping coefficient corresponding to the running state is greater than the third mapping coefficient corresponding to the jumping state.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention solves the problem of control signal interruption in optical gesture control under occlusion by introducing millimeter-wave radar and a depth camera for multimodal fusion. The system uses the camera extrinsic parameter matrix to project the radar point cloud and hand depth map into a spatial three-dimensional voxel grid in the camera coordinate system. By calculating the area variation coefficient of the hand foreground mask in real time, the system can detect occlusion events and trigger the radar data path when the optical signal deteriorates due to occlusion. Under occlusion, the system extracts the connected components with concentrated radar reflection intensity in the three-dimensional voxel grid, calculates the centroid coordinates and velocity vector of the connected components, and maps the direction component and magnitude component of the velocity vector to the game view rotation control signal and displacement step control signal, respectively. Taking advantage of the characteristic that radar signals are not affected by physical occlusion, the system maintains continuous output of control signals during occlusion.

[0016] 2. This invention employs multi-layered refinement in its fusion mechanism. It triggers online recalibration of the camera's extrinsic parameter matrix by statistically analyzing the number of overlapping voxels across multiple modalities, ensuring accurate spatial registration. By dividing local regions, calculating the area variation coefficient, and combining it with the displacement distance of the circumscribed rectangle's center point for boundary walk determination, it eliminates false triggers in non-occluded scenes such as hands rapidly moving out of the field of view. By using a 3D region growing algorithm and introducing reflection intensity gradient and edge smoothing thresholds to determine transitional edge voxels, it improves the completeness of connected component extraction. A Kalman filter is used to smooth the weighted centroid coordinates, reducing velocity vector calculation noise caused by radar point cloud jitter. Finally, by dynamically adjusting the object size of the 3D voxel mesh based on the minimum depth distance of the hand... The system optimizes the size of the hand, balancing spatial resolution and coverage at different distances; it dynamically updates the occlusion threshold using a sliding window queue to adapt to different hand sizes and movement habits; it suppresses interference from environmental clutter on centroid calculation during occlusion by introducing a 3D region for hand prediction and removing external voxels from connected domains; it differentiates the velocity vector magnitude component by retrieving corresponding mapping coefficients based on different motion state identifiers of the virtual character, making the output of the control signal compatible with the game's physics calculation logic; and when recovering from the occlusion state to the visual state, it avoids the step jump of the control signal during mode switching by calculating the spatial deviation vector between the joint coordinates and the smoothed centroid coordinates and superimposing it onto the visual control signal according to an exponential decay law. Attached Figure Description

[0017] Figure 1 This is the overall flowchart of the multimodal fusion control of the present invention; Figure 2 This is a flowchart of the three-dimensional voxel mesh construction, registration, and verification process of the present invention; Figure 3 This is a flowchart of the area variation coefficient calculation and dynamic threshold update process of the present invention; Figure 4 This is a flowchart of the radar connectivity extraction process of the present invention; Figure 5 This is a flowchart of the spatial constraints and centroid smoothing calculation of the present invention; Figure 6 This is a flowchart of the mode switching and adaptive control signal mapping of the present invention. Detailed Implementation

[0018] Please refer to the attached document. Figure 1 This embodiment provides a gesture-based game motion control system, including a depth camera, a millimeter-wave radar, a multimodal fusion processor, and a game console. The depth camera and millimeter-wave radar cover the same interaction area, and their acquisition timing is synchronized. The depth camera and millimeter-wave radar are connected to the multimodal fusion processor through data transmission interfaces, and the multimodal fusion processor establishes a data interaction link with the game console through a communication interface.

[0019] When an operator performs a gesture within the interactive area, a depth camera captures a hand depth map of the interactive space, and a millimeter-wave radar captures radar point cloud data of the interactive space. A multimodal fusion processor fuses the received hand depth map and radar point cloud data to generate corresponding game control signals and send them to the game console. The game console drives the virtual character in the game scene to perform corresponding actions based on the received control signals, thus completing the closed-loop control of the gesture interaction.

[0020] Please refer to the attached document. Figure 2 In a preferred embodiment, after receiving the hand depth map acquired by a depth camera and the radar point cloud data acquired by a millimeter-wave radar, the multimodal fusion processor first performs timestamp alignment on the two data streams, matching the hand depth map and radar point cloud data whose acquisition time difference is within a preset synchronization threshold range into a single frame of processed data. The multimodal fusion processor then uses a pre-calibrated camera extrinsic matrix to project the matched radar point cloud data from the radar coordinate system to the camera coordinate system containing the hand depth map. The camera extrinsic matrix includes a rotation matrix and a translation vector. The rotation matrix describes the rotation relationship between the radar coordinate system and the camera coordinate system, and the translation vector describes the translation of the origin of the radar coordinate system relative to the origin of the camera coordinate system. After coordinate transformation, the three-dimensional coordinates of each point cloud unit in the radar point cloud data are obtained in its corresponding three-dimensional coordinates in the camera coordinate system. The coordinate transformation operation satisfies the following formula:

[0021] in, Let be the three-dimensional coordinate vector of a point cloud unit in the radar coordinate system. It is a 3×3 rotation matrix. It is a 3×1 translation vector. This is the three-dimensional coordinate vector in the camera coordinate system after projection.

[0022] When the multimodal fusion processor preprocesses the received radar point cloud data, it first performs a range-dimensional fast Fourier transform on the original radar echo data to obtain the echo signal of each range gate. Then, it performs a Doppler-dimensional fast Fourier transform on the echo signal of each range gate to obtain the velocity information corresponding to each range gate. After that, it processes the echo signal through a digital beamforming algorithm to obtain the angle information of each target point, thus completing the calculation of three-dimensional information of range, velocity, and angle.

[0023] After completing the 3D information calculation, the multimodal fusion processor uses a constant false alarm rate (CFAR) detection algorithm to perform target detection on the calculated echo data, remove invalid signals corresponding to clutter, extract the 3D coordinates and reflection intensity values ​​corresponding to valid target points, and generate preprocessed radar point cloud data. The CFAR detection algorithm adopts a cell-average CFAR detection method. Taking the current detection cell as the center, it selects the guard cells and reference cells on both sides, calculates the average power of the reference cells, sets a detection threshold, and compares the power of the current detection cell with the detection threshold. When the power is greater than the detection threshold, the cell is determined to be a valid target cell, and the corresponding point cloud data is extracted.

[0024] Please refer to the attached document. Figure 3 In a preferred embodiment, the multimodal fusion processor constructs a spatial three-dimensional voxel mesh in the camera coordinate system. The spatial three-dimensional voxel mesh is centered at the origin of the camera coordinate system and divided into multiple uniformly distributed three-dimensional voxel units according to fixed physical dimensions. Each three-dimensional voxel unit corresponds to a fixed spatial region in the camera coordinate system. The multimodal fusion processor fills the corresponding three-dimensional voxel units with the hand foreground mask from the hand depth map and the reflection intensity values ​​from the radar point cloud, respectively.

[0025] The multimodal fusion processor preprocesses the received hand depth map. First, it fills the depth map with invalid pixels. Invalid pixels with a depth value of zero or that are outside the maximum acquisition range are filled by neighborhood interpolation. The filling process uses a bilinear interpolation algorithm, which uses a weighted average of the depth values ​​of the four valid neighboring pixels around the invalid pixel to obtain the filling depth value of the invalid pixel.

[0026] After invalid pixel filling is completed, the multimodal fusion processor performs bilateral filtering on the depth map. The kernel function of the bilateral filter includes a spatial domain kernel and a pixel value domain kernel. The spatial domain kernel calculates the weights based on the Euclidean distance between pixels, and the pixel value domain kernel calculates the weights based on the difference in pixel depth values. The pixel depth value of the filtered output is a weighted average of the depth values ​​of neighboring pixels. The bilateral filtering process retains the edge contour information of the hand region while removing random noise in the depth map, thus avoiding edge blurring during subsequent foreground segmentation.

[0027] After filtering, the multimodal fusion processor performs foreground segmentation on the preprocessed depth map using a preset depth threshold range. The lower limit of the depth threshold range corresponds to the nearest interaction distance in the interaction space, and the upper limit corresponds to the farthest interaction distance. Pixels with depth values ​​within the depth threshold range are marked as hand foreground pixels, and pixels with depth values ​​outside the depth threshold range are marked as background pixels, generating an initial binarized hand foreground mask. The multimodal fusion processor then performs morphological opening operations on the initial hand foreground mask. First, an erosion operation is performed to remove isolated small noise points from the mask, and then a dilation operation is performed to restore the contour of the hand foreground region. The mask after morphological opening is the final hand foreground mask.

[0028] The multimodal fusion processor projects the hand foreground pixels with valid depth values ​​from the hand foreground mask onto a 3D space in the camera coordinate system using the camera intrinsic parameter matrix, obtaining the spatial position corresponding to each hand foreground pixel. The 3D voxel unit to which this spatial position belongs is then marked as visually occupied. The camera intrinsic parameter matrix is ​​obtained through a pre-calibrated process using a checkerboard calibration board. Multiple frames of calibration board images from different angles are acquired, and the corner coordinates of the calibration board in each frame are extracted. A calibration algorithm is then used to solve for the camera's focal length, principal point coordinates, and distortion coefficients, generating the camera intrinsic parameter matrix and distortion correction parameters. Simultaneously, the multimodal fusion processor accumulates the reflection intensity value carried by each radar point cloud unit after coordinate projection into the 3D voxel unit to which the spatial position of that point cloud unit belongs, completing the filling of the 3D voxel grid with radar data.

[0029] To clearly define the state marking and data filling rules of the three-dimensional voxel unit, the filling logic is specified in the following table in this embodiment: Table 1. Status Marking and Data Filling Rules for Three-Dimensional Voxel Units ; After completing the state labeling and data filling of the 3D voxel mesh for each frame according to the rules described in Table 1, the multimodal fusion processor calculates the area variation coefficient of the hand foreground mask in real time. Specifically, the multimodal fusion processor divides the hand foreground mask of the current frame into multiple non-overlapping local regions on the image plane, counts the number of hand foreground pixels in each local region, obtains the area of ​​the foreground pixels corresponding to each local region, and calculates the ratio of the variance to the arithmetic mean of the foreground pixel areas of all local regions as the area variation coefficient of the current frame. The calculation process of the area variation coefficient satisfies the following formula:

[0030] in, This is the area variation coefficient of the hand foreground mask in the current frame. Let V be the variance of the foreground pixel area of ​​all local regions. This is the arithmetic mean of the foreground pixel areas of all local regions.

[0031] The multimodal fusion processor compares the calculated area variation coefficient with a preset occlusion determination threshold. When the area variation coefficient is lower than the preset occlusion determination threshold, an occlusion event is determined to have occurred in the current frame, and the system enters an occlusion state. In the occlusion state, the multimodal fusion processor extracts connected components with concentrated reflection intensity from the 3D voxel mesh.

[0032] The multimodal fusion processor uses a preset lower limit of reflection intensity as a filtering condition, traverses all voxel units in the three-dimensional voxel grid, and marks voxel units with reflection intensity values ​​greater than the lower limit of reflection intensity as effective reflection voxels. It performs three-dimensional connected component analysis on the effective reflection voxels, and uses the 6-neighborhood connectivity rule to group spatially adjacent effective reflection voxels into the same connected component. It extracts the connected component with the most voxel units among all connected components as the target connected component, which corresponds to the operator's hand area in the interaction space.

[0033] Furthermore, the multimodal fusion processor calculates the centroid coordinates and velocity vector of the target connected domain. Specifically, the multimodal fusion processor calculates the arithmetic mean of the spatial coordinates of all voxel units within the target connected domain to obtain the centroid coordinates of the target connected domain in the current frame. The multimodal fusion processor stores the centroid coordinates of multiple consecutive frames, calculates the difference between the centroid coordinates of the current frame and the centroid coordinates of the previous frame, and uses the ratio of this difference to the time interval between adjacent frames as the velocity vector of the current frame. The multimodal fusion processor maps the direction component of the velocity vector to the rotation control signal of the game viewpoint, and maps the magnitude component of the velocity vector to the displacement step control signal of the virtual character in the game scene.

[0034] The multimodal fusion processor maps the horizontal component of the velocity vector in the camera coordinate system to a horizontal rotation control signal for the game view, and the vertical component of the velocity vector in the camera coordinate system to a vertical rotation control signal for the game view. It then multiplies the magnitude component of the velocity vector with a preset displacement mapping coefficient to generate a displacement step control signal for the virtual character. The multimodal fusion processor sends the generated rotation control signal and displacement step control signal to the game console via a communication interface. The game console then drives the virtual character to perform the corresponding view rotation and displacement actions based on the received control signals.

[0035] During the processing of each frame, the multimodal fusion processor continuously calculates the area variation coefficient of the hand foreground mask. When the area variation coefficient recovers to above the occlusion determination threshold, the occlusion event is determined to end, the system exits the occlusion state, and the multimodal fusion processor switches to the hand joint extraction path based on the hand depth map to generate control signals and send them to the game console.

[0036] The multimodal fusion processor extracts features from the depth region corresponding to the foreground mask of the hand in the hand depth map. It then outputs the 3D coordinates of multiple hand joints in the camera coordinate system using a pre-built convolutional neural network model. The input to the convolutional neural network model is a normalized image of the hand depth region. This image passes through multiple convolutional and pooling layers to extract depth features. These extracted features are then mapped to joint coordinate vectors of corresponding dimensions via fully connected layers. Each joint coordinate output by the model corresponds to its 3D spatial position in the camera coordinate system. Based on the changes in hand joint coordinates between consecutive frames, the multimodal fusion processor generates corresponding game view rotation control signals and virtual character displacement step size control signals. These generated control signals are sent to the game console to complete gesture interaction control in unobstructed states.

[0037] In this embodiment, the system uses multimodal data fusion from a depth camera and millimeter-wave radar to switch to a control signal generation path based on radar point cloud data when the optically acquired hand foreground mask is occluded and degraded. This maintains the continuous output of game control signals during the occlusion period and avoids the stagnation of virtual character movements under occlusion.

[0038] Please refer to the attached document. Figure 4In a preferred embodiment, the multimodal fusion processor simultaneously performs spatial registration accuracy verification and online update of the camera extrinsic matrix during the construction of a spatial 3D voxel mesh and data filling in the camera coordinate system. After completing the state marking and data filling of each frame of the 3D voxel mesh, the multimodal fusion processor traverses all 3D voxel units, counts the total number of 3D voxel units that simultaneously possess visual occupancy markers and effective reflection intensity values, and marks these 3D voxel units as multimodal coincident voxels. The multimodal fusion processor compares the total number of multimodal coincident voxels obtained from the statistics with a preset registration verification threshold. If the total number of multimodal coincident voxels is lower than the preset registration verification threshold, the online recalibration process of the camera extrinsic matrix is ​​triggered.

[0039] In the online recalibration process, the multimodal fusion processor acquires continuous multi-frame synchronized hand depth maps and radar point cloud data. It extracts the 3D visual point cloud and effective radar point cloud corresponding to the hand foreground mask in each frame. The two sets of point clouds are then registered using an iterative nearest-point algorithm to obtain the optimized rotation matrix and translation vector. These optimized rotation and translation vectors are then used to update the camera extrinsic parameter matrix, completing the online recalibration of the extrinsic parameters. During the registration process of the iterative nearest-point algorithm, the two sets of point clouds are initially aligned. Then, corresponding point pairs between the two sets of point clouds are found iteratively. The rigid body transformation parameters that minimize the mean square error between corresponding point pairs are calculated until the mean square error is below a preset convergence threshold or the number of iterations reaches a preset upper limit. The final rigid body transformation parameters are then output as the optimized extrinsic parameter matrix.

[0040] Furthermore, the multimodal fusion processor employs a dynamic adjustment method to construct a spatial three-dimensional voxel mesh. Specifically, the multimodal fusion processor extracts the minimum depth distance of the effective depth values ​​in the current frame's hand depth map in real time. Based on the minimum depth distance, it queries a preset distance resolution mapping table to obtain the target physical size corresponding to the current minimum depth distance. The obtained target physical size is then used to divide the current frame's three-dimensional voxel mesh. The depth distance recorded in the distance resolution mapping table is positively correlated with the target physical size. When the minimum depth distance decreases, the target physical size of the three-dimensional voxel mesh decreases accordingly; when the minimum depth distance increases, the target physical size of the three-dimensional voxel mesh increases accordingly. To clearly define the mapping relationship between depth distance and voxel mesh size, this embodiment clarifies the mapping logic using the following table: Table 2 Mapping Table of Depth Distance and 3D Voxel Mesh Physical Dimensions ; In Table 2, the minimum depth distance in the first distance interval is less than the minimum depth distance in the second distance interval, the value of the first size parameter is less than the value of the second size parameter, the value of the second size parameter is less than the value of the third size parameter, and the value of the third size parameter is less than the value of the fourth size parameter. The multimodal fusion processor dynamically divides the 3D voxel mesh according to the mapping relationship described in Table 2. When the operator's hand is close to the depth camera, the spatial resolution is improved by reducing the physical size of the voxels; when the operator's hand is far from the depth camera, the spatial coverage is expanded by increasing the physical size of the voxels, thus balancing the spatial resolution and computational load at different interaction distances.

[0041] Furthermore, before determining that an occlusion event has occurred, the multimodal fusion processor performs a double condition check operation. Specifically, after calculating the area variation coefficient of the hand foreground mask in the current frame, the multimodal fusion processor simultaneously extracts the minimum bounding rectangle of the hand foreground mask on the image plane, calculates the coordinates of the center point of the minimum bounding rectangle, compares the center point coordinates of the current frame with the center point coordinates of the previous frame, and calculates the Euclidean displacement distance between the two center point coordinates.

[0042] The multimodal fusion processor compares the calculated displacement distance with a preset boundary walk threshold. Only when the displacement distance is less than the preset boundary walk threshold and the area variation coefficient is lower than the occlusion determination threshold is an occlusion event confirmed and the radar connectivity extraction process triggered. The dual-condition verification operation eliminates interference from non-occlusion scenarios such as the operator's hand quickly moving out of the acquisition field of view, thus avoiding erroneous mode switching in non-occlusion states.

[0043] Furthermore, the multimodal fusion processor adjusts the occlusion determination threshold dynamically. Specifically, the multimodal fusion processor maintains a sliding window queue of fixed length. During the initial operation phase when no occlusion event occurs, the area variation coefficients calculated from multiple consecutive frames are sequentially stored in the sliding window queue. When the sliding window queue is full, the multimodal fusion processor calculates the arithmetic mean and standard deviation of all area variation coefficients in the sliding window queue. The arithmetic mean is then subtracted from the standard deviation by a preset multiple to obtain the updated occlusion determination threshold.

[0044] During subsequent system operation, the sliding window queue continuously updates the stored area variation coefficients according to the first-in, first-out principle. Every preset update cycle, the multimodal fusion processor recalculates the arithmetic mean and standard deviation of all area variation coefficients in the sliding window queue, updating the occlusion detection threshold accordingly. This dynamically updated occlusion detection threshold adapts to different operator hand sizes, varying interaction distances, and different acquisition environments, reducing the probability of false occlusion detection caused by fixed thresholds.

[0045] In this embodiment, the system achieves online verification of spatial registration accuracy and update of extrinsic parameters through the statistics of multimodal overlapping voxels, balances spatial resolution and computational load through dynamically adjusted three-dimensional voxel mesh, improves the accuracy of occlusion determination through dual condition verification, and adapts to different application scenarios and operator differences through dynamically updated occlusion determination threshold.

[0046] Please refer to the attached document. Figure 5 In a preferred embodiment, when extracting connected components with concentrated reflection intensity in a 3D voxel mesh under occlusion conditions, the multimodal fusion processor employs a 3D region growing algorithm to achieve complete extraction of connected components. After determining that an occlusion event has occurred, the multimodal fusion processor locks the 3D voxel mesh region corresponding to the hand foreground mask in the previous frame as the search starting point. This region is the smallest spatial bounding box of all visual foreground voxel units corresponding to the hand foreground mask in the previous frame. Using a preset lower limit of reflection intensity as the seed point selection condition, the multimodal fusion processor traverses all 3D voxel units within the spatial region corresponding to the search starting point, marking 3D voxel units with reflection intensity values ​​greater than the lower limit of reflection intensity as seed points and adding them to the region growing queue.

[0047] The multimodal fusion processor employs a 3D region growing algorithm to expand the 3D voxel mesh. It traverses the 6-neighborhood 3D voxel cells of each seed point in the region growing queue. If the reflection intensity value of the current neighboring 3D voxel cell to be expanded is greater than the lower limit of reflection intensity, then the neighboring 3D voxel cell is assigned to the same connected component and added to the region growing queue. If the reflection intensity value of the current neighboring 3D voxel cell to be expanded is lower than the lower limit of reflection intensity, then the reflection intensity gradient between the 3D voxel cell to be expanded and the corresponding seed point is calculated. The calculation process of the reflection intensity gradient satisfies the following formula:

[0048] in, The reflection intensity gradient, This represents the reflection intensity value at the corresponding seed point. The reflection intensity value of the three-dimensional voxel unit in the neighborhood to be expanded. The spatial Euclidean distance between the seed point and the three-dimensional voxel unit in the neighborhood to be expanded is given.

[0049] The multimodal fusion processor compares the calculated reflection intensity gradient with a preset edge smoothing threshold. When the reflection intensity gradient is less than the preset edge smoothing threshold, the 3D voxel unit to be expanded is incorporated into the connected component as a transition edge voxel and added to the region growing queue. After the 3D region growing algorithm is completed, the multimodal fusion processor obtains the complete target connected component, which includes the high reflection intensity region of the hand body and the low reflection intensity transition region of the hand edge, improving the completeness of the connected component extraction.

[0050] Furthermore, before calculating the centroid coordinates of the connected components, the multimodal fusion processor performs a spatial constraint filtering operation. The multimodal fusion processor extracts the centroid positions of the hand foreground mask for a preset number of consecutive frames before the occlusion event occurs, obtaining a sequence of three-dimensional coordinates of the hand centroids for multiple consecutive frames. A linear extrapolation algorithm is used to fit the trajectory of the three-dimensional coordinate sequence of the centroids to predict the predicted three-dimensional region of the hand in the current frame under the occlusion state. This predicted three-dimensional region of the hand is a spatial cube of a preset size centered on the predicted centroid position.

[0051] The multimodal fusion processor uses the predicted 3D region of the hand as a spatial mask and performs spatial intersection operations with the previously extracted target connected components. It then removes 3D voxel units located outside the predicted 3D region of the hand, retaining only the remaining 3D voxel units located inside the predicted 3D region of the hand. This spatial constraint filtering operation eliminates invalid voxel units corresponding to environmental clutter and suppresses interference from environmental reflection signals during occlusion on connected component extraction.

[0052] Furthermore, the multimodal fusion processor calculates the centroid coordinates and velocity vectors of the target's connected components. Specifically, the multimodal fusion processor takes the spatial coordinates of the remaining three-dimensional voxel units after spatial constraint filtering, and performs a weighted average based on the corresponding reflection intensity values ​​to obtain the weighted centroid coordinates of the current frame. The calculation process for the weighted centroid coordinates satisfies the following formula:

[0053] in, The weighted centroid coordinate vector of the current frame. This represents the total number of remaining 3D voxel units after spatial constraint filtering. For the first The reflection intensity value of the remaining three-dimensional voxel units. For the first The spatial three-dimensional coordinate vector of the remaining three-dimensional voxel units.

[0054] The multimodal fusion processor incorporates a Kalman filter. It uses the weighted centroid coordinates of the current frame as the observation input to the Kalman filter, and combines the predicted centroid coordinates and prediction error covariance from the previous frame's output to calculate the smoothed centroid coordinates of the current frame. The Kalman filter's state vector is a 6-dimensional vector containing three-dimensional spatial coordinates and three-dimensional velocity components. The state transition matrix is ​​constructed based on a uniform motion model, and the observation matrix corresponds to the observation input of the three-dimensional centroid coordinates.

[0055] The Kalman filter first performs a prediction step, calculating the predicted state vector and prediction error covariance matrix of the current frame using the state transition matrix. Then, it performs an update step, calculating the Kalman gain based on the observed input values, updating the state vector and error covariance matrix, and outputting the smoothed centroid coordinates of the current frame. The multimodal fusion processor extracts a sequence of smoothed centroid coordinates from multiple consecutive frames and calculates the smoothed velocity vector of the current frame using the ratio of the difference in smoothed centroid coordinates between adjacent frames to the inter-frame time interval. The smoothing process of the Kalman filter reduces centroid coordinate jitter caused by random noise in the radar point cloud, improving the stability of velocity vector calculation.

[0056] Furthermore, the multimodal fusion processor maps the smoothed velocity vector into rotation control signals and displacement step control signals. The multimodal fusion processor inputs the direction component of the smoothed velocity vector into a preset dead-zone nonlinear mapping function to generate the rotation control signal from the game's perspective. The calculation process of the dead-zone nonlinear mapping function satisfies the following formula:

[0057] in, For rotation control signal, The direction component of the smooth velocity vector, The preset dead zone threshold, This is a preset nonlinear scaling factor. It is a symbolic function.

[0058] When the magnitude of the directional component is within the dead zone, the rotation control signal output is set to zero. When the magnitude of the directional component exceeds the dead zone, the rotation control signal is amplified according to a nonlinear scaling factor. Dead zone nonlinear mapping avoids unexpected shaking of the game view caused by radar point cloud noise when the operator's hand is stationary.

[0059] Furthermore, the multimodal fusion processor maps the magnitude component of the smoothed velocity vector to a displacement step control signal for the virtual character in the game scene. The multimodal fusion processor obtains the motion state identifier of the virtual character currently rendered by the game console through an inter-process communication interface. The motion state identifier includes stationary, running, and jumping states. Based on the obtained motion state identifier, the multimodal fusion processor retrieves the corresponding target mapping coefficient from a preset mapping coefficient set, multiplies the magnitude component of the smoothed velocity vector by the target mapping coefficient, and generates the displacement step control signal. To clearly define the correspondence between motion states and mapping coefficients, this embodiment clarifies the mapping logic using the following table: Table 3. Correspondence between Virtual Character Motion State and Mapping Coefficient ; In Table 3, the value of the first mapping coefficient is greater than the value of the second mapping coefficient, and the value of the second mapping coefficient is greater than the value of the third mapping coefficient. The multimodal fusion processor generates the displacement step size control signal according to the mapping relationship described in Table 3, ensuring that the output of the control signal adapts to the physical motion logic of the virtual character within the game scene, thus reducing the probability of erroneous operations under different motion states.

[0060] Furthermore, when the multimodal fusion processor detects that the area variation coefficient has recovered to above the occlusion determination threshold, it performs modal switching smoothing processing. The multimodal fusion processor records the hand joint coordinates of the first frame extracted by the depth camera at the switching moment, calculates the spatial deviation vector between the hand joint coordinates of the first frame and the smoothed centroid coordinates of the last frame before the switching moment, and in subsequent consecutive frame processing, adds the spatial deviation vector to the control signal generated by the hand joint extraction path according to an exponential decay law until the magnitude of the spatial deviation vector decays to zero. The exponential decay process of the spatial deviation vector satisfies the following formula:

[0061] in, After switching The spatial deviation vector at time, Let be the initial spatial deviation vector at the switching moment. The preset attenuation coefficient, This is the runtime calculated from the mode switching moment.

[0062] The exponential decay smoothing process avoids the step jump between the control signals generated by the two control paths during mode switching, thus maintaining the continuity of the control signal output.

[0063] In this embodiment, the system improves the completeness of connected component extraction through a three-dimensional region growing algorithm, suppresses environmental clutter interference through spatial constraint screening, enhances the stability of centroid coordinate and velocity vector calculation through a Kalman filter, avoids viewpoint jitter in a static state through dead zone nonlinear mapping, optimizes the adaptability of displacement control through mapping logic adapted to the virtual character's motion state, and avoids abrupt changes in control signals during mode switching through exponential decay smoothing processing.

[0064] Please refer to the attached document. Figure 6In a preferred embodiment, the multimodal fusion processor, under unobstructed conditions, simultaneously uses radar point cloud data to correct the joint coordinates extracted from the hand depth map. Specifically, the multimodal fusion processor compares the spatial region corresponding to the hand joint coordinates with the effective reflection region of the radar point cloud, calculates the deviation between the joint coordinates and the centroid of the radar point cloud, and adds the deviation to the joint coordinates according to a preset weighting coefficient, thus completing the joint coordinate correction and reducing the joint extraction error caused by illumination changes during optical acquisition.

Claims

1. A game motion-sensing operation control system based on gesture interaction, characterized in that, include: Depth cameras, millimeter-wave radar, multimodal fusion processors, and game consoles; A depth camera captures hand depth images in the interactive space, while a millimeter-wave radar captures radar point cloud data of the interactive space. The multimodal fusion processor uses a pre-calibrated camera extrinsic matrix to project the radar point cloud from the radar coordinate system to the camera coordinate system where the depth map is located. It then constructs a spatial three-dimensional voxel mesh in the camera coordinate system and fills the hand foreground mask of the hand depth map and the reflection intensity value of the radar point cloud into the corresponding three-dimensional voxels. The multimodal fusion processor calculates the area variation coefficient of the hand foreground mask in the current frame in real time. When the area variation coefficient is lower than the preset occlusion determination threshold, an occlusion event is determined to have occurred. In the occlusion state, the multimodal fusion processor extracts the connected domains where the reflection intensity is concentrated in the three-dimensional voxel, calculates the centroid coordinates and velocity vector of the connected domain, maps the direction component of the velocity vector to the rotation control signal of the game view, maps the magnitude component of the velocity vector to the displacement step control signal of the virtual character in the game scene, and sends the rotation control signal and displacement step control signal to the game host. When the area variation coefficient recovers to above the occlusion determination threshold, the multimodal fusion processor switches to the hand joint extraction path based on the hand depth map to generate control signals and send them to the game console.

2. The game motion-sensing operation control system based on gesture interaction according to claim 1, characterized in that, The process of constructing a spatial three-dimensional voxel grid and filling data by the multimodal fusion processor includes: taking the origin of the camera coordinate system as the center, dividing the three-dimensional voxel grid according to a fixed physical size; for each frame of hand depth map captured by the depth camera, using a depth threshold to remove background noise to generate a hand foreground mask; projecting the pixels of the depth value in the hand foreground mask to the corresponding three-dimensional voxel grid position and marking them as visual occupancy states. For radar point cloud data collected by millimeter-wave radar, the reflection intensity value carried by each radar point after projection transformation is accumulated to the corresponding three-dimensional voxel grid position. When the same three-dimensional voxel grid has both a visual occupancy marker and a reflection intensity value, the three-dimensional voxel grid is recorded as a multimodal overlapping voxel. The multimodal fusion processor counts the total number of multimodal overlapping voxels in each frame. If the total number of multimodal overlapping voxels is lower than a preset registration verification threshold, the online recalibration process of the camera extrinsic matrix is ​​triggered.

3. The game motion-sensing operation control system based on gesture interaction according to claim 1, characterized in that, The multimodal fusion processor calculates the area variation coefficient of the hand foreground mask as follows: the hand foreground mask of the current frame is divided into multiple non-overlapping local regions on the image plane, the pixel area of ​​the hand foreground mask in each local region is counted, and the ratio of the variance to the mean of the pixel area of ​​all local regions is calculated as the area variation coefficient. Before determining that an occlusion event has occurred, the multimodal fusion processor also extracts the bounding rectangle of the hand foreground mask on the image plane, calculates the displacement distance of the center point coordinates of the bounding rectangle relative to the center point coordinates of the bounding rectangle in the previous frame, and only when the displacement distance is less than the preset boundary walk threshold and the area variation coefficient is lower than the occlusion determination threshold, does it finally confirm that an occlusion event has occurred and trigger the radar connected domain extraction process.

4. The game motion-sensing operation control system based on gesture interaction according to claim 1, characterized in that, The process of the multimodal fusion processor extracting the connected components with concentrated reflection intensity in the three-dimensional voxels includes: after determining that an occlusion event has occurred, locking the three-dimensional voxel grid region corresponding to the hand foreground mask in the previous frame as the search starting point, using the preset lower limit of reflection intensity as the seed point screening condition, and using a three-dimensional region growing algorithm to expand in the three-dimensional voxel grid, and grouping the three-dimensional voxel grids with reflection intensity values ​​greater than the lower limit of reflection intensity and spatially adjacent into the same connected component. During the expansion process of the region growing algorithm, if the reflection intensity value of the current three-dimensional voxel mesh to be expanded is lower than the lower limit of reflection intensity, the reflection intensity gradient between the three-dimensional voxel mesh to be expanded and the seed point is calculated. When the reflection intensity gradient is less than the preset edge smoothing threshold, the three-dimensional voxel mesh to be expanded is incorporated into the connected region as a transition edge voxel.

5. The game motion-sensing operation control system based on gesture interaction according to claim 4, characterized in that, The process by which the multimodal fusion processor calculates the centroid coordinates and velocity vectors of the connected domain includes: weighting the spatial coordinates of all three-dimensional voxel meshes within the connected domain according to their corresponding reflection intensity values ​​to obtain the weighted centroid coordinates of the current frame; The multimodal fusion processor has a built-in Kalman filter. It uses the weighted centroid coordinates of the current frame as the observation input value of the Kalman filter, and calculates the smoothed centroid coordinates of the current frame by combining the predicted centroid coordinates and prediction error covariance of the Kalman filter output of the previous frame. The multimodal fusion processor extracts a smooth centroid coordinate sequence from multiple consecutive frames and calculates the smooth velocity vector of the current frame by using the ratio of the difference in smooth centroid coordinates between adjacent frames to the inter-frame time interval.

6. The game motion-sensing operation control system based on gesture interaction according to claim 5, characterized in that, The multimodal fusion processor maps the velocity vector into a rotation control signal and a displacement step control signal in the following ways: inputting the direction component of the smooth velocity vector into a preset dead-zone nonlinear mapping function; setting the rotation control signal output to zero when the magnitude of the direction component is within the dead-zone range; and amplifying the rotation control signal according to a nonlinear scaling factor when the magnitude of the direction component exceeds the dead-zone range. When the multimodal fusion processor detects that the area variation coefficient has recovered to above the occlusion determination threshold, it records the coordinates of the hand joint points extracted by the depth camera at the switching moment, calculates the spatial deviation vector between the coordinates of the hand joint points in the first frame and the smooth centroid coordinates of the last frame before the switching moment, and superimposes the spatial deviation vector into the control signal generated by the hand joint extraction path in subsequent consecutive frames according to the exponential decay law until the magnitude of the spatial deviation vector decays to 0.

7. The game motion-sensing operation control system based on gesture interaction according to claim 2, characterized in that, The method of dividing the three-dimensional voxel mesh according to a fixed physical size is as follows: the multimodal fusion processor extracts the minimum depth distance of the depth value in the hand depth map in real time, and queries a preset distance resolution mapping table according to the minimum depth distance to obtain the target physical size corresponding to the current minimum depth distance; The multimodal fusion processor dynamically divides the current frame into a three-dimensional voxel mesh based on the target physical size, where the depth distance recorded in the distance resolution mapping table is positively correlated with the target physical size; When the minimum depth distance decreases, the target physical size of the 3D voxel mesh decreases accordingly to improve spatial resolution; when the minimum depth distance increases, the target physical size of the 3D voxel mesh increases accordingly to expand the spatial coverage.

8. The game motion-sensing operation control system based on gesture interaction according to claim 3, characterized in that, The occlusion determination threshold is determined as follows: the multimodal fusion processor maintains a sliding window queue of fixed length. In the initial running phase when no occlusion event occurs, the area variation coefficients calculated from multiple consecutive frames are sequentially stored into the sliding window queue. When the sliding window queue is full, calculate the arithmetic mean and standard deviation of all area variation coefficients in the sliding window queue, and use the arithmetic mean minus the standard deviation by a preset multiple as the updated occlusion judgment threshold. During subsequent operation, the sliding window queue continuously updates the area variation coefficient according to the first-in-first-out principle, and the multimodal fusion processor recalculates and updates the occlusion determination threshold every preset update cycle.

9. The game motion-sensing operation control system based on gesture interaction according to claim 5, characterized in that, Before calculating the weighted centroid coordinates of the current frame, the multimodal fusion processor also performs a spatial constraint filtering process: the multimodal fusion processor extracts the centroid positions of the hand foreground mask of a preset number of consecutive frames before the occlusion event occurs, and uses a linear extrapolation algorithm to predict the predicted three-dimensional region of the hand in the current frame under the occlusion state. The multimodal fusion processor uses the predicted 3D region of the hand as a spatial mask to remove the 3D voxel mesh located outside the predicted 3D region of the hand in the connected domain, and retain only the remaining 3D voxel mesh located inside the predicted 3D region of the hand. The multimodal fusion processor recalculates the weighted average using the spatial coordinates of the remaining three-dimensional voxel mesh and the corresponding reflection intensity values ​​to obtain the constrained weighted centroid coordinates.

10. The game motion-sensing operation control system based on gesture interaction according to claim 6, characterized in that, The process by which the multimodal fusion processor maps the magnitude component of the velocity vector to the displacement step control signal of the virtual character in the game scene includes: the multimodal fusion processor obtains the motion state identifier of the virtual character currently rendered by the game host through the inter-process communication interface, and the motion state identifier includes the stationary state, the running state, and the jumping state. The multimodal fusion processor retrieves the corresponding target mapping coefficient from the preset mapping coefficient set according to the motion state identifier, multiplies the magnitude component of the velocity vector by the target mapping coefficient to generate a displacement step control signal, wherein the first mapping coefficient corresponding to the stationary state is greater than the second mapping coefficient corresponding to the running state, and the second mapping coefficient corresponding to the running state is greater than the third mapping coefficient corresponding to the jumping state.

Images