Multi-user holographic sand table collaborative resolving system and method based on 360-degree omni-directional perception gesture recognition

Abstract

The invention relates to a multi-user holographic sand table collaborative resolving system and method based on 360-degree omni-directional perception gesture recognition. The method comprises the following steps: acquiring continuous multi-frame multi-view hand image data; hand instance separation and key point extraction are carried out on the multi-view hand image data of each frame, and spatial 3D coordinates of 21 key points are independently solved for each hand instance; based on the space 3D coordinate sequence of the continuous frames, space-time action features are extracted, and a gesture template library is matched to recognize gesture types and motion parameters; analyzing the gesture type, matching an interaction intention in combination with a predefined gesture-instruction mapping table, and generating an interaction control instruction according to a matching result; generating a three-dimensional holographic sand table image cooperatively operated by multiple persons in real time according to the interaction control instruction; through a multi-hand parallel processing technology, different hand instance feature extraction paths are separated, the problem of multi-hand ID jump is solved, occlusion area coordinate prediction is optimized, and interactive use experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and human-computer interaction technology, and in particular to a multi-user holographic sandbox collaborative solution system and method based on 360-degree omnidirectional perception gesture recognition. Background Art

[0002] When building a three-dimensional holographic interactive sandbox, in addition to the holographic 3D stereoscopic imaging display of the image screen, human-computer interaction design is one of the most crucial links, which directly affects the user's acceptance and efficiency of the system.

[0003] Currently, existing spatial interaction solutions in the field of three-dimensional holographic interactive sandboxes mostly use fixed-viewing angle projection and physical button control, and the real-time interaction experience of natural gestures in the air is insufficient.

[0004] Existing hand interaction technologies primarily focus on single-hand or double-hand detection using monocular or binocular cameras, and generally suffer from the following drawbacks: They lack depth control and complete 3D displacement control; they also lack real-time multi-hand interaction capabilities; and traditional algorithms employ serial processing, significantly increasing latency in multi-user scenarios. These drawbacks contribute to a poor user experience for existing holographic interactive sandboxes. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings and provide a multi-user holographic sandbox collaborative solution system and method based on 360-degree omnidirectional perception gesture recognition.

[0006] To achieve the above object, the specific solutions of the present invention are as follows:

[0007] On one hand, the present invention provides a multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition, comprising the following steps:

[0008] Multi-view synchronous acquisition: obtain multi-view hand image data of multiple frames continuously;

[0009] Multi-hand instance separation and spatial coordinate calculation: Hand instance separation and key point extraction are performed on multi-view hand image data for each frame, and the spatial 3D coordinates of 21 key points are independently calculated for each hand instance. This solution combines the rotation and translation transformation with the wrist decentralization process into a single transformation model, combines it with the camera projection constraint equation, and uses least squares optimization to solve the spatial 3D coordinates.

[0010] Dynamic gesture recognition: Based on the spatial 3D coordinate sequence of continuous frames, it extracts spatiotemporal motion features and matches the gesture template library to identify gesture types and motion parameters; it also analyzes the gesture type to obtain a gesture description;

[0011] Multi-user command generation: Based on gesture descriptions and motion parameters, the predefined gesture-command mapping table is used to match interaction intentions and generate interactive control commands based on the matching results.

[0012] Holographic collaborative rendering: Generate three-dimensional holographic sandbox images for collaborative operation by multiple people in real time based on interactive control instructions.

[0013] The present invention further includes the following steps:

[0014] Build the hand key point transformation model:

[0015] WorldLandmarks[i]=R×SpaceLandmarks[i]+t;

[0016] Where i represents the i-th key point, R and t are rotation and translation matrices, SpaceLandmarks is the spatial 3D coordinate of the hand key point to be found, and WorldLandmarks is the original 3D coordinate of the hand key point;

[0017] Joint camera projection equation:

[0018] Landmarks[i]=Intrinsic×g(SpaceLandmarks[i]);

[0019] Among them, the g(x) function represents the distortion transformation applied to x, and Intrinsic is the intrinsic parameter matrix of the camera;

[0020] Using the 2D / 3D coordinate data of ≥3 key points, the rotation matrix R, translation vector t and spatial coordinates SpaceLandmarks[i] are solved through least squares optimization.

[0021] The present invention further adopts a multi-branch neural network model for the multi-hand instance separation, and each neural network model independently processes the key point detection of a single-hand instance.

[0022] The present invention further includes the step of acquiring multiple frames of multi-view hand image data.

[0023] The circularly arranged dynamic capture camera array completes the acquisition of multi-view images within the same time window, and the collected multi-view image data is time-synchronized. The circularly arranged dynamic capture camera array covers a horizontal 360° and vertical ±80° field of view.

[0024] The present invention further adopts a global timestamp synchronization mechanism to perform frame-level alignment on images of each perspective, thereby obtaining a multi-perspective image set with consistent time.

[0025] The present invention further performs hand instance separation and key point extraction on the multi-view image data of each frame, and independently solves the spatial 3D coordinates of 21 key points for each hand instance, specifically including: after obtaining the spatial 3D coordinates, using the Kalman filter algorithm to correct the 3D spatial coordinates of the hand instance; during the correction process, fusing historical trajectory information and predicted values ​​to compensate for missing or abnormal points.

[0026] Furthermore, the present invention provides that the gesture types include static gestures, dynamic gestures and combined gestures.

[0027] The present invention further includes parsing the gesture type to obtain the gesture description, specifically including:

[0028] For static gestures, key points are extracted directly based on single-frame images and classified and recognized;

[0029] For dynamic gestures, a spatiotemporal motion encoding mechanism is introduced into the feature extraction path. By modeling the motion information between consecutive frames, the temporal evolution characteristics of gestures are captured, thereby more accurately analyzing dynamic behaviors.

[0030] For combined gestures, the gesture semantic rule library and multi-target tracking results are combined to analyze the relative position and movement relationship between each hand to achieve the recognition and analysis of compound gestures.

[0031] The present invention further includes performing hand instance separation and key point extraction on the multi-view image data of each frame, and also assigning an independent hand ID to each hand of each user based on multi-target tracking; maintaining hand ID consistency in consecutive frames through motion trajectory prediction and feature matching to avoid hand ID jumps due to occlusion or posture changes.

[0032] Furthermore, the present invention adopts a time priority method when multiple users issue conflicting instructions to a gesture-affected object, and sets the first user as a temporary master user. The operation of the temporary master user has a high priority.

[0033] A second aspect of the present invention provides a multi-user holographic sandbox collaborative solution system based on 360-degree omnidirectional perception gesture recognition, comprising:

[0034] An image acquisition module, used to acquire multiple frames of multi-view hand image data;

[0035] A gesture separation and solution module is used to separate hand instances and extract key points from multi-view hand image data for each frame, independently solving the spatial 3D coordinates of 21 key points for each hand instance. This solution combines the rotation and translation transformation with the wrist decentralization process into a single transformation model, combines it with the camera projection constraint equation, and uses least squares optimization to solve the spatial 3D coordinates.

[0036] The gesture recognition module is used to extract spatiotemporal motion features based on the spatial 3D coordinate sequence of continuous frames, match the gesture template library to identify the gesture type and motion parameters; and parse the gesture type to obtain the gesture description;

[0037] The command generation module is used to match the interaction intention based on the gesture description and motion parameters in combination with the predefined gesture-command mapping table, and generate the interaction control command based on the matching result;

[0038] The holographic collaborative rendering module is used to generate three-dimensional holographic sandbox images for multi-person collaborative operation in real time according to interactive control instructions.

[0039] The beneficial effects of the present invention are as follows: the present invention overcomes the limitation of the prior art that users can only perform gesture recognition in a small range above the camera, realizes the 360° surround walking gesture interaction required for surrounding the three-dimensional holographic interactive sandbox table, greatly expanding the user's activity space and interaction range; realizes real-time interaction of natural gestures in the air with multiple hands (≥4 hands), separates the feature extraction paths of different hand instances through multi-hand parallel processing technology, solves the problem of multi-hand ID jumping, optimizes the coordinate prediction of the occluded area, effectively improves the accuracy and fluency of multi-hand real-time interaction, and enhances the interactive usage experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of a multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition of the present invention;

[0041] Figure 2 This is a schematic diagram of the application of the multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition of the present invention; DETAILED DESCRIPTION

[0042] like Figure 1 In this embodiment, a multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition may specifically include the following steps:

[0043] Step S100, multi-view synchronous acquisition: acquiring multiple frames of multi-view hand image data.

[0044] Specifically, a circular array of motion capture cameras captures multi-view images within the same time window and performs time-synchronized processing on the captured multi-view image data. The circular array covers a 360° horizontal field of view and a ±80° vertical field of view. Specifically, the circular layout consists of several groups of four-quadrant motion capture cameras. Each group of motion capture cameras utilizes a precision tilted installation based on a field of view coverage optimization strategy, achieving omnidirectional coverage for gesture recognition across an ultra-wide 360° horizontal field of view and ±80° vertical field of view.

[0045] Based on field of view overlap compensation, image information from adjacent camera fields of view is fused to eliminate data collection blind spots. Specifically, based on matching the overlapping areas of images between adjacent cameras, key point features in the common field of view (such as the length of adjacent key points, palm contours, etc.) are extracted; image data from different perspectives is aligned using feature matching; edges of non-overlapping areas are extended and gaps are filled to form a complete and continuous spatial perception map. This enables gesture recognition without blind spots, improves the robustness of key point detection in occluded scenarios, and forms a complete spatial perception area. The system design is capable of capturing and recognizing user gestures within a 1.5m radius around the sandbox with high precision.

[0046] This eliminates the need for users to wear markers or sensors, and a 360° multi-angle gesture recognition and dynamic capture camera ring array is installed to ensure coverage without blind spots, enabling all-round data collection of multiple users' gestures.

[0047] To ensure the consistency of images from different perspectives in the temporal dimension, the system adopts a global timestamp synchronization mechanism and achieves frame-level alignment through a network synchronization protocol to obtain a temporally consistent set of multi-perspective images, eliminating timing deviations caused by device response differences or transmission delays.

[0048] Step S200: Multiple hand instance separation and spatial coordinate solution: Hand instance separation and key point extraction are performed on the multi-view hand image data of each frame, and the spatial 3D coordinates of 21 key points are independently solved for each hand instance; the solution is achieved by combining the rotation and translation transformation with the wrist decentering process into a single transformation model, combining the camera projection constraint equation, and using least squares optimization to solve the spatial 3D coordinates.

[0049] Specifically, the system independently detects and extracts key points from multiple hand instances in each frame, and then calculates their 3D spatial coordinates. Based on the input multi-view hand image data, the system independently identifies and extracts features from each hand instance, ultimately outputting the 3D coordinates of the 21 key points corresponding to each hand instance.

[0050] The spatial coordinate calculation process of the 21 key points corresponding to each hand instance includes the following steps:

[0051] Step S201, analyze the generation process of the original hand key point 3D coordinates: Reverse analysis shows that the original hand key point 3D coordinates are usually obtained by rotating and translating the hand key point space 3D coordinates to obtain the intermediate transition 3D coordinates (TransLandmarks), and then subtracting the WRIST-numbered transition 3D coordinates from all the transition 3D coordinates.

[0052] The original hand keypoint space coordinates (SpaceLandmarks) need to undergo two steps of transformation:

[0053] Rotation and translation transformation: transform to intermediate transition 3D coordinates (TransLandmarks) through matrix R and translation vector t;

[0054] Wrist decentralization: Taking the wrist point (WRIST) as the origin, all coordinates are subtracted from TransLandmarks[0] to obtain the decentralized coordinates (WorldLandmarks).

[0055] The formula is as follows:

[0056] TransLandmarks[i]=R×SpaceLandmarks[i]+t

[0057] WorldLandmarks[i]=TransLandmarks[i]-TransLandmarks[0]

[0058] That is:

[0059] WorldLandmarks[i]=(R×SpaceLandmarks[i]+t)-(R×SpaceLandmarks[0]+t)

[0060] Among them, i represents the i-th key point, R and t are rotation and translation matrices, SpaceLandmarks is the spatial 3D coordinate of the hand key point to be found, and WorldLandmarks is the original 3D coordinate of the hand key point.

[0061] Step S202, simplifying the transformation process: without loss of generality, the "rotation and translation transformation" process and the "transition 3D coordinates minus the WRIST number" process can be merged into a "new rotation and translation transformation" process, because a new R and t can always be found to realize the combination of the two processes.

[0062] Therefore, the hand key point transformation model is obtained:

[0063] WorldLandmarks[i]=R×SpaceLandmarks[i]+t

[0064] Where i represents the i-th keypoint, R and t are the desired rotation and translation matrices, and SpaceLandmarks are the 3D coordinates of the desired hand keypoint. R and t are shared across all keypoints in each frame because the transformation is performed uniformly across all keypoints in the current frame, without involving transformations between different frames.

[0065] If the R and t rotation and translation matrices are obtained, the calculation formula for SpaceLandmarks is:

[0066] SpaceLandmarks[i] = R-1 * (WorldLandmarks[i] - t) (1)

[0067] Step S203: introduce the camera projection constraint transformation relationship: the 3D coordinates of the hand key points and the 2D coordinates of the hand key points satisfy the camera projection transformation relationship. Therefore, the joint camera projection equation is:

[0068] Landmarks[i] = Intrinsic × g(SpaceLandmarks[i]) (2)

[0069] Here, g(x) is the distortion transformation applied to x, and Intrinsic is the camera's intrinsic parameter matrix. The camera's intrinsic parameters and distortion parameters are pre-calibrated.

[0070] Step S204, solving the rotation and translation matrix and the spatial 3D coordinates: Combining formulas (1) and (2), it can be seen that at least the 2D coordinates and 3D coordinates of three key points of the hand are needed to solve the rotation and translation matrix R, t, and the corresponding spatial 3D coordinates.

[0071] Using the 2D / 3D coordinate data of ≥3 key points, the rotation matrix R, translation vector t and spatial coordinates SpaceLandmarks[i] are solved through least squares optimization.

[0072] In the least squares optimization, the objective function is:

[0073]

[0074] Among them, N≥3 and N≤21.

[0075] In this embodiment, the least squares optimization step includes the following steps:

[0076] Step A1, initializing the rotation and translation matrices R and t;

[0077] Step A2: Calculate the spatial 3D coordinates of the hand key points using formula (1) based on the current R and t;

[0078] Step A3, according to formula (2), combined with the camera intrinsic parameter matrix and distortion parameters, calculate the projected 2D coordinates of the hand key points;

[0079] Step A4, calculating the error between the projected 2D coordinates and the actually acquired 2D coordinates;

[0080] Step A5: Update the rotation and translation matrices R and t according to the error;

[0081] Step A6: Repeat steps A2 to A5 until the error meets a preset threshold or reaches a maximum number of iterations.

[0082] Through the above iterative optimization process, we finally obtain the stable rotation and translation matrix R, t, and the corresponding spatial 3D coordinates of the hand key points.

[0083] In this embodiment, for the same frame image, all key points share the same set of R and t, enforcing the rigid body motion consistency of the hand skeleton.

[0084] After obtaining the 3D spatial coordinates, the Kalman filter algorithm is used to correct the 3D spatial coordinates of the hand instance. During this correction process, historical trajectory information is integrated with the predicted values ​​to compensate for missing or abnormal points (occluded areas). This allows for the prediction of spatial 3D coordinates and smoothing of motion trajectories, reducing jitter and improving the robustness of key point positioning. Occluded areas refer to locations where certain key points cannot be directly observed from the current perspective due to being blocked by other hands or objects.

[0085] The specific processing of the Kalman filter algorithm includes:

[0086] Step B1, initialization: For each key point, initialize the state vector (including position and velocity) and covariance matrix.

[0087] Step B2, prediction: using the state prediction equation and covariance prediction equation to predict the state vector and covariance matrix of the current frame.

[0088] Step B3, Update: Calculate the Kalman gain using the keypoint coordinates detected in the current frame, and then update the state estimate and covariance matrix.

[0089] Step B4, output: Use the position in the updated state vector as the final spatial 3D coordinate of the key point in the current frame.

[0090] Furthermore, to solve the problem of multi-hand ID jumping in multi-user scenarios (that is, the same hand is mistakenly identified as different IDs in different frames due to occlusion or posture changes), an independent hand ID is assigned to each hand of each user based on multi-target tracking; the hand ID consistency is maintained through motion trajectory prediction and feature matching in consecutive frames to avoid hand ID jumping caused by occlusion or posture changes, and support at least 4 users to perform gesture interaction simultaneously.

[0091] For example, multiple users gather around the sandbox to edit its contents. For example, user A pinches and selects a part with his right hand and rotates and adjusts his posture with his left hand; user B places parts on the other side of the sandbox at the same time, achieving conflict-free collaborative editing.

[0092] Step S300, dynamic gesture recognition: Based on the spatial 3D coordinate sequence of continuous frames, spatiotemporal motion features (such as the gesture's starting point, end point, motion trajectory (such as linear motion, circular motion, etc.), speed change, etc.) are extracted, and the gesture type and motion parameters are identified by matching the gesture template library; the gesture type is analyzed to obtain a gesture description;

[0093] Specifically, the model architecture utilizes a multi-branch neural network model, each focused on key point detection and pose estimation for a single hand instance, ensuring effective differentiation of gestures between users in multi-person scenarios. A pre-built gesture template library is used to classify collected gestures and determine their type. Based on the gesture type, the gesture is parsed to generate a description of the gesture. Specifically, gesture types include static, dynamic, and combined gestures.

[0094] For example, for static gestures, key points are directly extracted based on a single-frame image and classified and recognized; gesture descriptions include pinching the thumb and index finger, and the index finger pointing vertically to a specific position on the sandbox.

[0095] For dynamic gestures, a spatiotemporal motion encoding mechanism is introduced into the feature extraction path. By modeling the motion information between consecutive frames, the temporal evolution characteristics of the gesture are captured, thereby more accurately analyzing the dynamic behavior. Specifically, a dynamic gesture template library can be constructed, and the motion characteristics of typical gestures (such as sliding trajectories, rotation angles, etc.) can be used as templates to match them with the gesture features detected in real time to improve the recognition accuracy. For example, taking the "rotation gesture" as an example, the rotation angle and rotation speed are calculated by fitting the trajectories of the palm key points in consecutive frames, and then compared with the preset "clockwise rotation" and "counterclockwise rotation" templates to determine whether it is a valid rotation gesture.

[0096] To improve the consistency and accuracy of dynamic gesture recognition, a feedback mechanism is implemented. Specifically, the relative position changes between key points are continuously monitored to determine whether they conform to the preset dynamic behavior characteristics. For example, when performing a grasping action, the distance change trend between the fingertip and palm key points is analyzed in real time as a basis for determining the grasping status. This feedback mechanism is embedded in the key point solution process, using the solution results of one or more previous frames as prior input to improve the consistency and accuracy of dynamic gesture recognition.

[0097] Gesture descriptions include gesture zooming, waving the palm left and right, wrist rotation, and drawing a straight line with one hand.

[0098] For combined gestures, the gesture semantic rule library and multi-target tracking results are combined to analyze the relative positions and movement relationships between the hands to achieve recognition and analysis of complex gestures; gesture descriptions include pinching the gesture first (static), maintaining the pinch state while rotating the wrist (dynamic), and pinching the gesture first (static), maintaining the pinch state while swinging the palm horizontally (dynamic).

[0099] Step S400, multi-user command generation: matching interaction intentions based on gesture descriptions and motion parameters in combination with a predefined gesture-command mapping table, and generating interaction control commands based on the matching results.

[0100] Specifically, when generating interactive control instructions, based on the gesture description (such as sliding, zooming, rotating, clenching fist, etc.) output by S300 and its related features (such as speed, direction, trajectory shape, etc.), combined with the predefined gesture semantic rule library, the corresponding interactive intention is judged, and the corresponding interactive control instruction is generated according to the gesture-instruction mapping table. For example, when the dynamic gesture is recognized as a zoom gesture, an instruction to adjust the size of the object (zoom in / out) is generated. Exemplarily, the predefined gesture-instruction mapping table can be included as follows:

[0101] Gesture-command mapping table

[0102]

[0103] Step S500, holographic collaborative rendering: generating a three-dimensional holographic sandbox image for collaborative operation by multiple people in real time according to interactive control instructions.

[0104] Specifically, in holographic rendering, the rendering effect is updated in real time according to the real-time changes of dynamic gestures, making the interaction more natural and smooth.

[0105] For example, the user makes a "two-finger open" gesture; step S200 calculates the spatial 3D coordinates of the hand instance, analyzes its opening trajectory and speed based on the spatial 3D coordinate sequence of continuous frames, determines it as a "zoom-in" gesture through the semantic rule library, and generates an interactive instruction of "zooming in interface elements". After receiving the instruction, the holographic rendering module enlarges the virtual object proportionally, and the user can see the instant linkage between the gesture operation and the holographic display, forming a closed-loop interactive experience.

[0106] Furthermore, when multiple users issue conflicting commands to an object affected by a gesture, a time-priority approach is adopted, with the first user set as the temporary master user. The temporary master user's operations are given priority, and higher-priority operations preempt lower-priority operations. Operations of the same priority level are sorted by timestamp. This effectively addresses the poor real-time multi-hand interaction capabilities of existing technologies, supporting simultaneous operations by ≥4 users and meeting the requirements for efficient interaction in multi-person collaborative scenarios.

[0107] For example, user C makes a "terrain lifting" gesture (palm moves upward in parallel) around the sandbox, and the Z-axis displacement change of 21 key points of user C's hand is detected; the gesture behavior is analyzed to generate interactive control instructions, and according to the interactive control instructions (Z-axis displacement ≥ 50mm triggers terrain modification), contour line changes are generated in real time (update frame rate 60fps).

[0108] The embodiment of the present invention further provides a multi-user holographic sandbox collaborative solution system based on 360-degree omnidirectional perception gesture recognition, comprising:

[0109] An image acquisition module, used to acquire multiple frames of multi-view hand image data;

[0110] A gesture separation and solution module is used to separate hand instances and extract key points from multi-view hand image data for each frame, independently solving the spatial 3D coordinates of 21 key points for each hand instance. This solution combines the rotation and translation transformation with the wrist decentralization process into a single transformation model, combines it with the camera projection constraint equation, and uses least squares optimization to solve the spatial 3D coordinates.

[0111] The gesture recognition module is used to extract spatiotemporal motion features based on the spatial 3D coordinate sequence of continuous frames, match the gesture template library to identify the gesture type and motion parameters; and parse the gesture type to obtain the gesture description;

[0112] The command generation module is used to match the interaction intention based on the gesture description and motion parameters in combination with the predefined gesture-command mapping table, and generate the interaction control command based on the matching result;

[0113] The holographic collaborative rendering module is used to generate three-dimensional holographic sandbox images for multi-person collaborative operation in real time according to interactive control instructions.

[0114] The above description is only a preferred embodiment of the present invention. Therefore, any equivalent changes or modifications made according to the structure, characteristics and principles described in the scope of the patent application of the present invention are included in the protection scope of the patent application of the present invention.

Claims

1. A multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition, characterized in that: The steps include: Multi-view synchronous acquisition: obtain multi-view hand image data of multiple frames continuously; Multi-hand instance separation and spatial coordinate calculation: Hand instance separation and key point extraction are performed on multi-view hand image data for each frame, and the spatial 3D coordinates of 21 key points are independently calculated for each hand instance. This solution combines the rotation and translation transformation with the wrist decentralization process into a single transformation model, combines it with the camera projection constraint equation, and uses least squares optimization to solve the spatial 3D coordinates. Dynamic gesture recognition: Based on the spatial 3D coordinate sequence of continuous frames, it extracts spatiotemporal motion features and matches the gesture template library to identify gesture types and motion parameters; it also analyzes the gesture type to obtain a gesture description; Multi-user command generation: Based on gesture descriptions and motion parameters, the predefined gesture-command mapping table is used to match interaction intentions and generate interactive control commands based on the matching results. Holographic collaborative rendering: Generate three-dimensional holographic sandbox images for collaborative operation by multiple people in real time based on interactive control instructions.

2. The multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition according to claim 1 is characterized in that: The spatial 3D coordinate solution specifically includes: Build the hand key point transformation model: WorldLandmarks[i]=R×SpaceLandmarks[i]+t; Where i represents the i-th key point, R and t are rotation and translation matrices, SpaceLandmarks is the spatial 3D coordinate of the hand key point to be found, and WorldLandmarks is the original 3D coordinate of the hand key point; Joint camera projection equation: Landmarks[i]=Intrinsic×g(SpaceLandmarks[i]); Among them, the g(x) function represents the distortion transformation applied to x, and Intrinsic is the intrinsic parameter matrix of the camera; Using the 2D / 3D coordinate data of ≥3 key points, the rotation matrix R, translation vector t and spatial coordinates SpaceLandmarks[i] are solved through least squares optimization.

3. The multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition according to claim 1 is characterized in that: The multi-hand instance separation adopts a multi-branch neural network model, and each neural network model independently processes the key point detection of a single-hand instance.

4. The multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition according to claim 1 is characterized in that: The step of acquiring multiple frames of multi-view hand image data specifically includes: The circularly arranged dynamic capture camera array completes the acquisition of multi-view images within the same time window, and the collected multi-view image data is time-synchronized. The circularly arranged dynamic capture camera array covers a horizontal 360° and vertical ±80° field of view.

5. The multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition according to claim 1 is characterized in that: The method performs hand instance separation and key point extraction on the multi-view image data of each frame, and independently solves the spatial 3D coordinates of 21 key points for each hand instance. Specifically, after obtaining the spatial 3D coordinates, the 3D spatial coordinates of the hand instance are corrected using the Kalman filter algorithm; during the correction process, the historical trajectory information and the predicted value are integrated to compensate for missing or abnormal points.

6. The multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition according to claim 1 is characterized in that: The gesture types include static gestures, dynamic gestures and combined gestures.

7. The multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition according to claim 6 is characterized in that: The gesture type is parsed to obtain a gesture description, specifically including: For static gestures, key points are extracted directly based on single-frame images and classified and recognized; For dynamic gestures, a spatiotemporal motion encoding mechanism is introduced into the feature extraction path. By modeling the motion information between consecutive frames, the temporal evolution characteristics of gestures are captured, thereby more accurately analyzing dynamic behaviors. For combined gestures, the gesture semantic rule library and multi-target tracking results are combined to analyze the relative position and movement relationship between each hand to achieve the recognition and analysis of compound gestures.

8. The multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional perception gesture recognition according to claim 6 is characterized in that: The hand instance separation and key point extraction are performed on the multi-view image data of each frame, and an independent hand ID is assigned to each hand of each user based on multi-target tracking; and the hand ID consistency is maintained through motion trajectory prediction and feature matching in consecutive frames.

9. The multi-user holographic sandbox collaborative solution method based on 360-degree omnidirectional gesture recognition according to claim 6 is characterized in that: When multiple users issue conflicting instructions to the gesture-affected object, the time priority method is adopted, and the first user is set as the temporary master user, and the temporary master user's operation has a high priority.

10. A multi-user holographic sandbox collaborative solution system based on 360-degree omnidirectional perception gesture recognition, characterized in that: include: An image acquisition module, used to acquire multiple frames of multi-view hand image data; A gesture separation and solution module is used to separate hand instances and extract key points from multi-view hand image data for each frame, independently solving the spatial 3D coordinates of 21 key points for each hand instance. This solution combines the rotation and translation transformation with the wrist decentralization process into a single transformation model, combines it with the camera projection constraint equation, and uses least squares optimization to solve the spatial 3D coordinates. The gesture recognition module is used to extract spatiotemporal motion features based on the spatial 3D coordinate sequence of continuous frames, match the gesture template library to identify the gesture type and motion parameters; and parse the gesture type to obtain the gesture description; The command generation module is used to match the interaction intention based on the gesture description and motion parameters in combination with the predefined gesture-command mapping table, and generate the interaction control command based on the matching result; The holographic collaborative rendering module is used to generate three-dimensional holographic sandbox images for multi-person collaborative operation in real time according to interactive control instructions.

Images