Screen interaction method and system based on somatosensory actions

Abstract

The invention belongs to the technical field of human-computer interaction, and aims to provide a screen interaction method and system based on somatosensory actions. Activation conditions are judged according to the bone state, then the wrist joint position at the activation moment serves as the circle center of a rudder, and a continuous three-dimensional motion track of the hand of a user is converted into a rotation angle relative to the circle center; when the angle exceeds a preset threshold value, the rudder automatically rotates and highlights the corresponding service module, so that the user can complete module switching through hand rotation action without memorizing a complex instruction; and finally, the system judges a business confirmation condition according to the real-time skeleton state and triggers a confirmation instruction to realize somatosensory interaction from module selection to business confirmation. Complex interaction can be supported, the memory burden of the user can be greatly reduced, the learning cost of the user is low, the computing power consumption is low, and the navigation efficiency is high.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, specifically relating to a screen interaction method and system based on motion sensing. Background Technology

[0002] With the rapid development of information and interactive technologies, large-screen intelligent interactive systems, as an important component of human-computer interaction products, have been widely used in various fields such as control rooms, conference systems, command and dispatch centers, data visualization platforms, exhibitions, virtual simulations, and entertainment experiences. Traditional large-screen interaction methods mainly rely on contact-based devices such as mice, keyboards, and touchscreens. However, these methods have limitations in ultra-large screen application scenarios, including limited operability, less intuitive and natural interaction, and insufficient operational accuracy.

[0003] In recent years, motion-sensing interaction technology has gradually become a research hotspot in large-screen contactless interaction, especially the technology route based on skeletal tracking and gesture recognition, which has shown significant advantages. Currently, various motion-sensing interaction solutions have emerged; however, these solutions are difficult to adapt to complex enterprise-level business scenarios, as detailed below: a. Triggering specific operations through specific gestures: For example, application CN201711428309.0 discloses a motion-sensing interaction method, device, and computer-readable storage medium, which calculates the skeletal information based on preset rules to obtain corresponding calculation results; determines the corresponding motion-sensing interaction action command based on the calculation results, and responds to the motion-sensing interaction action command. Application CN202410587576.6 discloses a motion-sensing interaction method using a motion-sensing interaction system, which pre-collects motion-sensing interaction domain data to construct a motion graph library, and identifies and classifies user actions through analysis and recognition models to identify user actions and postures.

[0004] However, this approach performs specific operations through predefined gestures, relying on a pre-defined static action library. Since enterprise business is relatively complex, there are usually many mapping relationships between the business operations and gestures, resulting in a heavy memory burden and high learning costs for users. In addition, each gesture action is calculated separately with high precision, which requires a large amount of computing resources.

[0005] b. Control via arm mapping: The coordinates of the user's arm end in three-dimensional space are linearly mapped to the two-dimensional coordinates on the screen to form a virtual cursor. Arm mapping is commonly used in games and entertainment scenarios. The system maps the user's arm movements to the screen in real time, and controls the content on the screen by the user moving their arm and the changes in their movements.

[0006] However, this solution is limited by the range of motion of the human arm. Long-term arm raising can easily lead to user fatigue. In addition, the distance between the user and the screen is far, and the user's arm movement range is limited. Furthermore, the content that enterprises need to operate on the ultra-large screen is relatively scattered. In order to expand the mapping range, the gain needs to be increased, which leads to a decrease in control precision and makes it difficult to operate the target module on the screen conveniently and quickly, resulting in low navigation efficiency.

[0007] c. Modular switching: Users can switch between modules using simple actions such as up, down, left, and right to select the target module on the screen.

[0008] However, this approach uses a linear, alternating switching method, which is inflexible. When there are many modules, users cannot quickly reach their target and need to switch multiple times, resulting in low navigation efficiency in complex business scenarios.

[0009] In summary, when the aforementioned existing technologies are applied to complex scenarios such as enterprise data dashboards, they suffer from high user learning costs, high computing power consumption, and low navigation efficiency, making it difficult to support users in quickly and efficiently accessing specific indicator details or functional modules. Summary of the Invention

[0010] The present invention aims to solve the above-mentioned technical problems to at least a certain extent. The present invention provides a screen interaction method, system, electronic device and computer program product based on motion sensing.

[0011] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a screen interaction method based on motion sensing, comprising: The system collects skeletal point data of the target user within the interactive area in real time, and preprocesses the skeletal point data to obtain preprocessed skeletal point data. The skeletal point data includes the three-dimensional coordinate data of multiple specified skeletal points, including shoulder joint, elbow joint, wrist joint and hand nodes. Based on the preprocessed skeletal point data, the system activation condition is detected, and the rudder is activated when the system activation condition is met. The position of the wrist joint at the moment of activation is taken as the center of the rudder, and the rudder interface is displayed on the target screen to visualize the rudder; wherein, the rudder interface includes a central control point and multiple business module icons set around the central control point. Based on the preprocessed skeletal point data at multiple time points, the three-dimensional motion trajectory of the hand node relative to the center of the circle is continuously acquired, and the rotation angle of the hand node relative to the center of the circle is obtained based on the three-dimensional motion trajectory. When the rotation angle exceeds a preset angle threshold, the rudder is driven to rotate, and the business module icon that the rudder points to after rotation is highlighted on the target screen. Based on the preprocessed skeletal point data at the current moment, business confirmation conditions are detected, and a business confirmation command is triggered when the business confirmation conditions are met. Then, the details page information corresponding to the current business module icon is visualized on the target screen.

[0012] In one possible design, the skeletal point data is preprocessed, including: coordinate system transformation, noise reduction, outlier removal, and / or smoothing.

[0013] In one possible design, a multi-condition joint decision mechanism is used for system activation condition detection. This mechanism includes detection of effective arm raising, palm orientation, hand closure, and motion stability. Correspondingly, based on the preprocessed skeletal point data, system activation condition detection is performed, including: Based on the preprocessed skeletal point data, the arm state, palm orientation, hand state, and hand movement stability coefficient are obtained. The effective lifting of the arm is detected based on the arm state, the palm orientation is detected based on the palm orientation, the hand state is detected based on the hand state, and the hand movement stability is detected based on the hand movement stability coefficient. When the system detects that the arm is effectively raised, the palm is facing the specified direction, the hand is in the specified state, the hand movement stability meets the standard, and the preset first time threshold is continuously reached, the system activation conditions are determined to be met.

[0014] In one possible design, the arm state is determined by the relative positional relationship between the wrist joint and the elbow joint. Correspondingly, detecting effective arm lifting based on the arm state includes: The arm length of the target user is obtained based on the three-dimensional coordinate data of the shoulder joint and the three-dimensional coordinate data of the elbow joint, and a dynamic threshold for arm length is obtained based on the arm length. When the height of the wrist joint in the vertical direction is higher than the height of the elbow joint, and the height difference between the wrist joint and the elbow joint is greater than the dynamic threshold for arm length, it is determined that the target user's arm is effectively raised. The hand node includes the palm, fingertips, and thumb. The palm orientation is the normal vector of the palm plane constructed based on the palm, fingertips, and thumb. Correspondingly, palm orientation detection is performed based on the palm orientation, including: Calculate the normal vector of the target screen. When the angle between the normal vector and the normal vector of the target screen is less than a preset angle threshold, determine that the palm is facing the specified direction. The hand state is determined by spatial distribution analysis of the hand nodes. Correspondingly, hand state detection is performed based on the hand state, including: When the hand changes from an open state to a clenched fist state based on the hand state, the hand is determined to be in a specified state. Hand movement stability detection is performed based on the aforementioned hand movement stability coefficient, including: When the change in the hand stability coefficient within a preset time window is less than a preset coefficient threshold, the hand movement stability is determined to be up to standard.

[0015] In one possible design, the rotation angle of the hand node relative to the center of the circle is obtained based on the three-dimensional motion trajectory, including: Based on the initial n local three-dimensional coordinate data in the three-dimensional motion trajectory, determine the optimal projection plane; where n is a natural number greater than 1. The subsequent local three-dimensional coordinate data in the three-dimensional motion trajectory are projected onto the optimal projection plane to obtain multiple local two-dimensional coordinate data. A polar coordinate system with the center of the circle as the pole is constructed on the optimal projection plane, and multiple local two-dimensional coordinate data are converted into polar coordinate data. Then, the rotation angle of the hand node relative to the center of the circle at each corresponding moment is obtained based on the multiple polar coordinate data.

[0016] In one possible design, the rotation angle of the rudder is mapped to the rotation angle of the hand node relative to the center of the circle, and the mapping is defined by a monotonically increasing linear function, a piecewise linear function, or a smooth nonlinear function.

[0017] In one possible design, based on the preprocessed skeletal point data at the current moment, business confirmation condition checks are performed, including: The current hand state is obtained based on the preprocessed skeletal point data at the current moment; When the hand changes from a clenched fist to an open state at the current moment and continues for a preset second time threshold, the business confirmation condition is determined to be met.

[0018] In one possible design, a pre-trained multimodal feature extraction network model is used to identify skeletal states, including arm state, hand state, and hand movement patterns. The multimodal feature extraction network model includes a temporal feature input module, a temporal feature extraction module, a spatial relationship encoding module, a feature fusion module, and a multi-task output module; The temporal feature input module is used to receive a temporal sequence of skeleton points to be identified, consisting of preprocessed skeleton point data from t consecutive frames; where t is a natural number greater than 1. The temporal feature extraction module is used to capture the temporal dependency of bone point motion from the temporal sequence of the bone points to be identified, so as to obtain a temporal feature vector. The spatial relationship encoding module is used to learn the spatial correlation between different bone points from the temporal sequence of the bone points to be identified, so as to obtain a spatial feature vector. The feature fusion module is used to fuse the temporal feature vector and the spatial feature vector to obtain a fused feature vector; The multi-task output module is used to output the classification results of arm state, hand state and movement pattern based on the fused feature vector.

[0019] In one possible design, the method further includes: When the target user does not perform any operation or leaves the interaction area for a preset period of time, the steering wheel interface on the target screen will be automatically closed.

[0020] Secondly, the present invention provides a screen interaction system based on motion sensing, comprising: The data acquisition unit is used to collect skeletal point data of the target user in the interactive area in real time; wherein, the skeletal point data includes the three-dimensional coordinate data of multiple specified skeletal points, and the multiple specified skeletal points include shoulder joint, elbow joint, wrist joint and hand nodes; A data processing unit, which is communicatively connected to the data acquisition unit, is used to preprocess the skeletal point data to obtain preprocessed skeletal point data. An interactive recognition unit, which is communicatively connected to the data processing unit, is used to detect system activation conditions based on the preprocessed skeletal point data, and activate the rudder when the system activation conditions are met. The interface rendering unit is communicatively connected to the interaction recognition unit. It is used to take the position of the wrist joint at the moment of activation as the center of the rudder and display the rudder interface through the target screen to visualize the rudder. The rudder interface includes a central control point and multiple business module icons set around the central control point. The interactive recognition unit is also used to continuously acquire the three-dimensional motion trajectory of the hand node relative to the center of the circle based on the preprocessed skeletal point data at multiple times, and to obtain the rotation angle of the hand node relative to the center of the circle based on the three-dimensional motion trajectory. The interface rendering unit is also used to drive the rudder to rotate when the rotation angle exceeds a preset angle threshold, and to highlight the business module icon that the rudder points to after rotation on the target screen. The interaction recognition unit is also used to detect business confirmation conditions based on the preprocessed skeletal point data at the current moment, and trigger a business confirmation instruction when the business confirmation conditions are met, and then visualize the details page information corresponding to the current business module icon through the target screen.

[0021] Thirdly, the present invention provides an electronic device, comprising: Memory, used to store computer program instructions; and, A processor is configured to execute the computer program instructions to perform the operation of a motion-based screen interaction method as described in any of the preceding claims.

[0022] Fourthly, the present invention provides a computer program product, including a computer program or instructions, wherein the computer program or instructions, when executed by a computer, implement a screen interaction method based on motion sensing as described in any of the preceding claims.

[0023] The beneficial effects of this invention are as follows: This invention discloses a screen interaction scheme based on motion sensing, which supports complex interactions, significantly reduces the user's memory burden, has low learning costs, low computational power consumption, and high navigation efficiency. Specifically, this invention uses a unified "rudder" interaction paradigm to replace a discrete, specific gesture library. Users only need to master three natural actions, such as "raising arm and clenching fist to activate - circular selection - opening palm to confirm," to complete the activation, business module selection, and business module confirmation operations. The interface provides corresponding visual guidance for intuitive judgment, significantly reducing the user's memory burden. Simultaneously, this invention uses the wrist joint position at the moment of activation as the center of the rudder, mapping a small-range circular motion to a large-range screen navigation, avoiding large-amplitude limb movements caused by large-screen operations. This balances the arm mapping range and operational accuracy, resulting in high navigation efficiency. Furthermore, the circular selection stage for different business modules based on this invention relies only on geometric angle calculations, eliminating the need for continuous gesture recognition, thereby significantly reducing the false trigger rate and computational load.

[0024] Other beneficial effects of the present invention will be further explained in the specific embodiments. Attached Figure Description

[0025] Figure 1 This is a flowchart of the screen interaction method based on motion sensing in Example 1; Figure 2 It is a diagram showing the positional relationship between the target screen's display interface and the human body's skeletal points. Figure 3 This is a diagram showing the palm in an open state and the hand in a clenched fist state; where state 1 is the palm in an open state and state 2 is the palm in a clenched fist state. Figure 4 This is a diagram showing the interaction between hand state changes and the rudder; among them, states 1 to 4 show the interaction between hand state changes and the rudder when the rudder is activated, continuously rotated, and when the service is confirmed. Figure 5 This is a block diagram of the multimodal feature extraction network model in Example 1; Figure 6 This is a block diagram of the screen interaction system based on motion sensing in Embodiment 2; Figure 7 This is a block diagram of an electronic device in Embodiment 3. Detailed Implementation

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0027] Example 1: This embodiment discloses a screen interaction method based on motion sensing, which can be executed by, but is not limited to, a computer device or virtual machine with certain computing resources, such as a personal computer, smartphone, personal digital assistant or wearable device, or by a virtual machine.

[0028] like Figures 1 to 4 As shown, a screen interaction method based on motion sensing may include, but is not limited to, the following steps: S1. Collect skeletal point data of the target user in real time within the interactive area, and preprocess the skeletal point data to obtain preprocessed skeletal point data; wherein, the skeletal point data includes three-dimensional coordinate data of multiple specified skeletal points, the multiple specified skeletal points include shoulder joint (SHOULDER), elbow joint (ELBOW), wrist joint (WRIST) and hand nodes, the hand nodes include palm (HAND), fingertip (HANDTIP) and thumb (THUMB).

[0029] In this embodiment, a depth sensing device is used to collect skeletal point data of the target user in real time. This depth sensing device may be a Kinect Azure (a small smart terminal device integrating multiple industry-leading artificial intelligence sensors) depth camera, connected to the computing device via a USB 3.0 interface. During implementation, the Kinect Azure depth camera collects depth image data of the target user and extracts skeletal point data from it. Furthermore, the computing device is connected to the target screen via an HDMI (High Definition Multimedia Interface) or DisplayPort (a digital video interface standard developed by the Video Electronics Standards Association). The target screen can dynamically render a steering interface based on an interface rendering unit and output the rendered steering interface, including a central control point and multiple surrounding business module icons, to achieve interactive feedback with the target user and enhance the interactive experience. All devices are powered by a unified power management system to ensure stable operation.

[0030] It should be noted that multiple specified bone points can also be set as multiple bone points of the legs, thereby enabling screen interaction through the legs.

[0031] In step S1, the skeletal point data is preprocessed, including coordinate system transformation, noise reduction, outlier removal, and / or smoothing. In this embodiment, a Kalman filter is used, employing the Kalman filtering algorithm for data noise reduction to eliminate noise interference and ensure data quality and stability; a quaternion interpolation algorithm is used for data smoothing.

[0032] S2. Based on the preprocessed skeletal point data, perform system activation condition detection, and activate the rudder when the system activation conditions are met. The user's initial state is as follows: Figure 2 As shown in (a), the target screen does not display the rudder at this time. Figure 2 (b) and Figure 4 In state 1, the user's action meets the system activation conditions. At this time, the rudder is activated and the rudder interface is displayed on the target screen.

[0033] In existing technologies, arm mapping control schemes are prone to misinterpreting unintentional non-target actions such as raising an arm or turning around as interaction commands, leading to false triggers and low interaction accuracy. Single sensors or simple thresholds are insufficient to accurately determine the start time of the interaction intent, while multiple sensors consume significant computing resources, have high calibration complexity, and significantly increase deployment and maintenance costs, limiting their widespread application. Therefore, this embodiment further makes the following improvements: In step S2, a multi-condition joint judgment mechanism is used to detect system activation conditions. This mechanism includes detection of effective arm raising, palm orientation, hand closure, and motion stability. Correspondingly, based on the preprocessed skeletal point data, system activation condition detection is performed, including: S201. Based on the preprocessed skeletal point data, the arm state, palm orientation, hand state, and hand movement stability coefficient are obtained; in this embodiment, the hand movement stability coefficient can be determined based on the rate of change of the position of the hand node within a predetermined time window.

[0034] S202. Detect effective arm lifting based on the arm state, detect palm orientation based on the palm orientation, detect hand state based on the hand state, and detect hand movement stability based on the hand movement stability coefficient. S203. When it is detected that the arm is effectively raised, the palm is facing the specified direction, the hand is in the specified state, the hand movement stability meets the standard, and the preset first time threshold is maintained, the system activation condition is determined to be met. The first time threshold is set to 400-600ms; however, a setting of 500ms is not specified here.

[0035] Specifically, in this embodiment, the arm state is determined by the relative positional relationship between the wrist joint and the elbow joint. Correspondingly, detecting effective arm lifting based on the arm state includes: The arm length of the target user (i.e., the horizontal distance between the shoulder joint and the elbow joint) is obtained based on the three-dimensional coordinate data of the shoulder joint and the elbow joint. A dynamic threshold T_lift for the arm length is then obtained based on the arm length. When the height of the wrist joint in the vertical direction (Y-axis) is higher than the height of the elbow joint, and the height difference ΔY between the wrist joint and the elbow joint is greater than the dynamic threshold T_lift for the arm length, it is determined that the target user's arm is effectively raised. The dynamic threshold T_lift is adaptively adjusted based on the target user's arm length (||SHOULDER-ELBOW||) to accommodate users of different heights.

[0036] The hand node includes the palm (HAND), fingertip (HANDTIP), and thumb (THUMB). The palm orientation is the normal vector of the palm plane constructed based on the palm, fingertip, and thumb. That is, the palm plane is determined by the spatial relationship between the palm, fingertip, and thumb. Correspondingly, palm orientation detection is performed based on the palm orientation, including: Calculate the normal vector of the target screen. When the angle between the normal vector and the normal vector of the target screen is less than a preset angle threshold, it is determined that the palm is facing the specified direction, that is, when the palm is facing the screen, it meets the interaction intention. The hand state is determined by spatial distribution analysis of the hand nodes. Correspondingly, hand state detection is performed based on the hand state, including: When the hand state is determined to change from an open to a clenched fist, the hand is considered to be in a designated state, meaning the change from an open to a clenched fist aligns with the interaction intent. It should be noted that during implementation, the user can face the target screen, raise their forearm and extend their palm while their upper arm is naturally hanging down or slightly raised, changing the palm from an open to a clenched fist. The system uses this clenching motion as a trigger signal for conscious interaction, requiring only a clear lifting motion of the forearm, thus reducing user fatigue. Figure 3 This is a diagram showing the palm in an open state and a clenched fist state.

[0037] Specifically, in this embodiment, the hand state transition from an open state to a clenched fist state is identified using a hand morphology feature function. As an example, the hand state is characterized by calculating the clustering degree of the hand nodes: Hand closure degree = g(HAND, HANDTIP, THUMB); Where g(·) is a feature function based on the relative distance of the hand nodes. When the degree of hand closure changes from greater than the opening threshold to less than the fist clenching threshold, it is determined that a fist clenching action has occurred.

[0038] Hand movement stability detection is performed based on the aforementioned hand movement stability coefficient, including: When the change in the hand stability coefficient within a preset time window is less than a preset coefficient threshold, the hand movement stability is deemed to have met the standard. Specifically, in this embodiment, the rate of change of the hand node's position within a predetermined time window is calculated using a trajectory smoothness function to assess the magnitude of the hand position change over a certain period of time and to perform hand movement stability detection.

[0039] Based on the above steps S201 to S203, this embodiment introduces a multi-dimensional joint judgment mechanism for activation detection to achieve a three-level anti-mistouch function, including physical constraint filtering: requiring the arm to be raised, directional consistency: the palm plane normal vector faces the screen, hand state matching: changing from a palm to a fist posture, and a continuous first time threshold. It can effectively distinguish between conscious interaction and unconscious hand raising without using multiple sensors, avoid accidental triggering, and improve the accuracy of interaction.

[0040] In another embodiment, a pre-trained multimodal feature extraction network model is used for skeletal state recognition, wherein the skeletal state includes arm state, hand state, and hand movement pattern; like Figure 5 As shown, the multimodal feature extraction network model includes a temporal feature input module, a temporal feature extraction module, a spatial relationship encoding module, a feature fusion module, and a multi-task output module; The temporal feature input module is used to receive a temporal sequence of skeleton points to be identified, consisting of preprocessed skeleton point data from t consecutive frames; where t is a natural number greater than 1. The temporal feature extraction module employs a temporal modeling unit based on an LSTM (Long Short-Term Memory) recurrent neural network to capture the temporal dependencies of skeletal point motion from the temporal sequence of the skeletal points to be identified, thereby obtaining a temporal feature vector T_feat. It should be noted that the temporal feature extraction module can learn and output feature vectors representing changes in motion state based on the three-dimensional coordinate sequence of hand nodes in consecutive frames.

[0041] The spatial relationship encoding module employs a spatial relationship encoder based on the Transformer (a deep learning model based on self-attention) self-attention mechanism to learn the spatial correlation between different skeleton points from the temporal sequence of the skeleton points to be identified, thereby obtaining the spatial feature vector S_feat. It should be noted that the spatial relationship encoding module calculates the correlation weights of different time steps and different skeleton points in the temporal sequence of the skeleton points to be identified, enabling the multimodal feature extraction network model to focus on key frames and key joints of state transitions.

[0042] The feature fusion module is used to fuse the temporal feature vector T_feat and the spatial feature vector S_feat to obtain a fused feature vector; The multi-task output module is used to output the classification results of arm state, hand state and movement pattern based on the fused feature vector.

[0043] It should be noted that before model training, the collected sample skeletal point sequences were pre-labeled, data augmented, and normalized. The labels used in the multi-labeling process included: arm extension state labels (fully extended / not fully extended), hand state labels (open / clenched / transitional), and movement pattern labels (circular / linear / stationary / unconscious movement). During data augmentation, rotation, translation, and scaling transformations were applied to the skeletal point coordinates to simulate different user body shapes and standing angles, enhancing the model's generalization ability. Sequence normalization involved unifying skeletal point sequences of different lengths into a fixed length through interpolation or truncation, ensuring consistent network input dimensions.

[0044] In this embodiment, the multi-task output module simultaneously outputs the classification results of arm state, hand state, and movement pattern through a multi-task learning framework. During the training of the multimodal feature extraction network model, three related tasks are learned simultaneously: Arm status recognition: a binary classification problem (raised, not raised); Hand state recognition: a three-class classification problem (open / clenched fist / transition); Motion pattern classification: multi-classification problem (circular motion / linear motion, etc.).

[0045] The multimodal feature extraction network model is trained end-to-end using the following weighted multi-task loss function to balance the importance of different tasks and differences in data distribution: Loss = α×L1 + β×L2 + γ×L3; In the formula, Loss represents the total loss, L1 represents the arm state loss, L2 represents the hand state loss, L3 represents the motion pattern loss, and α, β and γ are preset adjustable weight hyperparameters used to control the relative contributions of arm state loss, hand state loss and motion pattern loss to the total loss. During implementation, the overall recognition performance of the multimodal feature extraction network model can be improved by leveraging the correlation between tasks.

[0046] In this embodiment, the multimodal feature extraction network model, by combining the temporal modeling capability of LSTM and the spatial relationship learning of Transformer, can significantly improve the accuracy and robustness of skeletal point state recognition, providing a reliable technical foundation for haptic large-screen interaction.

[0047] S3. The position of the wrist joint at the moment of activation is taken as the center of the rudder, and the rudder interface is displayed on the target screen to visualize the rudder; wherein, the rudder interface includes a central control point and multiple business module icons arranged around the central control point. It should be understood that the target screen is the screen for interaction with the target user. In this embodiment, the layout of the multiple business module icons can be dynamically configured according to the current business scenario, and is not limited here. It should be noted that the center can also be described as an "anchor point" or "spatial reference point," which is the positioning reference point of the rudder.

[0048] In existing technologies, triggering specific operations through specific gestures has the drawbacks of a limited number of definable gestures and poor scalability. To address this, this embodiment adopts a radial menu layout, which can simultaneously present multiple core business modules. Each small sector area corresponds to a functional module, allowing for flexible expansion of business modules according to actual needs. The number, order, and sector angle of modules can all be dynamically configured. Furthermore, it supports priority for frequently used functions, a fast scanning mode, and automatic exit upon timeout, thereby improving interaction efficiency and system robustness.

[0049] In step S3, during the process of visually displaying the rudder on the target screen, a dynamic smoothing algorithm is also used to filter out minor hand tremors. This effectively reduces the shaking of the rudder interface caused by the user's hand tremors, thus maintaining the stability of the center position. This makes the subsequent selection of business modules more stable and accurate, improving the quality of user operation.

[0050] S4. Based on the preprocessed skeletal point data at multiple time points, continuously acquire the three-dimensional motion trajectory of the hand node relative to the center of the circle, and obtain the rotation angle of the hand node relative to the center of the circle based on the three-dimensional motion trajectory. In this embodiment, the rotation angle can be mapped to a rotation control command for the rudder, thereby driving the rudder to rotate and selecting the business module icon.

[0051] In step S4, the three-dimensional motion trajectory of the hand node relative to the center of the circle is obtained from the three-dimensional coordinate data of the center of the circle and the three-dimensional coordinate data of the hand node at multiple time points. Specifically, during implementation, the three-dimensional coordinate data of the center of the circle is subtracted from the three-dimensional coordinate data of the hand node in each frame, thereby uniformly transforming the original coordinates to a local coordinate system with the center of the circle as the origin, obtaining the local three-dimensional coordinate data of the hand node relative to the center of the circle. Further, based on the local three-dimensional coordinate data of the hand node at different time points, these local three-dimensional coordinate data are connected in chronological order using a trajectory reconstruction algorithm to obtain the three-dimensional motion trajectory of the hand node relative to the center of the circle (wrist joint position). This processing eliminates the global offset caused by the overall movement of the user's body and accurately reflects the true path of the hand node's own movement.

[0052] In step S4, the rotation angle of the hand node relative to the center of the circle is obtained based on the three-dimensional motion trajectory, including: S401. Determine the optimal projection plane based on the initial n local three-dimensional coordinate data in the three-dimensional motion trajectory; where n is a natural number greater than 1. As an example, in this embodiment, n is 2. At this time, the optimal projection plane is the plane containing the two local three-dimensional coordinate data and the three-dimensional coordinate data of the center of the circle. S402. Project the subsequent local three-dimensional coordinate data in the three-dimensional motion trajectory onto the optimal projection plane to obtain multiple local two-dimensional coordinate data; S403. Construct a polar coordinate system with the center of the circle as the pole on the optimal projection plane, and convert multiple local two-dimensional coordinate data into polar coordinate data. Then, obtain the rotation angle of the hand node relative to the center of the circle at each corresponding moment based on the multiple polar coordinate data. The rotation angle is also the absolute angle relative to the direction of the activation moment. In practical applications, after obtaining the rotation angle, it can be smoothed to reduce the impact of user hand tremors.

[0053] S5. When the rotation angle exceeds a preset angle threshold, drive the rudder to rotate, and highlight the business module icon that the rudder is pointing to on the target screen after rotation, such as... Figure 2 (c) and Figure 4 As shown in state 2 or 3. It should be noted that during implementation, the user maintains a clenched fist, using the elbow as an approximate fulcrum to drive the hand in a small arc-shaped movement. The system calculates the change in the angle of the wrist trajectory relative to the center of the circle, driving the rudder to rotate to select the target module. This process relies solely on position calculation and does not require continuous recognition of complex gestures, resulting in low computational load. It should also be noted that... Figure 2 (c) Figure 4Medium state 2 and Figure 4 In state 3, the rotation angle of the user's hand node relative to the center of the circle exceeds a preset angle threshold. At this time, the rudder rotates. In this embodiment, driving the rudder to rotate also drives the selection indicator (such as a pointer) on the rudder to move within the annular area to highlight the sectors corresponding to different business module icons. Highlighting also means visually highlighting the business module icon that the rudder is currently pointing to.

[0054] In step S5, the rotation angle of the rudder is mapped to the rotation angle of the hand node relative to the center of the circle. This mapping relationship is defined by a monotonically increasing linear function, a piecewise linear function, or a smooth nonlinear function. Specifically, in this embodiment, the mapping relationship can be linear or nonlinear to achieve interactive effects with different precision requirements.

[0055] Furthermore, in a preferred embodiment, the sensitivity coefficient of the mapping relationship can be dynamically adjusted according to the distance from the hand node to the center of the circle, thereby achieving a smart interactive effect that is more accurate for close-range operation and faster for long-range operation.

[0056] S6. Based on the preprocessed skeletal point data at the current moment, perform business confirmation condition detection, and trigger a business confirmation command when the business confirmation condition is met. Then, visualize the details page information corresponding to the current business module icon through the target screen, such as... Figure 4 As shown in state 4, this is a schematic diagram of triggering a business confirmation instruction when the business confirmation conditions are met. At this time, the target screen displays the details page information corresponding to the current business module icon.

[0057] In step S6, based on the preprocessed skeleton point data at the current moment, business confirmation condition detection is performed, including: S601. Obtain the current hand state based on the preprocessed skeletal point data at the current moment; S602. When the hand state changes from a clenched fist to an open state at the current moment and continues for a preset second time threshold, the business confirmation condition is determined to be met. It should be noted that when the user opens their fist to a palm, the system recognizes the palm state transition based on the spatial relationship change of the palm, fingertips and thumb nodes, and determines it as a confirmation operation to trigger a confirmation command in order to open the currently selected business module.

[0058] S7. When the target user does not perform any operation or leaves the interaction area for a preset period of time, the steering wheel interface on the target screen is automatically closed. Based on this, system resources can be released when the user does not perform any operation for a long time or leaves the interaction area.

[0059] In addition, this embodiment also has the ability to detect abnormal states. When unreasonable skeletal point data or motion trajectories are identified, the interaction state is automatically reset to ensure system stability.

[0060] This embodiment supports complex interactions, significantly reducing the user's memory burden, lowering the user's learning cost, minimizing computational consumption, and increasing navigation efficiency. Specifically, this embodiment achieves a natural mapping of user actions to a visual rudder interface by real-time acquisition and preprocessing of the three-dimensional coordinates of key skeletal points such as the shoulder, elbow, wrist, and hand nodes. During implementation, the activation condition is first determined based on the skeletal state. Then, using the wrist joint position at the moment of activation as the center of the rudder, the continuous three-dimensional motion trajectory of the user's hand is converted into a rotation angle relative to the center. When this angle exceeds a preset threshold, the rudder automatically rotates and highlights the corresponding business module, allowing the user to switch modules through hand rotation without memorizing complex commands. Finally, the system determines the business confirmation condition based on the real-time skeletal state and triggers the confirmation command, realizing a haptic interaction from module selection to business confirmation. Based on this, this embodiment adopts a unified "rudder" interaction paradigm to replace the discrete specific gesture library. Users only need to master three natural actions, such as "raising arm and clenching fist to activate - circular selection - opening palm to confirm," to complete the activation, business module selection, and business module confirmation operations. Moreover, the interface has corresponding visual guidance to allow users to make intuitive judgments, greatly reducing the user's memory burden. At the same time, this embodiment uses the wrist joint position at the moment of activation as the center of the rudder, and maps a small-range circular motion to a large-range screen navigation, avoiding large-scale limb movements caused by large-screen operation. It can balance the arm mapping range and operation accuracy, resulting in high navigation efficiency. In addition, the circular selection stage for selecting different business modules in this embodiment only relies on geometric angle calculations and does not require continuous gesture recognition, thereby significantly reducing the false trigger rate and computational load.

[0061] Example 2: This embodiment discloses a motion-based screen interaction system for implementing the motion-based screen interaction method in Embodiment 1; such as Figure 6 As shown, the motion-sensing-based screen interaction system includes: The data acquisition unit is used to collect skeletal point data of the target user in the interactive area in real time; wherein, the skeletal point data includes the three-dimensional coordinate data of multiple specified skeletal points, and the multiple specified skeletal points include shoulder joint, elbow joint, wrist joint and hand nodes; A data processing unit, which is communicatively connected to the data acquisition unit, is used to preprocess the skeletal point data to obtain preprocessed skeletal point data. An interactive recognition unit, which is communicatively connected to the data processing unit, is used to detect system activation conditions based on the preprocessed skeletal point data, and activate the rudder when the system activation conditions are met. The interface rendering unit is communicatively connected to the interaction recognition unit. It is used to take the position of the wrist joint at the moment of activation as the center of the rudder and display the rudder interface through the target screen to visualize the rudder. The rudder interface includes a central control point and multiple business module icons set around the central control point. The interactive recognition unit is also used to continuously acquire the three-dimensional motion trajectory of the hand node relative to the center of the circle based on the preprocessed skeletal point data at multiple times, and to obtain the rotation angle of the hand node relative to the center of the circle based on the three-dimensional motion trajectory. The interface rendering unit is also used to drive the rudder to rotate when the rotation angle exceeds a preset angle threshold, and to highlight the business module icon that the rudder points to after rotation on the target screen. The interaction recognition unit is also used to detect business confirmation conditions based on the preprocessed skeletal point data at the current moment, and trigger a business confirmation instruction when the business confirmation conditions are met, and then visualize the details page information corresponding to the current business module icon through the target screen.

[0062] It should be noted that the working process, working details and technical effects of the motion-based screen interaction system provided in this embodiment 2 can be found in embodiment 1, and will not be repeated here.

[0063] Example 3: Based on Embodiment 1 or 2, this embodiment discloses an electronic device, which may be a smartphone, tablet computer, laptop computer, or desktop computer, etc. The electronic device may be referred to as a user terminal, portable terminal, desktop terminal, etc. Figure 7 As shown, the electronic device includes: Memory, used to store computer program instructions; and, A processor is configured to execute the computer program instructions to perform the operation of a motion-based screen interaction method as described in any of Embodiment 1.

[0064] Specifically, processor 301 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 301 may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). Processor 301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen.

[0065] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 are used to store at least one instruction, which is executed by the processor 301 to implement the motion-sensing-based screen interaction method provided in Embodiment 1 of this application.

[0066] In some embodiments, the terminal may also optionally include a communication interface 303 and at least one peripheral device. The processor 301, memory 302, and communication interface 303 can be connected via a bus or signal line. Each peripheral device can be connected to the communication interface 303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 304, a display screen 305, and a power supply 306.

[0067] The communication interface 303 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0068] The radio frequency (RF) circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 304 communicates with communication networks and other communication devices via electromagnetic signals.

[0069] Display screen 305 is used to display the UI (User Interface). The UI may include any combination of graphics, text, icons, and video.

[0070] Power supply 306 is used to supply power to various components in electronic devices.

[0071] Example 4: Based on any one of Embodiments 1 to 3, this embodiment discloses a computer program product, including a computer program or instructions, which, when executed by a computer, implements a screen interaction method based on motion sensing as described in any one of Embodiments 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0072] Obviously, those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A somatosensory action-based screen interaction method, characterized by, The method comprises the following steps: Real-time acquisition of the skeletal point data of the target user in the interactive area, and preprocessing of the skeletal point data to obtain preprocessed skeletal point data; wherein the skeletal point data comprises three-dimensional coordinate data of a plurality of specified skeletal points, and the plurality of specified skeletal points comprise shoulder joints, elbow joints, wrist joints, and hand nodes; Based on the preprocessed skeletal point data, system activation condition detection is performed, and a rudder is activated when the system activation condition is met; The position of the wrist joint at the activation time is taken as the center of the rudder, and a rudder interface is displayed on the target screen to visually display the rudder; wherein the rudder interface comprises a center control point and a plurality of business module icons arranged around the center control point; According to the preprocessed skeletal point data at a plurality of times, the three-dimensional motion trajectory of the hand node relative to the center is continuously obtained, and the rotation angle of the hand node relative to the center is obtained according to the three-dimensional motion trajectory; When the rotation angle exceeds a preset angle threshold, the rudder is driven to rotate, and the business module icon pointed to by the rudder after rotation is highlighted on the target screen; Based on the preprocessed skeletal point data at the current time, business confirmation condition detection is performed, and a business confirmation instruction is triggered when the business confirmation condition is met, and then the detail page information corresponding to the current business module icon is visually displayed on the target screen.

2. The method of claim 1, wherein, The preprocessing of the skeletal point data comprises coordinate system conversion, noise reduction, outlier removal, and / or smoothing processing of the skeletal point data.

3. The method of claim 1, wherein, A multi-condition joint determination mechanism is used for system activation condition detection, and the multi-condition joint determination mechanism comprises arm effective lifting detection, palm orientation detection, hand closure degree detection, and motion stability detection; Correspondingly, based on the preprocessed skeletal point data, system activation condition detection is performed, which comprises: According to the preprocessed skeletal point data, the arm state, the palm orientation, the hand state, and the hand motion stability coefficient are obtained; According to the arm state, arm effective lifting detection is performed, according to the palm orientation, palm orientation detection is performed, according to the hand state, hand state detection is performed, and according to the hand motion stability coefficient, hand motion stability detection is performed; When the arm is effectively lifted, the palm is oriented in a specified direction, the hand is in a specified state, the hand motion stability meets the standard, and lasts for a preset first time threshold, it is determined that the system activation condition is met.

4. The method of claim 3, wherein, The arm state is determined by the relative positional relationship between the wrist joint and the elbow joint, and correspondingly, the arm effective lifting detection according to the arm state comprises: According to the three-dimensional coordinate data of the shoulder joint and the three-dimensional coordinate data of the elbow joint, the arm length of the target user is obtained, and an arm length dynamic threshold is obtained according to the arm length. When the height of the wrist joint in the vertical direction is higher than the height of the elbow joint, and the height difference between the wrist joint and the elbow joint is greater than the arm length dynamic threshold, it is determined that the arm of the target user is effectively lifted. The hand nodes include a palm center, a palm tip and a thumb, the palm orientation is a normal vector of a palm plane constructed based on the palm center, the palm tip and the thumb, and palm orientation detection is performed according to the palm orientation, including: calculating a normal vector of a target screen, and determining that the palm orientation is in a specified direction when an included angle between the normal vector and a normal vector of the target screen is less than a preset angle threshold; The hand state is determined by spatial distribution analysis of the hand nodes, and hand state detection is performed according to the hand state, including: determining that the hand is in a specified state when the palm changes from an open state to a fist state according to the hand state; hand movement stability detection is performed according to the hand movement stability coefficient, including: determining that the hand movement stability meets a standard when a change amplitude of the hand stability coefficient within a preset time window is less than a preset coefficient threshold.

5. The method of claim 1, wherein, The rotation angle of the hand node relative to the center of the circle is obtained according to the three-dimensional motion trajectory, including: determining an optimal projection plane according to initial n local three-dimensional coordinate data in the three-dimensional motion trajectory; wherein n is a natural number greater than 1; projecting subsequent local three-dimensional coordinate data in the three-dimensional motion trajectory onto the optimal projection plane to obtain a plurality of local two-dimensional coordinate data; constructing a polar coordinate system with the center of the circle as the pole on the optimal projection plane, and converting the plurality of local two-dimensional coordinate data into polar coordinate data, and then obtaining the rotation angle of the hand node relative to the center of the circle at each corresponding time according to the plurality of polar coordinate data.

6. The method of claim 1, wherein, The rotation angle of the rudder and the rotation angle of the hand node relative to the center of the circle are in a mapping relationship, and the mapping relationship is defined by a monotonically increasing linear function, a piecewise linear function or a smooth nonlinear function.

7. The method of claim 1, wherein, Based on the preprocessed bone point data at the current time, a business confirmation condition detection is performed, including: obtaining the hand state at the current time according to the preprocessed bone point data at the current time; determining that the business confirmation condition is met when the hand state at the current time changes from a fist state to an open state and lasts for a preset second time threshold.

8. The method according to claim 3 or 7, characterized in that, A pre-trained multi-modal feature extraction network model is used for bone state recognition, and the bone state includes an arm state, a hand state and a hand movement mode; The multi-modal feature extraction network model includes a time sequence feature input module, a time sequence feature extraction module, a spatial relationship encoding module, a feature fusion module and a multi-task output module; The time sequence feature input module is configured to receive a to-be-recognized bone point time sequence composed of continuous t frames of preprocessed bone point data; wherein t is a natural number greater than 1; The time sequence feature extraction module is configured to capture the time sequence dependence of bone point motion from the to-be-recognized bone point time sequence to obtain a time sequence feature vector; The spatial relationship encoding module is configured to learn the spatial correlation between different bone points from the to-be-recognized bone point time sequence to obtain a spatial feature vector; The feature fusion module is configured to fuse the time sequence feature vector and the spatial feature vector to obtain a fusion feature vector; and the multi-task output module is configured to output the fusion feature vector as a hand state recognition result. The multi-task output module is configured to output classification results of arm state, hand state and motion mode according to the fusion feature vector.

9. The method of claim 1, wherein, The method further includes: When the target user does not operate or leaves the interaction area for more than a preset time length, the rudder interface on the target screen is automatically closed.

10. A somatosensory motion based screen interaction system, characterized by, Comprise: A data acquisition unit is configured to acquire bone point data of a target user in an interaction area in real time, wherein the bone point data comprises three-dimensional coordinate data of a plurality of specified bone points, and the plurality of specified bone points comprise shoulder joints, elbow joints, wrist joints and hand nodes; A data processing unit is in communication connection with the data acquisition unit and is configured to pre-process the bone point data to obtain pre-processed bone point data; An interaction recognition unit is in communication connection with the data processing unit and is configured to detect a system activation condition based on the pre-processed bone point data and activate a rudder when the system activation condition is met; An interface rendering unit is in communication connection with the interaction recognition unit and is configured to take a position of a wrist joint at an activation time as a center of the rudder, display a rudder interface on a target screen, and visually display the rudder; wherein the rudder interface comprises a center control point and a plurality of business module icons arranged around the center control point; The interaction recognition unit is further configured to continuously acquire a three-dimensional motion trajectory of the hand node relative to the center based on pre-processed bone point data at a plurality of times, and obtain a rotation angle of the hand node relative to the center based on the three-dimensional motion trajectory; The interface rendering unit is further configured to drive the rudder to rotate when the rotation angle exceeds a preset angle threshold, and highlight a business module icon pointed to by the rudder after rotation on the target screen; The interaction recognition unit is further configured to detect a business confirmation condition based on pre-processed bone point data at a current time, trigger a business confirmation instruction when the business confirmation condition is met, and visually display detail page information corresponding to a current business module icon on the target screen.

Images