Data interaction method and system for handheld tablet computer

Abstract

The present invention discloses a data interaction method and system for a handheld tablet computer, which is applied to the field of human-computer interaction technology. The method detects the spatial posture of the device relative to the gravity direction through a built-in posture sensor, and intelligently divides the screen into multiple interaction areas based on this. Each area is associated with a specific data interaction function. A depth camera is used to obtain a depth image sequence of the movement of the user's finger in the space around the device in real time, and the time-series three-dimensional coordinate information is extracted through an image processing algorithm to construct a non-contact three-dimensional motion trajectory. The trajectory is pattern matched with a predefined non-contact gesture library to determine the target area and related functions of the user's intended interaction, and trigger execution. During the execution process, corresponding perceptual feedback, such as tactile, visual or auditory signals, is transmitted to the user through a multimodal feedback device. This method realizes intelligent non-contact interaction based on device posture and user gestures, improves the intuitiveness and efficiency of operation, and significantly enhances the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a data interaction method and system for a handheld tablet computer. Background Art

[0002] With the rapid development of mobile computing technology, handheld tablets have become indispensable smart devices in people's daily lives and work. Traditional handheld tablets rely primarily on touchscreens for human-computer interaction, but this approach has limitations in certain scenarios, such as when users have clumsy hands or the device's surface is dirty. Furthermore, traditional interaction methods often require direct user contact with the device, which can be inconvenient or unhygienic in some environments.

[0003] While some existing contactless interaction solutions exist, most suffer from space limitations. These solutions typically operate only within a fixed area directly in front of the device, making them difficult to adapt to users' varying postures and environments. For example, when using a device while lying down, standing, or walking, the fixed interaction area can lead to operational difficulties and accidental touches. This lack of flexibility significantly impacts the user experience and efficiency of the device.

[0004] Another significant issue is that existing contactless interaction solutions are often too limited in functionality. Most solutions only offer simple control functions, such as scrolling, zooming in and out, and other basic operations. However, as users' demands for device functionality become increasingly complex, this simple interaction method is no longer sufficient for performing complex tasks. For example, existing contactless interaction solutions often struggle when it comes to multitasking, complex document editing, or operating specialized software.

[0005] Insufficient interaction feedback is another significant issue facing existing contactless interaction solutions. In traditional touchscreen interactions, users can directly sense the execution of actions through touch. However, in contactless interactions, the lack of physical contact makes it difficult for users to accurately determine whether their actions have been correctly recognized and executed by the device. This lack of feedback not only reduces operational efficiency but can also cause user frustration, impacting the overall user experience.

[0006] Furthermore, existing contactless interaction solutions generally suffer from poor adaptability. Most fail to automatically adjust the interaction area and recognition algorithms based on the device's different postures. Users must deliberately maintain a specific posture or manually adjust device settings to ensure accurate interaction. This lack of intelligent adaptation severely limits the device's usage scenarios and flexibility.

[0007] Finally, existing technologies still need to improve the accuracy and stability of gesture recognition. Many contactless interaction solutions use standard cameras or simple infrared sensors, which often perform poorly in complex lighting environments or when recognizing rapid movements. This results in high gesture recognition error rates, forcing users to repeat their actions multiple times to complete desired actions, significantly reducing the smoothness of interaction and user experience.

[0008] Therefore, there is an urgent need for a data interaction method for a handheld tablet computer that can overcome the above problems and provide a more intelligent, flexible, intuitive and feature-rich interaction experience. Summary of the Invention

[0009] In view of this, the purpose of the embodiments of the present invention is to provide a data interaction method and system for a handheld tablet computer, so as to achieve more intelligent, flexible and intuitive human-computer interaction based on non-contact gesture recognition, thereby improving the use efficiency and user experience of the handheld tablet computer.

[0010] To achieve the above objectives, in a first aspect, a data interaction method for a handheld tablet computer is provided, which comprises the following steps:

[0011] S10: Detecting the spatial posture of the handheld tablet computer relative to the direction of gravity using a posture sensor built into the handheld tablet computer;

[0012] S20: dividing the screen of the handheld tablet computer into a plurality of interactive areas according to the detected spatial posture of the handheld tablet computer relative to the gravity direction, each interactive area being associated with a corresponding data interaction function;

[0013] S30: using the depth camera built into the handheld tablet computer to obtain a depth image sequence of the user's finger moving in the space around the handheld tablet computer in real time;

[0014] S40: extracting time-series three-dimensional coordinate information of the user's finger through an image processing algorithm based on the depth image sequence, and constructing a non-contact three-dimensional motion trajectory of the user's finger according to the time-series three-dimensional coordinate information;

[0015] S50: performing pattern matching on the non-contact three-dimensional motion trajectory of the user's finger with a predefined non-contact gesture library stored in the handheld tablet computer to obtain a pattern matching result;

[0016] S60: determining a target interaction area where the user intends to interact and a data interaction function associated with the target interaction area according to the pattern matching result, and triggering execution of the data interaction function associated with the target interaction area;

[0017] S70: In response to the execution of the data interaction function, a perceptual feedback corresponding to the executed data interaction function is delivered to the user through the multimodal feedback device built into the handheld tablet computer.

[0018] In a second aspect, a data interaction system for a handheld tablet computer is provided, comprising:

[0019] A posture sensor, used to detect the spatial posture of the handheld tablet computer relative to the direction of gravity;

[0020] A depth camera, configured to acquire in real time a depth image sequence of a user's finger movement in a space surrounding the handheld tablet computer;

[0021] a multimodal feedback device for delivering sensory feedback to the user corresponding to the data interaction function performed;

[0022] a memory, electrically connected to the processor, and configured to store program instructions for executing any one of the methods of the first aspect;

[0023] a processor, electrically connected to the gesture sensor, the depth camera, the memory, and the data storage device, configured to execute a program in the memory to implement any one of the methods described in the first aspect;

[0024] a screen, electrically connected to the processor, for displaying the plurality of interactive areas and results of executing the data interactive function;

[0025] The server is electrically connected to the processor and is used to store a predefined contactless gesture library, a data interaction function library and a multimodal feedback mode library.

[0026] The above technical solution has the following beneficial technical effects:

[0027] This invention uses a built-in gesture sensor on a handheld tablet to detect the device's spatial posture relative to gravity and dynamically divides the screen's interactive area accordingly, achieving adaptive adjustment of the interactive area. This approach significantly increases the device's operational flexibility in different postures, allowing users to interact conveniently whether standing, sitting, or lying down, significantly improving the device's adaptability and user experience.

[0028] This invention uses a depth camera to capture real-time information about the user's finger's movement in the space around the device. Using image processing algorithms, it extracts the finger's temporal three-dimensional coordinates to construct a contactless three-dimensional motion trajectory, significantly expanding the interactive space. This not only breaks through the flat surface limitations of traditional touchscreens but also provides users with a more natural and ergonomic operation method, reducing fatigue caused by prolonged use.

[0029] This invention matches the contactless, three-dimensional motion trajectory of a user's finger against a predefined library of contactless gestures, significantly improving the accuracy and richness of gesture recognition. This carefully designed library enables the system to recognize complex, multi-dimensional gestures, supporting a wider range of operational commands and meeting user demands for advanced functionality and complex tasks.

[0030] Based on pattern matching results, the present invention determines the target area of ​​user interaction and the associated data interaction function, and triggers the execution of the corresponding function. This intelligent interaction mechanism greatly improves operational efficiency. Users no longer need to navigate tedious menus or perform multi-step operations. Simple gestures can quickly complete complex tasks, greatly simplifying the operation process.

[0031] This invention introduces a multimodal feedback mechanism, which delivers sensory feedback corresponding to the executed function to the user through the device's built-in multimodal feedback device. This immediate and intuitive feedback greatly enhances the user's sense of operational certainty, effectively solving the problem of "hanging in mid-air" common in contactless interaction, and improving interaction accuracy and user confidence.

[0032] The data interaction method proposed in this paper organically combines device posture perception, three-dimensional gesture recognition, intelligent function mapping, and multimodal feedback, creating a comprehensive contactless interaction system. This not only significantly improves the operational convenience and hygiene of handheld tablet computers, but also provides solutions for use in specialized environments (such as medical and industrial settings where cleanliness is crucial), significantly expanding the device's application scope and scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0034] Figure 1 This is a flow chart of a data interaction method for a handheld tablet computer according to an embodiment of the present invention;

[0035] Figure 2 is a specific flow chart of step S10 in an embodiment of the present invention;

[0036] Figure 3 is a specific flow chart of step S20 in an embodiment of the present invention;

[0037] Figure 4 is a specific flow chart of step S30 of an embodiment of the present invention;

[0038] Figure 5 is a specific flow chart of step S40 in an embodiment of the present invention;

[0039] Figure 6 is a specific flow chart of step S50 in an embodiment of the present invention;

[0040] Figure 7 is a specific flow chart of step S60 in an embodiment of the present invention;

[0041] Figure 8 is a specific flow chart of step S70 in an embodiment of the present invention;

[0042] Figure 9 is a flow chart of another data interaction method for a handheld tablet computer according to an embodiment of the present invention;

[0043] Figure 10 is a specific flow chart of step S90 in an embodiment of the present invention;

[0044] Figure 11 This is a functional block diagram of a data interaction system for a handheld tablet computer according to an embodiment of the present invention;

[0045] Figure 12 Schematic diagram of the structure of a computer system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment provides a data interaction method for a handheld tablet computer, which includes the following steps:

[0049] S10: Detecting the spatial posture of the handheld tablet computer relative to the direction of gravity using a posture sensor built into the handheld tablet computer;

[0050] S20: dividing the screen of the handheld tablet computer into a plurality of interactive areas according to the detected spatial posture of the handheld tablet computer relative to the gravity direction, each interactive area being associated with a corresponding data interaction function;

[0051] S30: using the depth camera built into the handheld tablet computer to obtain a depth image sequence of the user's finger moving in the space around the handheld tablet computer in real time;

[0052] S40: extracting time-series three-dimensional coordinate information of the user's finger through an image processing algorithm based on the depth image sequence, and constructing a non-contact three-dimensional motion trajectory of the user's finger according to the time-series three-dimensional coordinate information;

[0053] S50: performing pattern matching on the non-contact three-dimensional motion trajectory of the user's finger with a predefined non-contact gesture library stored in the handheld tablet computer to obtain a pattern matching result;

[0054] S60: determining a target interaction area where the user intends to interact and a data interaction function associated with the target interaction area according to the pattern matching result, and triggering execution of the data interaction function associated with the target interaction area;

[0055] S70: In response to the execution of the data interaction function, a multimodal feedback device built into the handheld tablet computer is used to transmit sensory feedback corresponding to the executed data interaction function to the user. The sensory feedback may include visual, auditory, optical, or other non-contact sensory signals.

[0056] The handheld tablet computer data interaction method based on contactless gesture recognition, proposed in this embodiment of the invention, organically combines device posture perception, three-dimensional gesture recognition, intelligent function mapping, and multimodal feedback, creating a comprehensive contactless interaction system. This not only significantly improves the convenience and hygiene of handheld tablet computer operation, but also provides a solution for use in specialized environments (such as medical and industrial settings where cleanliness is crucial), significantly expanding the device's scope and scenarios.

[0057] like Figure 2 As shown, in a preferred embodiment of the present invention, step S10 specifically includes the following steps:

[0058] S11: Configuring operating parameters of a built-in attitude sensor of the handheld tablet computer, wherein the attitude sensor includes an accelerometer, a gyroscope, and a magnetometer.

[0059] Specifically, this step begins by properly configuring the handheld tablet's built-in attitude sensors. These sensors include an accelerometer, gyroscope, and magnetometer, which work together to provide comprehensive device attitude information. Configuring operating parameters includes setting the sampling rate, range, and filtering parameters. For example, the accelerometer sampling rate can be set to 100Hz with a range of ±2g; the gyroscope sampling rate can be set to 200Hz with a range of ±250° / s; and the magnetometer sampling rate can be set to 50Hz with a range of ±4800μT. These parameters require a balance between measurement accuracy and power consumption to suit different application scenarios.

[0060] S12: Acquire acceleration data of the handheld tablet computer in three orthogonal axes through the accelerometer.

[0061] Specifically, the accelerometer measures the device's acceleration along three orthogonal axes: X, Y, and Z. This data includes not only the acceleration due to device motion but also the component of gravity. By analyzing this data, we can make a preliminary estimate of the device's tilt angle. For example, when the device is stationary, the distribution of gravity acceleration along the three axes directly reflects the device's tilt.

[0062] S13: Acquiring angular velocity data of the handheld tablet computer around three orthogonal axes through the gyroscope;

[0063] Specifically, a gyroscope measures the device's rotational velocity around the X, Y, and Z axes. This angular velocity data is crucial for capturing rapid changes in the device's attitude, providing faster and more accurate short-term attitude information than an accelerometer. By integrating this angular velocity data, this step determines the change in the device's attitude.

[0064] S14: Obtaining direction data of the handheld tablet computer relative to the Earth's magnetic field through the magnetometer.

[0065] Specifically, a magnetometer measures the components of the Earth's magnetic field along three axes. This data is primarily used to determine the device's absolute orientation relative to the Earth, and is particularly crucial for calculating the yaw angle (the device's horizontal rotation angle relative to magnetic north). However, magnetometers are easily affected by metallic objects or electromagnetic interference in the surrounding environment, requiring proper calibration and compensation before use.

[0066] S15: Using a Kalman filter algorithm or a complementary filter algorithm, the acceleration data from the accelerometer, the angular velocity data from the gyroscope, and the direction data from the magnetometer are integrated to obtain optimized attitude estimation data.

[0067] Specifically, Kalman filtering or complementary filtering algorithms can effectively fuse data from different sensors, leveraging the strengths of each sensor while offsetting their weaknesses. For example, accelerometers provide accurate tilt angles when stationary but are sensitive to vibration; gyroscopes accurately capture rapid motion but suffer from integration drift; and magnetometers provide an absolute orientation reference but are susceptible to external interference. These algorithms can achieve more stable and accurate attitude estimation.

[0068] S16: Calculating the pitch angle, roll angle, and yaw angle of the handheld tablet computer relative to the direction of gravity based on the optimized posture estimation data.

[0069] Using the optimized pose estimation data, three key angles of the device can be calculated:

[0070] Pitch, which is the angle at which the device tilts forward and backward;

[0071] Roll angle (Roll), which is the angle at which the device tilts left or right;

[0072] Yaw, which is the rotation angle of the device in the horizontal plane.

[0073] These angles are obtained by transforming quaternions or rotation matrices. For example, when using quaternions (q0, q1, q2, q3), they can be calculated using the following formula:

[0074] ;

[0075] .

[0076] S17: Determine the spatial posture of the handheld tablet computer relative to the direction of gravity according to the calculated pitch angle, roll angle, and yaw angle.

[0077] Finally, this step uses the three calculated angles to determine the device's complete spatial pose. This pose can be represented by a 3x3 rotation matrix or quaternion, which fully describes the device's orientation in three-dimensional space. This representation is intuitive and facilitates subsequent calculations and processing.

[0078] Through the detailed steps described above, the embodiments of the present invention can accurately detect the spatial posture of a handheld tablet computer relative to the direction of gravity. This precise posture detection provides a reliable foundation for subsequent interaction area division and gesture recognition, enabling the system to dynamically adjust the interaction mode based on the actual usage of the device, thereby greatly improving the flexibility and adaptability of the user experience. For example, the system can automatically switch between landscape and portrait modes based on the detected posture, or adjust the screen brightness based on the device's tilt angle, achieving a more intelligent and user-friendly interactive experience.

[0079] like Figure 3 As shown, in a preferred embodiment of the present invention, step S20 specifically includes the following steps:

[0080] S21: Determine the tilt angle of the screen plane of the handheld tablet computer relative to the horizontal plane according to the spatial posture of the handheld tablet computer relative to the gravity direction.

[0081] In this step, the spatial posture information obtained in step S10 is used to calculate the tilt angle of the screen plane relative to the horizontal plane. Specifically, this tilt angle can be determined using the pitch angle and roll angle. For example, if the pitch angle is 30° and the roll angle is 0°, the tilt angle of the screen plane relative to the horizontal plane is 30°. This tilt angle information is crucial for subsequent screen segmentation, as different usage postures may require different interaction area layouts.

[0082] S22: Based on the tilt angle, a preset screen division template is selected, and the screen of the handheld tablet computer is divided into a plurality of interaction areas of geometric shapes according to the selected screen division template.

[0083] Based on the calculated tilt angle, the system selects the template from a library of pre-designed screen division templates that best suits the current posture. For example, when the tilt angle is between 0-15°, a quartered template is selected for desktop use; when the tilt angle is between 45-60°, a triangular division template is selected for one-handed use. After selecting a template, the system divides the screen into multiple geometric interaction areas based on the template. These geometric shapes can be rectangles, triangles, circles, and more, providing the optimal interaction experience for different usage scenarios.

[0084] S23: Selecting a corresponding data interaction function for each interaction area from a preset data interaction function library according to the current application scenario and user usage habits.

[0085] In this step, the system will consider the current application scenario (such as office, entertainment, reading, etc.) and the user's usage habits, and assign the most appropriate data interaction function to each divided interaction area. These functions come from a preset data interaction function library, which includes various commonly used operations such as clicking, sliding, zooming, rotating, etc. For example, in the reading scenario, the area on the right side of the screen can be assigned to the page turning function, while the area on the left side can be assigned to the return to directory function. The system can also learn the user's usage habits and adjust the function allocation over time to provide a more personalized experience.

[0086] S24: Generate and store association information between the interaction area and the data interaction function.

[0087] After function assignment, the system generates a mapping table or data structure to store the association between each interactive area and its corresponding data interaction function. This association information may include the area's geometric parameters (such as coordinates, size, and shape), the assigned function type, and trigger conditions. For example, the storage format might be: {Area ID: 1, Shape: Rectangle, Coordinates: (0,0,100,100), Function: Swipe to Flip, Trigger Condition: Single-Finger Swipe}. This information is stored in the device's memory or persistent storage for quick access and updates.

[0088] S25: Visually displaying the divided interaction areas and their associated data interaction functions on the screen of the handheld tablet computer.

[0089] Finally, the system visualizes the divided interactive areas and their associated functions on the screen, allowing users to intuitively understand the current interactive layout. This visualization can take various forms, such as subtle color differentiation or translucent borders. For example, in a reading app, the system might display a faint arrow icon on the right side of the screen to indicate to the user that this area can be used to turn pages.

[0090] In the context of contactless gesture interaction, the system uses intelligent and intuitive methods to visualize interaction areas and their functions. This can include the following methods: persistent light visualization, such as using extremely subtle visual cues to continuously mark different interaction areas; dynamic response visualization, which briefly enhances the visual cues of a certain area of ​​the screen when the user's hand is detected approaching that area; gesture-guided animations, which display the gestures and functions corresponding to each area; intelligent contextual prompts, which display floating function prompts based on the current scenario and user behavior; and customizable feedback intensity, which allows users to adjust the intensity of visual feedback. Through these methods, the system can effectively convey the layout and function information of the interaction area without interfering with the user's normal use, achieving an intuitive and efficient contactless gesture interaction experience.

[0091] Through the detailed steps above, the present invention intelligently divides the screen's interactive areas based on the device's spatial posture and assigns appropriate functions to each area. This dynamic interactive area division and function allocation mechanism improves the flexibility and efficiency of handheld tablets, enabling users to obtain the best interactive experience in various usage scenarios.

[0092] like Figure 4 As shown, in a preferred embodiment of the present invention, step S30 specifically includes the following steps:

[0093] S31: activating the depth camera built into the handheld tablet computer, and adaptively adjusting the operating parameters of the depth camera according to the current ambient light conditions and the user distance to obtain adjusted operating parameters; the adjusted operating parameters include frame rate, resolution and depth of field.

[0094] In this step, the system first activates the device's built-in depth camera. Then, by analyzing the current ambient light conditions (indoors, outdoors, bright or dim), and estimating the distance between the user and the device, the system automatically adjusts the depth camera's operating parameters. For example, in bright light, the frame rate is reduced to improve resolution; in low light, the frame rate is increased but the resolution is reduced to maintain image quality. Furthermore, based on the estimated user distance, the system adjusts the depth of field to ensure accurate capture of user gestures. This adaptive adjustment ensures optimal gesture recognition in various usage environments.

[0095] S32: defining a three-dimensional space detection area around the handheld tablet computer, where the three-dimensional space detection area covers a range where the user may perform gesture interaction.

[0096] Specifically, based on the device's size and typical usage distance, the system defines a three-dimensional detection area. This area is a hemispherical or conical space centered on the device, covering the range where users are most likely to perform gesture interactions. For example, for a 10-inch tablet, this detection area could be a hemispherical space with a radius of 50 cm. This definition ensures that the system can capture all possible gestures while avoiding processing unnecessary background information, improving processing efficiency.

[0097] S33: Based on the adjusted working parameters, continuously capture multiple frames of depth images within the three-dimensional space detection area within a preset time interval.

[0098] Specifically, using the operating parameters adjusted in step S31, the system begins continuously capturing depth images within the predefined three-dimensional detection area. This process is performed at a high frequency, capturing 30 frames per second or more, to ensure that rapid hand movements can be accurately captured. Each depth image frame contains distance information about objects within the detection area, enabling the system to accurately track the movement trajectory of the finger.

[0099] S34: Process each captured depth image frame based on an image enhancement algorithm to obtain a processed depth image.

[0100] Specifically, in this step, the system applies a series of advanced image enhancement algorithms to improve the quality of the depth image. The detailed processing process is as follows: First, the non-local means denoising algorithm is used to reduce random noise in the depth image. This algorithm finds similar pixel blocks in different regions of the image and then performs a weighted average of these blocks, effectively preserving image detail while removing noise. Second, depth images often contain holes—pixels without valid depth values. To fill these holes, the system uses an algorithm based on the fast marching method to ensure the continuity of the depth image. Next, a bilateral filter is applied to the image for edge-preserving smoothing. This bilateral filter takes into account both the spatial distance of pixels and the differences in depth values, smoothing the image while maintaining edge sharpness, which is crucial for accurate finger contour recognition. Finally, an adaptive histogram equalization algorithm is used to enhance local contrast in the image. This method divides the image into multiple small blocks, performs histogram equalization on each block, and uses bilinear interpolation to eliminate the boundaries between the blocks. Finally, according to the predefined gesture interaction space, the depth values ​​are linearly mapped to expand the depth values ​​of the area of ​​interest to the full dynamic range, thereby improving the depth resolution of the gesture area. Through this series of processing, the original depth image is converted into a clearer, higher-quality depth image that is more conducive to gesture recognition.

[0101] S35: Organizing the processed depth images into a depth image sequence in time sequence.

[0102] Specifically, each processed depth image frame is assigned a timestamp and organized into a continuous sequence based on the chronological order of capture. This sequence contains not only spatial information (i.e., the 3D position of the object in each frame) but also temporal information (i.e., the change in the object's position at different time points). This organization provides the foundation for subsequent motion analysis, enabling the system to understand the dynamic characteristics of gestures.

[0103] S36: Performing time synchronization and spatial registration on the depth image sequence to generate a temporally and spatially consistent depth image sequence as a real-time depth image sequence representing the movement of the user's finger in the space around the handheld tablet computer.

[0104] Specifically, this step ensures the consistency of the depth image sequence in time and space dimensions, including time synchronization and spatial registration. Time synchronization ensures that the time interval between each frame is uniform, which is crucial for accurately calculating the speed and acceleration of movement. Spatial registration ensures that the spatial reference system between different frames is consistent, so that the movement trajectory of objects in three-dimensional space can be accurately tracked. For example, if the user's hand gets closer and closer to the device during movement, spatial registration can ensure that this distance change is accurately recorded. After these processing, the final depth image sequence can accurately represent the movement of the user's fingers in the space around the device, providing high-quality input data for subsequent gesture recognition and interactive function implementation.

[0105] In some embodiments, for time synchronization, each depth image frame is assigned a precise timestamp with millisecond-level accuracy. The system uses linear interpolation to address possible frame rate fluctuations and ensure uniform time intervals between frames. If frame loss is detected, an optical flow estimation algorithm is applied to predict the content of the missing frame to maintain sequence continuity. For spatial registration, the iterative closest point (ICP) algorithm is used to align depth images between consecutive frames, eliminating spatial reference frame changes caused by slight device movement. The system also applies an affine transformation matrix to each frame to ensure that all frames are aligned to the same spatial coordinate system. Furthermore, data from the device's built-in gyroscope and accelerometer is used to compensate for changes in viewpoint caused by device movement. Finally, to enhance spatiotemporal consistency, the system employs a spatial-temporal bilateral filter, which considers similarities in both spatial and temporal dimensions to further improve sequence consistency. A Kalman filter is also used to smooth finger motion trajectories and reduce jitter caused by sensor noise. After the above processing, the final generated depth image sequence is smooth and continuous in time and accurately aligned in space, providing high-quality input data for subsequent gesture recognition algorithms.

[0106] like Figure 5 As shown, in a preferred embodiment of the present invention, step S40 specifically includes the following steps:

[0107] S41: Preprocessing the acquired depth image sequence to obtain a preprocessed depth image sequence.

[0108] Specifically, this step is primarily intended to improve the efficiency and accuracy of subsequent processing. Preprocessing includes operations such as image denoising, contrast enhancement, and depth normalization. For example, a Gaussian filter can be used to remove high-frequency noise from the image, followed by histogram equalization to enhance image contrast. Finally, the depth values ​​are mapped to a range of 0-255 for normalization. These operations can significantly improve the recognizability of the finger area.

[0109] S42: In each frame of the preprocessed depth image, extract the user's finger area using a depth threshold segmentation algorithm.

[0110] Specifically, depth threshold segmentation is a depth-based binarization method that effectively separates the foreground (finger) from the background. In implementation, a depth range (e.g., 30cm to 60cm from the device) is set based on a predefined gesture interaction space, with pixels within this range considered potential finger areas. The Otsu algorithm is then used to adaptively determine the optimal threshold and further refine the segmentation results. This method effectively handles varying lighting conditions and background interference in various usage environments.

[0111] S43: Perform morphological processing on the user's finger area to obtain a finger outline.

[0112] Specifically, morphological processing is a shape-based image processing technique primarily used to improve segmentation results. In this step, an opening operation (erosion followed by dilation) is first applied to remove small noisy areas and disconnect fine connections. A closing operation (dilation followed by erosion) is then used to fill small holes and smooth the edges. Finally, an edge detection algorithm (such as the Canny edge detector) is used to extract the finger contour. This series of operations produces a clear, continuous finger contour, providing a reliable foundation for subsequent feature extraction.

[0113] S44: Calculate the centroid of the finger contour in each frame of the depth image as the three-dimensional coordinates of the finger in the depth image of this frame.

[0114] Specifically, the center of mass calculation uses the image moment method, with the formula: x = M10 / M00, y = M01 / M00, where M00 is the area, and M10 and M01 are the first-order moments in the x and y directions, respectively. For the z coordinate (depth), the average or median depth of the finger area can be used. This method effectively represents the overall position of the finger and is insensitive to slight changes in finger posture, thereby improving tracking stability.

[0115] S45: Arranging the three-dimensional coordinates of the finger extracted from the continuous multi-frame depth image in time sequence to form a time-series three-dimensional coordinate sequence representing the movement of the user's finger as the time-series three-dimensional coordinate information of the user's finger.

[0116] Specifically, this step combines spatial and temporal information to form a basic data structure describing finger motion. Each coordinate point is accompanied by a timestamp, indicating its precise position within the motion sequence. This representation not only incorporates the spatial position of the finger but also implicitly captures dynamic characteristics such as velocity and acceleration, providing rich information for subsequent trajectory analysis and gesture recognition.

[0117] S46: performing smoothing processing on the time-series three-dimensional coordinate sequence to obtain a smoothed time-series three-dimensional coordinate sequence.

[0118] Specifically, the goal of smoothing is to reduce noise caused by measurement errors and minor hand tremors. A Savitzky-Golay filter can be used to effectively remove noise while preserving the signal's high-order moments. For each dimension (x, y, z), a 5-point Savitzky-Golay filter with a polynomial order of 2 is used. This method effectively smooths the trajectory while preserving rapidly changing details, making it suitable for processing complex signals such as finger motion, which contain both smooth and abrupt changes.

[0119] S47: Based on the smoothed time-series three-dimensional coordinate sequence, a three-dimensional spline interpolation algorithm is used to construct a continuous non-contact three-dimensional motion trajectory.

[0120] Specifically, cubic B-spline interpolation is used here, which generates smooth, continuous curves and offers local controllability. In its implementation, the smoothed coordinate sequence is used as control points, and multiple interpolation points are inserted between every two adjacent control points (for example, nine points are inserted between each two control points, increasing the sampling rate tenfold). This method not only produces visually smooth trajectories but also accurately reflects subtle changes in finger movement, providing high-quality input data for subsequent gesture recognition.

[0121] S48: Extracting features from the non-contact three-dimensional motion trajectory to obtain trajectory features. The purpose of feature extraction is to convert complex trajectory data into a set of discriminative feature vectors.

[0122] Specifically, the trajectory features extracted in this step include geometric features such as the trajectory's total length, average velocity, maximum acceleration, curvature, and torsion, as well as spatial features such as the trajectory's main direction and the normal vector to the plane of motion. Furthermore, Fourier descriptors can be used to capture the trajectory's overall shape. These features comprehensively describe various aspects of finger motion, providing rich information for subsequent gesture classification and recognition.

[0123] S49: Combining the trajectory feature with the smoothed time-series three-dimensional coordinate sequence to form a complete non-contact three-dimensional motion trajectory of the user's finger.

[0124] Specifically, this step integrates the discrete coordinate data and extracted high-level features to form a data structure that comprehensively describes the finger motion. Specifically, trajectory features can be attached as metadata to the time-series coordinate sequence. Each time point contains not only the x, y, and z coordinates but also the local and global feature values ​​corresponding to that point. This representation method preserves the original spatiotemporal information while also including an abstract feature description.

[0125] like Figure 6 As shown, in another preferred embodiment of the present invention, step S50 specifically includes the following steps:

[0126] S51: Retrieving a predefined contactless gesture library from the memory of the handheld tablet computer, wherein the predefined contactless gesture library includes a plurality of predefined three-dimensional gesture patterns and their corresponding interaction meanings.

[0127] Specifically, the predefined contactless gesture library stores a series of standardized three-dimensional gesture patterns. Each gesture pattern is associated with a specific interactive meaning, for example, "swiping right" can correspond to the "next page" operation, and "drawing a circle" can correspond to the "refresh" operation. The design of the gesture library needs to consider intuitiveness, ease of learning, and differentiation to ensure that users can easily master and accurately use these gestures. In practical applications, the gesture library may contain 20-30 commonly used gestures, each with its own unique three-dimensional trajectory characteristics and clear interactive meaning.

[0128] S52: Extract features of the non-contact three-dimensional motion trajectory of the user's finger to obtain a trajectory feature vector.

[0129] Specifically, feature extraction is the process of converting complex trajectory data into computable numerical features. This step can extract a variety of features, including but not limited to: the total length of the trajectory, average speed, acceleration change, main motion direction, curvature and torsion of the trajectory, bounding box size of the trajectory, etc. In addition, more advanced features can also be used, such as Fourier descriptors or wavelet transform coefficients. These features together constitute a high-dimensional feature vector that comprehensively describes the geometric and dynamic characteristics of the finger movement. For example, a feature vector can contain 50-100 elements, each element representing a specific attribute of the trajectory.

[0130] S53: performing similarity calculation on the trajectory feature vector and each predefined three-dimensional gesture pattern in the predefined contactless gesture library to obtain a similarity calculation result.

[0131] Specifically, the similarity calculation measures the degree of match between the gesture input by the user and the predefined gestures. Similarity calculation methods include Euclidean distance, cosine similarity, dynamic time warping (DTW), etc. In this embodiment, weighted Euclidean distance can be used as the similarity metric, and the weights of different features can be optimized by machine learning algorithms (such as support vector machines or random forests). The calculation process traverses each predefined gesture in the gesture library and calculates its similarity score with the input gesture. Assuming that the gesture library contains 25 predefined gestures, this step will generate a result vector containing 25 similarity scores.

[0132] S54: Based on the similarity calculation result, select the predefined three-dimensional gesture pattern with the highest similarity as the best matching gesture.

[0133] Specifically, this step finds the highest similarity score from the previous step and determines the corresponding predefined gesture as the best match. For example, if the "swipe right" gesture has the highest similarity score of 0.92, it is selected as the best match.

[0134] S55: Determine whether the similarity of the selected best matching gesture exceeds a preset threshold.

[0135] Specifically, this step introduces a safety mechanism to prevent false positives. The selection of the preset threshold requires a balance between recognition rate and rejection rate, determined through extensive experimental data. For example, the threshold can be set to 0.85. The system considers recognition reliable only when the similarity of the best matching gesture exceeds 0.85. The threshold setting can be adjusted based on the application scenario; higher thresholds can be used for scenarios with high security requirements.

[0136] S56: If the preset threshold is exceeded, the gesture recognition is confirmed to be successful, and the interaction meaning corresponding to the best matching gesture is obtained.

[0137] Specifically, when recognition is successful, the system extracts the corresponding interactive meaning of the gesture from a predefined gesture library. For example, if a "swipe right" gesture is recognized, the system retrieves the corresponding "next page" action. This interactive meaning is then passed to the corresponding module in the application to perform specific actions or trigger specific functions.

[0138] S57: If the gesture does not exceed the preset threshold, it is determined to be an unrecognizable gesture, and a prompt message is sent to the user, requiring the user to re-enter the gesture.

[0139] Specifically, this is the system's fault tolerance mechanism. When user input cannot be reliably recognized, the system will provide feedback. Prompt information can be visual (such as prompt text or icons on the screen) or auditory (such as a short beep). For example, the system may display a prompt saying "Your gesture was not recognized. Please try again" and provide some example illustrations of common gestures to help users improve their gesture input.

[0140] S58: Combining the best matching gesture, the corresponding similarity value and the corresponding interaction meaning to form a final pattern matching result.

[0141] Specifically, this step integrates key information from the recognition process into a complete result data structure. For example, a sample result includes the following information: best matching gesture = "Swipe right", similarity value = 0.92, interaction meaning = "Next page". This comprehensive result is not only used for the current interaction operation but also for continuous system optimization. For example, by recording these results, the system can analyze user gesture usage patterns, adjust recognition algorithms, or customize user gesture settings.

[0142] like Figure 7 As shown, in another preferred embodiment of the present invention, step S60 specifically includes the following steps:

[0143] S61: Analyze the pattern matching results and extract the best matching gesture and its corresponding interaction meaning.

[0144] Specifically, this step is a further processing of the recognition results of the previous stage. The system will extract key information from the pattern matching results, mainly including the best matching gesture and the corresponding interaction meaning. For example, if the pattern matching results show that the best matching gesture is "swipe up" and the corresponding interaction meaning is "scroll the page", then this information will be extracted to prepare for subsequent interaction operations. This process involves parsing the data structure, such as extracting specific fields from results in JSON or XML format. In actual applications, the system will also record this extracted information for user behavior analysis or system performance optimization.

[0145] S62: Determine a target area for user intended interaction based on the interaction meaning and the current spatial posture of the handheld tablet computer.

[0146] Specifically, this step combines the semantic information of the gesture with the physical state of the device to infer the specific area that the user wants to interact with. The spatial posture of a handheld tablet can be obtained through the built-in accelerometer and gyroscope. For example, if the user makes a "swipe up" gesture and the device is in a horizontal position, the system will infer that the user wants to scroll the entire currently displayed page. But if the device is tilted, the system will infer that the user only wants to scroll a part of the page. For another example, if the gesture is "zoom in" and the device is facing the user's face, the system will infer that the user wants to zoom in on the content in the center of the screen. This method of interpreting gestures and device postures can improve the accuracy and naturalness of interactions.

[0147] S63: Based on the association information between the interaction area and the data interaction function generated and stored in step S24, the data interaction function associated with the target area is acquired.

[0148] Specifically, this step utilizes the system's predefined mapping relationship between interactive areas and functions. In step S24, the system has already assigned specific data interaction functions to different interactive areas. Now, the system will search this mapping relationship to find the specific function corresponding to the target area. For example, if the target area is determined to be the upper right corner of the screen, and this area is defined as the "Close current application" function area in the mapping, then the system will obtain the "Close current application" function. This mapping relationship can be static or dynamic, changing according to the currently running application or the user's personalized settings.

[0149] S64: triggering execution of a data interaction function associated with the target area.

[0150] Specifically, this is the last step of the entire interaction process and the part that the user can directly perceive. The system will call the corresponding program interface or function to execute the function associated with the target area. Continuing with the above example, if the determined function is "close the current application", the system will trigger the application's shutdown process. This includes a series of operations such as saving the current work, releasing resources, and updating the user interface. When executing functions, the system can also provide visual, auditory, or tactile feedback to enhance the user's interactive experience. For example, when closing an application, a short animation or sound effect will be played. In addition, the system also records the results of this interaction for subsequent user behavior analysis or system optimization.

[0151] Through the above steps, the system implements a complete process from gesture recognition to specific function execution, providing users with an intuitive and natural contactless interaction experience. This approach not only considers the meaning of the gesture itself, but also incorporates the physical state of the device and predefined interaction areas, thereby achieving a more intelligent and contextually relevant interaction method.

[0152] like Figure 8 As shown, in another preferred embodiment of the present invention, step S70 specifically includes the following steps:

[0153] S71: According to the executed data interaction function, select the corresponding feedback mode from the preset multimodal feedback mode library. The multimodal feedback mode library is a comprehensive data set that contains multiple feedback modes such as tactile, visual, and auditory corresponding to various interactive functions. For example, for the slide to unlock function, it can correspond to a combination of tactile vibration and visual animation; and for the function of receiving new messages, it can correspond to a short sound prompt plus screen flashing and slight vibration. The design of this library takes into account the characteristics of human multi-sensory perception and provides a rich, intuitive and non-intrusive feedback experience. In actual applications, the multimodal feedback mode library can contain 50-100 basic modes, which can be flexibly combined or changed according to different interactive functions.

[0154] S72: Generate a corresponding multimodal feedback control signal based on the selected feedback mode. This step converts the abstract feedback mode into a specific control signal. The control signal may include vibration parameters for tactile feedback (e.g., frequency, amplitude, duration), animation parameters for visual feedback (e.g., color, brightness, animation duration), audio parameters for auditory feedback (e.g., volume, pitch, sound effect type), etc. For example, for a multimodal feedback of "successfully completed task", the control signal may specify a 50ms tactile vibration (at a frequency of 180Hz), trigger a green checkmark animation (lasting 0.5 seconds), and play a short "ding" sound effect (at a volume of 70% of the system volume). These precisely controlled parameters can create a rich and diverse perceptual experience, thereby enhancing the user's understanding and memory of different interactive operations.

[0155] S73: Sending the multimodal feedback control signal to the multimodal feedback device built into the handheld tablet computer. The multimodal feedback device may include multiple components, such as a linear vibration motor (tactile), a display (visual), and a speaker or piezoelectric buzzer (auditory). These devices are capable of generating corresponding feedback effects based on the received control signal. Signal transmission is accomplished via the device's internal bus or dedicated hardware interface, involving the coordination of multiple subsystems. During this process, the system must ensure the synchronization and consistency of various feedback signals to create a coordinated and consistent multimodal feedback experience.

[0156] S74: Trigger the multimodal feedback device to perform a corresponding feedback action. This is a step that the user can directly perceive. The multimodal feedback device generates corresponding tactile, visual, and auditory feedback simultaneously or in a specific order based on the received control signal. For example, for a notification of "receiving an important email", the device can first generate a short vibration, then display an animation of an envelope icon, and finally play a unique prompt sound. This multimodal feedback not only enhances the richness of the interaction, but also provides users with multi-level information transmission, especially in situations where certain single-modal feedback may be ignored (for example, in a noisy environment, you may not hear the sound prompt, but you can still feel the vibration).

[0157] S75: Monitor the execution status of the feedback action. This step introduces a comprehensive feedback monitoring mechanism to ensure the quality and consistency of multimodal feedback. Monitoring can include measuring actual vibration parameters, detecting changes in screen brightness, verifying audio output, etc. For example, the system can use an accelerometer to measure the actual vibration intensity, a light sensor to detect changes in screen brightness, and an audio loop to verify the quality of the sound output. These data can be used to evaluate the overall effect of multimodal feedback and provide a basis for subsequent adjustments.

[0158] S76: Dynamically adjust the parameters of the feedback action according to the execution status of the feedback action. This is an adaptive optimization process that is conducive to providing the best user experience. For example, if the ambient light is detected to be very strong, the system can increase the brightness of the visual feedback; if it is detected that the device is in silent mode, the system can enhance the intensity of the tactile feedback to compensate for the missing auditory feedback. This dynamic adjustment can adapt to different usage environments, hardware status or user preferences to ensure that multimodal feedback is always kept in the best state. In addition, the system can also learn the user's response pattern and gradually optimize the feedback strategy. For example, if it is found that the user often ignores a certain type of feedback, the system will adjust the parameters of the feedback or switch to other more effective feedback modes.

[0159] Through the above steps, the embodiment of the present invention implements an intelligent, adaptive multimodal feedback system that provides rich, intuitive sensory feedback for different interactive operations and dynamically adjusts to actual conditions to ensure an optimal user experience. This refined multimodal feedback design greatly enhances the effectiveness and usability of contactless interaction, enabling users to accurately understand and respond to system feedback through multiple sensory channels, thereby improving interaction efficiency and satisfaction.

[0160] like Figure 9 As shown, in another preferred embodiment of the present invention, the method further comprises the following steps:

[0161] S80: Utilizing the built-in memory of the handheld tablet computer, record user interaction data, including interaction frequency, interaction duration, target interaction area, executed data interaction functions, and delivered tactile feedback. This step establishes a comprehensive user behavior data collection mechanism. For example, the system records the number of times a user opens an app per day (interaction frequency), the duration of each app session (interaction duration), the screen areas most frequently clicked (target interaction areas), the functions most frequently used (executed data interaction functions), and the user's response to various tactile feedback. This data provides rich raw material for subsequent deep learning and optimization. Specifically, the system records data such as: users open social apps an average of 15 times per day, each session lasting approximately 5 minutes; they most frequently click the "New Message" icon in the upper right corner of the screen; they frequently use the "Send Emoji" function; and they react more positively to strong vibration feedback.

[0162] S90: Based on the user's interactive operation data, a deep learning algorithm is employed to dynamically optimize the data interaction functions associated with each interactive area. This step utilizes advanced artificial intelligence technology to transform the collected user data into an actual optimization strategy. Deep learning algorithms can discover underlying patterns and trends from large amounts of interaction data, thereby customizing the optimal interactive experience for each user.

[0163] like Figure 10 As shown, in some embodiments, step S90 specifically includes the following sub-steps:

[0164] S91: Input the user's interactive operation data into a pre-trained deep learning model, and output a usage feature vector for the data interaction functions within each interactive area. The deep learning model here is a multi-layer neural network that, trained on a large amount of user data, can understand complex interaction patterns. For example, the deep learning model finds that users tend to check the news in the morning and use social functions more in the evening. The output feature vector can include multi-dimensional information such as frequency of use, time period of use, and correlation with other functions.

[0165] S92: Based on the usage feature vector output by the deep learning model, calculate the priority index of each data interaction function in each interaction area. The priority index is a comprehensive score that reflects the importance and convenience of a function to the user. The calculation can take into account multiple factors, such as frequency of use, recent use time, and duration of use. For example, if the user frequently uses the "Document Editing" function and each use time is long, it will receive a higher priority index, while the rarely used "System Settings" function will receive a lower priority index.

[0166] S93: Based on the priority index, generate a priority ranking of data interaction functions for each interactive area. This step converts the priority index calculated in S92 into a specific ranking. For example, for the home screen's shortcut function area, the ranking might be: Messages, Mail, Calendar, Camera, Settings. This ranking reflects user usage habits and preferences, providing direct guidance for subsequent interface optimization.

[0167] S94: Based on the priority ranking, adjust the layout, display order, or triggering method of the data interaction functions associated with each interactive area. This is a key step in converting the analysis results into actual optimization. For example, the system will place the top-ranked functions in a more accessible location (such as the bottom or edge of the screen), increase their icon size, or set a more convenient triggering method for them (such as a swipe gesture). For less frequently used functions, they will be moved to a secondary menu or have their icon size reduced. This adjustment is intended to improve user operation efficiency and comfort.

[0168] S95: Update the adjusted data interaction function configuration to each interaction area on the screen of the handheld tablet computer to achieve dynamic optimization of the data interaction functions associated with each interaction area. This step actually applies the adjustments in S94 to the device interface. Updates may include rearranging application icons on the home screen, adjusting the order of options in the quick settings panel, changing the default functions of certain gestures, etc. For example, if the system finds that the user often checks the weather information in the morning, it will automatically push the weather application to a prominent position on the home screen every morning. This dynamic optimization ensures that the user interface can continuously adapt to the user's usage habits and provide a personalized and efficient interactive experience.

[0169] This approach can continuously learn users' usage habits and adjust the interface layout and function configuration in real time, thereby providing each user with a highly personalized and efficient interactive experience.

[0170] Example 2

[0171] like Figure 11 As shown, an embodiment of the present invention further provides a data interaction system for a handheld tablet computer, which includes the following components:

[0172] The attitude sensor is used to detect the spatial attitude of the handheld tablet computer relative to the direction of gravity. This sensor is an integrated inertial measurement unit (IMU) that includes an accelerometer, a gyroscope, and possibly a magnetometer. It can measure the device's tilt angle, rotation speed, and direction in real time. For example, when the user rotates the tablet computer from a horizontal to a vertical position, the attitude sensor can accurately detect this change and provide important contextual information to the system. This is crucial for implementing features such as automatic screen rotation and attitude-based interactive control.

[0173] A depth camera is used to acquire a sequence of depth images of the user's finger movements in the space around the handheld tablet in real time. This camera uses structured light or Time of Flight (ToF) technology to capture three-dimensional spatial information. It can not only identify the position of the user's finger, but also measure its distance from the device. For example, the depth camera can capture a "swipe" gesture made by the user 10 cm above the device, even if the finger does not touch the screen. This provides the basis for precise contactless interaction, allowing users to perform intuitive operations in the space around the device.

[0174] A multimodal feedback device is used to provide users with sensory feedback corresponding to the data interaction function performed. This device may include a tactile feedback device (such as a linear vibration motor), a visual feedback system (such as an LED indicator or screen animation), and an auditory feedback system (such as a speaker or piezoelectric buzzer). For example, when a user successfully completes a contactless gesture, the device will generate a short vibration, a confirmation icon will flash on the screen, and a gentle sound effect will play. This multimodal feedback can provide users with a rich and intuitive interactive experience, enhancing the certainty and satisfaction of the operation.

[0175] A memory electrically connected to the processor for storing and executing the above Figures 1 to 9 The memory can be a high-speed flash memory or solid-state drive with sufficient storage capacity to accommodate the operating system, applications, and user data. It not only stores the core algorithms and programs for contactless interaction, but also includes information such as user personalized settings and usage history. For example, the memory may contain key data such as gesture recognition algorithms, user interface layout configurations, and deep learning models.

[0176] A processor is electrically connected to the posture sensor, the depth camera, the memory and the data storage device, and is used to execute the program in the memory to achieve the above Figures 1 to 9Any of the methods described above. This processor can be a high-performance multi-core ARM processor or similar mobile processor with sufficient computing power to handle complex gesture recognition and deep learning tasks in real time. For example, it can complete the entire process from receiving sensor data, analyzing user gestures, and triggering corresponding functions in milliseconds, ensuring smooth and responsive interaction.

[0177] A screen, electrically connected to the processor, is configured to display the multiple interactive areas and the results of executing data interaction functions. This high-resolution, high-refresh-rate screen is capable of accurately displaying complex user interface elements and smooth animations. For example, the screen can clearly display the divided interactive areas and provide real-time visual feedback when a user performs contactless gestures, such as highlighting the activated area or displaying the results of the executed function.

[0178] A server, electrically connected to the processor, is used to store a library of predefined contactless gestures, a library of data interaction functions, and a library of multimodal feedback patterns. This server, which can be a cloud-based system, not only stores a large number of predefined interaction patterns and functions but also continuously optimizes and expands these libraries through machine learning. For example, when the system recognizes a new, valid user gesture, it can be added to the gesture library; or the priority of certain functions can be dynamically adjusted based on usage data from user groups. This cloud-based storage and processing mechanism enables the system to continuously learn and evolve, providing users with an increasingly intelligent and personalized interaction experience.

[0179] Through the collaborative operation of these components, the handheld tablet computer data interaction system of the present invention enables highly intuitive, natural, and personalized contactless interaction. The system leverages advanced sensing technology to accurately capture user operational intent, employs powerful processing capabilities to analyze and respond in real time, and provides users with a rich sensory experience through multimodal feedback. Furthermore, leveraging the storage and computing power of cloud servers, the system continuously learns and optimizes to adapt to the habits and preferences of diverse users. This design not only improves the efficiency and accuracy of interaction, but also significantly enhances the comfort and enjoyment of the user experience.

[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0181] Example 3

[0182] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements any one of the above methods when executed by a processor.

[0183] If the integrated modules / units are implemented as software functional units and sold or used as standalone products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. Of course, there are other types of readable storage media, such as quantum memory and graphene memory. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0184] Example 4

[0185] The present invention also provides an electronic device. The electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the present invention.

[0186] Reference below Figure 12 , which shows a schematic structural diagram of a computer system 800 of an electronic device suitable for implementing an embodiment of the present invention. Figure 12 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0187] like Figure 12 As shown, computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage unit 208 into a random access memory (RAM) 203. Various programs and data required for the operation of computer system 200 are also stored in RAM 203. CPU 201, ROM 202, and RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to bus 204.

[0188] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, and the like; an output section 207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 208 including devices such as a hard disk; and a communication section 209 including a network interface card such as a LAN card or a modem. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. Removable media 211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 210 as needed, so that computer programs read from the media can be installed in the storage section 208 as needed.

[0189] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0190] The units involved in the embodiments of the present invention may be implemented in software or hardware. The units described may also be provided in a processor, and the names of these units do not, in some cases, limit the units themselves.

[0191] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data interaction method for a handheld tablet computer, characterized in that: The following steps are involved: S10: Detecting the spatial posture of the handheld tablet computer relative to the gravity direction by using a posture sensor built into the handheld tablet computer; S20: dividing the screen of the handheld tablet computer into a plurality of interactive areas according to the detected spatial posture of the handheld tablet computer relative to the gravity direction, each interactive area being associated with a corresponding data interactive function; S30: using the depth camera built into the handheld tablet computer to obtain in real time a depth image sequence of the user's finger moving in the space around the handheld tablet computer; S40: extracting the time-series three-dimensional coordinate information of the user's finger through an image processing algorithm based on the depth image sequence, and constructing a non-contact three-dimensional motion trajectory of the user's finger according to the time-series three-dimensional coordinate information; S50: performing pattern matching on the non-contact three-dimensional motion trajectory of the user's finger and a predefined non-contact gesture library stored in the handheld tablet computer to obtain a pattern matching result; S60: determining a target interaction area for user intended interaction and a data interaction function associated with the target interaction area according to the pattern matching result, and triggering execution of the data interaction function associated with the target interaction area; S70: In response to the execution of the data interaction function, transmitting perception feedback corresponding to the executed data interaction function to the user through the multimodal feedback device built into the handheld tablet computer; Step S20 specifically includes: S21: determining an inclination angle of a screen plane of the handheld tablet computer relative to a horizontal plane according to a spatial posture of the handheld tablet computer relative to a gravity direction; S22: selecting a preset screen division template based on the tilt angle, and dividing the screen of the handheld tablet computer into a plurality of interaction areas of geometric shapes according to the selected screen division template; S23: selecting a corresponding data interaction function for each interaction area from a preset data interaction function library according to the current application scenario and user usage habits; S24: Generate and store association information between the interaction area and the data interaction function; S25: Visually displaying the divided interactive areas and their associated data interactive functions on the screen of the handheld tablet computer; Step S30 specifically includes: S31: activating the depth camera built into the handheld tablet computer, and adaptively adjusting the working parameters of the depth camera according to the current ambient light conditions and the user distance to obtain adjusted working parameters; the adjusted working parameters include frame rate, resolution and depth of field; S32: defining a three-dimensional space detection area around the handheld tablet computer, where the three-dimensional space detection area covers a range in which a user may perform gesture interaction; S33: Based on the adjusted working parameters, continuously capturing multiple frames of depth images within the three-dimensional space detection area within a preset time interval; S34: Processing each captured depth image frame based on an image enhancement algorithm to obtain a processed depth image; S35: organizing the processed depth images into a depth image sequence in time order; S36: Performing time synchronization and space registration on the depth image sequence to generate a temporally and spatially consistent depth image sequence as a real-time depth image sequence representing the movement of the user's finger in the surrounding space of the handheld tablet computer.

2. The method according to claim 1, characterized in that Step S10 specifically includes: S11: configuring working parameters of a posture sensor built into the handheld tablet computer, wherein the posture sensor includes an accelerometer, a gyroscope, and a magnetometer; S12: Acquiring acceleration data of the handheld tablet computer in three orthogonal axes through the accelerometer; S13: Acquiring angular velocity data of the handheld tablet computer around three orthogonal axes through the gyroscope; S14: obtaining direction data of the handheld tablet computer relative to the geomagnetic field by using the magnetometer; S15: using a Kalman filter algorithm or a complementary filter algorithm, fusing the acceleration data from the accelerometer, the angular velocity data from the gyroscope, and the direction data from the magnetometer to obtain optimized attitude estimation data; S16: Calculating the pitch angle, roll angle and yaw angle of the handheld tablet computer relative to the gravity direction based on the optimized posture estimation data; S17: Determine the spatial posture of the handheld tablet computer relative to the gravity direction according to the calculated pitch angle, roll angle and yaw angle.

3. The method according to claim 1, characterized in that Step S40 specifically includes: S41: preprocessing the acquired depth image sequence to obtain a preprocessed depth image sequence; S42: extracting the user's finger area in each frame of the preprocessed depth image by using a depth threshold segmentation algorithm; S43: performing morphological processing on the user's finger area to obtain a finger contour; S44: Calculate the centroid of the finger contour in each frame of the depth image as the three-dimensional coordinates of the finger in the depth image of this frame; S45: arranging the three-dimensional coordinates of the finger extracted from the continuous multiple-frame depth image in time order to form a time-series three-dimensional coordinate sequence representing the movement of the user's finger as the time-series three-dimensional coordinate information of the user's finger; S46: performing smoothing processing on the time series three-dimensional coordinate sequence to obtain a smoothed time series three-dimensional coordinate sequence; S47: Based on the smoothed time-series three-dimensional coordinate sequence, a three-dimensional spline interpolation algorithm is used to construct a continuous non-contact three-dimensional motion trajectory; S48: extracting features of the non-contact three-dimensional motion trajectory to obtain trajectory features; S49: combining the trajectory feature with the smoothed time-series three-dimensional coordinate sequence to form a complete non-contact three-dimensional motion trajectory of the user's finger.

4. The method according to claim 1, characterized in that Step S50 specifically includes: S51: Retrieving a predefined contactless gesture library from the memory of the handheld tablet computer, wherein the predefined contactless gesture library includes a plurality of predefined three-dimensional gesture patterns and their corresponding interaction meanings; S52: extracting features of the non-contact three-dimensional motion trajectory of the user's finger to obtain a trajectory feature vector; S53: performing similarity calculation on the trajectory feature vector and each predefined three-dimensional gesture pattern in the predefined non-contact gesture library to obtain a similarity calculation result; S54: Based on the similarity calculation result, select the predefined three-dimensional gesture pattern with the highest similarity as the best matching gesture; S55: Determine whether the similarity of the selected best matching gesture exceeds a preset threshold; S56: If the preset threshold is exceeded, confirming that the gesture recognition is successful, and obtaining the interaction meaning corresponding to the best matching gesture; S57: If the value does not exceed the preset threshold, the gesture is determined to be unrecognizable, and a prompt message is sent to the user to request the user to re-enter the gesture; S58: Combining the best matching gesture, the corresponding similarity value and the corresponding interaction meaning to form a final pattern matching result.

5. The method according to claim 4, characterized in that Step S60 specifically includes: S61: parsing the pattern matching result, extracting the best matching gesture and its corresponding interaction meaning; S62: determining a target area for the user's intended interaction according to the interaction meaning and the current spatial posture of the handheld tablet computer; S63: acquiring the data interaction function associated with the target area based on the association information between the interaction area and the data interaction function generated and stored in step S24; S64: triggering execution of a data interaction function associated with the target area.

6. The method according to claim 1, characterized in that Step S70 specifically includes: S71: selecting a corresponding feedback mode from a preset multi-modal feedback mode library according to the executed data interaction function; S72: generating a corresponding multi-modal feedback control signal according to the selected feedback mode; S73: sending the multimodal feedback control signal to a multimodal feedback device built into the handheld tablet computer; S74: triggering the multimodal feedback device to perform a corresponding feedback action; S75: monitoring the execution status of the feedback action; S76: Dynamically adjust the parameters of the feedback action according to the execution status of the feedback action.

7. The method according to claim 1, characterized in that Also includes: S80: using the built-in memory of the handheld tablet computer, recording the user's interactive operation data, which includes: interaction frequency, interaction duration, target interaction area, executed data interaction function, and delivered tactile feedback; S90: Based on the interactive operation data of the user, a deep learning algorithm is used to dynamically optimize the data interaction functions associated with each interactive area.

8. A data interaction system for a handheld tablet computer, characterized in that: include: A posture sensor, used for detecting the spatial posture of the handheld tablet computer relative to the direction of gravity; A depth camera, used for acquiring in real time a depth image sequence of the movement of the user's finger in the space around the handheld tablet computer; A multimodal feedback device for delivering to the user perceptual feedback corresponding to the data interaction function performed; A memory, electrically connected to the processor, for storing program instructions for executing the method according to any one of claims 1 to 7; A processor, electrically connected to the gesture sensor, the depth camera and the memory, and configured to execute a program in the memory to implement the method according to any one of claims 1 to 7; a screen, electrically connected to the processor, for displaying the plurality of interactive areas and the results of executing the data interactive function; The server is electrically connected to the processor and is used to store a predefined contactless gesture library, a data interaction function library and a multimodal feedback mode library.

Images