Data interaction method and system of handheld tablet personal computer

Abstract

The invention discloses a data interaction method and system of a handheld tablet personal computer, which are applied to the technical field of man-machine interaction, the method detects the spatial attitude of equipment relative to the gravity direction through a built-in attitude sensor, a screen is intelligently divided into a plurality of interaction areas according to the spatial attitude, and each area is associated with a specific data interaction function. A depth camera is used for acquiring a motion depth image sequence of a finger of a user in a space around equipment in real time, time sequence three-dimensional coordinate information is extracted through an image processing algorithm, and a non-contact three-dimensional motion track is constructed. And performing mode matching on the track and a predefined non-contact gesture library, determining a target area and related functions of user intention interaction, and triggering execution. In the execution process, corresponding perceptual feedback, such as tactile, visual or auditory signals, is transmitted to the user through the multi-mode feedback device. According to the method, intelligent non-contact interaction based on the equipment posture and the user gesture is realized, the intuition and the efficiency of operation are improved, and the user experience is remarkably enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a data interaction method and system for a handheld tablet computer. Background Art

[0002] With the rapid development of mobile computing technology, handheld tablet computers have become an indispensable smart device in people's daily life and work. Traditional handheld tablet computers mainly rely on touch screens for human-computer interaction. This interaction method has limitations in some scenarios, such as when the user's hands are inconvenient or the device surface is dirty. In addition, traditional interaction methods often require users to directly touch the device, which may not be convenient or hygienic in some special environments.

[0003] Although there are some non-contact interaction solutions in the prior art, most of them have the problem of limited interaction space. These solutions can usually only be operated in a fixed area directly in front of the device, and it is difficult to adapt to the user's usage needs in different postures and environments. For example, when the user uses the device in different states such as lying, standing or walking, the fixed interaction area may cause problems such as inconvenient operation or accidental touch. This lack of flexibility seriously affects the user experience and efficiency of the device.

[0004] Another significant problem is that the functions of existing contactless interaction solutions are often too simple. Most solutions can only achieve simple control functions, such as page scrolling, zooming in and out, and other basic operations. However, as users' demands for device functions become increasingly complex, this simple interaction method can no longer meet users' needs for complex task operations. For example, when performing multitasking, complex document editing, or professional software operations, existing contactless interaction solutions are often unable to cope with the situation.

[0005] Insufficient interaction feedback is another important problem faced by existing non-contact interaction solutions. In traditional touch screen interactions, users can directly sense the execution of operations through touch. However, in non-contact interactions, due to the lack of physical contact, users often find it difficult to accurately judge whether their operations are correctly recognized and executed by the device. This lack of feedback not only leads to reduced operating efficiency, but may also cause user frustration, thus affecting the overall user experience.

[0006] In addition, existing contactless interaction solutions generally have the problem of poor adaptability. Most solutions cannot automatically adjust the interaction area and recognition algorithm according to the different usage postures of the device. Users need to deliberately maintain a certain usage posture or manually adjust the device settings to ensure the accuracy of the interaction. This lack of intelligent adaptation mechanism seriously limits the usage scenarios and flexibility of the device.

[0007] Finally, the accuracy and stability of gesture recognition in existing technologies still need to be improved. Many contactless interaction solutions use ordinary cameras or simple infrared sensors, which often perform poorly in complex lighting environments or when recognizing fast movements. This leads to a high error rate in gesture recognition, and users need to repeat the action multiple times to complete the required operation, which greatly reduces the smoothness of the interaction and the user experience.

[0008] Therefore, there is an urgent need for a handheld tablet computer data interaction method that can overcome the above problems to provide a more intelligent, flexible, intuitive and feature-rich interaction experience. Summary of the invention

[0009] In view of this, the purpose of the embodiments of the present invention is to provide a data interaction method and system for a handheld tablet computer, so as to achieve more intelligent, flexible and intuitive human-computer interaction based on non-contact gesture recognition, thereby improving the use efficiency and user experience of the handheld tablet computer.

[0010] To achieve the above object, in a first aspect, a data interaction method for a handheld tablet computer is provided, which comprises the following steps: S10: Detecting the spatial posture of the handheld tablet computer relative to the gravity direction by using a posture sensor built into the handheld tablet computer; S20: dividing the screen of the handheld tablet computer into a plurality of interactive areas according to the detected spatial posture of the handheld tablet computer relative to the gravity direction, each interactive area being associated with a corresponding data interactive function; S30: using the depth camera built into the handheld tablet computer to obtain in real time a depth image sequence of the user's finger moving in the space around the handheld tablet computer; S40: extracting the time-series three-dimensional coordinate information of the user's finger through an image processing algorithm based on the depth image sequence, and constructing a non-contact three-dimensional motion trajectory of the user's finger according to the time-series three-dimensional coordinate information; S50: performing pattern matching on the non-contact three-dimensional motion trajectory of the user's finger and a predefined non-contact gesture library stored in the handheld tablet computer to obtain a pattern matching result; S60: determining a target interaction area for user intended interaction and a data interaction function associated with the target interaction area according to the pattern matching result, and triggering execution of the data interaction function associated with the target interaction area; S70: In response to the execution of the data interaction function, a perceptual feedback corresponding to the executed data interaction function is transmitted to the user through a multimodal feedback device built into the handheld tablet computer.

[0011] In a second aspect, a data interaction system for a handheld tablet computer is provided, comprising: A posture sensor, used for detecting the spatial posture of the handheld tablet computer relative to the direction of gravity; A depth camera, used for acquiring in real time a depth image sequence of the movement of the user's finger in the surrounding space of the handheld tablet computer; a multimodal feedback device for delivering to the user perceptual feedback corresponding to the data interaction function performed; A memory, electrically connected to the processor, and configured to store program instructions for executing any one of the methods of the first aspect; a processor, electrically connected to the gesture sensor, the depth camera, the memory and the data storage device, and configured to execute a program in the memory to implement any one of the methods described in the first aspect; a screen, electrically connected to the processor, for displaying the plurality of interactive areas and the results of executing the data interactive function; The server is electrically connected to the processor and is used to store a predefined contactless gesture library, a data interaction function library and a multimodal feedback mode library.

[0012] The above technical solution has the following beneficial technical effects: The present invention detects the spatial posture of the device relative to the gravity direction through the built-in posture sensor of the handheld tablet computer, and dynamically divides the screen interaction area based on it, thereby realizing adaptive adjustment of the interaction area. This method greatly improves the operational flexibility of the device under different usage postures, allowing users to interact conveniently whether they are standing, sitting or lying, and significantly improves the adaptability of the device and user experience.

[0013] The present invention uses a depth camera to obtain the motion information of the user's finger in the space around the device in real time, and extracts the time-series three-dimensional coordinate information of the finger through an image processing algorithm to construct a non-contact three-dimensional motion trajectory, greatly expanding the interactive space. This not only breaks through the plane limitation of traditional touch screens, but also provides users with a more natural and ergonomic operation mode, reducing fatigue caused by long-term use.

[0014] The present invention matches the non-contact three-dimensional motion trajectory of the user's finger with a predefined non-contact gesture library, which significantly improves the accuracy and richness of gesture recognition. Through a carefully designed gesture library, the system can recognize complex multi-dimensional gestures, thereby supporting more diverse operating instructions and meeting users' needs for advanced functions and complex tasks.

[0015] The present invention determines the target area of ​​user's intended interaction and the associated data interaction function according to the pattern matching result, and triggers the execution of the corresponding function. This intelligent interaction mechanism greatly improves the operation efficiency. Users do not need to perform cumbersome menu navigation or multi-step operations, but can quickly complete complex tasks with simple gestures, which greatly simplifies the operation process.

[0016] The present invention introduces a multimodal feedback mechanism, which transmits perceptual feedback corresponding to the executed function to the user through the multimodal feedback device built into the device. This instant and intuitive feedback greatly enhances the user's sense of certainty in operation, effectively solves the common problem of suspended feeling in contactless interaction, and improves the accuracy of interaction and the user's confidence in operation.

[0017] The data interaction method proposed in the present invention realizes the organic combination of device posture perception, three-dimensional gesture recognition, intelligent function mapping and multi-modal feedback, and builds an all-round contactless interaction system. This not only significantly improves the operational convenience and hygiene of handheld tablet computers, but also provides solutions for use in special environments (such as medical and industrial scenarios where the equipment needs to be kept clean), greatly expanding the application scope and scenarios of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. Figure 1 is a flow chart of a data interaction method for a handheld tablet computer according to an embodiment of the present invention; Figure 2 is a specific flow chart of step S10 of an embodiment of the present invention; Figure 3 is a specific flow chart of step S20 of an embodiment of the present invention; Figure 4 is a specific flow chart of step S30 of an embodiment of the present invention; Figure 5 is a specific flow chart of step S40 of an embodiment of the present invention; Figure 6 is a specific flow chart of step S50 of an embodiment of the present invention; Figure 7 is a specific flow chart of step S60 of an embodiment of the present invention; Figure 8 is a specific flow chart of step S70 of an embodiment of the present invention; Fig. 9 is a flow chart of another data interaction method for a handheld tablet computer according to an embodiment of the present invention; Fig.10 is a specific flow chart of step S90 of an embodiment of the present invention; Fig.11It is a functional block diagram of a data interaction system for a handheld tablet computer according to an embodiment of the present invention; Fig.12 It is a schematic diagram of the structure of a computer system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0020] Embodiment 1 like Figure 1 As shown, this embodiment provides a data interaction method for a handheld tablet computer, which includes the following steps: S10: Detecting the spatial posture of the handheld tablet computer relative to the gravity direction by using a posture sensor built into the handheld tablet computer; S20: dividing the screen of the handheld tablet computer into a plurality of interactive areas according to the detected spatial posture of the handheld tablet computer relative to the gravity direction, each interactive area being associated with a corresponding data interactive function; S30: using the depth camera built into the handheld tablet computer to obtain in real time a depth image sequence of the user's finger moving in the space around the handheld tablet computer; S40: extracting the time-series three-dimensional coordinate information of the user's finger through an image processing algorithm based on the depth image sequence, and constructing a non-contact three-dimensional motion trajectory of the user's finger according to the time-series three-dimensional coordinate information; S50: performing pattern matching on the non-contact three-dimensional motion trajectory of the user's finger and a predefined non-contact gesture library stored in the handheld tablet computer to obtain a pattern matching result; S60: determining a target interaction area for user intended interaction and a data interaction function associated with the target interaction area according to the pattern matching result, and triggering execution of the data interaction function associated with the target interaction area; S70: In response to the execution of the data interaction function, a multimodal feedback device built into the handheld tablet computer is used to transmit sensory feedback corresponding to the executed data interaction function to the user. The sensory feedback may include visual, auditory, optical or other non-contact sensory signals.

[0021] The handheld tablet computer data interaction method based on non-contact gesture recognition proposed in the embodiment of the present invention realizes the organic combination of device posture perception, three-dimensional gesture recognition, intelligent function mapping and multi-modal feedback, and builds an all-round non-contact interaction system. This not only significantly improves the convenience and hygiene of handheld tablet computers, but also provides solutions for use in special environments (such as medical and industrial scenarios where the equipment needs to be kept clean), greatly expanding the application scope and scenarios of the equipment.

[0022] like Figure 2 As shown, in a preferred embodiment of the present invention, step S10 specifically includes the following steps: S11: configuring working parameters of a posture sensor built into the handheld tablet computer, wherein the posture sensor includes an accelerometer, a gyroscope, and a magnetometer.

[0023] Specifically, this step first configures the built-in attitude sensors of the handheld tablet appropriately. These sensors include accelerometers, gyroscopes, and magnetometers, which work together to provide comprehensive attitude information of the device. Configuring working parameters includes setting sampling rate, range, and filtering parameters. For example, the sampling rate of the accelerometer can be set to 100Hz, and the range is ±2g; the sampling rate of the gyroscope can be set to 200Hz, and the range is ±250° / s; the sampling rate of the magnetometer can be set to 50Hz, and the range is ±4800μT. The selection of these parameters requires a balance between measurement accuracy and power consumption to adapt to different application scenarios.

[0024] S12: Acquire acceleration data of the handheld tablet computer in three orthogonal axes through the accelerometer.

[0025] Specifically, the accelerometer can measure the acceleration of the device in three orthogonal axes: X, Y, and Z. These data not only include the acceleration generated by the movement of the device, but also the component of gravity acceleration. By analyzing these data, the tilt angle of the device can be preliminarily estimated. For example, when the device is stationary, the distribution of gravity acceleration in the three axes directly reflects the tilt state of the device.

[0026] S13: Acquiring angular velocity data of the handheld tablet computer around three orthogonal axes through the gyroscope; Specifically, the gyroscope measures the rotation speed of the device around the three axes of X, Y, and Z. These angular velocity data are very important for capturing the rapid attitude changes of the device, and can provide faster and more accurate short-term attitude change information than the accelerometer. By integrating the angular velocity data, this step can obtain the change in the device's attitude.

[0027] S14: Acquire direction data of the handheld tablet computer relative to the Earth's magnetic field through the magnetometer.

[0028] Specifically, a magnetometer measures the components of the Earth's magnetic field in three axes. These data are mainly used to determine the absolute orientation of the device relative to the Earth, especially when calculating the yaw angle (the horizontal rotation angle of the device relative to magnetic north). However, magnetometers are easily affected by metal objects or electromagnetic interference in the surrounding environment, so proper calibration and compensation are required when using them.

[0029] S15: Using a Kalman filter algorithm or a complementary filter algorithm, the acceleration data from the accelerometer, the angular velocity data from the gyroscope, and the direction data from the magnetometer are integrated to obtain optimized attitude estimation data.

[0030] Specifically, the Kalman filter algorithm or complementary filter algorithm can effectively fuse data from different sensors, making full use of the advantages of each sensor while offsetting their disadvantages. For example, the accelerometer provides accurate tilt angles in a stationary state, but is sensitive to vibration; the gyroscope can accurately capture fast movements, but there is integration drift; the magnetometer provides an absolute direction reference, but is susceptible to external interference. Through these algorithms, a more stable and accurate attitude estimation can be obtained.

[0031] S16: Based on the optimized posture estimation data, the pitch angle, roll angle and yaw angle of the handheld tablet computer relative to the gravity direction are calculated.

[0032] Using the optimized posture estimation data, three key angles of the device can be calculated: Pitch, which is the angle at which the device tilts forward and backward; Roll angle, which is the angle at which the device tilts left or right; Yaw, which is the rotation angle of the device in the horizontal plane.

[0033] These angles are obtained by converting quaternions or rotation matrices. For example, when using quaternions (q0, q1, q2, q3), they can be calculated using the following formula: ; .

[0034] S17: Determine the spatial posture of the handheld tablet computer relative to the gravity direction according to the calculated pitch angle, roll angle and yaw angle.

[0035] Finally, this step uses the three calculated angles to determine the complete spatial pose of the device. This spatial pose can be represented by a 3x3 rotation matrix or quaternion, which fully describes the orientation of the device in three-dimensional space. This representation method is not only intuitive, but also convenient for subsequent calculations and processing.

[0036] Through the above detailed steps, the embodiment of the present invention can accurately detect the spatial posture of the handheld tablet computer relative to the direction of gravity. This precise posture detection provides a reliable basis for the subsequent interactive area division and gesture recognition, so that the system can dynamically adjust the interactive mode according to the actual use status of the device, thereby greatly improving the flexibility and adaptability of the user experience. For example, the system can automatically switch between horizontal and vertical screen modes according to the detected posture, or adjust the screen brightness according to the tilt angle of the device, to achieve a more intelligent and humanized interactive experience.

[0037] like Figure 3 As shown, in a preferred embodiment of the present invention, step S20 specifically includes the following steps: S21: Determine the inclination angle of the screen plane of the handheld tablet computer relative to the horizontal plane according to the spatial posture of the handheld tablet computer relative to the gravity direction.

[0038] In this step, the spatial posture information obtained in step S10 is used to calculate the tilt angle of the screen plane relative to the horizontal plane. Specifically, the pitch angle and the roll angle can be used to determine this tilt angle. For example, if the pitch angle is 30° and the roll angle is 0°, the tilt angle of the screen plane relative to the horizontal plane is 30°. This tilt angle information is crucial for subsequent screen division, because different usage postures may require different interactive area layouts.

[0039] S22: Based on the tilt angle, a preset screen division template is selected, and the screen of the handheld tablet computer is divided into a plurality of interaction areas of geometric shapes according to the selected screen division template.

[0040] Based on the calculated tilt angle, the system will select the template that best suits the current usage posture from the pre-designed screen division template library. For example, when the tilt angle is between 0-15°, select a four-division template suitable for desktop use; when the tilt angle is between 45-60°, select a triangular division template suitable for one-handed holding. After selecting a template, the system will divide the screen into multiple geometric interaction areas based on the template. These geometric shapes can be rectangles, triangles, circles, etc., providing the best interaction experience for different usage scenarios.

[0041] S23: According to the current application scenario and user usage habits, a corresponding data interaction function is selected for each interaction area from a preset data interaction function library.

[0042] In this step, the system will consider the current application scenario (such as office, entertainment, reading, etc.) and the user's usage habits, and assign the most appropriate data interaction function to each divided interaction area. These functions come from a preset data interaction function library, which contains various commonly used operations such as clicking, sliding, zooming, rotating, etc. For example, in the reading scenario, the area on the right side of the screen can be assigned to the page turning function, and the area on the left side can be assigned to the return to directory function. The system can also learn the user's usage habits and adjust the function allocation over time to provide a more personalized experience.

[0043] S24: Generate and store association information between the interaction area and the data interaction function.

[0044] After the function assignment is completed, the system will generate a mapping table or data structure to store the association information between each interactive area and its corresponding data interaction function. This association information can include the geometric parameters of the area (such as coordinates, size, shape), the assigned function type, trigger conditions, etc. For example, the storage format is: {area ID: 1, shape: rectangle, coordinates: (0,0,100,100), function: slide to turn pages, trigger condition: single finger slide}. This information is stored in the device's memory or persistent storage for subsequent quick access and update.

[0045] S25: Visually displaying the divided interactive areas and their associated data interactive functions on the screen of the handheld tablet computer.

[0046] Finally, the system will present the divided interactive areas and their associated functions on the screen in a visual way so that users can intuitively understand the current interactive layout. This visualization can take many forms, such as slight color distinction, semi-transparent boundary lines, etc. For example, in a reading application, the system can display a faint arrow icon on the right side of the screen to prompt the user that this area can be used to turn pages.

[0047] In the scenario of contactless gesture interaction, the system uses intelligent and intuitive ways to visualize the interaction area and its functions. This can include the following methods: continuous light visualization, such as using extremely slight visual cues to continuously mark different interaction areas; dynamic response visualization, briefly enhancing the visual cues of a certain area of ​​the screen when the user's hand is detected approaching that area; gesture-guided animation, showing the gestures and functions corresponding to each area; intelligent contextual prompts, showing floating function prompts based on the current scenario and user behavior; and customizable feedback intensity, allowing users to adjust the intensity of visual feedback. Through these methods, the system can effectively convey the layout and functional information of the interaction area without interfering with the user's normal use, and achieve an intuitive and efficient contactless gesture interaction experience.

[0048] Through the above detailed steps, the present invention can intelligently divide the screen interaction area according to the spatial posture of the device and assign appropriate functions to each area. This dynamic interaction area division and function allocation mechanism improves the use flexibility and efficiency of the handheld tablet computer, allowing users to obtain the best interaction experience in various usage scenarios.

[0049] like Figure 4 As shown, in a preferred embodiment of the present invention, step S30 specifically includes the following steps: S31: activating the depth camera built into the handheld tablet computer, and adaptively adjusting the working parameters of the depth camera according to the current ambient light conditions and the user distance to obtain adjusted working parameters; the adjusted working parameters include frame rate, resolution and depth of field.

[0050] In this step, the system first activates the built-in depth camera of the device. Then, by analyzing the current ambient light conditions (indoors, outdoors, bright or dim, etc.) and estimating the distance between the user and the device, the system automatically adjusts the working parameters of the depth camera. For example, when there is sufficient light, the frame rate is reduced to improve the resolution; when there is insufficient light, the frame rate is increased but the resolution is reduced to ensure image quality. At the same time, based on the estimated user distance, the system will adjust the depth of field range to ensure that the user's gestures can be accurately captured. This adaptive adjustment ensures the best gesture recognition effect in different usage environments.

[0051] S32: defining a three-dimensional space detection area around the handheld tablet computer, where the three-dimensional space detection area covers a range in which the user may perform gesture interaction.

[0052] Specifically, based on the size of the device and the typical usage distance, the system defines a three-dimensional space detection area. This area is a hemispherical or conical space centered on the device, covering the range where the user is most likely to perform gesture interaction. For example, for a 10-inch tablet, this detection area can be a hemispherical space with a radius of 50 cm. This definition ensures that the system can capture all possible gestures while avoiding processing unnecessary background information and improving processing efficiency.

[0053] S33: Based on the adjusted working parameters, continuously capture multiple frames of depth images within the three-dimensional space detection area within a preset time interval.

[0054] Specifically, using the adjusted working parameters in step S31, the system begins to continuously capture depth images within the predefined three-dimensional space detection area. This process is high-frequency, capturing 30 frames per second or more to ensure that fast hand movements can be accurately captured. Each frame of the depth image contains the distance information of the object in the detection area, enabling the system to accurately track the movement trajectory of the finger.

[0055] S34: Process each captured depth image frame based on an image enhancement algorithm to obtain a processed depth image.

[0056] Specifically, in this step, the system applies a series of advanced image enhancement algorithms to improve the quality of the depth image. The specific processing process is as follows: First, the non-local means denoising algorithm is used to reduce the random noise in the depth image. This algorithm effectively retains the image details and removes the noise by finding similar pixel blocks in different areas of the image and then performing weighted averaging on these similar blocks. Secondly, holes often appear in the depth image, that is, some pixels do not have valid depth values. To fill these holes, the system uses an algorithm based on the Fast Marching Method to ensure the continuity of the depth image. Next, a bilateral filter is applied to the image for edge-preserving smoothing. The bilateral filter takes into account the spatial distance and depth value differences of pixels, and can smooth the image while maintaining the sharpness of the edges, which is very important for accurately identifying the finger contour. Then, the adaptive histogram equalization algorithm is used to enhance the local contrast of the image. This method divides the image into multiple small blocks, performs histogram equalization on each of them, and uses bilinear interpolation to eliminate the boundaries between blocks. Finally, according to the predefined gesture interaction space, the depth values ​​are linearly mapped to expand the depth values ​​of the area of ​​interest to the full dynamic range and improve the depth resolution of the gesture area. Through this series of processing, the original depth image is converted into a high-quality depth image that is clearer and more conducive to gesture recognition.

[0057] S35: Organizing the processed depth images into a depth image sequence in time sequence.

[0058] Specifically, each processed depth image frame is assigned a timestamp and organized into a continuous sequence according to the time order of capture. This sequence contains not only spatial information (i.e. the 3D position of the object in each frame), but also temporal information (i.e. the position change of the object at different time points). This organization provides the basis for subsequent action analysis, enabling the system to understand the dynamic characteristics of gestures.

[0059] S36: Performing time synchronization and space registration on the depth image sequence to generate a temporally and spatially consistent depth image sequence as a real-time depth image sequence representing the movement of the user's finger in the surrounding space of the handheld tablet computer.

[0060] Specifically, this step ensures the consistency of the depth image sequence in time and space dimensions, including time synchronization and spatial registration. Time synchronization ensures that the time interval between each frame is uniform, which is crucial for accurately calculating the speed and acceleration of movement. Spatial registration ensures that the spatial reference system between different frames is consistent, so that the movement trajectory of objects in three-dimensional space can be accurately tracked. For example, if the user's hand gets closer and closer to the device during movement, spatial registration can ensure that this distance change is accurately recorded. After these processes, the final depth image sequence can accurately characterize the movement of the user's fingers in the space around the device, providing high-quality input data for subsequent gesture recognition and interactive function implementation.

[0061] In some embodiments, in terms of time synchronization, an accurate timestamp is assigned to each frame of the depth image, with an accuracy of up to milliseconds. The system uses linear interpolation technology to handle possible frame rate fluctuations to ensure uniform time intervals between frames. If frame loss is detected, the optical flow estimation algorithm is applied to predict the content of the missing frame to maintain the continuity of the sequence. In terms of spatial alignment, the iterative closest point (ICP) algorithm is used to align the depth images between consecutive frames to eliminate the changes in the spatial reference system caused by slight movements of the device. The system also applies an affine transformation matrix to correct each frame of the image to ensure that all frames are aligned to the same spatial coordinate system. In addition, the built-in gyroscope and accelerometer data of the device are used to compensate for changes in perspective caused by device movement. Finally, in order to enhance spatiotemporal consistency, the system uses a spatial-temporal bilateral filter (Spatial-Temporal Bilateral Filter), which takes into account the similarity of spatial and temporal dimensions at the same time to further improve sequence consistency, and uses a Kalman filter to smooth the finger motion trajectory to reduce jitter caused by sensor noise. After the above processing, the final generated depth image sequence is smooth and continuous in time and accurately aligned in space, providing high-quality input data for the subsequent gesture recognition algorithm.

[0062] like Figure 5 As shown, in a preferred embodiment of the present invention, step S40 specifically includes the following steps: S41: preprocessing the acquired depth image sequence to obtain a preprocessed depth image sequence.

[0063] Specifically, this step is mainly to improve the efficiency and accuracy of subsequent processing. Preprocessing includes operations such as image denoising, contrast enhancement, and depth value normalization. For example, a Gaussian filter can be used to remove high-frequency noise in the image, and then histogram equalization can be applied to enhance the image contrast, and finally the depth value can be mapped to the range of 0-255 for normalization. These operations can significantly improve the recognizability of the finger area.

[0064] S42: In each frame of the preprocessed depth image, extract the user's finger area by using a depth threshold segmentation algorithm.

[0065] Specifically, depth threshold segmentation is a binarization method based on depth values, which can effectively separate the foreground (finger) from the background. In specific implementation, a depth range (for example, 30cm to 60cm from the device) can be set according to the predefined gesture interaction space, and the pixels within the range can be regarded as potential finger areas. Then, the Otsu algorithm is used to adaptively determine the optimal threshold to further refine the segmentation results. This method can effectively cope with lighting changes and background interference in different usage environments.

[0066] S43: Perform morphological processing on the user's finger area to obtain a finger contour.

[0067] Specifically, morphological processing is a shape-based image processing technique that is mainly used to improve segmentation results. In this step, an opening operation (erosion followed by dilation) is first applied to remove small noise areas and disconnect small connections, and then a closing operation (dilation followed by erosion) is used to fill small holes and smooth the boundaries. Finally, an edge detection algorithm (such as the Canny edge detector) is used to extract the finger contour. This series of operations can obtain a clear and continuous finger contour, providing a reliable basis for subsequent feature extraction.

[0068] S44: Calculate the centroid of the finger contour in each frame of the depth image as the three-dimensional coordinates of the finger in the depth image of this frame.

[0069] Specifically, the centroid calculation uses the image moment method, and the specific formula is: x = M10 / M00, y = M01 / M00, where M00 is the area of ​​the region, and M10 and M01 are the first-order moments in the x and y directions, respectively. For the z coordinate (depth value), the average depth or median depth of the finger area can be taken. This method can effectively represent the overall position of the finger and is insensitive to small changes in the finger posture, thereby improving the stability of tracking.

[0070] S45: Arrange the three-dimensional coordinates of the finger extracted from the continuous multi-frame depth image in time sequence to form a time-series three-dimensional coordinate sequence representing the movement of the user's finger as the time-series three-dimensional coordinate information of the user's finger.

[0071] Specifically, this step combines spatial information and temporal information to form a basic data structure that describes finger motion. Each coordinate point carries a timestamp, indicating its precise position in the motion sequence. This representation method not only contains the spatial position information of the finger, but also implies dynamic characteristics such as velocity and acceleration, providing rich information for subsequent trajectory analysis and gesture recognition.

[0072] S46: performing smoothing processing on the time series three-dimensional coordinate sequence to obtain a smoothed time series three-dimensional coordinate sequence.

[0073] Specifically, the purpose of smoothing is to reduce the noise caused by measurement errors and slight hand shaking. A Savitzky-Golay filter can be used, which can effectively remove noise while maintaining the high-order moments of the signal. For each dimension (x, y, z), a 5-point Savitzky-Golay filter with a polynomial order of 2 is used. This method can effectively smooth the trajectory while retaining the details of fast changes, and is suitable for processing complex signals such as finger movements that contain both smooth movements and mutations.

[0074] S47: Based on the smoothed time-series three-dimensional coordinate sequence, a three-dimensional spline interpolation algorithm is used to construct a continuous non-contact three-dimensional motion trajectory.

[0075] Specifically, cubic B-spline interpolation is used here, which can generate smooth and continuous curves and has local controllability. In specific implementation, the smoothed coordinate sequence is used as the control point, and multiple interpolation points are inserted between every two adjacent control points (for example, 9 points are inserted between every two control points to increase the sampling rate by 10 times). This method can not only generate visually smooth trajectories, but also accurately reflect the subtle changes in finger movements, providing high-quality input data for subsequent gesture recognition.

[0076] S48: Extracting features from the non-contact three-dimensional motion trajectory to obtain trajectory features. The purpose of feature extraction is to convert complex trajectory data into a set of discriminative feature vectors.

[0077] Specifically, the trajectory features extracted in this step include: geometric features such as the total length of the trajectory, average speed, maximum acceleration, curvature, torsion, and spatial features such as the main direction of the trajectory and the normal vector of the motion plane. In addition, Fourier descriptors can be used to capture the overall shape characteristics of the trajectory. These features comprehensively describe various aspects of finger movement and provide rich information for subsequent gesture classification and recognition.

[0078] S49: combining the trajectory feature with the smoothed time-series three-dimensional coordinate sequence to form a complete non-contact three-dimensional motion trajectory of the user's finger.

[0079] Specifically, this step integrates the discrete coordinate data and the extracted high-level features to form a data structure that fully describes the finger movement. Specifically, the trajectory features can be attached to the time-series coordinate sequence as metadata. In addition to the x, y, and z coordinates, each time point also contains the local and global feature values ​​corresponding to the point. This representation method retains the original spatiotemporal information and includes an abstract feature description.

[0080] like Figure 6 As shown, in another preferred embodiment of the present invention, step S50 specifically includes the following steps: S51: Retrieving a predefined contactless gesture library from the memory of the handheld tablet computer, wherein the predefined contactless gesture library includes a plurality of predefined three-dimensional gesture patterns and their corresponding interaction meanings.

[0081] Specifically, the predefined contactless gesture library stores a series of standardized three-dimensional gesture patterns. Each gesture pattern is associated with a specific interactive meaning, for example, "swiping right" can correspond to the "next page" operation, "drawing a circle" can correspond to the "refresh" operation, etc. The design of the gesture library needs to consider intuitiveness, ease of learning, and distinguishability to ensure that users can easily master and accurately use these gestures. In actual applications, the gesture library can contain 20-30 commonly used gestures, each of which has its own unique three-dimensional trajectory characteristics and clear interactive meaning.

[0082] S52: Extract features of the non-contact three-dimensional motion trajectory of the user's finger to obtain a trajectory feature vector.

[0083] Specifically, feature extraction is the process of converting complex trajectory data into computable numerical features. This step can extract a variety of features, including but not limited to: the total length of the trajectory, average speed, acceleration change, main direction of motion, curvature and torsion of the trajectory, bounding box size of the trajectory, etc. In addition, more advanced features can also be used, such as Fourier descriptors or wavelet transform coefficients. Together, these features form a high-dimensional feature vector that fully describes the geometric and dynamic characteristics of the finger motion. For example, a feature vector can contain 50-100 elements, each representing a specific attribute of the trajectory.

[0084] S53: performing similarity calculation on the trajectory feature vector and each predefined three-dimensional gesture pattern in the predefined non-contact gesture library to obtain a similarity calculation result.

[0085] Specifically, the similarity calculation measures the degree of match between the gesture input by the user and the predefined gesture. Similarity calculation methods include Euclidean distance, cosine similarity, dynamic time warping (DTW), etc. In this embodiment, weighted Euclidean distance can be used as a similarity metric, and the weights of different features can be optimized by a machine learning algorithm (such as a support vector machine or a random forest). The calculation process traverses each predefined gesture in the gesture library and calculates its similarity score with the input gesture. Assuming that the gesture library contains 25 predefined gestures, this step will generate a result vector containing 25 similarity scores.

[0086] S54: Based on the similarity calculation result, select the predefined three-dimensional gesture pattern with the highest similarity as the best matching gesture.

[0087] Specifically, this step is to find the highest value from the similarity scores obtained in the previous step and determine the corresponding predefined gesture as the best match. For example, if the similarity score of the "swipe right" gesture is the highest, reaching 0.92, then it is selected as the best matching gesture.

[0088] S55: Determine whether the similarity of the selected best matching gesture exceeds a preset threshold.

[0089] Specifically, this step introduces a safety mechanism to prevent false recognition. The selection of the preset threshold requires a trade-off between the recognition rate and the rejection rate, which is determined by a large amount of experimental data. For example, the threshold can be set to 0.85, and the system considers the recognition to be reliable only when the similarity of the best matching gesture exceeds 0.85. The setting of the threshold can be adjusted according to different application scenarios, and a higher threshold can be used for scenarios with high security requirements.

[0090] S56: If it exceeds the preset threshold, confirm that the gesture recognition is successful, and obtain the interaction meaning corresponding to the best matching gesture.

[0091] Specifically, when recognition is successful, the system will extract the interactive meaning corresponding to the gesture from the predefined gesture library. For example, if the "swipe right" gesture is recognized, the system will obtain its corresponding "next page" operation instruction. This interactive meaning will be passed to the corresponding module of the application to perform specific operations or trigger specific functions.

[0092] S57: If the value does not exceed the preset threshold, it is determined as an unrecognizable gesture, and a prompt message is sent to the user to request the user to re-enter the gesture.

[0093] Specifically, this is the system's fault tolerance mechanism. When the user input cannot be reliably recognized, the system will give feedback. The prompt information can be visual (such as prompt text or icons on the screen) or auditory (such as a short prompt sound). For example, the system can display a prompt "Failed to recognize your gesture, please try again", and can also provide some example illustrations of common gestures to help users improve their gesture input.

[0094] S58: Combining the best matching gesture, the corresponding similarity value and the corresponding interaction meaning to form a final pattern matching result.

[0095] Specifically, this step integrates the key information of the recognition process to form a complete result data structure. For example, the result of an example contains the following information: best matching gesture = "swipe right", similarity value = 0.92, interaction meaning = "next page". This comprehensive result is not only used for the current interaction operation, but also for continuous optimization of the system. For example, by recording these results, the system can analyze the user's gesture usage pattern, adjust the recognition algorithm, or personalize the user's gesture settings.

[0096] like Figure 7 As shown, in another preferred embodiment of the present invention, step S60 specifically includes the following steps: S61: Analyze the pattern matching results and extract the best matching gesture and its corresponding interaction meaning.

[0097] Specifically, this step is a further processing of the recognition results of the previous stage. The system will extract key information from the pattern matching results, mainly including the best matching gesture and the corresponding interactive meaning. For example, if the pattern matching results show that the best matching gesture is "swipe up" and the corresponding interactive meaning is "scroll the page", then this information will be extracted to prepare for subsequent interactive operations. This process involves parsing data structures, such as extracting specific fields from results in JSON or XML format. In actual applications, the system will also record this extracted information for user behavior analysis or system performance optimization.

[0098] S62: Determine a target area where the user intends to interact based on the interaction meaning and the current spatial posture of the handheld tablet computer.

[0099] Specifically, this step combines the semantic information of the gesture and the physical state of the device to infer the specific area that the user wants to interact with. The spatial posture of a handheld tablet can be obtained through the built-in accelerometer and gyroscope. For example, if the user makes a "swipe up" gesture and the device is in a horizontal position, the system will infer that the user wants to scroll the entire currently displayed page. But if the device is tilted, the system will infer that the user only wants to scroll a part of the page. For another example, if the gesture is "zoom in" and the device is facing the user's face, the system will infer that the user wants to zoom in on the content in the center of the screen. This interpretation method that combines gestures and device posture can improve the accuracy and naturalness of the interaction.

[0100] S63: Based on the association information between the interaction area and the data interaction function generated and stored in step S24, the data interaction function associated with the target area is acquired.

[0101] Specifically, this step utilizes the interactive area and function mapping relationship predefined by the system. In step S24, the system has assigned specific data interaction functions to different interactive areas. Now, the system will look up this mapping relationship to find out the specific function corresponding to the target area. For example, if the target area is determined to be the upper right corner of the screen, and this area is defined as the "close current application" function area in the mapping, then the system will obtain the "close current application" function. This mapping relationship can be static or dynamic, and changes according to the currently running application or the user's personalized settings.

[0102] S64: triggering execution of a data interaction function associated with the target area.

[0103] Specifically, this is the last step of the entire interaction process and the part that the user can directly perceive. The system will call the corresponding program interface or function to execute the function associated with the target area. Continuing with the above example, if the determined function is "close the current application", the system will trigger the application's closing process. This includes a series of operations such as saving the current work, releasing resources, and updating the user interface. When executing functions, the system can also provide visual, auditory, or tactile feedback to enhance the user's interactive experience. For example, when closing an application, a short animation or sound effect will be played. In addition, the system also records the results of this interaction for subsequent user behavior analysis or system optimization.

[0104] Through the above steps, the system realizes the complete process from gesture recognition to specific function execution, providing users with an intuitive and natural contactless interactive experience. This method not only takes into account the meaning of the gesture itself, but also combines the physical state of the device and the predefined interactive area, thus achieving a more intelligent and context-related interaction method.

[0105] like Figure 8 As shown, in another preferred embodiment of the present invention, step S70 specifically includes the following steps: S71: According to the executed data interaction function, select the corresponding feedback mode from the preset multimodal feedback mode library. The multimodal feedback mode library is a comprehensive data set that contains multiple feedback modes such as tactile, visual, and auditory corresponding to various interactive functions. For example, for the slide to unlock function, it can correspond to a combination of a tactile vibration plus a visual animation; and for the function of receiving new messages, it can correspond to a short sound prompt plus screen flashing and slight vibration. The design of this library takes into account the characteristics of human multi-sensory perception and provides a rich, intuitive and non-intrusive feedback experience. In actual applications, the multimodal feedback mode library can contain 50-100 basic modes, which can be flexibly combined or changed according to different interactive functions.

[0106] S72: Generate a corresponding multimodal feedback control signal according to the selected feedback mode. This step converts the abstract feedback mode into a specific control signal. The control signal may include vibration parameters of tactile feedback (such as frequency, amplitude, duration), animation parameters of visual feedback (such as color, brightness, animation duration), audio parameters of auditory feedback (such as volume, pitch, sound effect type), etc. For example, for a multimodal feedback of "successfully completing a task", the control signal may specify a 50ms tactile vibration (frequency of 180Hz), trigger a green check mark animation (lasting 0.5 seconds), and play a short "ding" sound effect (volume is 70% of the system volume). These precisely controlled parameters can create a rich and diverse perceptual experience, thereby enhancing the user's understanding and memory of different interactive operations.

[0107] S73: Send the multimodal feedback control signal to the multimodal feedback device built into the handheld tablet computer. The multimodal feedback device may include multiple components such as a linear vibration motor (tactile), a display screen (visual), a speaker or a piezoelectric buzzer (auditory). These devices can generate corresponding feedback effects based on the received control signal. The transmission of the signal is completed through the internal bus of the device or a dedicated hardware interface, involving the coordination of multiple subsystems. In this process, the system needs to ensure the synchronization and consistency of various feedback signals to create a coordinated and consistent multimodal feedback experience.

[0108] S74: Trigger the multimodal feedback device to perform a corresponding feedback action. This is a step that the user can directly perceive. The multimodal feedback device generates corresponding tactile, visual, and auditory feedback simultaneously or in a specific order according to the received control signal. For example, for a notification of "receiving important emails", the device may first generate a short vibration, then display an animation of an envelope icon, and finally play a unique prompt sound. This multimodal feedback not only enhances the richness of the interaction, but also provides users with multi-level information transmission, especially in situations where some single-modal feedback may be ignored (for example, in a noisy environment, you may not hear the sound prompt, but you can still feel the vibration).

[0109] S75: Monitor the execution status of the feedback action. This step introduces a comprehensive feedback monitoring mechanism to ensure the quality and consistency of multimodal feedback. Monitoring may include measuring actual vibration parameters, detecting changes in screen brightness, verifying audio output, etc. For example, the system may use an accelerometer to measure the actual vibration intensity, a light sensor to detect changes in screen brightness, and an audio loop to verify the quality of the sound output. These data can be used to evaluate the overall effect of multimodal feedback and provide a basis for subsequent adjustments.

[0110] S76: Dynamically adjust the parameters of the feedback action according to the execution state of the feedback action. This is an adaptive optimization process that is conducive to providing the best user experience. For example, if the ambient light is detected to be strong, the system can increase the brightness of the visual feedback; if the device is detected to be in silent mode, the system can enhance the intensity of the tactile feedback to compensate for the missing auditory feedback. This dynamic adjustment can adapt to different usage environments, hardware conditions or user preferences to ensure that multimodal feedback is always kept in the best state. In addition, the system can also learn the user's response pattern and gradually optimize the feedback strategy. For example, if it is found that the user often ignores a certain type of feedback, the system will adjust the parameters of the feedback or switch to other more effective feedback modes.

[0111] Through the above steps, the embodiment of the present invention realizes an intelligent, adaptive multi-modal feedback system, which can provide rich and intuitive perceptual feedback for different interactive operations, and can dynamically adjust according to actual conditions to ensure the best user experience. This refined multi-modal feedback design greatly enhances the effect and usability of contactless interaction, enabling users to accurately understand and respond to system feedback through multiple sensory channels, thereby improving the efficiency and satisfaction of interaction.

[0112] like Fig. 9 As shown, in another preferred embodiment of the present invention, the method further comprises the following steps: S80: Using the built-in memory of the handheld tablet computer, record the user's interactive operation data, which includes: interaction frequency, interaction duration, target interaction area, executed data interaction function, and delivered tactile feedback. This step establishes a comprehensive user behavior data collection mechanism. For example, the system will record the number of times the user opens the application every day (interaction frequency), the duration of each use of the application (interaction duration), which areas of the screen the user most often clicks (target interaction area), the functions the user most often uses (executed data interaction functions), and the user's response to different tactile feedback. These data provide rich raw materials for subsequent deep learning and optimization. Specifically, the system will record data such as: users open social applications 15 times a day on average, each time for about 5 minutes, most often click the "new message" icon in the upper right corner of the screen, often use the "send emoticon" function, and react more positively to strong vibration feedback.

[0113] S90: Based on the user's interactive operation data, a deep learning algorithm is used to dynamically optimize the data interaction functions associated with each interactive area. This step uses advanced artificial intelligence technology to convert the collected user data into actual optimization strategies. The deep learning algorithm can discover potential patterns and trends from a large amount of interactive data, thereby customizing the optimal interactive experience for each user.

[0114] like Fig.10 As shown, in some embodiments, step S90 specifically includes the following sub-steps: S91: Input the user's interactive operation data into a pre-trained deep learning model, and output the usage feature vector of the data interaction function in each interactive area. The deep learning model here is a multi-layer neural network that can understand complex interaction patterns after being trained with a large amount of user data. For example, the deep learning model finds that users tend to check the news in the morning and use social functions more in the evening. The output feature vector can contain multi-dimensional information such as frequency of use, time period of use, and correlation with other functions.

[0115] S92: Calculate the priority index of each data interaction function in each interaction area based on the usage feature vector output by the deep learning model. The priority index is a comprehensive score that reflects the importance and convenience of a function to the user. The calculation can take into account multiple factors, such as frequency of use, last use time, duration of use, etc. For example, if the user frequently uses the "document editing" function and the usage time is long each time, it will get a higher priority index, while the "system settings" function that is rarely used will get a lower priority index.

[0116] S93: Generate a priority ranking of data interaction functions for each interaction area according to the priority index. This step converts the priority index calculated in S92 into a specific ranking. For example, for the shortcut function area of ​​the home screen, the ranking may be: message, mail, calendar, camera, setting. This ranking reflects the user's usage habits and preferences, and provides direct guidance for subsequent interface optimization.

[0117] S94: Based on the priority ranking, adjust the layout, display order or triggering method of the data interaction functions associated with each interaction area. This is a key step in converting the analysis results into actual optimization. For example, the system will place the top-ranked functions in a more accessible position (such as the bottom or edge of the screen), increase their icon size, or set a more convenient triggering method for them (such as a swipe gesture). For infrequently used functions, they will be moved to the secondary menu or their icon size will be reduced. This adjustment is intended to improve the user's operating efficiency and comfort.

[0118] S95: Update the adjusted data interaction function configuration to each interaction area on the screen of the handheld tablet computer to achieve dynamic optimization of the data interaction functions associated with each interaction area. This step actually applies the adjustment in S94 to the device interface. The update may include rearranging application icons on the home screen, adjusting the order of options in the quick settings panel, changing the default functions of certain gestures, etc. For example, if the system finds that the user often checks the weather information in the morning, it will automatically push the weather application to a prominent position on the home screen every morning. This dynamic optimization ensures that the user interface can continuously adapt to the user's usage habits and provide a personalized and efficient interactive experience.

[0119] This method can continuously learn users' usage habits and adjust the interface layout and function configuration in real time, thereby providing each user with a highly personalized and efficient interactive experience.

[0120] Embodiment 2 like Fig.11 As shown, an embodiment of the present invention further provides a data interaction system for a handheld tablet computer, the system comprising the following components: The attitude sensor is used to detect the spatial attitude of the handheld tablet computer relative to the direction of gravity. This sensor is an integrated inertial measurement unit (IMU) that includes an accelerometer, a gyroscope, and possibly a magnetometer. It can measure the tilt angle, rotation speed, and direction of the device in real time. For example, when the user turns the tablet computer from a horizontal position to a vertical position, the attitude sensor can accurately detect this change and provide important contextual information to the system. This is crucial for realizing functions such as automatic screen rotation and attitude-based interactive control.

[0121] A depth camera is used to obtain a sequence of depth images of the movement of the user's finger in the space around the handheld tablet in real time. This camera uses structured light or Time of Flight (ToF) technology to capture three-dimensional spatial information. It can not only identify the position of the user's finger, but also measure its distance from the device. For example, the depth camera can capture a "slide" gesture made by the user 10 cm above the device, even if the finger does not touch the screen. This provides the basis for precise contactless interaction, allowing users to perform intuitive operations in the space around the device.

[0122] A multimodal feedback device is used to deliver sensory feedback to the user corresponding to the data interaction function performed. This device may include a tactile feedback device (such as a linear vibration motor), a visual feedback system (such as an LED indicator or screen animation), and an auditory feedback system (such as a speaker or a piezoelectric buzzer). For example, when the user successfully completes a contactless gesture, the device will generate a short vibration, a confirmation icon will flash on the screen, and a gentle sound effect will be played. This multimodal feedback can provide users with a rich and intuitive interactive experience, enhancing the certainty and satisfaction of the operation.

[0123] A memory electrically connected to the processor for storing and executing the above Figures 1 to 9 The memory here can be a high-speed flash memory or solid-state hard disk, and the storage capacity is sufficient to accommodate the operating system, application programs and user data. It not only stores the core algorithms and programs for realizing contactless interaction, but also contains information such as the user's personalized settings and usage history. For example, the memory contains key data such as gesture recognition algorithms, user interface layout configurations, and deep learning models.

[0124] A processor is electrically connected to the gesture sensor, the depth camera, the memory and the data storage device, and is used to execute the program in the memory to achieve the above Figures 1 to 9 The processor can be a high-performance multi-core ARM processor or a similar mobile processor with sufficient computing power to handle complex gesture recognition and deep learning tasks in real time. For example, it can complete the entire process from receiving sensor data, analyzing user gestures to triggering corresponding functions within milliseconds to ensure smooth interaction and responsiveness.

[0125] A screen is electrically connected to the processor and is used to display the multiple interactive areas and the results of executing the data interactive function. This is a high-resolution, high-refresh rate screen that can accurately display complex user interface elements and smooth animation effects. For example, the screen can clearly display the divided interactive areas and display visual feedback in real time when the user performs a contactless gesture, such as highlighting the activated area or showing the results of the function execution.

[0126] The server is electrically connected to the processor and is used to store a predefined contactless gesture library, a data interaction function library, and a multimodal feedback mode library. This server can be a cloud system that not only stores a large number of predefined interaction modes and functions, but also continuously optimizes and expands these libraries through machine learning. For example, when the system recognizes a new and valid user gesture, it can be added to the gesture library; or the priority of certain functions can be dynamically adjusted based on the usage data of the user group. This cloud storage and processing mechanism enables the system to continue to learn and evolve, providing users with an increasingly intelligent and personalized interaction experience.

[0127] Through the collaborative work of these components, the handheld tablet computer data interaction system of the embodiment of the present invention can achieve highly intuitive, natural and personalized non-contact interaction. The system uses advanced sensing technology to accurately capture the user's operating intentions, analyzes and responds in real time through powerful processing capabilities, and provides users with a rich perceptual experience through multimodal feedback. At the same time, with the storage and computing capabilities of the cloud server, the system can continuously learn and optimize to adapt to the usage habits and preferences of different users. This design not only improves the efficiency and accuracy of the interaction, but also greatly enhances the comfort and fun of the user experience.

[0128] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0129] Embodiment 3 An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the above methods is implemented.

[0130] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. Of course, there are other forms of readable storage media, such as quantum memory, graphene memory, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0131] Embodiment 4 The present invention also provides an electronic device. The electronic device of an embodiment of the present invention comprises: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the present invention.

[0132] Reference below Fig.12 , which shows a schematic diagram of the structure of a computer system 800 of an electronic device suitable for implementing an embodiment of the present invention. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0133] like Fig.12As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage part 208 to a random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the computer system 200 are also stored. The CPU 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0134] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed so that a computer program read therefrom is installed into the storage section 208 as needed.

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The units involved in the embodiments of the present invention may be implemented by software or hardware. The units described may also be arranged in a processor, and the names of these units do not constitute limitations on the units themselves in some cases.

[0137] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data interaction method for a handheld tablet computer, characterized in that: The following steps are involved: S10: Detecting the spatial posture of the handheld tablet computer relative to the gravity direction by using a posture sensor built into the handheld tablet computer; S20: dividing the screen of the handheld tablet computer into a plurality of interactive areas according to the detected spatial posture of the handheld tablet computer relative to the gravity direction, each interactive area being associated with a corresponding data interactive function; S30: using the depth camera built into the handheld tablet computer to obtain in real time a depth image sequence of the user's finger moving in the space around the handheld tablet computer; S40: extracting the time-series three-dimensional coordinate information of the user's finger through an image processing algorithm based on the depth image sequence, and constructing a non-contact three-dimensional motion trajectory of the user's finger according to the time-series three-dimensional coordinate information; S50: performing pattern matching on the non-contact three-dimensional motion trajectory of the user's finger and a predefined non-contact gesture library stored in the handheld tablet computer to obtain a pattern matching result; S60: determining a target interaction area where the user intends to interact and a data interaction function associated with the target interaction area according to the pattern matching result, and triggering execution of the data interaction function associated with the target interaction area; S70: In response to the execution of the data interaction function, a perceptual feedback corresponding to the executed data interaction function is transmitted to the user through a multimodal feedback device built into the handheld tablet computer.

2. The method according to claim 1, characterized in that Step S10 specifically includes: S11: configuring working parameters of a posture sensor built into the handheld tablet computer, wherein the posture sensor includes an accelerometer, a gyroscope, and a magnetometer; S12: Acquiring acceleration data of the handheld tablet computer in three orthogonal axes through the accelerometer; S13: Acquiring angular velocity data of the handheld tablet computer around three orthogonal axes through the gyroscope; S14: obtaining direction data of the handheld tablet computer relative to the geomagnetic field by using the magnetometer; S15: using a Kalman filter algorithm or a complementary filter algorithm, fusing the acceleration data from the accelerometer, the angular velocity data from the gyroscope, and the direction data from the magnetometer to obtain optimized attitude estimation data; S16: Calculating the pitch angle, roll angle and yaw angle of the handheld tablet computer relative to the gravity direction based on the optimized posture estimation data; S17: Determine the spatial posture of the handheld tablet computer relative to the gravity direction according to the calculated pitch angle, roll angle and yaw angle.

3. The method according to claim 1, characterized in that Step S20 specifically includes: S21: determining an inclination angle of a screen plane of the handheld tablet computer relative to a horizontal plane according to a spatial posture of the handheld tablet computer relative to a gravity direction; S22: selecting a preset screen division template based on the tilt angle, and dividing the screen of the handheld tablet computer into a plurality of interaction areas of geometric shapes according to the selected screen division template; S23: selecting a corresponding data interaction function for each interaction area from a preset data interaction function library according to the current application scenario and user usage habits; S24: Generate and store association information between the interaction area and the data interaction function; S25: Visually displaying the divided interactive areas and their associated data interactive functions on the screen of the handheld tablet computer.

4. The method according to claim 1, characterized in that: Step S30 specifically includes: S31: activating the depth camera built into the handheld tablet computer, and adaptively adjusting the working parameters of the depth camera according to the current ambient light conditions and the user distance to obtain adjusted working parameters; the adjusted working parameters include frame rate, resolution and depth of field; S32: defining a three-dimensional space detection area around the handheld tablet computer, where the three-dimensional space detection area covers a range in which a user may perform gesture interaction; S33: Based on the adjusted working parameters, continuously capturing multiple frames of depth images within the three-dimensional space detection area within a preset time interval; S34: Processing each captured depth image frame based on an image enhancement algorithm to obtain a processed depth image; S35: organizing the processed depth images into a depth image sequence in time order; S36: Performing time synchronization and space registration on the depth image sequence to generate a temporally and spatially consistent depth image sequence as a real-time depth image sequence representing the movement of the user's finger in the surrounding space of the handheld tablet computer.

5. The method according to claim 1, characterized in that Step S40 specifically includes: S41: preprocessing the acquired depth image sequence to obtain a preprocessed depth image sequence; S42: extracting the user's finger area in each frame of the preprocessed depth image by using a depth threshold segmentation algorithm; S43: performing morphological processing on the user's finger area to obtain a finger contour; S44: Calculate the centroid of the finger contour in each frame of the depth image as the three-dimensional coordinates of the finger in the depth image of this frame; S45: arranging the three-dimensional coordinates of the finger extracted from the continuous multiple-frame depth image in time order to form a time-series three-dimensional coordinate sequence representing the movement of the user's finger as the time-series three-dimensional coordinate information of the user's finger; S46: performing smoothing processing on the time series three-dimensional coordinate sequence to obtain a smoothed time series three-dimensional coordinate sequence; S47: constructing a continuous non-contact three-dimensional motion trajectory using a three-dimensional spline interpolation algorithm based on the smoothed time-series three-dimensional coordinate sequence; S48: extracting features of the non-contact three-dimensional motion trajectory to obtain trajectory features; S49: combining the trajectory feature with the smoothed time-series three-dimensional coordinate sequence to form a complete non-contact three-dimensional motion trajectory of the user's finger.

6. The method according to claim 3, characterized in that Step S50 specifically includes: S51: Retrieving a predefined contactless gesture library from the memory of the handheld tablet computer, wherein the predefined contactless gesture library includes a plurality of predefined three-dimensional gesture patterns and their corresponding interaction meanings; S52: extracting features of the non-contact three-dimensional motion trajectory of the user's finger to obtain a trajectory feature vector; S53: performing similarity calculation on the trajectory feature vector and each predefined three-dimensional gesture pattern in the predefined non-contact gesture library to obtain a similarity calculation result; S54: Based on the similarity calculation result, select the predefined three-dimensional gesture pattern with the highest similarity as the best matching gesture; S55: Determine whether the similarity of the selected best matching gesture exceeds a preset threshold; S56: If the preset threshold is exceeded, confirming that the gesture recognition is successful, and obtaining the interaction meaning corresponding to the best matching gesture; S57: If the value does not exceed the preset threshold, the gesture is determined to be unrecognizable, and a prompt message is sent to the user to request the user to re-enter the gesture; S58: Combining the best matching gesture, the corresponding similarity value and the corresponding interaction meaning to form a final pattern matching result.

7. The method according to claim 6, characterized in that Step S60 specifically includes: S61: parsing the pattern matching result, extracting the best matching gesture and its corresponding interaction meaning; S62: determining a target area for the user's intended interaction according to the interaction meaning and the current spatial posture of the handheld tablet computer; S63: acquiring the data interaction function associated with the target area based on the association information between the interaction area and the data interaction function generated and stored in step S24; S64: triggering execution of a data interaction function associated with the target area.

8. The method according to claim 1, characterized in that Step S70 specifically includes: S71: selecting a corresponding feedback mode from a preset multi-modal feedback mode library according to the executed data interaction function; S72: generating a corresponding multi-modal feedback control signal according to the selected feedback mode; S73: sending the multimodal feedback control signal to a multimodal feedback device built into the handheld tablet computer; S74: triggering the multimodal feedback device to perform a corresponding feedback action; S75: monitoring the execution status of the feedback action; S76: Dynamically adjust the parameters of the feedback action according to the execution status of the feedback action.

9. The method according to claim 1, characterized in that: Also includes: S80: using the built-in memory of the handheld tablet computer to record the user's interactive operation data, which includes: interaction frequency, interaction duration, target interaction area, executed data interaction function, and delivered tactile feedback; S90: Based on the interactive operation data of the user, a deep learning algorithm is used to dynamically optimize the data interaction functions associated with each interactive area.

10. A data interaction system for a handheld tablet computer, characterized in that: include: A posture sensor, used for detecting the spatial posture of the handheld tablet computer relative to the direction of gravity; A depth camera, used for acquiring in real time a depth image sequence of the movement of the user's finger in the space around the handheld tablet computer; A multimodal feedback device for delivering to the user perceptual feedback corresponding to the data interaction function performed; A memory, electrically connected to the processor, for storing program instructions for executing the method according to any one of claims 1 to 9; A processor, electrically connected to the gesture sensor, the depth camera, the memory and the data storage device, configured to execute a program in the memory to implement the method according to any one of claims 1 to 9; a screen, electrically connected to the processor, for displaying the plurality of interactive areas and the results of executing the data interactive function; The server is electrically connected to the processor and is used to store a predefined contactless gesture library, a data interaction function library and a multimodal feedback mode library.

Images