Three-dimensional force touch operation recognition method and device

Abstract

The application provides a three-dimensional force touch operation recognition method and device. It is applied to the field of human-computer interaction technology. The method comprises the following steps: obtaining pressure distribution data corresponding to a touch operation; calculating basic feature information of the touch operation according to the pressure distribution data; calculating a pressure center displacement according to a pressure center position at a current moment and a pressure center position at a starting moment; calculating a pressure distribution form change amount according to a pressure distribution form descriptor at the current moment and a pressure distribution form descriptor at the starting moment; judging whether the pressure center displacement is less than a displacement threshold value; if the pressure center displacement is less than the displacement threshold value, judging whether the pressure distribution form change amount is greater than a form change amount threshold value; if the pressure distribution form change amount is greater than the form change amount threshold value, determining that the current touch operation is a tangential force operation, and calculating a force vector according to the pressure distribution form change amount; and outputting a corresponding control signal according to the force vector. The application can accurately recognize complex gestures of the tangential force operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, in particular to a three-dimensional force touch operation recognition method and device. BACKGROUND

[0002] In the rapid development of human-computer interaction technology, touch interaction devices have become mainstream, and the capacitive touch screen widely used in smart phones is a typical representative. The traditional capacitive touch screen works based on a specific sensing principle, and its main function is coordinate measurement, which can sense the two-dimensional point coordinates when the finger contacts the screen.

[0003] The limitations of existing touch interaction technology have caused a series of obvious drawbacks: first, the interaction experience is poor. Since the traditional touch screen can only measure point coordinates, it loses rich three-dimensional force information, making it difficult to distinguish operations of different forces, resulting in a lack of realism and force in the interaction. Second, the control is limited. The loss of three-dimensional force information makes it difficult for the device to achieve fine one-handed control and recognize complex gestures, and the cumulative error caused by touch further interferes with the accuracy of control, making it more difficult to control. Third, some improved solutions are not good. Some technologies infer user intent by integrating multiple sensors and complex models, resulting in a complex system, high cost, poor real-time performance, and unsuitability for large-scale applications. SUMMARY

[0004] Therefore, the present application provides a three-dimensional force touch operation recognition method, comprising:

[0005] obtaining pressure distribution data corresponding to a touch operation;

[0006] calculating the basic feature information of the touch operation according to the pressure distribution data, including the pressure center position and the pressure distribution pattern descriptor;

[0007] calculating the pressure center displacement according to the pressure center position at the current moment and the pressure center position at the starting moment;

[0008] calculating the pressure distribution pattern change amount according to the pressure distribution pattern descriptor at the current moment and the pressure distribution pattern descriptor at the starting moment;

[0009] determining whether the pressure center displacement is less than a displacement threshold;

[0010] if the pressure center displacement is less than the displacement threshold, determining whether the pressure distribution pattern change amount is greater than a pattern change amount threshold;

[0011] if the pressure distribution pattern change amount is greater than the pattern change amount threshold, determining that the current touch operation is a tangential force operation, and calculating a force vector according to the pressure distribution pattern change amount;

[0012] According to the force vector, a corresponding control signal is outputted.

[0013] Optionally, the basic feature information further comprises a contact area; and before the calculation of the displacement of the pressure center, the method further comprises:

[0014] Determining whether the contact area is greater than a contact area threshold value;

[0015] If the contact area is greater than the contact area threshold value, it is determined that the current touch operation is a region pressing operation.

[0016] Optionally, when it is determined to be a region pressing operation, a total pressure is obtained by calculating a sum of pressure values of all pixel points according to the pressure distribution data, and a corresponding control signal is outputted according to the contact area and the total pressure.

[0017] Optionally, if the displacement of the pressure center is greater than the displacement threshold value, it is determined that the current touch operation is a sliding operation.

[0018] Optionally, when it is determined to be a sliding operation, a corresponding control signal is outputted according to the displacement of the pressure center.

[0019] Optionally, the calculation of the position of the pressure center according to the pressure distribution data comprises:

[0020] Filtering and normalizing the pressure distribution data to obtain preprocessed pressure distribution data;

[0021] Screening pixel points with pressure values greater than a first pressure threshold value in the preprocessed pressure distribution data to obtain effective pressure points;

[0022] Performing weighted centroid calculation on positions and corresponding pressure values of the effective pressure points to obtain the position of the pressure center.

[0023] Optionally, the pressure distribution pattern descriptor is calculated from a covariance matrix, comprising:

[0024] Calculating a weighted covariance matrix of the pressure distribution according to the pressure distribution data and the position of the pressure center;

[0025] Performing eigenvalue decomposition on the weighted covariance matrix to obtain a maximum eigenvalue and a minimum eigenvalue, and corresponding eigenvectors;

[0026] Determining a principal axis direction according to the maximum eigenvalue and the corresponding eigenvectors;

[0027] Calculating an eccentricity according to the minimum eigenvalue and the corresponding eigenvectors;

[0028] Obtaining the pressure distribution pattern descriptor according to the principal axis direction and the eccentricity.

[0029] Optionally, the pressure distribution pattern descriptor is calculated by double centroid offset, comprising:

[0030] The pressure distribution data is binarized and the geometric centroid position is calculated;

[0031] The offset vector is calculated according to the geometric centroid position and the pressure center position, as the pressure distribution pattern descriptor.

[0032] Optionally, the pressure distribution pattern descriptor is calculated by gradient domain and frequency domain, comprising:

[0033] The gradient field of the pressure distribution data is calculated by using a gradient operator, and the weighted average direction is calculated according to the gradient amplitude of each pixel point in the gradient field, to obtain a first pressure distribution pattern descriptor component;

[0034] The pressure distribution profile in the binarized pressure distribution data is extracted, the coordinates of the pressure distribution profile are converted into a complex sequence, the fast Fourier transform is performed on the complex sequence to obtain frequency domain coefficients, and the amplitude of the low frequency coefficient in the frequency domain coefficients is selected to obtain a second pressure distribution pattern descriptor component;

[0035] The pressure distribution data is filtered by using a plurality of filters in different directions, the response total energy of each filter after filtering is calculated, and the sum of all response total energies is calculated to obtain a third pressure distribution pattern descriptor component;

[0036] The pressure distribution pattern descriptor is obtained according to the first pressure distribution pattern descriptor component, the second pressure distribution pattern descriptor component and the third pressure distribution pattern descriptor component.

[0037] Optionally, the pressure distribution pattern descriptor is calculated according to the pressure distribution data, further comprising:

[0038] The pressure distribution data is Gaussian smoothed in different scales, and the pressure distribution pattern descriptor is calculated on each scale layer, wherein the pressure distribution pattern descriptor of each scale layer is any one of a covariance matrix, a double centroid offset, a gradient domain and a frequency domain;

[0039] The basic weight of the pressure distribution pattern descriptor of each scale layer is assigned, and the confidence is calculated;

[0040] The basic weight of the pressure distribution pattern descriptor of each scale layer is modified according to the confidence;

[0041] The pressure distribution pattern descriptors of each scale layer are fused according to the modified weights to obtain the pressure distribution pattern descriptor.

[0042] Optionally, determining the tangential force operation further includes:

[0043] The modulus is calculated based on the offset vector to obtain the morphological change.

[0044] Determine whether the amount of morphological change is greater than the morphological change threshold;

[0045] If the amount of shape change is greater than the threshold value of shape change, then the current touch operation is determined to be a tangential force operation.

[0046] A second aspect of the present invention provides a three-dimensional force touch operation recognition device, the device comprising: a processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to cause the processor to perform the above-described three-dimensional force touch operation recognition method.

[0047] This invention acquires pressure distribution data from touch operations. This data includes not only the coordinates of each measurement point but also rich information such as the magnitude of the pressure at each point and the rate of pressure change. Based on this comprehensive data, key information such as the pressure center location, pressure distribution morphology descriptor, pressure center displacement, and pressure distribution morphology change are calculated, and threshold judgments are performed. In particular, by comprehensively considering both pressure center displacement and pressure distribution morphology change, tangential force operations are determined, enabling accurate recognition of complex three-dimensional force gestures using tangential force. This two-factor judgment method makes the interaction more realistic and responsive, allowing for precise one-handed operation and significantly improving the accuracy and reliability of the system's recognition of three-dimensional force gestures. Furthermore, this method eliminates the need for multi-sensor and complex models, reducing costs. Its simplified processing flow allows for rapid response to touch operations, making it suitable for large-scale applications. By generating force vectors and outputting control signals, precise control of related devices or systems can be achieved, comprehensively improving the performance and user experience of touch interaction. Attached Figure Description

[0048] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a three-dimensional force touch operation recognition method according to an embodiment of the present invention;

[0050] Figure 2 This is a diagram showing the pressure distribution pattern of the tangential force operation in an embodiment of the present invention;

[0051] Figure 3 This is a flowchart of another three-dimensional force touch operation recognition method in an embodiment of the present invention. Detailed Implementation

[0052] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0054] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] like Figure 1 As shown, this embodiment of the invention provides a three-dimensional force touch operation recognition method, which is executed by an electronic device such as a computer or server, and specifically includes:

[0056] S1, obtain the pressure distribution data corresponding to the touch operation.

[0057] This embodiment uses a pressure sensor array to collect pressure distribution data in real time during touch operations. A 64×64 pixel resistive or capacitive matrix is ​​preferred, and the sampling frequency can be configured within the range of 50Hz-200Hz. It supports USB, serial port, CAN, and Bluetooth protocols. Each collected pressure distribution data is presented as an m×n array. Taking a 64×64 pixel pressure sensor array as an example, m=n=64, where... This represents the pressure value of pixel (i,j).

[0058] S2 calculates the basic feature information of the touch operation based on the pressure distribution data, including the pressure center location and pressure distribution shape descriptor.

[0059] Because the pressure applied during touch operation is unevenly distributed and has a specific shape on the contact surface, the pressure center position (COP) and pressure distribution shape descriptor are calculated. This study extracts key information reflecting touch operation characteristics from pressure distribution data. The pressure center location is the equivalent concentration point of touch pressure, represented by coordinates; the pressure distribution morphology descriptor is used to quantify the shape characteristics of the pressure distribution. These parameters are calculated to extract information from the pressure data for identifying touch operation types.

[0060] S3, calculate the pressure center displacement based on the current pressure center position and the initial pressure center position.

[0061] Record the position of the pressure center at the current moment. and the position of the pressure center at the start time In this embodiment, the starting time can be the moment when pressure distribution data was first collected, used to capture the change in the pressure center throughout the entire touch period, or it can be the previous moment, facilitating the focus on real-time changes. Specifically, the pressure center displacement can be calculated by subtracting the corresponding coordinates at the starting time from the current pressure center position coordinates. Alternatively, other calculation methods suitable for specific scenarios can be used. The purpose of displacement calculation is to quantify the change in the position of the pressure center during a touch.

[0062] S4. Calculate the change in pressure distribution pattern based on the current pressure distribution pattern descriptor and the initial pressure distribution pattern descriptor.

[0063] Record the pressure distribution pattern descriptor at the current moment. and pressure distribution morphology descriptor at the initial moment Specifically, the change in pressure distribution pattern can be calculated by subtracting the pressure distribution pattern descriptor at the current moment from the pressure distribution pattern descriptor at the beginning moment; that is, the change in pressure distribution pattern. Of course, other suitable calculation methods can also be used to more accurately capture the changes in pressure distribution over time. The change in pressure distribution can quantify the changes in the geometric shape of pressure sensor data caused by physical deformation (such as elongation or concentration along the direction of force), and can be used to analyze the dynamic process of touch operation and identify different types of touch operation.

[0064] S5. Determine if the displacement of the pressure center is less than the displacement threshold. If the displacement of the pressure center is less than the displacement threshold, proceed to step S6. If the displacement of the pressure center is greater than the displacement threshold, it means that the pressure center has moved beyond the expected range. The corresponding touch operation may not be the main type of recognition. Therefore, the data can be determined to be invalid, or other operations can be performed.

[0065] S6. Determine whether the change in pressure distribution pattern is greater than the pattern change threshold. If the change in pressure distribution pattern is greater than the pattern change threshold, proceed to step S7.

[0066] S7, determine that the current touch operation is a tangential force operation, and calculate the force vector based on the change in pressure distribution.

[0067] like Figure 2 The left-hand image shows the initial pressure distribution, while the right-hand image shows the pressure distribution after applying tangential force (such as pushing, twisting, or kneading). The images show that the pressure center position (marked by a white circle) changes very little, while the pressure distribution pattern descriptor (marked by a black circle) changes significantly. This demonstrates that accurately identifying tangential force operations requires considering both the pressure center displacement and the change in pressure distribution pattern. This comprehensive approach allows for more precise recognition of complex hand gestures, improving the system's accuracy and reliability in gesture recognition.

[0068] The tangential direction described in this application refers to the direction relative to the surface where the pressure sensor (used to collect the pressure of touch operations) is located. This surface can be simply referred to as the operating surface, which can be a plane or a curved surface. During operations such as pushing, twisting, and kneading, the operating surface is simultaneously subjected to forces perpendicular to and parallel to the plane. This application defines these two types of forces as tangential forces.

[0069] Based on the change in pressure distribution pattern A force vector capable of accurately simulating tangential force is generated. This force vector not only reflects the magnitude and direction of the tangential force but also key parameters such as rotational torque.

[0070] S8 outputs the corresponding control signal based on the force vector.

[0071] Specifically, after obtaining the force vector, a force-control mapping is first performed according to pre-set rules, that is, the physical information represented by the force vector is converted into corresponding control information. Next, to avoid abrupt changes in the control signal that could lead to system instability, the mapped control signal is smoothed. The smoothed control signal is the final force control signal, which is then output for precise control of relevant devices or systems. Taking racing games as an example, the direction and magnitude of the force vector can simulate the steering and steering force of a racing car. When the user applies a tangential force to the game's touch area, the output control signal will control the car's steering according to the direction of the force vector, and will also control the steering angle according to the magnitude of the force vector.

[0072] This embodiment acquires pressure distribution data from touch operations. This data not only measures the coordinates of each measurement point but also includes rich information such as the magnitude of pressure at each point and the rate of pressure change. Based on this comprehensive data, key information such as the pressure center location, pressure distribution morphology descriptor, pressure center displacement, and pressure distribution morphology change are calculated, and threshold judgments are performed. In particular, by comprehensively considering both pressure center displacement and pressure distribution morphology change, tangential force operations are determined, enabling accurate recognition of complex three-dimensional force gestures involving tangential force. This two-factor judgment method makes the interaction more realistic and responsive, suitable for precise one-handed operation, and greatly improves the accuracy and reliability of the system's recognition of three-dimensional force gestures. Simultaneously, this method eliminates the need for multi-sensor and complex models, reducing costs. Its simplified processing flow allows for rapid response to touch operations, making it suitable for large-scale applications. By generating force vectors and outputting control signals, precise control of related devices or systems can be achieved, comprehensively improving the performance and user experience of touch interaction.

[0073] The above embodiments demonstrate that the recognition of tangential force operations can be applied to various scenarios, such as game controllers, VR / AR interactive applications, and automotive smart cockpits. In game controller applications, when a user applies a push force in place on the joystick area, the tangential force mode enables extremely precise aiming adjustments or simulates the acceleration effect of "half-pressing the trigger," which is difficult to achieve with traditional joystick potentiometers. In VR / AR interactive applications, when a user "rubs" or "pushes" the surface of a virtual object, even without actual hand movement, the system can provide precise force feedback through tangential force recognition, greatly enhancing the immersiveness and physical realism of the operation. In automotive smart cockpit applications, the tangential force operation mode can be used in map navigation, allowing for stepless zooming or tilting of the map by applying a push force in all directions without moving the center point of the finger, resulting in more stable and safer operation; it can also be used for media and environmental control, enabling precise stepless adjustments on virtual knobs, where users only need to apply or increase / decrease the push force at a single point, eliminating the need to search for sliders.

[0074] In some optional embodiments of this example, the basic feature information also includes the contact area; such as Figure 3 As shown, before calculating the displacement of the pressure center, the following is also included:

[0075] Sa1 determines whether the contact area is greater than the contact area threshold. If the contact area is greater than the contact area threshold, then proceed to step Sa2.

[0076] Sa2 indicates that the current touch operation is a region press operation.

[0077] In this embodiment, the area-based press mode is determined by the contact area, which is calculated from pressure distribution data, specifically the number of pixels whose pressure values ​​exceed a threshold. Then, it is determined whether the contact area is greater than the threshold. If the contact area is greater than the threshold, the current touch operation is identified as an area-based press operation. This embodiment is suitable for scenarios simulating large-area presses and palm control. This embodiment can quickly and accurately identify area-based press operations based on the contact area. By recognizing different types of touch operations, the device can implement diverse functions based on different touch operations, greatly improving the interaction efficiency between the device and the user.

[0078] In some optional implementations of this embodiment, such as Figure 3 As shown, when a region pressing operation is determined, the total pressure, Sa3, is obtained by calculating the sum of the pressure values ​​of all pixels based on the pressure distribution data. The corresponding control signal is then output based on the contact area and the total pressure.

[0079] In this embodiment, after determining that a region-based pressing operation is being performed, the pressure distribution data is further processed by summing the pressure values ​​of all pixels in the current pressure distribution data to obtain the total pressure. Finally, combining the calculated contact area and total pressure, a corresponding control signal is output to achieve the corresponding control function. This embodiment combines the contact area and total pressure to output control signals, bringing richer interactive functions to the device. Users can adjust the pressing contact area and the applied total pressure to achieve various different control effects, improving the interactive experience.

[0080] The above embodiments demonstrate that the recognition of area-based pressing operations can be applied to various scenarios, such as game controllers, VR / AR interactive applications, and automotive smart cockpits. In game controller applications, users press the touch area with a larger area (such as the thumb knuckle), and the area-based pressing mode can trigger preset macro commands or special skills. In VR / AR interactive applications, when a user grasps a virtual object, the area-based pressing mode simulates the gripping force by recognizing changes in the contact area and total pressure. In automotive smart cockpit applications, the area-based pressing mode can realize shortcut command functions. For example, lightly tapping the armrest area with the palm can be defined as a custom shortcut command, such as "mute," "answer a call," or "navigate home," enabling efficient blind operation.

[0081] In some preferred embodiments of this example, such as Figure 3 As shown, if the displacement of the pressure center is greater than the displacement threshold in step S5, step S9 can also be executed.

[0082] S9, determine that the current touch operation is a swipe operation.

[0083] When executing step S5, the displacement of the pressure center is judged. Once it is determined that the displacement of the pressure center exceeds the preset displacement threshold, step S9 will be executed first, which means that the current touch operation is a sliding operation.

[0084] Specifically, in the scenario described above, if the current touch operation does not meet the conditions for either a region press operation or a tangential force operation, it will further distinguish between tangential force operations and sliding operations. At this point, the displacement of the pressure center becomes the key criterion: if the displacement is less than a displacement threshold, the system will enter the tangential force operation recognition process; if the displacement is greater than the displacement threshold, the current touch action can be accurately determined as a sliding operation. This method, which relies on comparing the displacement of the pressure center with a displacement threshold to accurately distinguish between sliding operations and tangential force operations, greatly enhances the accuracy of touch operation recognition. It allows the device to more accurately capture the user's operational intent and improve the user's interactive experience.

[0085] In some optional implementations of this embodiment, such as Figure 3 As shown, when the operation is determined to be sliding, S10 outputs a corresponding control signal based on the displacement of the pressure center.

[0086] In this embodiment, when the current operation is determined to be a sliding operation, the corresponding control signal is output based on the displacement of the pressure center. Taking a traditional touchpad as an example, the displacement of a user's finger sliding on the touchpad is converted into a control signal for the cursor's position or speed on the screen. Similarly, in this embodiment, when a sliding operation is determined, the displacement generated by the pressure center is like the sliding displacement of a finger on the touchpad. The system will generate a corresponding position control signal (such as controlling an element on the screen to move to a specific position) or speed control signal (such as controlling the scrolling speed of the scroll bar) based on this pressure center displacement.

[0087] This embodiment outputs a sliding operation control signal based on the displacement of the pressure center, which can quickly and accurately convert the user's sliding action into a position or speed control signal, thereby improving the user's interactive experience.

[0088] The above embodiments demonstrate that the recognition of swipe gestures can be applied to various scenarios, such as game controllers, VR / AR interactive applications, and automotive smart cockpits. In game controller applications, the swipe mode can be used to quickly switch perspectives or move the cursor. In VR / AR interactive applications, the swipe mode can be used to scroll or move within virtual menus. In automotive smart cockpit applications, the swipe mode is used to pan the map in map navigation interfaces; and in media and environmental control, it can be used to quickly switch tracks or functions in a list.

[0089] In some optional embodiments of this example, step S2, which calculates the pressure center location based on the pressure distribution data, specifically includes:

[0090] S211, the pressure distribution data is filtered and normalized to obtain preprocessed pressure distribution data.

[0091] In this embodiment, Gaussian filtering with a standard deviation σ = 1.0 can be used for filtering. Gaussian filtering reduces noise in the data, making the pressure distribution data smoother and preventing inaccurate calculations due to noise interference. Normalization maps the pressure distribution data to a specific range, making pressure data collected under different conditions comparable and facilitating subsequent calculations and analysis. After filtering and normalization, the preprocessed pressure distribution data is obtained.

[0092] S212, filter the pixels in the preprocessed pressure distribution data whose pressure values ​​are greater than the first pressure threshold to obtain the effective pressure points.

[0093] In the preprocessed pressure distribution data of this embodiment, each pixel has a corresponding pressure value. By comparing it with a first pressure threshold, pixels with pressure values ​​greater than the threshold are selected. These pixels represent areas with relatively high pressure that may be of practical significance for subsequent calculations, and are defined as valid pressure points. This eliminates some pixels with too low pressure, which may be interfering or have no practical effect, reducing unnecessary computation.

[0094] S213, calculate the weighted centroid of the effective pressure point and the corresponding pressure value to obtain the pressure center position.

[0095] For example, the weighted centroid formula is used only for the effective pressure point ( Calculations are performed using the first pressure threshold.

[0096] ,

[0097] ,

[0098] in, This represents the coordinates of the center of pressure along the x-axis. This indicates the coordinates of the pressure center along the y-axis.

[0099] This embodiment first performs Gaussian filtering and normalization preprocessing on the pressure distribution data, effectively reducing data noise and making the data smoother. Next, it filters valid pressure points, eliminating possible interference and pixels with no practical effect, greatly reducing unnecessary computation and improving computational efficiency. Finally, it uses the weighted centroid formula to calculate and accurately determine the coordinates of the pressure center by comprehensively considering the position of the valid pressure points and the pressure value, providing a reliable basis for subsequent identification operations.

[0100] In some optional embodiments of this example, step S2, which calculates the pressure distribution morphology descriptor based on the pressure distribution data using the covariance matrix, further includes:

[0101] S221a, calculate the weighted covariance matrix of the pressure distribution based on the pressure distribution data and the location of the pressure center.

[0102] Based on pressure distribution data and the COP (pressure center location) The weighted covariance matrix is ​​calculated according to a specific formula, and this matrix reflects the statistical characteristics of the pressure distribution. For example, the weighted covariance matrix... ,in For pixels Pressure value, and It is a non-negative integer used to define the order of the central moment.

[0103] S222a, perform eigenvalue decomposition on the weighted covariance matrix to obtain the maximum and minimum eigenvalues, as well as the corresponding eigenvectors.

[0104] The weighted covariance matrix calculated earlier Perform eigenvalue decomposition to obtain the largest eigenvalue of the matrix. Minimum eigenvalue And their respective eigenvectors.

[0105] S223a, determine the direction of the principal axis based on the largest eigenvalue and the corresponding eigenvector.

[0106] Using the largest eigenvalue The corresponding eigenvector determines the principal axis direction θ, which represents the most dominant extension direction of the pressure distribution.

[0107] S224a, calculate the eccentricity based on the minimum eigenvalue and the corresponding eigenvector.

[0108] Through the minimum eigenvalue The eccentricity e is calculated using the corresponding eigenvector. Eccentricity reflects the ellipticity or flattening of the pressure distribution. For example, e... .

[0109] S225a, a pressure distribution morphology descriptor is obtained based on the principal axis direction and eccentricity.

[0110] The pressure distribution morphology descriptor can be defined as a two-dimensional vector: .

[0111] This embodiment proposes a method for calculating a pressure distribution morphology descriptor based on covariance matrix analysis. First, the pressure-weighted centroid is calculated as the analytical benchmark. Then, a covariance matrix is ​​constructed to describe the statistical characteristics of the pressure distribution, and key features are obtained through eigenvalue decomposition. Finally, a descriptor is constructed to quantify the elliptical morphology, encompassing the extension direction and flattening degree. When a tangential force is applied, the key parameters of the descriptor change significantly. This method can accurately quantify the morphological characteristics of pressure distribution, providing an effective tool for analyzing situations such as fingertip deformation, and helping to deepen the study of pressure distribution variation patterns, thereby improving the accuracy of tangential force operation identification.

[0112] In some optional embodiments of this example, step S2, which calculates the pressure distribution morphology descriptor based on the pressure distribution data using a dual centroid offset, further includes:

[0113] S221b performs binarization on the pressure distribution data and calculates the position of the geometric centroid.

[0114] This embodiment performs binarization processing on the pressure distribution data (e.g., using the Otsu adaptive thresholding method), converting the pressure data into a binary form. In the obtained binarized contact region, the weight of all pixels is set to 1, thereby calculating the position of the geometric centroid. .

[0115] S222b calculates the offset vector based on the position of the geometric centroid and the position of the pressure center, which serves as a descriptor for the pressure distribution morphology.

[0116] Based on the position of the geometric centroid and the center of pressure COP ( The offset vector between the two is calculated, and this offset vector serves as the pressure distribution morphology descriptor. For example, the pressure distribution morphology descriptor... .

[0117] This embodiment proposes a method for calculating the pressure distribution morphology descriptor based on dual-centroid offset analysis. Utilizing the phenomenon that applying a tangential force causes the pressure center to separate from the geometric center, a simple binarization process and centroid calculation are used to obtain the offset vector as the morphology descriptor. This descriptor not only directly indicates the direction of the tangential force, but its magnitude also reflects the force's magnitude. In determining the tangential force, with the initial contact offset vector close to (0,0) as a reference, when applying a tangential force significantly increases the magnitude of the offset vector and exceeds a threshold, it can be accurately determined as a tangential force pattern. This method provides an intuitive and effective approach to tangential force determination, helping to simplify related detection processes and improve detection accuracy and efficiency. It has significant application value in fields involving pressure distribution analysis and tangential force detection.

[0118] Furthermore, determining the tangential force operation also includes:

[0119] Step 1: Calculate the modulus based on the offset vector to obtain the morphological change.

[0120] Due to the initial pressure distribution morphology descriptor at initial contact. It is close to (0,0). When the user applies a tangential force, the pressure peak concentrates in the direction of the force, causing the pressure center position COP to deviate from the geometric centroid position GC. The modulus is calculated based on the offset vector of the pressure distribution morphology descriptor calculated in step S222b, and this modulus is the morphological change.

[0121] Step two: Determine if the amount of morphological change is greater than the morphological change threshold. If the amount of morphological change is greater than the morphological change threshold, proceed to step three.

[0122] Step 3: Determine that the current touch operation is a tangential force operation.

[0123] Based on the morphological change calculated from the offset vector, when the morphological change is greater than this threshold, it means that the degree of change in pressure distribution has exceeded the range of change that may be brought about by normal fluctuations or non-tangential force operations. At this time, it can be determined that the current operation may be a tangential force operation. If the morphological change is less than or equal to the threshold, it means that the change in pressure distribution is not significant, and the current operation may not be a tangential force operation.

[0124] This embodiment calculates morphological changes based on offset vectors, comprehensively reflecting pressure distribution and unaffected by local fluctuations. This improves the accuracy of tangential force operation recognition, enhancing user experience and equipment usability.

[0125] In some optional embodiments of this example, step S2, which calculates the pressure distribution morphology descriptor based on the pressure distribution data by calculating in the gradient domain and frequency domain, further includes:

[0126] S221c uses the gradient operator to calculate the gradient field of the pressure distribution data, and calculates the weighted average direction based on the gradient magnitude of each pixel in the gradient field to obtain the first pressure distribution morphology descriptor component.

[0127] First, a pressure distribution morphological descriptor is constructed by calculating gradient domain features: Gradient operators (such as the Sobel operator, Scharr operator, and Laplacian operator) are used to calculate the gradient field on the pressure distribution data. The gradient field reflects the changes in pressure at various locations. Then, the weighted average direction is calculated based on the gradient magnitude of each pixel in the gradient field. This direction represents the overall trend of the most drastic pressure changes and is used as the first component of the pressure distribution morphological descriptor.

[0128] S222c: Extract the pressure distribution contour from the binarized pressure distribution data, convert the coordinates of the pressure distribution contour into a complex sequence, perform a fast Fourier transform on the complex sequence to obtain frequency domain coefficients, select the amplitude of the low-frequency coefficients in the frequency domain coefficients to obtain the second pressure distribution morphology descriptor component.

[0129] Then, the frequency domain features are calculated to construct a pressure distribution morphology descriptor: First, the pressure distribution data is binarized, and then the pressure distribution contour is extracted. The coordinates of the contour are transformed into a complex sequence, and then a Fast Fourier Transform is performed on this complex sequence to obtain the frequency domain coefficients. The amplitudes of the low-frequency coefficients (e.g., the first 16) are selected from the frequency domain coefficients. These amplitudes can robustly represent the contour shape and are used as the second pressure distribution morphology descriptor component.

[0130] S223c uses multiple filters in different directions to filter the pressure distribution data, calculates the total energy of the response after filtering each filter, and sums all the total energy of the response to obtain the third pressure distribution morphology descriptor component.

[0131] The pressure distribution data is filtered using multiple filters in different directions. The total energy of the response after filtering each filter is calculated. The total energy of all responses is then added together to obtain the third pressure distribution morphology descriptor component. This component can reflect the comprehensive characteristics of the pressure distribution under different directional filtering.

[0132] S224c, The pressure distribution morphology descriptor is obtained based on the first pressure distribution morphology descriptor component, the second pressure distribution morphology descriptor component, and the third pressure distribution morphology descriptor component.

[0133] The first, second, and third pressure distribution morphology descriptor components obtained earlier are combined to form the final pressure distribution morphology descriptor, thereby comprehensively describing the morphological characteristics of the pressure distribution.

[0134] This embodiment constructs a pressure distribution morphology descriptor based on gradient domain and frequency domain analysis. In the gradient domain, by calculating the gradient field and weighted average direction, the overall trend of the most dramatic pressure changes can be captured, providing effective information for analyzing the dynamic changes of pressure distribution. In the frequency domain, the low-frequency coefficient amplitudes of the pressure distribution contour are extracted using the Fourier descriptor, which can robustly represent the contour shape and enhance the stability and versatility of the descriptor. In addition, the third component obtained by filtering with filters in different directions and summing the energy further enriches the feature dimensions of the descriptor. The pressure distribution morphology descriptor constructed by combining these features can comprehensively and accurately describe the morphology of pressure distribution from multiple perspectives, which helps to analyze the edge, texture, and contour features of pressure distribution in greater depth, thereby improving the accuracy of tangential force operation recognition.

[0135] In some optional embodiments of this example, step S2, which calculates the pressure distribution morphology descriptor based on the pressure distribution data, further includes:

[0136] S221d performs Gaussian smoothing on the pressure distribution data at different scales and calculates the pressure distribution morphology descriptor at each scale level. The pressure distribution morphology descriptor at each scale level is any one of the following: covariance matrix, double centroid offset, gradient domain, and frequency domain.

[0137] Gaussian smoothing of the pressure distribution data at different scales (e.g., standard deviations of 0.5, 1.0, 2.0, and 4.0) is applied to construct an image pyramid. Then, at each smoothed scale layer, one of the methods mentioned in the previous embodiments (such as covariance matrix, dual centroid shift, gradient domain and frequency domain analysis) is selected to calculate the pressure distribution morphological descriptor.

[0138] S222d assigns basic weights to the pressure distribution morphology descriptors for each scale layer and calculates confidence levels.

[0139] A basic weight is assigned to each pressure distribution morphology descriptor calculated at each scale level, reflecting the importance of the descriptor in the initial stage. Simultaneously, a fixed-length (e.g., 10-frame) time series is maintained for each pressure distribution morphology descriptor at each scale level. Confidence scores are calculated based on these time series; confidence scores can be understood as an assessment of the reliability of the descriptor results. In addition to confidence scores, angular consistency (the standard deviation of the historical angle sequence) can be analyzed. A smaller standard deviation indicates a more stable method. Temporal continuity (the average angle change between adjacent frames) is also analyzed. A smoother change indicates better continuity.

[0140] S223d, adjusts the base weights of the pressure distribution morphology descriptor for each scale layer based on the confidence level.

[0141] Based on the confidence level calculated in step S222d, the basic weights of the pressure distribution morphology descriptors for each scale layer are adjusted. If a descriptor has a high confidence level, its weight may increase; conversely, its weight may decrease.

[0142] S224d, the pressure distribution morphology descriptors of each scale layer are fused according to the corrected weights to obtain the pressure distribution morphology descriptors.

[0143] Using the corrected weights, the pressure distribution morphological descriptors of each scale layer are fused together to obtain a comprehensive pressure distribution morphological descriptor.

[0144] This embodiment constructs a multi-scale, multi-modal fusion and temporal consistency analysis method. Multi-scale analysis effectively addresses different contact areas and morphologies by applying Gaussian smoothing to pressure data at different scales and calculating descriptors at each scale level, comprehensively capturing pressure distribution characteristics. In terms of intelligent weight fusion, basic weight allocation assigns higher weights to more stable global features in the initial stage, ensuring the basic reliability of the fusion results. Dynamic weight adjustment is based on temporal consistency; through historical data management, reliability assessment, and weight updates, weights are dynamically optimized according to the actual performance of the descriptors, making the results more consistent with reality. Finally, by fusing the descriptors through weighted averaging, the output tangential force direction and intensity exhibit high robustness and accuracy.

[0145] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A three-dimensional force touch operation recognition method, characterized by, The method comprises: obtaining pressure distribution data corresponding to a touch operation; calculating basic feature information of the touch operation according to the pressure distribution data, including a pressure center position and a pressure distribution pattern descriptor; calculating a pressure center displacement according to the pressure center position at a current time and the pressure center position at a starting time; calculating a pressure distribution pattern change amount according to the pressure distribution pattern descriptor at the current time and the pressure distribution pattern descriptor at the starting time; judging whether the pressure center displacement is less than a displacement threshold value; if the pressure center displacement is less than the displacement threshold value, judging whether the pressure distribution pattern change amount is greater than a pattern change amount threshold value; if the pressure distribution pattern change amount is greater than the pattern change amount threshold value, determining that the current touch operation is a tangential force operation, and calculating a force vector according to the pressure distribution pattern change amount; outputting a corresponding control signal according to the force vector.

2. The method of claim 1, wherein, The basic feature information further includes a contact area; before calculating the pressure center displacement, the method further comprises: judging whether the contact area is greater than a contact area threshold value; if the contact area is greater than the contact area threshold value, determining that the current touch operation is a region pressing operation.

3. The method of claim 2, wherein, When it is determined to be a region pressing operation, calculating a total pressure by calculating a sum of pressure values of all pixel points according to the pressure distribution data, and outputting a corresponding control signal according to the contact area and the total pressure.

4. The method of claim 1, wherein, If the pressure center displacement is greater than the displacement threshold value, it is determined that the current touch operation is a sliding operation.

5. The method of claim 4, wherein, When it is determined to be a sliding operation, outputting a corresponding control signal according to the pressure center displacement.

6. The method of claim 1, wherein, The pressure center position is calculated according to the pressure distribution data, comprising: performing filtering and normalization processing on the pressure distribution data to obtain preprocessed pressure distribution data; screening pixel points with pressure values greater than a first pressure threshold value in the preprocessed pressure distribution data to obtain effective pressure points; performing weighted centroid calculation on positions of the effective pressure points and corresponding pressure values to obtain the pressure center position.

7. The method of claim 1, wherein, The pressure distribution pattern descriptor is calculated by a covariance matrix, comprising: calculating a weighted covariance matrix of pressure distribution according to the pressure distribution data and the pressure center position; performing eigenvalue decomposition on the weighted covariance matrix to obtain a maximum eigenvalue and a minimum eigenvalue, and corresponding eigenvectors; determining a principal axis direction according to the maximum eigenvalue and the corresponding eigenvectors; calculating an eccentricity according to the minimum eigenvalue and the corresponding eigenvectors; obtaining the pressure distribution pattern descriptor according to the principal axis direction and the eccentricity.

8. The method of claim 1, wherein, The pressure distribution pattern descriptor is calculated by a double centroid offset, comprising: performing binaryzation processing on the pressure distribution data, and calculating a geometric centroid position; calculating an offset vector according to the geometric centroid position and the pressure center position, to serve as the pressure distribution pattern descriptor.

9. The method of claim 1, wherein, The pressure distribution pattern descriptor is calculated by a gradient domain and a frequency domain, comprising: calculating a gradient field of the pressure distribution data by using a gradient operator, and calculating a weighted average direction according to gradient amplitudes of pixel points in the gradient field to obtain a first pressure distribution pattern descriptor component; calculating a second pressure distribution pattern descriptor component according to a frequency domain of the pressure distribution data; extracting a pressure distribution profile in the pressure distribution data after binarization, converting coordinates of the pressure distribution profile into a complex sequence, performing fast Fourier transform on the complex sequence to obtain frequency domain coefficients, selecting amplitudes of low frequency coefficients in the frequency domain coefficients to obtain a second pressure distribution pattern descriptor component; filtering the pressure distribution data using a plurality of filters in different directions, respectively calculating response total energy after filtering of each filter, summing all response total energy to obtain a third pressure distribution pattern descriptor component; obtaining the pressure distribution pattern descriptor according to the first pressure distribution pattern descriptor component, the second pressure distribution pattern descriptor component and the third pressure distribution pattern descriptor component.

10. The method according to any one of claims 7-9, characterized in that, calculating a pressure distribution pattern descriptor according to the pressure distribution data, further comprising: performing Gaussian smoothing on the pressure distribution data in different scales, and calculating a pressure distribution pattern descriptor on each scale layer, the pressure distribution pattern descriptor on each scale layer being any one of a covariance matrix, a double centroid offset, a gradient domain and a frequency domain; allocating a basic weight to the pressure distribution pattern descriptor of each scale layer, and performing confidence calculation; correcting the basic weight of the pressure distribution pattern descriptor of each scale layer according to the confidence; fusing the pressure distribution pattern descriptor of each scale layer according to the corrected weight to obtain the pressure distribution pattern descriptor.

11. The method of claim 8, wherein, determining the tangential force operation further comprises: performing modulus length calculation according to the offset vector to obtain a pattern change amount; judging whether the pattern change amount is greater than a pattern change amount threshold; if the pattern change amount is greater than the pattern change amount threshold, determining that the current touch operation is a tangential force operation.

12. A three-dimensional force touch operation recognition device, characterized by comprising: comprising: a processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to make the processor execute the three-dimensional force touch operation recognition method in any one of claims 1-11.

Images