Three-dimensional relative touch force estimation and man-machine interaction method based on touch sensor image

Abstract

The invention provides a three-dimensional relative touch force estimation and man-machine interaction method based on touch sensor images. The method comprises the following steps: acquiring continuous touch sensor images in real time; multiple adjacent frames of touch sensor images are selected from the continuous touch sensor images for alignment each time, then the trained three-dimensional relative touch force estimation model is input to output a three-dimensional relative touch force estimation value corresponding to the multiple adjacent frames of touch sensor images each time, and the three-dimensional relative touch force is the variable quantity of the three-dimensional touch force. The three-dimensional touch force comprises pressure perpendicular to the surface of the touch sensor, and transverse force and longitudinal force parallel to the surface of the touch sensor; and executing an action according to a preset interaction mode based on the three-dimensional relative touch force estimated value obtained in the set time period. By means of the method, force measurement based on touch control can be more accurate, and then the accuracy of man-machine interaction is improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction, and in particular to a three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images. Background Art

[0002] With the improvement of the performance of smart electronic devices and the advancement of production technology, touch input devices have been widely used in various consumer electronic products, such as laptops, smart phones, smart watches and virtual reality (VR) devices. Traditional touch input devices use low-resolution capacitive images to determine the contact point between the finger and the sensor, and move the cursor or recognize gesture operations according to the change in the contact point position. However, since the finger surface is prone to deformation, the contact area with the sensor changes greatly, which will cause errors in the judgment of the contact point position, thereby reducing the accuracy of finger touch. In addition, in order to consider the ease of use of the above products, touch input devices are usually designed to be smaller in area, which requires users to slide their fingers on the sensor multiple times to move the cursor or screen to a specific position, thereby reducing the operating efficiency. At the same time, the capacitive images generated by traditional touch input devices have low resolution and contain limited information. They are usually only used to determine the coordinates of the contact point position, which greatly limits the diversity of interaction methods.

[0003] Some mobile phones have built-in pressure-sensitive sensors to measure normal pressure. Some researchers have proposed installing pressure-sensitive sensors on the sides of mobile phones to measure horizontal forces. However, these solutions increase hardware costs and the size of mobile phones, which also brings challenges to the hardware design of mobile phones.

[0004] Some researchers have proposed using touch images to measure force, but such methods only use a single touch image for pressure estimation. Due to the different sizes and shapes of different users and fingers, this method has large errors and poor robustness when estimating force, and cannot be used for continuous interactive operations. In addition, the existing method only estimates the normal force perpendicular to the plane, which can only provide one operating dimension, and has limited efficiency improvement on existing touch interaction methods. In addition, the existing method does not solve the compatibility of new gestures with traditional interactive gestures, and cannot accurately distinguish between multiple gestures.

[0005] Therefore, more accurate touch-based force measurement and interaction technologies still need to be developed. Summary of the invention

[0006] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0007] To this end, a first object of the present invention is to propose a three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images, so as to make the touch-based force measurement more accurate, thereby improving the accuracy of human-computer interaction.

[0008] The second objective of the present invention is to provide a three-dimensional relative touch force estimation and human-computer interaction system based on touch sensor images.

[0009] A third objective of the present invention is to provide an electronic device.

[0010] A fourth object of the present invention is to provide a computer-readable storage medium.

[0011] To achieve the above-mentioned purpose, the first aspect of the present invention proposes a three-dimensional relative touch force estimation and human-computer interaction method based on a touch sensor image, comprising:

[0012] Acquire continuous touch sensor images in real time;

[0013] Selecting adjacent multiple frames of touch sensor images from the continuous touch sensor images for alignment each time, and then inputting a trained three-dimensional relative touch force estimation model to output a three-dimensional relative touch force estimation value corresponding to each adjacent multiple frames of touch sensor images, wherein the three-dimensional relative touch force is a change in the three-dimensional touch force, and the three-dimensional touch force includes a pressure perpendicular to a touch sensor surface, and a lateral force and a longitudinal force parallel to the touch sensor surface;

[0014] An action is performed in a preset interactive manner based on the three-dimensional relative touch force estimation value obtained within a set time period.

[0015] In the method of the first aspect of the present invention, each time adjacent multiple frames of touch sensor images are selected from the continuous touch sensor images for alignment, the method includes: each time adjacent multiple frames of touch sensor images are selected from the continuous touch sensor images, the center points of the finger areas of the adjacent multiple frames of touch sensor images are respectively calculated, and the adjacent multiple frames of touch sensor images are aligned according to the center points of the finger areas.

[0016] In the method of the first aspect of the present invention, the aligning of the adjacent multiple frames of touch sensor images according to the center point of the finger area includes: if each frame of the adjacent multiple frames of touch sensor images has a center point of a finger area, the adjacent multiple frames of touch sensor images are aligned based on the center point of the one finger area; if each frame of the adjacent multiple frames of touch sensor images has multiple finger areas, each alignment processing is for one target finger, and the adjacent multiple frames of touch sensor images are aligned based on the center point of the finger area of ​​the target finger in each frame of the touch sensor image.

[0017] In the method of the first aspect of the present invention, the three-dimensional relative touch force estimation model includes a force estimation module and multiple feature extraction modules, the number of feature extraction modules is equal to the number of frames of the adjacent multiple frames of touch sensor images, the input of each feature extraction module is a touch sensor image in the adjacent multiple frames of touch sensor images, and the output of all feature extraction modules is the input of the force estimation module.

[0018] In the method of the first aspect of the present invention, the three-dimensional relative touch force estimation model is constructed based on a deep neural network.

[0019] In the method of the first aspect of the present invention, the preset interaction mode includes a first mapping relationship between multiple scene signals and multiple interaction modes, wherein the multiple interaction modes include a discrete mode and a continuous mode, and when the interaction mode is a discrete mode, the preset interaction mode also includes a second mapping relationship between multiple operation intention signals and multiple functions, and when the interaction mode is a continuous mode, the preset interaction mode also includes a third mapping relationship between three-dimensional relative touch force estimation values ​​and multiple functions.

[0020] In the method of the first aspect of the present invention, the performing of an action according to a preset interaction method based on the three-dimensional relative touch force estimation value obtained within a set time period includes: determining a current interaction mode based on a detected current scene signal and the corresponding preset interaction method; if the current interaction mode is a discrete mode, determining a current operation intention signal based on the three-dimensional relative touch force estimation value obtained within a set time period; determining a required function based on a second mapping relationship between the current operation intention signal and the corresponding preset interaction method; and performing an action according to the required function.

[0021] To achieve the above-mentioned purpose, the second aspect of the present invention proposes a three-dimensional relative touch force estimation and human-computer interaction system based on touch sensor images, comprising:

[0022] A touch sensor image acquisition module, used for acquiring continuous touch sensor images in real time;

[0023] a three-dimensional relative touch force estimation module, configured to select adjacent multiple frames of touch sensor images from the continuous touch sensor images for alignment each time, and then input a trained three-dimensional relative touch force estimation model to output a three-dimensional relative touch force estimation value corresponding to each adjacent multiple frames of touch sensor images, wherein the three-dimensional relative touch force is a variation of the three-dimensional touch force, and the three-dimensional touch force includes a pressure perpendicular to a touch sensor surface, and a lateral force and a longitudinal force parallel to the touch sensor surface;

[0024] The interactive control module is used to perform an action in a preset interactive manner based on the three-dimensional relative touch force estimation value obtained within a set time period.

[0025] To achieve the above-mentioned purpose, the third aspect of the present invention proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method proposed in the first aspect of the present invention.

[0026] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method proposed in the first aspect of the present invention.

[0027] The three-dimensional relative touch force estimation and human-computer interaction method, system, electronic device and storage medium based on touch sensor images provided by the present invention obtain continuous touch sensor images in real time; select adjacent multiple frames of touch sensor images from the continuous touch sensor images for alignment each time, then input the trained three-dimensional relative touch force estimation model to output the three-dimensional relative touch force estimation value corresponding to each adjacent multiple frames of touch sensor images, wherein the three-dimensional relative touch force is the variation of the three-dimensional touch force, and the three-dimensional touch force includes the pressure perpendicular to the touch sensor surface, the lateral force and the longitudinal force parallel to the touch sensor surface; based on the three-dimensional relative touch force estimation value obtained within a set time period, an action is performed in a preset interaction mode. In this case, the pressure perpendicular to the touch sensor surface, the lateral force and the longitudinal force parallel to the touch sensor surface are comprehensively considered, and the three-dimensional relative touch force is estimated using adjacent multiple frames of touch sensor images, so that the force measurement based on touch is more accurate. In addition, the action is performed in a preset interaction mode based on the three-dimensional relative touch force estimation value obtained within a set time period, which makes full use of the three-dimensional contact force and improves the accuracy of human-computer interaction.

[0028] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A schematic flow chart of a method for three-dimensional relative touch force estimation and human-computer interaction based on touch sensor images provided by an embodiment of the present invention;

[0031] Figure 2 This is an example diagram of the touch sensor image when the touch sensor is pressed with different strengths and directions provided by the embodiment of the present invention;

[0032] Figure 3A schematic diagram of finger detection provided by an embodiment of the present invention;

[0033] Figure 4 An example diagram of a three-dimensional relative touch force estimation model provided by an embodiment of the present invention;

[0034] Figure 5 An example diagram of a custom gesture operation in a drawing application provided by an embodiment of the present invention;

[0035] Figure 6 An example diagram of a multi-level turntable menu interaction provided by an embodiment of the present invention;

[0036] Figure 7 An example diagram of continuous variable control of a palette provided in an embodiment of the present invention;

[0037] Figure 8 A schematic diagram of multi-finger interaction provided by an embodiment of the present invention;

[0038] Fig. 9 A block diagram of a three-dimensional relative touch force estimation and human-computer interaction system based on touch sensor images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0040] The following describes a three-dimensional relative touch force estimation and human-computer interaction method, system, electronic device, and storage medium based on touch sensor images according to embodiments of the present invention with reference to the accompanying drawings.

[0041] Touch input devices for human-computer interaction, such as laptop trackpads, touchpads in controllers, and screens of smartphones and smartwatches, are limited by the small touch area, resulting in low control accuracy and limited interaction methods, which affects the user's efficiency and experience. Force plays an important role in people's daily interactions with their surroundings, but most touch interactions only use touch position, ignoring the potential of using force as an additional input method.

[0042] Based on this, an embodiment of the present invention provides a three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images, so as to make the touch-based force measurement more accurate, thereby improving the accuracy of human-computer interaction, and at the same time solving the problems of low control accuracy and limited interaction methods due to the limitation of a small touch area.

[0043] Figure 1A schematic flow chart of a three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images provided by an embodiment of the present invention.

[0044] like Figure 1 As shown, the three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images includes the following steps:

[0045] Step S101, acquiring continuous touch sensor images in real time.

[0046] In step S101, a touch sensor image is collected in real time through a touch sensor. The touch sensor may be, for example, a touch sensor used in a common touch input device (such as a mobile phone touch screen, a computer touch pad, etc.). The touch sensor image may be referred to as a touch image.

[0047] In step S101, the type of the touch sensor may be a capacitive touch sensor, and the touch sensor image is a touch sensor capacitive image. The touch sensor capacitive image may be referred to as a capacitive image. It should be noted that the type of touch sensor used in the embodiment of the present invention may also be an acoustic, electrical, optical or other touch sensor. An acoustic touch sensor may be, for example, an ultrasonic fingerprint sensor, and an optical touch sensor may be, for example, an optical fingerprint sensor.

[0048] In step S101 , considering that human-computer interaction has the characteristics of continuity and real-time, the present invention needs to acquire continuous touch sensor images in real time.

[0049] Figure 2 This is an example of a touch sensor image when the touch sensor is pressed with different forces and directions provided by an embodiment of the present invention. Figure 2 (a) and Figure 2 (b) is two frames of touch sensor images obtained when the touch sensor is pressed with different forces and directions. Each frame of the touch sensor image takes the upper left corner of the image as the origin of the pixel coordinate system, and the horizontal and vertical coordinates of each pixel in each frame of the touch sensor image are the column and row numbers of the pixel, respectively.

[0050] In step S101, finger regions need to be detected for each frame of the touch sensor image. Specifically, firstly, whether the touch sensor image contains a finger region is determined by threshold detection; then, the number of finger regions and the position of each finger region are determined by the maximum connected domain. When multiple finger regions are detected, each finger region is segmented so that the relative contact force can be estimated separately later. Figure 3 This is a schematic diagram of finger detection provided by an embodiment of the present invention. Figure 3 The area within the red frame in the touch sensor image shown is the detected finger area. Figure 3The touch sensor image shown contains 2 finger areas.

[0051] Step S102 , each time selecting multiple adjacent touch sensor image frames from continuous touch sensor images for alignment, and then inputting a trained 3D relative touch force estimation model to output 3D relative touch force estimation values ​​corresponding to each multiple adjacent touch sensor image frames.

[0052] Specifically, considering that the touch sensor is affected by the size, force, posture and other conditions of the contacting finger, different touch sensor images are output. The traditional absolute force estimation method based on a single-frame image cannot distinguish the factors affecting the image change, resulting in a large force estimation error. Therefore, in step S102 of the present invention, adjacent multi-frame touch sensor images are used to accurately estimate the three-dimensional relative touch force by extracting the changes in image features.

[0053] In step S102, the three-dimensional relative touch force is the change in the three-dimensional touch force. Specifically, the relative touch force is defined as the change in the touch force applied by the finger on the touch sensor at any two time points during the process of the finger contacting the touch sensor. It should be noted that the method of the present invention cannot directly measure the absolute force, but can measure it indirectly. For example, when a terminal device such as a mobile phone is equipped with a sensor for measuring absolute pressure, the three-dimensional relative touch force measured by the present invention can be mapped to an absolute force according to a certain mapping function.

[0054] In step S102, the three-dimensional touch force includes pressure perpendicular to the touch sensor surface, lateral force and longitudinal force parallel to the touch sensor surface. The pressure is the normal force, and the lateral force and longitudinal force are both tangential forces and are orthogonal in direction. For example, the lateral force can be considered as the tangential force in the left and right directions of the touch screen, and the lateral force includes the left tangential force and the right tangential force; the longitudinal force can be considered as the tangential force in the up and down directions of the touch screen, and the longitudinal force includes the upward tangential force and the downward tangential force.

[0055] In step S102, the trained three-dimensional relative touch force estimation model is used to extract image features and estimate the force of adjacent multi-frame touch sensor images to obtain a three-dimensional relative touch force estimation value. The three-dimensional relative touch force estimation model includes a force estimation module and a plurality of feature extraction modules, the number of feature extraction modules is equal to the number of frames of adjacent multi-frame touch sensor images, the input of each feature extraction module is a touch sensor image in the adjacent multi-frame touch sensor images, and the output of all feature extraction modules is the input of the force estimation module. In some embodiments, the three-dimensional relative touch force estimation model is constructed based on a deep neural network. The feature extraction module can be composed of a convolutional layer, a residual module, a residual module and a fully connected layer in a deep neural network, and the force estimation module can be composed of a plurality of fully connected layers in a deep neural network. For example, the deep neural network can select Resnet (deep residual network), VGG-Net (deep convolutional neural network), Transformer (neural network model based on self-attention mechanism), etc.

[0056] In step S102, the input data of the trained 3D relative touch force estimation model is aligned adjacent multi-frame touch sensor images, and the output data is 3D relative touch force estimation values, wherein the 3D relative touch force estimation values ​​include relative touch force estimation values ​​in three dimensions: pressure, lateral force, and longitudinal force.

[0057] In step S102, considering the continuity and real-time characteristics of human-computer interaction, the trained three-dimensional relative touch force estimation model is used to continuously output the three-dimensional relative touch force estimation value. Each output three-dimensional relative touch force estimation value corresponds to a group of aligned adjacent multi-frame touch sensor images. Here, each time a group of adjacent multi-frame touch sensor images is selected from the continuous touch sensor images obtained in step S101 for alignment, a group of aligned adjacent multi-frame touch sensor images can be obtained.

[0058] Each time, selecting adjacent multiple frames of touch sensor images from continuous touch sensor images for alignment specifically includes: selecting adjacent multiple frames of touch sensor images from continuous touch sensor images each time, respectively calculating the center points of the finger areas of the adjacent multiple frames of touch sensor images, and aligning the adjacent multiple frames of touch sensor images according to the center points of the finger areas. Thus, the three-dimensional relative touch force is estimated using the aligned adjacent multiple frames of touch sensor images, which can improve the accuracy of the estimation result.

[0059] It should be noted that the three-dimensional relative touch force estimation model trained in step S102 estimates the three-dimensional relative touch force of a finger each time. The method of the present invention can support the three-dimensional relative touch force estimation of multiple fingers at the same time. Specifically, adjacent multiple frames of touch sensor images are aligned according to the center point of the finger area, including: if each frame of the touch sensor image in the adjacent multiple frames of touch sensor images has a center point of a finger area, then the adjacent multiple frames of touch sensor images are aligned based on the center point of the one finger area; if each frame of the touch sensor image in the adjacent multiple frames of touch sensor images has multiple finger areas, then each alignment process is for one target finger, and the adjacent multiple frames of touch sensor images are aligned based on the center point of the finger area of ​​the target finger in each frame of the touch sensor image.

[0060] Therefore, if it is detected in step S101 that the touch sensor image includes multiple finger areas, step S102 performs multiple alignments for the multiple finger areas, and inputs the trained three-dimensional relative touch force estimation model into the aligned adjacent multiple frames of touch sensor images obtained for the same finger to obtain the three-dimensional relative touch force estimation value of the finger, and then obtains the three-dimensional relative touch force estimation value of each finger.

[0061] In step S102, the trained three-dimensional relative touch force estimation model is obtained by training the three-dimensional relative touch force estimation model using the training data set. In the offline training stage, the parameters of the three-dimensional relative touch force estimation model are optimized using a supervised machine learning method. The training data set can be obtained through a three-dimensional force sensor. Specifically, the touch input device is connected to the three-dimensional force sensor through an external bracket, and it is ensured that there is no relative sliding between the two. When a person's finger touches the touch input device with various forces, the data acquisition program simultaneously records the touch image sequence of the touch sensor and the three-dimensional force sequence of the three-dimensional force sensor. The touch image sequence includes multiple frames of continuous touch images, and the three-dimensional force sequence is a collection of three-dimensional absolute touch forces. The three-dimensional relative touch force sequence is calculated based on the three-dimensional absolute touch force, and the touch image sequence is used as input data for model training, and the three-dimensional relative touch force sequence is used as a label, thereby constructing a training data set.

[0062] Figure 4 This is an example diagram of a three-dimensional relative touch force estimation model provided by an embodiment of the present invention.

[0063] In some embodiments, the number of feature extraction modules is 2, and the adjacent multiple frames of touch sensor images are 2 adjacent frames of touch sensor images. Figure 4As shown, the three-dimensional relative touch force estimation model includes two feature extraction modules and one force estimation module. Each feature extraction module (also called feature extraction network) includes a convolution layer, a residual module, a residual module and a fully connected layer. The feature extraction module is used to extract features of the input touch sensor image and send the extracted features to the force estimation module (also called force estimation network). The force estimation module includes multiple fully connected layers. The force estimation module performs force estimation based on the features output by the two feature extraction modules to obtain a three-dimensional relative touch force estimation value. During training, only one feature extraction module can be trained, and then the weights can be shared with other feature extraction modules. XYZ is the predicted value of the three-dimensional relative touch force during training. Based on the predicted value and the true value of the relative force (i.e., the label in the training data set), the model parameters are iterated using a loss function.

[0064] use Figure 4 The three-dimensional relative touch force estimation model shown uses two adjacent frames of touch sensor images to accurately estimate the three-dimensional relative touch force by extracting changes in image features.

[0065] Specifically, two adjacent touch sensor image frames are selected from the continuous touch sensor image, the center points of the finger areas of the two touch sensor image frames are calculated respectively, and the two touch sensor image frames are aligned according to the center points. Then, the aligned images are input into two feature extraction modules respectively, and finally input into the force estimation module to perform three-dimensional relative force estimation to obtain a three-dimensional relative touch force estimation value. By continuously acquiring two adjacent touch sensor image frames within a set time period and repeating the above process, a continuous three-dimensional relative touch force estimation value can be obtained.

[0066] Step S103: performing an action in a preset interactive manner based on the three-dimensional relative touch force estimation value obtained within a set time period.

[0067] In step S103, the preset interaction method includes a first mapping relationship between multiple scene signals and multiple interaction modes, wherein the multiple interaction modes include a discrete mode and a continuous mode; when the interaction mode is a discrete mode, the preset interaction method also includes a second mapping relationship between multiple operation intention signals and multiple functions; when the interaction mode is a continuous mode, the preset interaction method also includes a third mapping relationship between three-dimensional relative touch force estimation values ​​and multiple functions.

[0068] The scene signals include but are not limited to the first-level scene signals, the second-level scene signals, and the third-level scene signals. The first-level scene signals are scene signals when no application is running; the second-level scene signals are scene signals when any application is running, such as editing document signals, drawing program signals, map program signals, etc.; the third-level scene signals are scene signals when subprograms under various applications are running, such as two-dimensional joystick program signals and multi-dimensional joystick program signals under various applications.

[0069] The various functions include but are not limited to primary functions, secondary functions, and tertiary functions. The primary functions are the functions corresponding to the primary scene signals, such as calling out the main interface menu of a touch input device; the secondary functions are the functions corresponding to the secondary scene signals, such as copying, pasting, selecting, deleting, etc. corresponding to the editing document signal; brush selection, brush thickness, etc. corresponding to the painting program signal; translation, zooming, rotation, etc. corresponding to the map program signal; and the tertiary functions are the functions corresponding to the tertiary scene signals, such as the direction and speed of the target object corresponding to the 2D joystick program signal.

[0070] In step S103, an action is performed according to a preset interaction method based on the three-dimensional relative touch force estimation value obtained within a set time period, including: determining a current interaction mode based on a detected current scene signal and a corresponding preset interaction method; if the current interaction mode is a discrete mode, determining a current operation intention signal based on the three-dimensional relative touch force estimation value obtained within a set time period; determining a required function based on a second mapping relationship between the current operation intention signal and the corresponding preset interaction method; and performing an action according to the required function.

[0071] Among them, determining the current operation intention signal based on the three-dimensional relative touch force estimation value obtained within a set time period includes: within the set time period, determining the current operation intention signal based on the comparison result of the relative touch force estimation value in at least one direction of the three-dimensional relative touch force estimation value and the corresponding threshold.

[0072] The three-dimensional relative touch force estimation values ​​include but are not limited to estimation values ​​of relative forces in five directions: pressure, left tangential force, right tangential force, upward tangential force, and downward tangential force.

[0073] The operation intention signal of a single finger includes but is not limited to six gesture signals: a heavy pressure gesture signal and a light pressure gesture signal for pressure; a left rubbing gesture signal for a left tangential force; a right rubbing gesture signal for a right tangential force; an upward rubbing gesture signal for an upward tangential force; and a downward rubbing gesture signal for a downward tangential force.

[0074] When judging different gesture signals of a single finger, the maximum value of the estimated value of the relative force in the four tangential force directions of the finger in the same time period can be selected first. If the maximum value exceeds the set threshold of the corresponding direction, the direction corresponding to the maximum value is the rubbing direction of the finger (that is, the gesture signal of the finger is a rubbing gesture signal in this direction). If the estimated values ​​of the relative force in the four tangential force directions of the finger are all less than the set threshold of the corresponding direction, it is judged whether the pressure is greater than the pressure threshold. If it is greater than the pressure threshold, it is a heavy pressure gesture signal, otherwise it is a light pressure gesture signal.

[0075] In step S103, an action is performed according to a preset interaction mode based on the three-dimensional relative touch force estimation value obtained within a set time period, and also includes: if the current interaction mode is a continuous mode, a required function is determined based on a third mapping relationship between the obtained three-dimensional relative touch force estimation value and the corresponding preset interaction mode; and an action is performed according to the required function.

[0076] It should be noted that, if there are multiple fingers, the three-dimensional relative touch force estimation value and operation intention signal of each finger are determined separately.

[0077] The interactive control of the present invention is described below in conjunction with some specific scene signals, interactive modes and functions.

[0078] Considering that the interaction mode includes discrete mode and continuous mode, examples are given from two aspects respectively.

[0079] (1) Discrete mode:

[0080] a) Single-finger gesture interaction:

[0081] On the basis of traditional gesture operation, the single-finger gesture operation interaction of the present invention is a customizable gesture operation interaction based on multi-dimensional force. Compared with the existing menu operation, there is no need to move the cursor and click the menu continuously, which reduces misoperation and improves efficiency. In addition, users can customize different operation intention signals according to needs and scenarios.

[0082] For example, if the current scene signal is detected to be a document editing signal, it indicates that the current state is in the document editing state. Based on the corresponding preset interaction method, it is determined that the interaction mode under the document editing signal is a discrete mode, and then the current operation intention signal is determined; if the current operation intention signal is a light pressure gesture signal, based on the corresponding preset interaction method, it is determined that the required function is to move the cursor, and then the corresponding program is called to realize the cursor movement; if the current operation intention signal is a heavy pressure gesture signal, the menu is called out; if the current operation intention signal is an upward rubbing gesture signal, copying is performed; if the current operation intention signal is a downward rubbing gesture signal, pasting is performed; if the current operation intention signal is a right rubbing gesture signal, text selection is performed; if the current operation intention signal is a left rubbing gesture signal, deletion is performed.

[0083] Figure 5 This is an example diagram of a custom gesture operation in a drawing application provided by an embodiment of the present invention. For example, if the current scene signal is detected to be a drawing program signal, it indicates that the current state is drawing editing. Based on the corresponding preset interaction mode, the interaction mode under the drawing program signal is determined to be a discrete mode, and then the current operation intention signal is determined; if the current operation intention signal is an upward rubbing gesture signal or a downward rubbing gesture signal, a different brush type is selected; if the current operation intention signal is a rightward rubbing gesture signal or a leftward rubbing gesture signal, the brush thickness is selected (see Figure 5 ).

[0084] b) Multi-finger gesture interaction:

[0085] The present invention expands the traditional multi-finger discrete gesture operation into a multi-finger force gesture by estimating the three-dimensional relative touch force. The multi-finger gesture refers to determining the operation intention signal for each finger, and then integrating the operation intention signals of each finger to realize the corresponding function. The operation intention signal corresponding to the mapping relationship in the preset interaction mode corresponding to the multi-finger force gesture is a combination of multiple operation intention signals.

[0086] For example, traditional multi-finger pinch gestures and multi-finger sliding (sweeping) gestures require a large operating space, which often causes screen occlusion. It is difficult to use in small-screen touch input devices such as smart watches. In an embodiment of the present invention, for example, if the current scene signal is detected as a document editing signal, it indicates that the document is currently in the editing state, and the interaction mode under the document editing signal is determined to be a discrete mode based on the corresponding preset interaction mode, and then the current operation intention signal is determined; if the current operation intention signal is a combination of a left rubbing gesture signal and a right rubbing gesture signal, the required function is determined to be document enlargement based on the corresponding preset interaction mode, and the document enlargement function is executed. For example, under any scene signal, if the current operation intention signal is a combination of multiple downward rubbing gesture signals, a screenshot is executed. Since the finger can generate a rubbing gesture signal when the finger applies force but does not move when the tangential force direction corresponds to the set threshold, the finger does not need to move on the screen, does not require a large operating space, and reduces the occlusion of the screen content, solving the problem of low control accuracy and limited interaction methods due to the small touch area.

[0087] (2) Continuous mode:

[0088] a) Two-dimensional joystick:

[0089] Considering that the traditional two-dimensional joystick relies on mechanical structure to realize input control, it often fails as the internal structure wears out. The existing virtual joystick of touch screen mobile phone requires a relatively large working area. The method of the present invention simulates the operation mode of the joystick by the size and direction of the relative touch force estimation value, uses the touch sensor and the touch force measurement algorithm to realize the joystick function, and requires a very small working area while maintaining the control accuracy.

[0090] For example, if the current scene signal is detected as a two-dimensional joystick program signal under a certain application, and the interaction mode under the two-dimensional joystick program signal is determined to be a continuous mode based on the corresponding preset interaction mode, then based on the obtained three-dimensional relative touch force estimate and the determined required function, the action is performed according to the required function. Specifically, the relative tangential force estimate in each direction can simulate the direction and distance of pushing the joystick, and then map to the moving speed of the target object in the system. For example, the function corresponding to the relative tangential force estimate to the left is to control the target object's leftward and leftward movement speed. The larger the relative tangential force estimate to the left, the faster the target object moves to the left.

[0091] b) Multi-dimensional joystick:

[0092] The three-dimensional relative touch force estimation value obtained by the present invention can be used as a force input method in the interaction to combine the traditional touch input method based on the contact point position, thereby further expanding the input dimension, realizing a movable joystick, and supporting continuous value input of multiple degrees of freedom, such as 4 degrees of freedom. Considering that a certain downward pressure needs to be applied at the same time when applying tangential force, pressure can be used as a basis for switching touch input and force input, and tangential force is used as an additional two-degree-of-freedom input. For example, the application method in a 4-degree-of-freedom map is as follows:

[0093] Usually, multi-degree-of-freedom map control of touch input devices is generally controlled by single-finger plus multi-finger gesture control. This operation mode does not conform to human intuitive feelings and has low operation accuracy. Especially when holding a mobile phone in one hand, only the thumb is convenient to use, and it is difficult to implement multi-finger gestures. The present invention uses a three-dimensional relative touch force estimation value combined with traditional single-finger touch input to achieve 4-degree-of-freedom map precision control, including translation, rotation and zoom operations.

[0094] For example, if it is detected that the current scene signal is a multi-dimensional joystick program signal under a 3D map, the interaction mode under the multi-dimensional joystick program signal is determined to be a continuous mode based on the corresponding preset interaction mode, and the pressure in the three-dimensional relative touch force estimation value corresponds to the switching function. If the pressure is less than the pressure threshold, it corresponds to the operation under the traditional touch input mode. At this time, the unified touch input mode is entered, and the single-finger sliding operation of the traditional touch is used to control the translation of the map up, down, left and right; if the pressure is greater than the pressure threshold, it corresponds to the operation under the force input mode. At this time, the force input mode is entered. In the force input mode, if the upward or downward relative tangential force in the three-dimensional relative touch force estimation value is non-zero, it corresponds to the map zoom function, and the map zoom is executed. In the force input mode, if the left or right relative tangential force in the three-dimensional relative touch force estimation value is non-zero, it corresponds to the map rotation, and the map is rotated clockwise or counterclockwise.

[0095] c) Quick menu:

[0096] Menu operation is often used in human-computer interaction. The menus in computers and mobile phones are generally called out through the menu bar at the top of the page, clicking the right button, or long pressing the screen, and then the operation is completed by continuously clicking different items, which requires many operation steps, is cumbersome to operate, and is prone to misoperation. The present invention controls the shortcut menu through the three-dimensional relative touch force estimation value, which can reduce the operation steps and reduce the possibility of misoperation.

[0097] For example, if it is detected that the current scene signal is a font software program signal, the interaction mode is determined to be a continuous mode based on the corresponding preset interaction mode. If the pressure in the received three-dimensional relative touch force estimation value is greater than the pressure threshold, the corresponding turntable menu pop-up function is implemented to open the turntable menu. If the relative tangential force in the received three-dimensional relative touch force estimation value is non-zero, the pointer in the turntable is controlled. During the rotation of the pointer, if it is detected in real time that the pressure is less than the pressure threshold, the option pointed to by the pointer can be determined as the currently selected menu item.

[0098] In addition, multi-level menus can be defined. For example, the navigation of the secondary menu can be controlled by a relative tangential force in another direction (that is, the primary menu and the secondary menu are controlled by relative tangential forces in different directions), so that the user can still complete the menu selection in one finger press.

[0099] Figure 6 This is an example diagram of the multi-level turntable menu interaction provided by an embodiment of the present invention. Figure 6 As shown, the turntable menu includes a first-level menu, which includes five options: pencil, ink, crayon, brush, and pen. There is a second-level menu under the first-level menu, for example, there are options of different hardness under the pencil. If the current scene signal is detected to be a font software program signal, the interaction mode is determined to be a continuous mode, and when the pressure in the three-dimensional relative touch force estimation value is detected to be greater than the pressure threshold, the touch screen is turned on. Figure 6 In the turntable menu shown, if the left relative tangential force in the received three-dimensional relative touch force estimation value is non-zero, the pointer in the turntable is controlled. During the pointer rotation process, if the pressure detected is less than the pressure threshold, the option pointed by the pointer can be determined as the currently selected menu item, such as Figure 6 The middle pointer points to the pencil option. The pencil option includes a secondary menu with two hardness options. If the upward relative tangential force in the received 3D relative touch force estimate is non-zero, the desired hardness can be selected.

[0100] Figure 7 This is an example diagram of the palette continuous variable control provided by an embodiment of the present invention.

[0101] In addition, in some scenarios, if a first-level menu item is activated, a continuous control may pop up. By using the method of the present invention, the interaction from the first-level menu selection to the continuous control can be completed in one finger press. For example, in some scenario signals, if a first-level menu item is activated, a one-dimensional continuous control pops up. The one-dimensional continuous control (such as a progress bar for controlling brightness or volume) is controlled based on the relative tangential force in one direction of the received three-dimensional relative touch force estimation value, and the two-dimensional continuous control is controlled based on the relative tangential force in another direction of the received three-dimensional relative touch force estimation value. A two-dimensional continuous control is, for example, Figure 7 The color palette shown can achieve chromaticity control.

[0102] d) Drag operation:

[0103] The drag operation of existing mobile phones uses actual position mapping. For example, to move a file from position A to position B, the finger also needs to move from position A to position B. When used on large-area touch-screen devices (such as tablets), foldable screen mobile phones (or foldable screen tablets), and across devices (such as moving a file from a mobile phone to a certain location on a computer), this direct position mapping dragging method is very inconvenient (because the finger needs to move a long distance) or cannot be implemented (because it cannot cross two devices or cannot cross the folding area).

[0104] By using the method of the present invention, if the current scene signal is detected as a file selection scene signal, the interaction mode is determined to be a continuous mode, and when the pressure in the detected three-dimensional relative touch force estimation value is greater than the pressure threshold, the dragging operation is triggered, and the file dragging is controlled based on the relative tangential force in the detected three-dimensional relative touch force estimation value, wherein the magnitude of the relative tangential force controls the dragging speed, and the direction of the relative tangential force controls the dragging direction, and when the pressure in the detected three-dimensional relative touch force estimation value is less than the pressure threshold, the dragging is stopped. Such a dragging operation can be used to move files, icons, text, web pages, and maps, and is suitable for moving within touch devices of various areas, within folding screen devices, and across devices.

[0105] e) Multi-finger operation:

[0106] The pressure sensors currently used in mobile devices cannot distinguish the force of each finger in multi-finger operation and can only measure the overall pressure of the hand. The present invention can measure the three-dimensional relative touch force estimation values ​​of multiple touching fingers at the same time, so it can support richer interactions. When the user uses multiple fingers to operate at the same time, the system will perform a three-dimensional relative touch force estimation for each finger separately, and can realize multi-finger independent force input.

[0107] Figure 8Schematic diagram of multi-finger interaction provided by an embodiment of the present invention. For example, if the current scene signal is detected to be a first-person or third-person perspective game scene signal, the interaction mode is determined to be a continuous mode, and the three-dimensional relative touch force estimation values ​​of the two fingers are obtained. The three-dimensional relative touch force estimation value of one finger corresponds to the character movement control, and the three-dimensional relative touch force estimation value of the other finger corresponds to the perspective rotation control. In drone control, two fingers are used to control the joysticks at both ends of the screen (see Figure 8 ), achieving four-degree-of-freedom control.

[0108] In order to implement the above embodiment, the present invention further proposes a three-dimensional relative touch force estimation and human-computer interaction system based on touch sensor images.

[0109] Fig. 9 A block diagram of a three-dimensional relative touch force estimation and human-computer interaction system based on touch sensor images provided by an embodiment of the present invention.

[0110] like Fig. 9 As shown, the three-dimensional relative touch force estimation and human-computer interaction system based on touch sensor images includes a touch sensor image acquisition module 11, a three-dimensional relative touch force estimation module 12 and an interaction control module 13, wherein:

[0111] A touch sensor image acquisition module 11, used for acquiring continuous touch sensor images in real time;

[0112] a three-dimensional relative touch force estimation module 12, configured to select adjacent multiple frames of touch sensor images from continuous touch sensor images for alignment each time, and then input a trained three-dimensional relative touch force estimation model to output a three-dimensional relative touch force estimation value corresponding to each adjacent multiple frames of touch sensor images, wherein the three-dimensional relative touch force is a variation of the three-dimensional touch force, and the three-dimensional touch force includes a pressure perpendicular to the touch sensor surface, and a lateral force and a longitudinal force parallel to the touch sensor surface;

[0113] The interaction control module 13 is used to perform an action in a preset interaction mode based on the three-dimensional relative touch force estimation value obtained within a set time period.

[0114] Furthermore, in a possible implementation of an embodiment of the present invention, in the three-dimensional relative touch force estimation module 12, adjacent multiple frames of touch sensor images are selected from continuous touch sensor images for alignment each time, including: each time adjacent multiple frames of touch sensor images are selected from continuous touch sensor images, center points of finger areas of adjacent multiple frames of touch sensor images are respectively calculated, and adjacent multiple frames of touch sensor images are aligned according to the center points of the finger areas.

[0115] Furthermore, in a possible implementation of an embodiment of the present invention, in the three-dimensional relative touch force estimation module 12, adjacent multiple frames of touch sensor images are aligned according to the center point of the finger area, including: if each frame of the touch sensor image in the adjacent multiple frames of touch sensor images has a center point of a finger area, then the adjacent multiple frames of touch sensor images are aligned based on the center point of the one finger area; if each frame of the touch sensor image in the adjacent multiple frames of touch sensor images has multiple finger areas, then each alignment processing is for one target finger, and the adjacent multiple frames of touch sensor images are aligned based on the center point of the finger area of ​​the target finger in each frame of the touch sensor image.

[0116] Furthermore, in a possible implementation of an embodiment of the present invention, in the three-dimensional relative touch force estimation module 12, the three-dimensional relative touch force estimation model includes a force estimation module and multiple feature extraction modules, the number of feature extraction modules is equal to the number of frames of adjacent multiple frames of touch sensor images, the input of each feature extraction module is a frame of touch sensor images in the adjacent multiple frames of touch sensor images, and the output of all feature extraction modules is the input of the force estimation module.

[0117] Furthermore, in a possible implementation manner of the embodiment of the present invention, in the three-dimensional relative touch force estimation module 12, a three-dimensional relative touch force estimation model is constructed based on a deep neural network.

[0118] Furthermore, in a possible implementation of an embodiment of the present invention, in the interaction control module 13, the preset interaction method includes a first mapping relationship between multiple scene signals and multiple interaction modes, wherein the multiple interaction modes include a discrete mode and a continuous mode; when the interaction mode is a discrete mode, the preset interaction method also includes a second mapping relationship between multiple operation intention signals and multiple functions; when the interaction mode is a continuous mode, the preset interaction method also includes a third mapping relationship between three-dimensional relative touch force estimation values ​​and multiple functions.

[0119] Furthermore, in a possible implementation of an embodiment of the present invention, in the interaction control module 13, an action is performed in a preset interaction mode based on the three-dimensional relative touch force estimation value obtained within a set time period, including: determining a current interaction mode based on a detected current scene signal and a corresponding preset interaction mode; if the current interaction mode is a discrete mode, determining a current operation intention signal based on the three-dimensional relative touch force estimation value obtained within a set time period; determining a required function based on a second mapping relationship between the current operation intention signal and the corresponding preset interaction mode; and performing an action according to the required function.

[0120] It should be noted that the aforementioned explanation of the embodiment of the three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images is also applicable to the three-dimensional relative touch force estimation and human-computer interaction system based on touch sensor images of this embodiment, and will not be repeated here.

[0121] In an embodiment of the present invention, continuous touch sensor images are acquired in real time; adjacent multiple frames of touch sensor images are selected from the continuous touch sensor images for alignment each time, and then a trained three-dimensional relative touch force estimation model is input to output a three-dimensional relative touch force estimation value corresponding to each adjacent multiple frames of touch sensor images, wherein the three-dimensional relative touch force is the variation of the three-dimensional touch force, and the three-dimensional touch force includes the pressure perpendicular to the touch sensor surface, the lateral force and the longitudinal force parallel to the touch sensor surface; an action is performed in a preset interactive mode based on the three-dimensional relative touch force estimation value obtained within a set time period. In this case, the pressure perpendicular to the touch sensor surface, the lateral force and the longitudinal force parallel to the touch sensor surface are comprehensively considered, and the three-dimensional relative touch force is estimated using adjacent multiple frames of touch sensor images, so that the force measurement based on touch is more accurate. In addition, the action is performed in a preset interactive mode based on the three-dimensional relative touch force estimation value obtained within a set time period, which makes full use of the three-dimensional contact force and improves the accuracy of human-computer interaction.

[0122] The method and system of the present invention relate to the field of human-computer interaction, and in particular to the expansion of touch input and its use in the field of human-computer interaction. The three-dimensional relative touch force is estimated through multiple frames of touch images. The three-dimensional relative touch force includes the pressure perpendicular to the sensor surface and the tangential force parallel to the surface, thereby realizing precise control of the input. The present invention further proposes a human-computer interaction technology based on three-dimensional force, and realizes the expansion of traditional interaction methods by using three-dimensional contact force.

[0123] In order to implement the above embodiments, the present invention also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0124] In order to implement the above embodiments, the present invention further proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0125] In order to implement the above embodiments, the present invention further provides a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0126] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0127] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0128] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0129] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0130] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0131] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0132] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0133] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images, characterized in that: include: Acquire continuous touch sensor images in real time; Selecting adjacent multiple frames of touch sensor images from the continuous touch sensor images for alignment each time, and then inputting a trained three-dimensional relative touch force estimation model to output a three-dimensional relative touch force estimation value corresponding to each adjacent multiple frames of touch sensor images, wherein the three-dimensional relative touch force is a change in the three-dimensional touch force, and the three-dimensional touch force includes a pressure perpendicular to a touch sensor surface, and a lateral force and a longitudinal force parallel to the touch sensor surface; An action is performed in a preset interactive manner based on the three-dimensional relative touch force estimation value obtained within a set time period.

2. The three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images according to claim 1, characterized in that: The selecting adjacent multiple frames of touch sensor images from the continuous touch sensor images for alignment each time comprises: Each time, multiple adjacent touch sensor image frames are selected from the continuous touch sensor images, center points of finger regions of the multiple adjacent touch sensor image frames are calculated respectively, and the multiple adjacent touch sensor image frames are aligned according to the center points of the finger regions.

3. The three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images according to claim 2, characterized in that: The aligning the adjacent multiple frames of touch sensor images according to the center point of the finger area includes: If each of the plurality of adjacent touch sensor image frames has a center point of a finger area, aligning the plurality of adjacent touch sensor image frames based on the center point of the finger area; If each of the adjacent multiple touch sensor image frames has multiple finger areas, each alignment process is performed for one target finger, and the adjacent multiple touch sensor image frames are aligned based on the center point of the finger area of ​​the target finger in each touch sensor image frame.

4. The three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images according to claim 1, characterized in that: The three-dimensional relative touch force estimation model includes a force estimation module and multiple feature extraction modules, the number of feature extraction modules is equal to the number of frames of the adjacent multiple frames of touch sensor images, the input of each feature extraction module is a touch sensor image frame in the adjacent multiple frames of touch sensor images, and the output of all feature extraction modules is the input of the force estimation module.

5. The three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images according to claim 1 or 4, characterized in that: The three-dimensional relative touch force estimation model is constructed based on a deep neural network.

6. The three-dimensional relative touch force estimation and human-computer interaction method based on touch sensor images according to claim 1, characterized in that: The preset interaction mode includes a first mapping relationship between multiple scene signals and multiple interaction modes, wherein the multiple interaction modes include a discrete mode and a continuous mode. When the interaction mode is a discrete mode, the preset interaction mode also includes a second mapping relationship between multiple operation intention signals and multiple functions. When the interaction mode is a continuous mode, the preset interaction mode also includes a third mapping relationship between three-dimensional relative touch force estimation values ​​and multiple functions.

7. The method for 3D relative touch force estimation and human-computer interaction based on touch sensor images according to claim 6, wherein the step of performing an action in a preset interaction mode based on the 3D relative touch force estimation value obtained within a set time period comprises: Determining a current interaction mode based on the detected current scene signal and the corresponding preset interaction mode; If the current interaction mode is a discrete mode, a current operation intention signal is determined based on the three-dimensional relative touch force estimation value obtained within a set time period; a required function is determined based on a second mapping relationship between the current operation intention signal and the corresponding preset interaction mode; and an action is performed according to the required function.

8. A three-dimensional relative touch force estimation and human-computer interaction system based on touch sensor images, characterized in that: include: A touch sensor image acquisition module, used for acquiring continuous touch sensor images in real time; a three-dimensional relative touch force estimation module, configured to select adjacent multiple frames of touch sensor images from the continuous touch sensor images for alignment each time, and then input a trained three-dimensional relative touch force estimation model to output a three-dimensional relative touch force estimation value corresponding to each adjacent multiple frames of touch sensor images, wherein the three-dimensional relative touch force is a variation of the three-dimensional touch force, and the three-dimensional touch force includes a pressure perpendicular to a touch sensor surface, and a lateral force and a longitudinal force parallel to the touch sensor surface; The interactive control module is used to perform an action in a preset interactive manner based on the three-dimensional relative touch force estimation value obtained within a set time period.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images