A haptic interaction information prediction method, system, terminal and storage medium

Abstract

The application discloses a kind of tactile interaction information prediction method, system, terminal and storage medium, the method includes: obtaining tactile data, tactile data is extracted, and tactile frame sequence and tactile feature sequence are obtained;Visual data is identified in space, and scene graph feature sequence is obtained, and tactile feature sequence and scene graph feature sequence are inferred in semantics, and operation semantic label sequence is obtained;According to time window, tactile frame sequence and operation semantic label sequence are encoded respectively, and first input matrix and second input matrix are obtained;Through tactile prediction model, tactile data prediction is carried out according to first input matrix and second input matrix, and future tactile data is obtained.The application carries out semantic inference to tactile data and visual data, obtains operation semantic label, and predicts future tactile signal according to tactile data and operation semantic label by depth learning model, improve the prediction accuracy of tactile signal in human-computer interaction.

Description

Technical Field

[0001] This invention relates to the field of information prediction technology, and in particular to a tactile interactive information prediction method, system, terminal, and computer-readable storage medium. Background Technology

[0002] With the development of the tactile internet, the demand for immersive human-computer interaction in high-value scenarios (such as precision industrial manufacturing and remote surgery) is increasing. These tasks require extremely high real-time performance, and the latency of tactile feedback must be controlled within the human perception threshold; otherwise, it will lead to operational errors. Therefore, it is necessary to predict the tactile signals during the human-computer interaction process and send the tactile signals a certain period in advance to the receiving end, thereby compensating for the latency caused by network transmission.

[0003] Existing tactile interaction information prediction methods typically predict tactile signal sequences directly using deep learning models. However, these methods rely heavily on the autocorrelation of data, which can lead to prediction failures when predicting tactile signals with abrupt changes, resulting in reduced accuracy in predicting tactile signals in human-computer interaction.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a tactile interaction information prediction method, system, terminal, and computer-readable storage medium. This invention aims to solve the problem that existing tactile interaction information prediction methods rely heavily on data autocorrelation, which leads to prediction failures when predicting tactile signals with abrupt changes, resulting in reduced accuracy in predicting tactile signals in human-computer interaction.

[0006] To achieve the above objectives, the present invention provides a method for predicting tactile interaction information, comprising the following steps: Acquire tactile data, perform feature extraction processing on the tactile data, and obtain a tactile frame sequence and a tactile feature sequence; Visual data is acquired, spatial recognition processing is performed on the visual data to obtain a scene graph feature sequence, and semantic reasoning is performed on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence. A time window is obtained, and the haptic frame sequence and the operation semantic label sequence are encoded according to the time window to obtain a first input matrix and a second input matrix. A tactile prediction model is determined, and tactile data prediction processing is performed using the tactile prediction model based on the first input matrix and the second input matrix to obtain future tactile data.

[0007] Optionally, the tactile interaction information prediction method, wherein acquiring tactile data and performing feature extraction processing on the tactile data to obtain a tactile frame sequence and a tactile feature sequence specifically includes: Acquire tactile data, segment the tactile data to obtain multiple tactile frames, and obtain a tactile frame sequence based on the multiple tactile frames; Obtain the time span, and perform time division processing on multiple tactile frames according to the time span to obtain multiple tactile segments; A tactile feature sequence is obtained based on multiple tactile segments.

[0008] Optionally, the tactile interaction information prediction method, wherein acquiring visual data, performing spatial recognition processing on the visual data to obtain a scene graph feature sequence, and performing semantic reasoning on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence, specifically includes: Acquire visual data, determine the target detection algorithm, and perform object recognition processing on the visual data using the target detection algorithm to obtain multiple target objects; A spatial scene map is obtained by performing scene map generation processing on multiple target objects; A grouping normalization algorithm is determined, and the spatial scene graph is processed by the grouping normalization algorithm to extract scene graph features, thereby obtaining a scene graph feature sequence; A temporal convolutional network is determined, and cross-modal inference processing is performed on the tactile feature sequence and the scene graph feature sequence through the temporal convolutional network to obtain an operational semantic label sequence.

[0009] Optionally, the tactile interaction information prediction method, wherein the step of generating a spatial scene map based on multiple target objects specifically includes: The state extraction process is performed on the multiple target objects to obtain multiple object states, and the position extraction process is performed on the multiple target objects to obtain multiple position vectors; A scene graph is constructed based on multiple target objects, multiple object states, and multiple position vectors to obtain a spatial scene graph.

[0010] Optionally, in the tactile interaction information prediction method, the time window includes a long time window and a short time window; The acquisition of the time window involves encoding the haptic frame sequence and the operation semantic label sequence according to the time window to obtain a first input matrix and a second input matrix, specifically including: The long time window and the short time window are obtained, and the haptic frame sequence is encoded to obtain the first input matrix, wherein the number of rows of the first input matrix is ​​equal to the length of the long time window, and the number of columns of the first input matrix is ​​equal to the length of the short time window. The operation semantic label sequence is encoded to obtain the second input matrix, wherein the number of rows of the second input matrix is ​​equal to the length of the long time window, and the number of columns of the second input matrix is ​​equal to the length of the short time window.

[0011] Optionally, in the tactile interaction information prediction method, the tactile prediction model includes a memory learning model and a regression prediction model; The determination of the tactile prediction model involves performing tactile data prediction processing based on the first input matrix and the second input matrix to obtain future tactile data, specifically including: Determine the memory learning model, and perform a first reasoning process on the first input matrix and the second input matrix through the memory learning model to obtain long-term semantic memory lexical units; The regression prediction model is determined, and the regression prediction model is used to perform regression prediction processing based on the long-term semantic memory lexical units to obtain future tactile data.

[0012] Optionally, in the tactile interaction information prediction method, the memory learning model includes a tactile learning model and a semantic learning model; The step of determining the memory learning model and performing a first inference process on the first input matrix and the second input matrix through the memory learning model to obtain long-term semantic memory lexical units specifically includes: Determine the tactile learning model, and perform a first inference process on the first input matrix using the tactile learning model to obtain a first output vector; Determine the semantic learning model, and perform a second inference process on the second input matrix through the semantic learning model to obtain a second output vector; The first output vector and the second output vector are concatenated to obtain long-term semantic memory lexical units.

[0013] Furthermore, to achieve the above objectives, the present invention also provides a tactile interaction information prediction system, wherein the tactile interaction information prediction system comprises: The tactile feature extraction module is used to acquire tactile data, perform feature extraction processing on the tactile data, and obtain a tactile frame sequence and a tactile feature sequence. The semantic label generation module is used to acquire visual data, perform spatial recognition processing on the visual data to obtain a scene graph feature sequence, and perform semantic reasoning on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence. A matrix encoding module is used to obtain a time window and encode the haptic frame sequence and the operation semantic label sequence according to the time window to obtain a first input matrix and a second input matrix. The data prediction module is used to determine a tactile prediction model, and to perform tactile data prediction processing based on the first input matrix and the second input matrix using the tactile prediction model to obtain future tactile data.

[0014] In addition, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a tactile interaction information prediction program stored in the memory and executable on the processor, wherein when the tactile interaction information prediction program is executed by the processor, it implements the steps of the tactile interaction information prediction method as described above.

[0015] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a tactile interaction information prediction program, which, when executed by a processor, implements the steps of the tactile interaction information prediction method as described above.

[0016] In this invention, tactile data is acquired, and feature extraction processing is performed on the tactile data to obtain a tactile frame sequence and a tactile feature sequence. Visual data is acquired, and spatial recognition processing is performed on the visual data to obtain a scene graph feature sequence. Semantic reasoning is then performed on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence. A time window is acquired, and the tactile frame sequence and the operation semantic label sequence are encoded according to the time window to obtain a first input matrix and a second input matrix. A tactile prediction model is determined, and the tactile prediction model is used to predict future tactile data based on the first input matrix and the second input matrix to obtain future tactile data. This invention performs semantic reasoning on tactile and visual data to obtain operation semantic labels, and uses a deep learning model to predict future tactile signals based on the tactile data and operation semantic labels, thereby improving the accuracy of tactile signal prediction in human-computer interaction. Attached Figure Description

[0017] Figure 1 This is a flowchart of a preferred embodiment of the tactile interaction information prediction method of the present invention; Figure 2 This is a system diagram of the tactile interaction information prediction method of the present invention; Figure 3This is a flowchart of the operational semantic understanding of the tactile interaction information prediction method of the present invention; Figure 4 This is a flowchart of the tactile data prediction method of the present invention for predicting tactile interaction information; Figure 5 This is a comparison chart of the tactile signal prediction results of the tactile interaction information prediction method of the present invention with the performance of common models; Figure 6 This is a comparison diagram of the tactile signal prediction results and the true values ​​of the tactile interaction information prediction method of the present invention; Figure 7 This is a structural diagram of a preferred embodiment of the tactile interaction information prediction system of the present invention; Figure 8 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] With the development of the tactile internet, the demand for immersive human-computer interaction in high-value scenarios (such as precision industrial manufacturing and remote surgery) is increasing. These tasks require extremely high real-time performance, and the latency of tactile feedback must be controlled within the human perception threshold; otherwise, it will lead to operational errors. Therefore, it is necessary to predict the tactile signals during the human-computer interaction process and send the tactile signals a certain period in advance to the receiving end, thereby compensating for the latency caused by network transmission.

[0020] Existing tactile interaction information prediction methods typically predict tactile signal sequences directly using deep learning models. However, these methods rely heavily on the autocorrelation of data, which can lead to prediction failures when predicting tactile signals with abrupt changes, resulting in reduced accuracy in predicting tactile signals in human-computer interaction.

[0021] To address the aforementioned issues, this invention proposes a method for predicting tactile interaction information. This method performs semantic reasoning on tactile and visual data to obtain operational semantic labels, and then uses a deep learning model to predict future tactile signals based on the tactile data and operational semantic labels, thereby improving the accuracy of predicting tactile signals in human-computer interaction.

[0022] The tactile interaction information prediction method described in the preferred embodiment of the present invention, such as... Figure 1 As shown, the tactile interaction information prediction method includes the following steps: Step S10: Acquire tactile data, perform feature extraction processing on the tactile data, and obtain a tactile frame sequence and a tactile feature sequence.

[0023] like Figure 2 As shown, this invention collects data; obtains operational semantic information by semantic reasoning from visual and tactile data; constructs a semantic-tactile signal by combining tactile data and operational semantic information; performs multi-historical 2D encoding on the semantic-tactile signal to construct an input matrix; and inputs the input matrix into a neural network for prediction to obtain future prediction data.

[0024] Specifically, tactile data is acquired, the tactile data is segmented to obtain multiple tactile frames, and a tactile frame sequence is obtained based on the multiple tactile frames; a time span is acquired, the multiple tactile frames are divided into multiple tactile segments based on the time span, and a tactile feature sequence is obtained based on the multiple tactile segments.

[0025] In acquiring tactile data, tactile sensors (including but not limited to triaxial force sensors, pressure sensors, and array sensors) are used to collect triaxial force signals from the operator's hand. , , As tactile data, in which , , These represent force signals on the X, Y, and Z axes, respectively.

[0026] like Figure 3 As shown, the collected tactile data is segmented according to the time series to aid in reasoning.

[0027] Specifically, tactile data is divided into continuous tactile frames according to time. f 1, f 2, ..., f time=T}; Set a time span △, for a certain time point i, collect all tactile frames within △ before and after time point i { f i-△ , ..., f i+△ As a tactile fragment c i Collect all tactile fragments within time intervals {1, 2, ..., T} to obtain the tactile feature sequence U. F ={ c 1, c 2, ..., c T}

[0028] Step S20: Acquire visual data, perform spatial recognition processing on the visual data to obtain a scene graph feature sequence, and perform semantic reasoning on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence.

[0029] Specifically, visual data is acquired, an object detection algorithm is determined, and the visual data is processed by the object detection algorithm to obtain multiple target objects; the multiple target objects are processed to extract states to obtain multiple object states, and the multiple target objects are processed to extract positions to obtain multiple position vectors; a scene graph is constructed based on the multiple target objects, the multiple object states, and the multiple position vectors to obtain a spatial scene graph.

[0030] like Figure 3 As shown, visual images are collected to construct a scene graph G, which is then used in cross-modal inference along with tactile data.

[0031] While collecting tactile data, an RGB-D camera is used simultaneously to capture visual images of the operation scene.

[0032] Object detection algorithms (such as YOLO) are used to identify objects (e.g., hands, tools, parts, etc.) in a scene, extracting the relative positions and motion vectors between objects. Based on the relative positions and motion vectors between objects, the position vectors of the objects are obtained. To avoid missed or over-detection due to occlusion in the object detection algorithm, the detection results are post-processed.

[0033] The specific process of post-processing includes: Confidence filtering is used to remove redundant detections; By utilizing temporal causal relationships (positional correlation between consecutive frames), missing tags caused by occlusion can be supplemented.

[0034] The pre-trained model outputs the state of objects in the scene at each moment (such as approaching, touching, grabbing, moving, releasing, etc.).

[0035] According to the state of the object u ,object nodes and the position vector of the object edges Construct spatial scene graph G={ u , nodes , edges}

[0036] Furthermore, a grouping normalization algorithm is determined, and the spatial scene graph is processed by the grouping normalization algorithm to extract scene graph features, thereby obtaining a scene graph feature sequence.

[0037] like Figure 3 As shown, features of the spatial scene map are extracted based on the Group Normalization (GN) algorithm, resulting in a scene map feature sequence. U v ={ u 1, u2, ..., u T}

[0038] Furthermore, a temporal convolutional network is determined, and cross-modal inference processing is performed on the tactile feature sequence and the scene graph feature sequence through the temporal convolutional network to obtain an operational semantic label sequence.

[0039] like Figure 3 As shown, the tactile feature sequence and the scene graph feature sequence are input into a multi-stage temporal convolutional network (MS-TCN). By analyzing tactile abrupt changes and visual scene changes, the operation semantic label at each moment is output, giving physical meaning to the meaningless force signal.

[0040] like Figure 3 As shown, the obtained operational semantic label sequence will be input together with the tactile feature sequence into the subsequent tactile data prediction model for data prediction.

[0041] Step S30: Obtain the time window, and encode the tactile frame sequence and the operation semantic label sequence according to the time window to obtain the first input matrix and the second input matrix.

[0042] Specifically, the time window includes a long time window and a short time window; the long time window and the short time window are obtained, and the haptic frame sequence is encoded to obtain a first input matrix, wherein the number of rows in the first input matrix is ​​equal to the length of the long time window, and the number of columns in the first input matrix is ​​equal to the length of the short time window; the operation semantic label sequence is encoded to obtain a second input matrix, wherein the number of rows in the second input matrix is ​​equal to the length of the long time window, and the number of columns in the second input matrix is ​​equal to the length of the short time window.

[0043] like Figure 4 As shown, the one-dimensional data sequence is reorganized into a two-dimensional matrix, which serves as the input for the subsequent model.

[0044] For components in tactile data , , The tactile frame sequences on different coordinate axes are encoded to obtain input matrices composed of tactile data on the X, Y, and Z axes respectively. The three input matrices are combined together as the first input matrix.

[0045] Specifically, for each data sequence, a long time window of length L is first set, which is a relatively long time window, such as 64 frames, to capture the global task logic; then a short time window of length M is set, which is a relatively short time window, such as 20 frames, to capture local contact mechanical features.

[0046] A one-dimensional sequence can be reassembled into a two-dimensional matrix using a sliding window method. For example, the first row of data after 2D encoding is from t0 to t... -M Where t0 is the current frame, t -M It is the previous M frames in history, and so on, to obtain a two-dimensional matrix with height L and width M.

[0047] The semantic label sequence of the operation is also encoded to obtain the second input matrix.

[0048] This encoding method can capture dependencies at different time scales.

[0049] Step S40: Determine the tactile prediction model, and use the tactile prediction model to perform tactile data prediction processing based on the first input matrix and the second input matrix to obtain future tactile data.

[0050] Specifically, the tactile prediction model includes a memory learning model and a regression prediction model; the memory learning model includes a tactile learning model and a semantic learning model; the tactile learning model is determined, and the first input matrix is ​​subjected to a first inference process through the tactile learning model to obtain a first output vector; the semantic learning model is determined, and the second input matrix is ​​subjected to a second inference process through the semantic learning model to obtain a second output vector; the first output vector and the second output vector are concatenated to obtain long-term semantic memory lexical units.

[0051] like Figure 4 As shown, this invention utilizes a dual-stream long short-term memory (LSTM) network to process force signals and semantic signals separately, and then uses a multi-head attention mechanism to fuse force features and semantic features (coupling).

[0052] Specifically, LSTM includes a semantic LSTM and a tactile LSTM. The matrix encoded from tactile data features is input into the tactile LSTM to obtain the first output vector, and the matrix encoded from the semantic labels is input into the semantic LSTM to obtain the second output vector. The first and second output vectors are concatenated and fused to obtain a long-term semantic token with short-to-medium-term memory.

[0053] The role of fusing force and semantic features through the dual-stream long short-term memory network is to integrate continuous tactile fluctuations and discrete semantics within window M into a high-dimensional vector. This high-dimensional vector not only contains the numerical value of the force, but also information about "what action produced the force".

[0054] Furthermore, the regression prediction model is determined, and future tactile data is obtained by performing regression prediction processing based on the long-term semantic memory lexical units through the regression prediction model.

[0055] like Figure 4 As shown, the present invention inputs the generated long-term semantic tokens (length L) into the Transformer encoder-decoder structure to infer the haptic tokens for the next N frames (e.g., 10 frames) and maps them back to triaxial force values.

[0056] Specifically, the model analyzes the dependencies between tokens through a self-attention mechanism and learns the global logic of the task (e.g., "grab" is usually followed by "move"); after the decoder infers, it infers the three-dimensional force value of the first future frame through a linear layer; the prediction result of the first future frame is concatenated to the previous input matrix, and new tokens are extracted for the prediction of the second future frame, and so on to form an autoregressive prediction; finally, the tactile data of the next N frames is obtained.

[0057] The system immediately sends the predicted data for the next N frames to the receiver at time T. Even if there is a 50ms jitter in the network, the receiver can receive the data at the same time as (or even slightly earlier) the physical action occurs because the predicted future data is sent 50ms earlier (e.g., 51.32ms), thus eliminating the perceived network delay.

[0058] In a 3C computer motherboard assembly scenario, a camera was used to capture RGB and depth images of the human hand during the demonstration. All captured images were three-channel, with resolutions of 640px x 480px, a frame rate of 20fps, and saved as PNGs. A 3D tactile sensor was incorporated into the assembly dataset acquisition platform to capture changes in 3D fingertip force during the assembly process. The tactile sensor can detect tangential and normal force, with a 33Hz sampling rate for 3D force information, and was worn on the subject's thumb. In this scenario, the semantics of touch included stillness, approach, grasping, moving, inserting, releasing, pulling, twisting (forward), twisting (backward), and leaving.

[0059] Performance tests were conducted on the framework proposed in this invention based on the aforementioned platform. The sequences generated by the assembly task were divided according to action type using expert annotation, and action labels, start and end timestamps were added. The data was then divided into training and test sets (4:1) to train the TOSUM model's semantic annotation capabilities for multimodal, multi-task data. The annotated tactile data was used to train the Tacformer time series prediction model.

[0060] like Figure 5As shown, the present invention can accurately understand the meaning of the subject's operation, and the edit score of the test set can reach 99.88%. There will be no misunderstanding of the operation semantics, and the understanding ability is significantly higher than that of the visual solution based on graph network.

[0061] like Figure 6 As shown, the present invention achieves an average mean squared error (MSE) of 0.004N for the ten-frame haptic prediction task, ensuring the accuracy of immersive haptic interaction.

[0062] During the performance test, taking the Video Graphic Array (VGA) task as an example, as shown in Table 1, although each computing module takes 248.18ms to complete the prediction of 10 frames, Tacformer can generate a prediction window of 299.5ms, providing a solid theoretical basis for advance transmission and delay cancellation.

[0063] Table 1: Running time of each module in the prediction transmission model during the VGA insertion task

[0064] The innovative aspects of this invention include: 1. Cross-modal semantic understanding (TOSUM): Instead of blindly predicting data, it introduces visual information and its semantics. Through the TOSUM model, it utilizes visual RGB-D stream and scene graph, combined with multi-stage temporal convolutional network (MS-TCN), to explicitly identify the current operation action (such as plugging, unplugging, and turning), and establish a mapping between physical action (cause) and tactile pattern (effect). 2. Strong Coupling of Operational Semantics and Tactile Sensation (Coupling Tokenizer): A coupled tokenizer mechanism (implemented in the mid-term memory module) is proposed. It tightly combines continuous, non-linear tactile fluctuations with discrete, symbolic semantic tags through a two-stream LSTM to generate a unified "action-force" token. This solves the problem of unpredictable tactile mutations because the model knows that "if the action changes, the force must also change." 3. Hierarchical Historical Feature Reasoning (Tacformer): Tactile signals have strong abrupt changes and nonlinearity. When the operation action changes, the tactile signal will change drastically. Existing models often fail to predict or degenerate into random guessing because they do not understand this action logic. This invention adopts a hybrid architecture (LSTM+Transformer), using LSTM to process short-term contact mechanical features and Transformer to process long-term task logic (i.e., the sequence of actions). This hierarchical structure enables the model to capture both microscopic mechanical details and macroscopic task flow, thereby achieving high-precision advance prediction.

[0065] This invention performs semantic reasoning on tactile and visual data to obtain operational semantic labels, and uses a deep learning model to predict tactile signals at future moments based on tactile data and operational semantic labels, thereby improving the accuracy of tactile signal prediction in human-computer interaction.

[0066] Furthermore, such as Figure 7 As shown, based on the above-described tactile interaction information prediction method, the present invention also provides a tactile interaction information prediction system, wherein the tactile interaction information prediction system includes: The tactile feature extraction module 51 is used to acquire tactile data and perform feature extraction processing on the tactile data to obtain a tactile frame sequence and a tactile feature sequence. The semantic label generation module 52 is used to acquire visual data, perform spatial recognition processing on the visual data to obtain a scene graph feature sequence, and perform semantic reasoning on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence. The matrix encoding module 53 is used to obtain a time window and encode the tactile frame sequence and the operation semantic label sequence according to the time window to obtain a first input matrix and a second input matrix. The data prediction module 54 is used to determine the tactile prediction model, and to perform tactile data prediction processing based on the first input matrix and the second input matrix to obtain future tactile data.

[0067] Furthermore, such as Figure 8 As shown, based on the above-mentioned tactile interaction information prediction method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 8 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0068] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a tactile interaction information prediction program 40, which can be executed by the processor 10 to implement the tactile interaction information prediction method of this application.

[0069] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the tactile interaction information prediction method.

[0070] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface.

[0071] In one embodiment, when the processor 10 executes the tactile interaction information prediction program 40 in the memory 20, the following steps are performed: Acquire tactile data, perform feature extraction processing on the tactile data, and obtain a tactile frame sequence and a tactile feature sequence; Visual data is acquired, spatial recognition processing is performed on the visual data to obtain a scene graph feature sequence, and semantic reasoning is performed on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence. A time window is obtained, and the haptic frame sequence and the operation semantic label sequence are encoded according to the time window to obtain a first input matrix and a second input matrix. A tactile prediction model is determined, and tactile data prediction processing is performed using the tactile prediction model based on the first input matrix and the second input matrix to obtain future tactile data.

[0072] The step of acquiring tactile data and performing feature extraction processing on the tactile data to obtain a tactile frame sequence and a tactile feature sequence specifically includes: Acquire tactile data, segment the tactile data to obtain multiple tactile frames, and obtain a tactile frame sequence based on the multiple tactile frames; Obtain the time span, and perform time division processing on multiple tactile frames according to the time span to obtain multiple tactile segments; A tactile feature sequence is obtained based on multiple tactile segments.

[0073] The process of acquiring visual data, performing spatial recognition processing on the visual data to obtain a scene graph feature sequence, and performing semantic reasoning on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence specifically includes: Acquire visual data, determine the target detection algorithm, and perform object recognition processing on the visual data using the target detection algorithm to obtain multiple target objects; A spatial scene map is obtained by performing scene map generation processing on multiple target objects; A grouping normalization algorithm is determined, and the spatial scene graph is processed by the grouping normalization algorithm to extract scene graph features, thereby obtaining a scene graph feature sequence; A temporal convolutional network is determined, and cross-modal inference processing is performed on the tactile feature sequence and the scene graph feature sequence through the temporal convolutional network to obtain an operational semantic label sequence.

[0074] The step of generating a spatial scene map based on multiple target objects specifically includes: The state extraction process is performed on the multiple target objects to obtain multiple object states, and the position extraction process is performed on the multiple target objects to obtain multiple position vectors; A scene graph is constructed based on multiple target objects, multiple object states, and multiple position vectors to obtain a spatial scene graph.

[0075] The time window includes a long time window and a short time window; The acquisition of the time window involves encoding the haptic frame sequence and the operation semantic label sequence according to the time window to obtain a first input matrix and a second input matrix, specifically including: The long time window and the short time window are obtained, and the haptic frame sequence is encoded to obtain the first input matrix, wherein the number of rows of the first input matrix is ​​equal to the length of the long time window, and the number of columns of the first input matrix is ​​equal to the length of the short time window. The operation semantic label sequence is encoded to obtain the second input matrix, wherein the number of rows of the second input matrix is ​​equal to the length of the long time window, and the number of columns of the second input matrix is ​​equal to the length of the short time window.

[0076] The tactile prediction model includes a memory learning model and a regression prediction model. The determination of the tactile prediction model involves performing tactile data prediction processing based on the first input matrix and the second input matrix to obtain future tactile data, specifically including: Determine the memory learning model, and perform a first reasoning process on the first input matrix and the second input matrix through the memory learning model to obtain long-term semantic memory lexical units; The regression prediction model is determined, and the regression prediction model is used to perform regression prediction processing based on the long-term semantic memory lexical units to obtain future tactile data.

[0077] The tactile prediction model includes a memory learning model and a regression prediction model. The determination of the tactile prediction model involves performing tactile data prediction processing based on the first input matrix and the second input matrix to obtain future tactile data, specifically including: Determine the memory learning model, and perform a first reasoning process on the first input matrix and the second input matrix through the memory learning model to obtain long-term semantic memory lexical units; The regression prediction model is determined, and the regression prediction model is used to perform regression prediction processing based on the long-term semantic memory lexical units to obtain future tactile data.

[0078] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a tactile interaction information prediction program, which, when executed by a processor, implements the steps of the tactile interaction information prediction method as described above.

[0079] In summary, this invention provides a method, system, and terminal for predicting tactile interaction information. The method includes: acquiring tactile data; performing feature extraction processing on the tactile data to obtain a tactile frame sequence and a tactile feature sequence; acquiring visual data; performing spatial recognition processing on the visual data to obtain a scene graph feature sequence; and performing semantic reasoning on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence; acquiring a time window; encoding the tactile frame sequence and the operation semantic label sequence according to the time window to obtain a first input matrix and a second input matrix; determining a tactile prediction model; and performing tactile data prediction processing based on the first input matrix and the second input matrix using the tactile prediction model to obtain future tactile data. This invention performs semantic reasoning on tactile and visual data to obtain operation semantic labels, and uses a deep learning model to predict future tactile signals based on the tactile data and operation semantic labels, thereby improving the accuracy of tactile signal prediction in human-computer interaction.

[0080] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0081] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0082] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for predicting tactile interaction information, characterized in that, The tactile interaction information prediction method includes: Acquire tactile data, perform feature extraction processing on the tactile data, and obtain a tactile frame sequence and a tactile feature sequence; Visual data is acquired, spatial recognition processing is performed on the visual data to obtain a scene graph feature sequence, and semantic reasoning is performed on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence. A time window is obtained, and the haptic frame sequence and the operation semantic label sequence are encoded according to the time window to obtain a first input matrix and a second input matrix. A tactile prediction model is determined, and tactile data prediction processing is performed using the tactile prediction model based on the first input matrix and the second input matrix to obtain future tactile data.

2. The tactile interaction information prediction method according to claim 1, characterized in that, The process of acquiring tactile data and performing feature extraction processing on the tactile data to obtain a tactile frame sequence and a tactile feature sequence specifically includes: Acquire tactile data, segment the tactile data to obtain multiple tactile frames, and obtain a tactile frame sequence based on the multiple tactile frames; Obtain the time span, and perform time division processing on multiple tactile frames according to the time span to obtain multiple tactile segments; A tactile feature sequence is obtained based on multiple tactile segments.

3. The tactile interaction information prediction method according to claim 1, characterized in that, The process of acquiring visual data, performing spatial recognition processing on the visual data to obtain a scene graph feature sequence, and performing semantic reasoning on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence specifically includes: Acquire visual data, determine the target detection algorithm, and perform object recognition processing on the visual data using the target detection algorithm to obtain multiple target objects; A spatial scene map is obtained by performing scene map generation processing on multiple target objects; A grouping normalization algorithm is determined, and the spatial scene graph is processed by the grouping normalization algorithm to extract scene graph features, thereby obtaining a scene graph feature sequence; A temporal convolutional network is determined, and cross-modal inference processing is performed on the tactile feature sequence and the scene graph feature sequence through the temporal convolutional network to obtain an operational semantic label sequence.

4. The tactile interaction information prediction method according to claim 3, characterized in that, The step of generating a spatial scene map based on multiple target objects specifically includes: The state extraction process is performed on the multiple target objects to obtain multiple object states, and the position extraction process is performed on the multiple target objects to obtain multiple position vectors; A scene graph is constructed based on multiple target objects, multiple object states, and multiple position vectors to obtain a spatial scene graph.

5. The tactile interaction information prediction method according to claim 1, characterized in that, The time window includes a long time window and a short time window; The acquisition of the time window involves encoding the haptic frame sequence and the operation semantic label sequence according to the time window to obtain a first input matrix and a second input matrix, specifically including: The long time window and the short time window are obtained, and the haptic frame sequence is encoded to obtain the first input matrix, wherein the number of rows of the first input matrix is ​​equal to the length of the long time window, and the number of columns of the first input matrix is ​​equal to the length of the short time window. The operation semantic label sequence is encoded to obtain the second input matrix, wherein the number of rows of the second input matrix is ​​equal to the length of the long time window, and the number of columns of the second input matrix is ​​equal to the length of the short time window.

6. The tactile interaction information prediction method according to claim 1, characterized in that, The tactile prediction model includes a memory learning model and a regression prediction model; The determination of the tactile prediction model involves performing tactile data prediction processing based on the first input matrix and the second input matrix to obtain future tactile data, specifically including: Determine the memory learning model, and perform a first reasoning process on the first input matrix and the second input matrix through the memory learning model to obtain long-term semantic memory lexical units; The regression prediction model is determined, and the regression prediction model is used to perform regression prediction processing based on the long-term semantic memory lexical units to obtain future tactile data.

7. The tactile interaction information prediction method according to claim 6, characterized in that, The memory learning model includes a tactile learning model and a semantic learning model; The step of determining the memory learning model and performing a first inference process on the first input matrix and the second input matrix through the memory learning model to obtain long-term semantic memory lexical units specifically includes: Determine the tactile learning model, and perform a first inference process on the first input matrix using the tactile learning model to obtain a first output vector; Determine the semantic learning model, and perform a second inference process on the second input matrix through the semantic learning model to obtain a second output vector; The first output vector and the second output vector are concatenated to obtain long-term semantic memory lexical units.

8. A tactile interaction information prediction system, characterized in that, The tactile interaction information prediction system includes: The tactile feature extraction module is used to acquire tactile data, perform feature extraction processing on the tactile data, and obtain a tactile frame sequence and a tactile feature sequence. The semantic label generation module is used to acquire visual data, perform spatial recognition processing on the visual data to obtain a scene graph feature sequence, and perform semantic reasoning on the tactile feature sequence and the scene graph feature sequence to obtain an operation semantic label sequence. A matrix encoding module is used to obtain a time window and encode the haptic frame sequence and the operation semantic label sequence according to the time window to obtain a first input matrix and a second input matrix. The data prediction module is used to determine the tactile prediction model, and to perform tactile data prediction processing based on the first input matrix and the second input matrix to obtain future tactile data.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a tactile interaction information prediction program stored in the memory and executable on the processor. When the processor executes the tactile interaction information prediction program, it implements the steps of the tactile interaction information prediction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a tactile interaction information prediction program, which, when executed by a processor, implements the steps of the tactile interaction information prediction method as described in any one of claims 1-7.

Images