A robot multi-part tactile perception and human-computer interaction action recognition method

Abstract

The application discloses a robot multi-part tactile perception and human-computer interaction action recognition method, first, an identification system is built, and a human-computer interaction tactile information database is constructed; then a sliding window algorithm is used to extract a key frame sequence from the human-computer interaction tactile information database to construct two interaction action data sets, which are respectively used as inputs of a neural network model; then a neural network architecture is constructed, a light-weight neural network model and a complex neural network model are constructed according to the neural network architecture, and the neural network model is trained and quantized; finally, different interaction action information applied by a participant on an array type tactile sensor is collected, a light-weight neural network model deployed by a signal collection module is used for bottom layer processing, an embedded control terminal collects tactile information of all signal collection modules, and a multi-thread program is executed in the embedded control terminal to realize parallel communication and model reasoning with the signal collection modules, so that the interaction action applied by the participant to the carrier is recognized.

Description

Technical Field

[0001] This invention relates to the field of robot tactile perception, specifically a method for multi-part tactile perception and human-computer interaction action recognition in robots. Background Technology

[0002] With the rapid development of intelligent robots, more and more of them are being applied in real-world scenarios, such as industry, healthcare, and domestic services. Humans are gradually entering an era of social interaction with robots, which requires mature scientific and technological support. How to endow robots with numerous human-like functions is a problem currently facing engineers. Among these challenges, robot perception technology is an area worthy of exploration and innovation.

[0003] Visual and auditory technologies have already achieved significant practical applications, such as facial recognition, posture recognition, and speech recognition. Tactile perception technology has become a hot topic among engineers in recent years, as physical contact is essential for human-robot interaction. How to give robots tactile perception like human skin and the ability to recognize human intentions presents a series of technical challenges that await resolution.

[0004] Currently, tactile sensors used in robots include single-point and array types. Single-point tactile sensors suffer from complex wiring and difficult installation, making them less convenient to wear than array-type sensors, and they cannot meet higher resolution requirements. Considering that intelligent robots should possess embodied intelligence in the future, current tactile perception and human-computer interaction action recognition systems only target tactile perception of a single part of the robot, making it difficult to handle scenarios where array-type tactile sensors cover multiple parts of the robot. For example, document application number 2020112535084 discloses a robot tactile action recognition system and method, which is mainly characterized by the robot not being equipped with an intelligent embedded control terminal, yet still capable of deploying complex artificial intelligence algorithms and simultaneously processing tactile information collected by multiple information acquisition modules. Furthermore, recognition methods for dynamic interactive actions are limited by traditional feature extraction and single-information recognition technologies, making it difficult to achieve real-time recognition of dynamic actions.

[0005] In conclusion, current technological development requires a method for multi-part tactile perception and human-computer interaction action recognition in robots to overcome the above limitations. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a method for multi-part tactile perception and human-computer interaction action recognition in robots.

[0007] The technical solution of the present invention to solve the aforementioned technical problem is to provide a method for multi-part tactile perception and human-computer interaction action recognition in a robot, characterized in that the method includes the following steps:

[0008] Step 1: Build the recognition system: Wear n array-type tactile sensors on various parts of the robot's body; the row and column electrodes of the n array-type tactile sensors are respectively connected to their respective signal acquisition modules; each of the n signal acquisition modules is connected to an embedded control terminal.

[0009] Start the embedded control terminal and n signal acquisition modules, and set the sampling frequency of each signal acquisition module to RHz, where R is not greater than the upper limit of the physical sampling frequency of the signal acquisition module;

[0010] The array-type tactile sensor is an array formed by i rows and j columns of flexible electrodes, with sensing units formed at the intersections of rows and columns and a dielectric layer disposed thereon.

[0011] Step 2: Construct a human-computer interaction tactile information database:

[0012] S2.1 The motion designer demonstrates to M participants L types of interactive actions applied to an array of tactile sensors using standard actions. The L types of interactive actions consist of L1 first interactive actions and L2 second interactive actions. The first interactive action is generated by the participant contacting a single array of tactile sensors, and the second interactive action is generated by the participant contacting n arrays of tactile sensors simultaneously.

[0013] S2.2. Next, have M participants apply L types of interactive actions to the array-type tactile sensor according to a preset standard. Each interactive action is repeated N times. The output signal of each participant applying an interactive action to the array-type tactile sensor once is recorded as an action sample. Then, K = (L1 + n × L2) × M × N interactive action samples are obtained, forming a human-computer interaction tactile information database. Among them, the first interactive action generates K1 = L1 × M × N first interactive action samples, and the second interactive action generates K2 = L2 × M × N × n second interactive action samples.

[0014] S2.3. Then, based on the sampling frequency R Hz of the signal acquisition module and the duration Ts of each participant's interaction action, determine the X samples contained in each first interaction action sample. p =R×T p (p∈[1,K1]) frames, each second interaction action sample contains X q =R×T q(q∈[1,K2]) frames, where each frame is defined as an i×j pressure matrix A, and the matrix elements correspond to the pressure mapping z∈[0,255] at the intersection of each row and column of the array of tactile sensors in row i and column j.

[0015] Step 3: Construct the first interactive action dataset and the second interactive action dataset from the interactive action samples;

[0016] For each of the K1 first interaction action samples, a sliding window algorithm is used for data preprocessing. The window size is set to a fixed value of Y = S(F-1) + 1 consecutive frames, where S is the number of frames in the keyframe span and F is the number of frames in the keyframe sequence. The window is slid from beginning to end in chronological order, with a span of G frames, to obtain the corresponding data. A window, thus obtaining A sequence of keyframes is generated, and each frame in the keyframe sequence is normalized to obtain the first interactive action dataset, which is used as the input for training a lightweight neural network model.

[0017] The first interactive action dataset has dimensions V1×F×i×j×1, where the first dimension V1 represents the number of keyframe sequences, the second dimension F represents the number of frames in the keyframe sequence, the third dimension i and the fourth dimension j represent the keyframes in row i and column j, and the last dimension 1 represents the grayscale channel.

[0018] The K2 second interaction action samples are preprocessed using the same sliding window algorithm as the first interaction action samples, resulting in... A window, thus obtaining A sequence of key frames for each interactive action is generated, and each frame in the key frame sequence is normalized. Then, the key frames in the key frame sequences generated by the n array-type tactile sensors at the same moment when the same interactive action is touched are superimposed on the grayscale channel to obtain a second interactive action dataset as input for training a complex neural network model.

[0019] The second interactive action dataset has the following dimensions: Among them, the first dimension The first dimension represents the number of keyframe sequences, the second dimension F represents the number of frames in the keyframe sequence, the third dimension i and the fourth dimension j represent the keyframes in row i and column j, and the last dimension n represents the number of touch sensors corresponding to the interactive action in the grayscale channel.

[0020] Step 4: Construct the neural network architecture for lightweight and complex neural network models:

[0021] The neural network architecture consists of a CNN module group, a GRU module, and an FCNN module connected in sequence; the CNN module group consists of F parallel CNN modules;

[0022] Step 5: Build and train lightweight neural network models and complex neural network models based on the neural network architecture;

[0023] Constructing a lightweight neural network model: The lightweight neural network model consists of a CNN module group, a time-distributed layer, a GRU module, and an FCNN module connected in sequence. The CNN module group consists of F parallel CNN modules. Each CNN module consists of a cascaded convolutional blocks and a global max pooling layer. Each convolutional block consists of a depthwise separable convolutional layer and a convolutional layer connected in sequence. The GRU module consists of one GRU layer. The FCNN module consists of b cascaded fully connected layers and one fully connected (FC) output layer.

[0024] Constructing a complex neural network model: The complex neural network model consists of a CNN module group, a time-distributed layer, a GRU module, and an FCNN module connected in sequence. The CNN module group consists of F parallel CNN modules. Each CNN module consists of a cascaded convolutional blocks and a global max-pooling layer. Each convolutional block consists of two identical convolutional layers and a max-pooling layer connected in sequence. The GRU module consists of one GRU layer. The FCNN module consists of b cascaded fully connected layers and one fully connected (FC) output layer.

[0025] Step 6: Deploy the trained lightweight neural network model and the trained complex neural network model, and design corresponding quantization methods to accelerate inference and set quantization levels for each;

[0026] Step 7: The signal acquisition module identifies the first interactive action and the embedded control terminal identifies the second interactive action.

[0027] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0028] (1) The present invention uses a method combining a distributed lightweight algorithm deployed by a signal acquisition module and a parallel algorithm of an embedded control terminal. First, a recognition system is built and a human-computer interaction tactile information database is constructed. Then, a sliding window algorithm is used to extract key frame sequences from the human-computer interaction tactile information database to construct two interactive action datasets, which are used as inputs for training neural network models. Next, a neural network architecture is constructed, and a lightweight neural network model and a complex neural network model are constructed based on the neural network architecture. The neural network models are then trained and quantized. Finally, information on different interactive actions applied by participants to array-type tactile sensors is collected. The lightweight neural network model deployed by the signal acquisition module performs low-level processing. The embedded control terminal collects tactile information from all signal acquisition modules and executes a multi-threaded program to achieve parallel communication and model inference with multiple signal acquisition modules, thereby recognizing the interactive actions applied by participants to any or several parts of the carrier.

[0029] (2) The sliding window algorithm designed in this invention performs data preprocessing on the collected tactile information and provides input data to the model in real time during the training and inference of the neural network model, so as to extract key frame sequences in real time for recognition during action interaction. In addition, by adjusting the parameters of the sliding window algorithm, a neural network model with better recognition performance can be trained.

[0030] (3) The neural network architecture designed in this invention takes into account the temporal variability of tactile information. Using the pressure matrix sequence as input, it can learn the spatial-temporal features of the tactile information input, thereby realizing the recognition of key frame sequence input of dynamic actions.

[0031] (4) The neural network model quantization method used in this invention can run on platforms with limited computational complexity and memory resources without losing the recognition accuracy of the neural network model.

[0032] (5) In this invention, the embedded control terminal is connected and communicates in parallel with each signal acquisition module. In the scenario where the robot is equipped with a tactile sensor, it can realize the acquisition and action recognition based on a single sensor signal, as well as the parallel acquisition and collaborative interactive action recognition of distributed tactile information. It can freely schedule the operation of each signal acquisition module, thereby recognizing the interactive actions applied by the participant to any part or several parts of the carrier and judging their interactive intention.

[0033] (6) The present invention achieves real-time acquisition and action recognition of a single sensor signal by deploying a signal acquisition module and a lightweight neural network model, and can meet the conditions of low computing power and low memory resources, thereby reducing the power consumption and cost of the signal acquisition module.

[0034] (7) This invention can be widely applied in the field of robot tactile perception. It breaks through many limitations of robot tactile perception of a single part and the use of traditional feature extraction and single information recognition methods for dynamic feature actions. It can recognize interaction actions based on a single sensor (push, pull, pinch, drag) or interaction actions based on multiple sensors (hug, hug, back hug, support). The designed tactile perception and human-computer interaction action recognition system and recognition method have the characteristics of good versatility and high recognition rate, and have extremely high social application value. Attached Figure Description

[0035] Figure 1 This is a block diagram of the overall structure of the identification system of the present invention;

[0036] Figure 2 This is a schematic diagram of the sliding window algorithm of the present invention;

[0037] Figure 3 This is a structural diagram of the neural network architecture of the present invention;

[0038] Figure 4 This is a structural diagram of the lightweight neural network model of the present invention;

[0039] Figure 5 This is a flowchart of the signal acquisition module program of the present invention;

[0040] Figure 6 This is a flowchart of the multi-threaded program for the embedded control terminal of the present invention. Detailed Implementation

[0041] Specific embodiments of the present invention are given below. These specific embodiments are only used to further illustrate the present invention in detail and do not limit the scope of protection of the claims of the present invention.

[0042] This invention provides a method for multi-part tactile perception and human-computer interaction action recognition in robots (hereinafter referred to as the method), characterized by the following steps:

[0043] Step 1: Build the recognition system: Wear n array-type tactile sensors on various parts of the robot's body; the row and column electrodes of the n array-type tactile sensors are connected to their respective signal acquisition modules via wired connections; the n signal acquisition modules are all connected to an embedded control terminal via wired (serial or parallel port communication) or wireless (wireless network or Bluetooth) connections.

[0044] Start the embedded control terminal and n signal acquisition modules, and set the sampling frequency of each signal acquisition module to RHz, where R is not greater than the upper limit of the physical sampling frequency of the signal acquisition module;

[0045] The array-type tactile sensor is an array formed by i rows and j columns of flexible electrodes, with sensing units formed at the intersections of rows and columns and a dielectric layer disposed thereon.

[0046] Preferably, in step 1, the array-type tactile sensor is used to sense the participant's touch action, and its principle is based on piezoresistive, capacitive, supercapacitive, piezoelectric, organic transistor and other sensing principles.

[0047] Preferably, in step 1, the robot's body surface includes the outer shells of each joint and link, which are used to support and mount the array-type tactile sensors, so that the array-type tactile sensors cover the entire body surface of the robot.

[0048] Preferably, in step 1, the signal acquisition module is an embedded microcontroller. Due to limitations such as low computing power and low memory resources, it can achieve real-time acquisition of tactile sensor contact pressure distribution information through ADC scanning by programming, deploy a lightweight neural network model, and identify interactive actions received by a single tactile sensor. In this embodiment, a micro embedded platform such as STM32 is used to meet the requirements of low cost and low power consumption.

[0049] Preferably, in step 1, the embedded control terminal has high computing power and high memory resources for deploying complex artificial intelligence algorithms. It undertakes tasks such as deploying and executing multi-threaded programs to achieve parallel communication with various signal acquisition modules, receiving human-computer interaction tactile information, data preprocessing, and complex neural network model inference and recognition. In this embodiment, the embedded control terminal uses an AI embedded platform such as NVIDIA Jetson Xavier NX or HUAWEI Atlas 200DK.

[0050] Preferably, in step 1, when the signal acquisition module and the embedded control terminal are connected by a wire, the power supply of the embedded control terminal powers the entire system; when the signal acquisition module and the embedded control terminal are connected by a wireless connection, both the embedded control terminal and the signal acquisition module are powered by external power supplies, and the signal acquisition module powers the array-type tactile sensor.

[0051] Step 2: Construct a human-computer interaction tactile information database:

[0052] S2.1 The motion designer demonstrates to M participants L types of interactive actions applied to an array of tactile sensors using standard actions. The L types of interactive actions consist of L1 first interactive actions and L2 second interactive actions. The first interactive action is generated by the participant contacting a single array of tactile sensors, and the second interactive action is generated by the participant contacting n arrays of tactile sensors simultaneously.

[0053] S2.2. Next, have M participants apply L types of interactive actions to the array-type tactile sensor according to a preset standard. Each interactive action is repeated N times. The output signal of each participant applying an interactive action to the array-type tactile sensor once is recorded as an action sample. Then, K = (L1 + n × L2) × M × N interactive action samples are obtained, forming a human-computer interaction tactile information database. Among them, the first interactive action generates K1 = L1 × M × N first interactive action samples, and the second interactive action generates K2 = L2 × M × N × n second interactive action samples.

[0054] S2.3. Then, based on the sampling frequency R Hz of the signal acquisition module and the duration Ts of each participant's interaction action, determine the X samples contained in each first interaction action sample. p =R×T p (p∈[1,K1]) frames, each second interaction action sample contains X q =R×T q (q∈[1,K2]) frames, where each frame is defined as an i×j pressure matrix A, and the matrix elements correspond to the pressure mapping z∈[0,255] at the intersection of each row and column of the array of tactile sensors in row i and column j.

[0055] Step 3: Construct the first interactive action dataset and the second interactive action dataset from the interactive action samples;

[0056] Apply the sliding window algorithm to each of the K1 first interaction action samples (see...) Figure 2 Data preprocessing is performed, setting the window size to a fixed value of Y = S(F-1) + 1 consecutive frames, where S is the number of frames in the keyframe span and F is the number of frames in the keyframe sequence. The window is then slid from beginning to end in chronological order with a span of G frames, yielding the corresponding data. A window, thus obtaining A sequence of keyframes for each interactive action is generated, and each frame in the keyframe sequence is normalized to obtain the first interactive action dataset, which is used as input for training a lightweight neural network model.

[0057] The first interactive action dataset has dimensions V1×F×i×j×1, where the first dimension V1 represents the number of keyframe sequences, the second dimension F represents the number of frames in the keyframe sequence, the third dimension i and the fourth dimension j represent the keyframes in row i and column j, and the last dimension 1 represents the grayscale channel.

[0058] The K2 second interaction action samples are preprocessed using the same sliding window algorithm as the first interaction action samples, resulting in... A window, thus obtaining A sequence of key frames for each interactive action is generated, and each frame in the key frame sequence is normalized. Then, the key frames in the key frame sequences generated by the n array-type tactile sensors at the same moment when the same interactive action is touched are superimposed on the grayscale channel to obtain a second interactive action dataset as input for training a complex neural network model.

[0059] The second interactive action dataset has the following dimensions: Among them, the first dimension The first dimension represents the number of keyframe sequences, the second dimension F represents the number of frames in the keyframe sequence, the third dimension i and the fourth dimension j represent the keyframes in row i and column j, and the last dimension n represents the number of touch sensors corresponding to the interactive action in the grayscale channel.

[0060] Preferably, in step 3, the data preprocessing of the K2 second interaction action samples using the sliding window algorithm specifically involves: setting the fixed size of the window to Y = S(F-1) + 1 consecutive frames, where S is the number of frames in the keyframe span and F is the number of frames in the keyframe sequence; sliding the window from beginning to end in chronological order for a span of G frames, corresponding to... A window, thus obtaining A sequence of keyframes for interactive actions.

[0061] Step 4: Construct the neural network architecture for lightweight and complex neural network models:

[0062] The neural network architecture consists of a CNN (Convolutional Neural Network) module group, a GRU (Gated Recurrent Unit) module, and an FCNN (Fully Connected Neural Network) module connected in sequence; the CNN module group consists of F parallel CNN modules;

[0063] Preferably, in step 4, the keyframe sequence is used as input (see...). Figure 3 The CNN module extracts the spatial features of each keyframe, the GRU module extracts the temporal features between the spatial features of the keyframes, and the FCNN module classifies the spatial-temporal feature information of each keyframe sequence to obtain the probability distribution of the interactive action category corresponding to this keyframe as the output.

[0064] Step 5: Build and train lightweight neural network models and complex neural network models based on the neural network architecture;

[0065] Constructing a lightweight neural network model: The lightweight neural network model consists of a CNN module group, a time-distributed layer, a GRU module, and an FCNN module connected in sequence. The CNN module group consists of F parallel CNN modules. Each CNN module consists of a cascaded convolutional blocks and a global max-pooling layer. Each convolutional block consists of a depthwise separable convolutional layer (DWConv) and a convolutional layer (Conv) connected in sequence. The GRU module consists of one GRU layer. The FCNN module consists of b cascaded fully connected layers (FC) and one FC output layer (see [link to documentation]). Figure 4 First, each keyframe in the keyframe sequence is convolved by a convolutional blocks and pooled by a global max pooling layer to obtain spatial features. This process is repeated F times for the number of frames in the keyframe sequence. Then, the time distributed layer packages the spatial features into a temporal sequence and learns the temporal features through the GRU module. Finally, the temporal feature classification information is learned through the FCNN module.

[0066] A complex neural network model is constructed: the complex neural network model consists of a CNN module group, a time-distributed layer, a GRU module, and an FCNN module connected in sequence. The CNN module group consists of F parallel CNN modules. Each CNN module consists of a cascaded convolutional blocks and a global max pooling layer. Each convolutional block consists of two identical convolutional layers (Conv) and a max pooling layer connected in sequence. The GRU module consists of one GRU layer. The FCNN module consists of b cascaded fully connected layers (FC) and one FC output layer. First, each keyframe in the keyframe sequence is convolved by a convolutional blocks and pooled by the global max pooling layer to obtain spatial features. This process is repeated F times for the number of frames in the keyframe sequence. Then, the time-distributed layer packages the spatial features into a temporal sequence, and the GRU module learns the temporal features. Finally, the FCNN module learns the classification information of the temporal features.

[0067] The lightweight neural network model and the complex neural network model were trained separately: the first interactive action dataset was divided into training, validation, and test sets for training the lightweight neural network model, and the second interactive action dataset was divided into training, validation, and test sets for training the complex neural network model. The Hyperband hyperparameter optimization method was used to set up a hyperparameter search space. The training and validation sets were used for hyperparameter search. The hyperparameter combination with the highest accuracy on the validation set was selected from the search results to train the neural network model. The neural network model with the highest accuracy on the test set was selected from the resulting model set and saved.

[0068] Preferably, in step 5, the first interactive action dataset is divided into a training set, a validation set, and a test set in a ratio of 60:20:20; the second interactive action dataset is divided into a training set, a validation set, and a test set in a ratio of 60:20:20.

[0069] Preferably, in step 5, a lightweight neural network model is trained using the Keras framework. Each keyframe sequence has a dimension of 1×F×i×j×1 and is used as input. The keyframes have a dimension of i×j×1. The CNN module extracts spatial features from the F keyframes. The DWConv convolutional kernel has a size of 3×3, a stride of 2, and the same padding. The Conv convolutional kernel has α kernels, a size of 1×1, a stride of 1, and the same padding. Batch Normalization and ReLU activation are used after each layer. The GRU module has β neurons and 1 layer. In the FCNN module, b is 1 and the number of neurons is χ. ReLU activation and Dropout layers are used after each layer, with a dropout rate of d. The FC output layer has L1 output channels and is activated using the Softmax activation function. The loss function is the cross-entropy loss function, and the learning rate is r.

[0070] Preferably, in step 5, a complex neural network model is trained using the Keras framework. Each keyframe sequence has a dimension of 1×F×i×j×n and is used as input. The keyframe dimension is i×j×n. The CNN module extracts spatial features from the F keyframes respectively. The Conv convolutional kernel has α kernels, a size of 3×3, a stride of 1, and the same padding. Batch Normalization and ReLU activation function are used after the layer. The Max Pooling pooling window has a size of 2×2, a stride of 2, and valid padding. Dropout layer is used after the layer with a dropout rate of d. The GRU module has β neurons and 1 layer. The FCNN module has b FC layers and χ neurons. ReLU activation function and Dropout layer are used after the layer with a dropout rate of d. The number of output channels of the FC output layer is set to L2, and the Softmax activation function is used. The loss function is the cross-entropy loss function, and the learning rate is r.

[0071] Preferably, in step 5, in the Hyperband hyperparameter optimization method, the search space of the lightweight neural network model is defined as a∈[1,3], α∈[32,128], β∈[32,128], χ∈[32,128], d∈[0,0.5], r∈[1e-4,1e-1], and the search space of the complex neural network model is defined as a∈[2,5], α∈[32,256], β∈[32,256], b∈[2,5], χ∈[32,256], d∈[0,0.5], r∈[1e-4,1e-1].

[0072] Step 6: Deploy the trained lightweight neural network model and the trained complex neural network model, and design corresponding quantization methods to accelerate inference and set quantization levels for each;

[0073] Preferably, in step 6, after analyzing the trained lightweight neural network model using the initialization code generator and confirming that the model successfully meets the RAM resource requirements of the signal acquisition module, a programmable model file for the signal acquisition module is generated using the AI ​​extension package. This file contains interface functions that the signal acquisition module software program can call, allowing the input and output data spaces of the model to be passed in. The trained lightweight neural network model reads keyframe sequences from the input space, performs calculations, and writes the calculation results into the output space. In this embodiment, the SEM32CubeMX initialization code generator is used to analyze the trained lightweight neural network model, and the X-CUBE-AI extension package generates the programmable model file for the STM32 signal acquisition module.

[0074] Preferably, in step 6, the trained complex neural network model is loaded into the embedded control terminal. Using an AI acceleration framework, any quantization level (float32, float16, int8) is set to obtain a reasoning-capable model file of a specific data type, which is then saved in the embedded control terminal. In this embodiment, in the NVIDIA GPU embedded control terminal, the trained complex neural network model is saved as a TensorFlow model file, and the TensorFlow-TensorRT (TF-TRT) acceleration framework is used to convert it into an accelerated inference model file before saving it.

[0075] Step 7: The signal acquisition module identifies the first interactive action and the embedded control terminal identifies the second interactive action;

[0076] Scenario 1: When a participant applies an interactive action to any of the array-type tactile sensors, the specific steps are as follows (see...). Figure 5 ):

[0077] A7.1 The signal acquisition module initializes the ADC scanning algorithm in the signal acquisition module and scans and acquires tactile information in a loop. It uses a noise reduction algorithm to filter out noise signals, obtains the pressure matrix tactile information of row i and column j, and uploads it to the embedded control terminal.

[0078] A7.2 Set a threshold for the average value of the pressure matrix; when the average value of the pressure matrix is ​​greater than the threshold, the frame is determined to have tactile motion information; when the average value of the pressure matrix is ​​equal to the threshold, the frame is determined to have no tactile motion information.

[0079] Preferably, in step A7.2, the threshold value of the pressure matrix mean is set to 0.

[0080] A7.3 When there is tactile motion information, keyframe sequences are collected as input to the lightweight neural network model. The data dimensions of the keyframe sequences and the data preprocessing of the sliding window algorithm are consistent with the construction of the first interactive action dataset in step 3. Whenever a keyframe sequence is collected, the lightweight neural network model interface function is called to execute the lightweight neural network model inference and upload the interactive action recognition result, and then the keyframe sequence is updated. When there is no tactile motion information, the lightweight neural network model inference is not executed, the uploaded recognition result is no action, and finally it is sent to the embedded control terminal via serial communication.

[0081] Scenario 2: When the participant applies interactive actions to at least two arrayed tactile sensors, the specific steps are as follows (see...). Figure 6 ):

[0082] B7.1 Import the program execution dependency library into the main thread of the embedded control terminal, initialize the global variables FrameList, ResultList, and Result, and load the complex neural network model. Result represents the second interactive action recognition result. FrameList and ResultList are list types, with list indices corresponding to n I / O sub-threads, and each of the n I / O sub-threads corresponding to n signal acquisition modules. The user visual operation interface includes n tactile information visualization interfaces, n labels for the first interactive action recognition result, and n labels for the second interactive action recognition result.

[0083] B7.2. Start n I / O sub-threads, and establish serial communication with the signal acquisition module in the I / O sub-threads to read the tactile information and the recognition results of the signal acquisition module in real time, and store them into the corresponding subscripts of the FrameList and ResultList lists respectively;

[0084] B7.3. Start a recognition sub-thread and read the FrameList list in real time. If the mean of each index element matrix is ​​greater than 0, it indicates that there is a second interaction action. Start collecting keyframe sequences as input to the complex neural network model. The data dimensions of the keyframe sequences and the data preprocessing of the sliding window algorithm are consistent with the construction of the second interaction action dataset in step 3. Whenever a keyframe sequence is collected, the complex neural network model interface function is called to execute the complex neural network model inference and the recognition result is written to Result. Then the keyframe sequence is updated. If there is no second interaction action, the complex neural network model inference is not executed and the no-action recognition result is written to Result.

[0085] B7.4 The main thread reads FrameList, ResultList and Result in real time and updates the corresponding user visual operation interface with the current value respectively; participants can observe the tactile information and recognition results of the robot's interactive actions on any part or several parts in real time through the user visual operation interface.

[0086] Preferably, in step 7, the main thread is used to import program execution dependency libraries, define and initialize global variables, and define and initialize the user visual operation interface; the I / O sub-thread is used to communicate with the signal acquisition module in real time; and the recognition sub-thread is used to perform complex neural network model inference tasks.

[0087] The working principle of this invention is as follows: A person comes into contact with an array of tactile sensors worn on the robot's body, generating pressure. This pressure causes changes in the voltage values ​​of the sensing units at the intersections of rows and columns, thus generating electrical signals. A signal acquisition module receives these electrical signals, performs analog-to-digital conversion (ADC) to convert them into digital signals, and uses denoising and sliding window algorithms for signal filtering and tactile information preprocessing. A lightweight neural network model infers the interaction action recognition results with a single sensor within the signal acquisition module. These results are then input to an embedded control terminal for feature group construction and cognitive recognition processing, yielding the interaction action recognition results with multiple sensors.

[0088] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A method for multi-part tactile perception and human-computer interaction action recognition in a robot, characterized in that, The method includes the following steps: Step 1: Build the recognition system: Wear n array-type tactile sensors on various parts of the robot's body; the row and column electrodes of the n array-type tactile sensors are respectively connected to their respective signal acquisition modules; each of the n signal acquisition modules is connected to an embedded control terminal. Start the embedded control terminal and n signal acquisition modules, and set the sampling frequency of each signal acquisition module to RHz, where R is not greater than the upper limit of the physical sampling frequency of the signal acquisition module; The array-type tactile sensor is an array formed by i rows and j columns of flexible electrodes, with sensing units formed at the intersections of rows and columns and a dielectric layer disposed thereon. Step 2: Construct a human-computer interaction tactile information database: S2.1 The motion designer demonstrates L interactive actions applied to an array of haptic sensors to M participants using standard actions, where the L interactive actions are determined by... The first interactive action and The second interactive action is constituted by the first interactive action being generated by the participant contacting a single array of tactile sensors, and the second interactive action being generated by the participant contacting n arrays of tactile sensors simultaneously. S2.2 Next, have M participants apply L types of interactive actions to the array-type tactile sensor according to a preset standard, with each interactive action repeated N times; the output signal of each participant applying one interactive action to the array-type tactile sensor once is recorded as an action sample, thus obtaining... A database of human-computer interaction tactile information is formed from samples of interactive actions; among them, the first interactive action generates... A first interaction action sample, the second interaction action is generated. A second interactive action sample; S2.

3. Then, based on the sampling frequency R Hz of the signal acquisition module and the duration T s of each participant's interaction action, determine the sample of each first interaction action. Each frame contains a second interaction action sample. Frames, where each frame is defined as The pressure matrix A, whose elements correspond to the pressure mappings at the intersections of the rows and columns of the array of tactile sensors in row i and column j. ; Step 3: Construct the first interactive action dataset and the second interactive action dataset from the interactive action samples; right Each of the first interaction action samples was preprocessed using a sliding window algorithm, with the window size set to a fixed value. Given a series of consecutive frames, where S is the number of frames in the keyframe span and F is the number of frames in the keyframe sequence, slide the window across G frames sequentially from beginning to end to obtain the corresponding... A window, thus obtaining A sequence of keyframes is generated, and each frame in the keyframe sequence is normalized to obtain the first interactive action dataset, which is used as the input for training a lightweight neural network model. The first interactive action dataset has the following dimensions: The first dimension The second dimension represents the number of keyframe sequences. The third dimension represents the number of frames in the keyframe sequence. and the fourth dimension This represents the keyframe in row i and column j, the last dimension. Represents the grayscale channel; right The second interaction action sample was preprocessed using the same sliding window algorithm as the first interaction action sample, resulting in... A window, thus obtaining A sequence of key frames for each interactive action is generated, and each frame in the key frame sequence is normalized. Then, the key frames in the key frame sequences generated by the n array-type tactile sensors at the same moment when the same interactive action is touched are superimposed on the grayscale channel to obtain a second interactive action dataset as input for training a complex neural network model. The second interactive action dataset has the following dimensions: The first dimension The second dimension represents the number of keyframe sequences. The third dimension represents the number of frames in the keyframe sequence. and the fourth dimension This represents the keyframe in row i and column j, the last dimension. This indicates the number of sensor contacts for the interactive action corresponding to the grayscale channel; Step 4: Construct the neural network architecture for lightweight and complex neural network models: The neural network architecture consists of a CNN module group, a GRU module, and an FCNN module connected in sequence; the CNN module group consists of F parallel CNN modules; Step 5: Build and train lightweight neural network models and complex neural network models based on the neural network architecture; Constructing a lightweight neural network model: The lightweight neural network model consists of a CNN module group, a time-distributed layer, a GRU module, and an FCNN module connected in sequence. The CNN module group consists of F parallel CNN modules. Each CNN module consists of a cascaded convolutional blocks and a global max pooling layer. Each convolutional block consists of a depthwise separable convolutional layer and a convolutional layer connected in sequence. The GRU module consists of one GRU layer. The FCNN module consists of b cascaded fully connected layers and one fully connected (FC) output layer. Constructing a complex neural network model: The complex neural network model consists of a CNN module group, a Timedistributed layer, a GRU module, and an FCNN module connected in sequence. The CNN module group consists of F parallel CNN modules. Each CNN module consists of a cascaded convolutional blocks and a global max pooling layer. Each convolutional block consists of two identical convolutional layers and a max pooling layer connected in sequence. The GRU module consists of one GRU layer. The FCNN module consists of b cascaded fully connected layers and one FC output layer. Step 6: Deploy the trained lightweight neural network model and the trained complex neural network model, and design corresponding quantization methods to accelerate inference and set quantization levels for each; Step 7: The signal acquisition module identifies the first interactive action and the embedded control terminal identifies the second interactive action.

2. The method for multi-part tactile perception and human-computer interaction action recognition of a robot according to claim 1, characterized in that, In step 3, for Each of the second interaction action samples underwent data preprocessing using the sliding window algorithm. Specifically, the window size was set to a fixed value. Given a series of consecutive frames, where S is the number of frames in the keyframe span and F is the number of frames in the keyframe sequence, slide the window across G frames sequentially from beginning to end to obtain the corresponding... A window, thus obtaining A sequence of keyframes for interactive actions.

3. The method for multi-part tactile perception and human-computer interaction action recognition of a robot according to claim 1, characterized in that, In step 4, the keyframe sequence is used as input, the CNN module extracts the spatial features of each keyframe, the GRU module extracts the temporal features between the spatial features of the keyframes, and the FCNN module classifies the spatial-temporal feature information of each keyframe sequence to obtain the probability distribution of the interactive action category corresponding to this keyframe as the output.

4. The method for multi-part tactile perception and human-computer interaction action recognition of a robot according to claim 1, characterized in that, In step 5, firstly, each keyframe in the keyframe sequence is convolved by a convolutional blocks and pooled by a global max pooling layer to obtain spatial features. This process is repeated F times for the number of frames in the keyframe sequence. Then, the time distributed layer packages the spatial features into a temporal sequence and learns the temporal features through the GRU module. Finally, the temporal feature classification information is learned through the FCNN module.

5. The method for multi-part tactile perception and human-computer interaction action recognition of a robot according to claim 1, characterized in that, In step 5, a lightweight neural network model is trained using the Keras framework, and the dimension of each keyframe sequence is... And take it as input, the dimension of the keyframe is The CNN module extracts spatial features from F keyframes, where the DWConv convolution kernel size is... The stride is 2, the padding is the same, and the number of Conv kernels is [number missing]. The dimensions are The stride is 1, the padding is the same, and all layers use Batch Normalization and ReLU activation functions; the number of neurons in the GRU module is... The number of layers is 1; in the FCNN module, b is 1, and the number of neurons is... The layer is followed by a ReLU activation function and a Dropout layer with a dropout rate of d; the number of output channels in the FC output layer is set to... The softmax activation function is used; the cross-entropy loss function is used, and the learning rate is r.

6. The method for multi-part tactile perception and human-computer interaction action recognition of a robot according to claim 1, characterized in that, In step 5, a complex neural network model is trained using the Keras framework, where the dimension of each keyframe sequence is... And use it as input, the keyframe dimension is The CNN module extracts spatial features from F keyframes, where the number of Conv convolution kernels is... The dimensions are The stride is 1, padding is the same, and BatchNormalization regularization and ReLU activation function are used after the layer; the Max Pooling window size is [missing value]. The step size is 2, padding is valid, a Dropout layer is used after the first layer, and the dropout rate is d; the number of neurons in the GRU module is... The number of layers is 1; the number of fully connected (FC) layers in the FCNN module is b, and the number of neurons is... The layer is followed by a ReLU activation function and a Dropout layer with a dropout rate of d; the number of output channels in the FC output layer is set to... The softmax activation function is used; the loss function is the cross-entropy loss function, and the learning rate is r.

7. The method for multi-part tactile perception and human-computer interaction action recognition of a robot according to claim 1, characterized in that, In step 5, the lightweight neural network model and the complex neural network model are trained respectively: the first interactive action dataset is divided into training set, validation set and test set for training the lightweight neural network model, and the second interactive action dataset is divided into training set, validation set and test set for training the complex neural network model; the Hyperband hyperparameter optimization method is used to set the hyperparameter search space, the training set and validation set are used for hyperparameter search, the hyperparameter combination with the highest accuracy on the validation set is selected from the search results to train the neural network model, and the neural network model with the highest accuracy on the test set is selected from the obtained model set and saved.

8. The method for multi-part tactile perception and human-computer interaction action recognition of a robot according to claim 1, characterized in that, In step 6, the trained lightweight neural network model is analyzed using the initialization code generator. After the analysis successfully meets the memory resource requirements of the signal acquisition module, the programmable model file of the signal acquisition module is generated through the AI ​​extension package. The file contains interface functions that can be called by the software program of the signal acquisition module. The function can pass in the model input and output data space. The trained lightweight neural network model reads keyframe sequences from the input space and performs calculations, then writes the calculation results into the output space. A pre-trained complex neural network model is loaded into the embedded control terminal. Through an AI acceleration framework, any quantization level of float32, float16, or int8 is set to obtain a reasonable model file of a specific data type, which is then saved in the embedded control terminal.

9. The method for multi-part tactile perception and human-computer interaction action recognition in a robot according to claim 1, characterized in that, Step 7 specifically involves the following steps: Condition 1: When a participant applies an interactive action to any of the array-type tactile sensors, the specific steps are as follows: A7.1 The signal acquisition module initializes the ADC scanning algorithm in the signal acquisition module and scans and acquires tactile information in a loop. It uses a noise reduction algorithm to filter out noise signals, obtains the pressure matrix tactile information of row i and column j, and uploads it to the embedded control terminal. A7.2 Set a threshold for the average value of the pressure matrix; when the average value of the pressure matrix is ​​greater than the threshold, the frame is determined to have tactile motion information; when the average value of the pressure matrix is ​​equal to the threshold, the frame is determined to have no tactile motion information. A7.3 When there is tactile motion information, keyframe sequences are collected as input to the lightweight neural network model. The data dimensions of the keyframe sequences and the data preprocessing of the sliding window algorithm are consistent with the construction of the first interactive action dataset in step 3. Whenever a keyframe sequence is collected, the lightweight neural network model interface function is called to execute the lightweight neural network model inference and upload the interactive action recognition result, and then the keyframe sequence is updated. When there is no tactile motion information, the lightweight neural network model inference is not executed, the uploaded recognition result is no action, and finally it is sent to the embedded control terminal via serial communication. Scenario 2: When a participant interacts with at least two arrayed tactile sensors, the specific steps are as follows: B7.1 Import the program execution dependency library into the main thread of the embedded control terminal, initialize the global variables FrameList, ResultList and Result, and load the complex neural network model. Among them, Result is the recognition result of the second interaction action, FrameList and ResultList are list types, and the list subscripts correspond to n I / O sub-threads, and the n I / O sub-threads correspond to n signal acquisition modules. B7.

2. Start n I / O sub-threads, and establish serial communication with the signal acquisition module in the I / O sub-threads to read the tactile information and the recognition results of the signal acquisition module in real time, and store them into the corresponding subscripts of the FrameList and ResultList lists respectively; B7.

3. Start a recognition sub-thread and read the FrameList list in real time. If the mean of each index element matrix is ​​greater than 0, it indicates that there is a second interaction action. Start collecting keyframe sequences as input to the complex neural network model. The data dimensions of the keyframe sequences and the data preprocessing of the sliding window algorithm are consistent with the construction of the second interaction action dataset in step 3. Whenever a keyframe sequence is collected, the complex neural network model interface function is called to execute the complex neural network model inference and the recognition result is written to Result. Then the keyframe sequence is updated. If there is no second interaction action, the complex neural network model inference is not executed and the no-action recognition result is written to Result. B7.4 The main thread reads FrameList, ResultList and Result in real time and updates the corresponding user visual operation interface with the current value respectively; participants can observe the tactile information and recognition results of the robot's interactive actions on any part or several parts in real time through the user visual operation interface.

Images