Intelligent identification method and device for force application action based on AI and multiple sensors

Abstract

The embodiment of the invention provides an intelligent identification method and device for a force application action based on AI and multiple sensors, and relates to the technical field of computers. The method comprises the steps that multi-dimensional data corresponding to array nodes contained in a contact surface array in target massage operation are obtained, the array nodes are arranged at equal intervals, and each array node except boundary nodes in the array nodes at least comprises a preset number of neighbor nodes; performing feature extraction on the respective multi-dimensional data of each array node to obtain the respective initial feature vector of each array node, and according to the spatial adjacency relation of each array node, updating each obtained initial feature vector to obtain an embedded vector of the target massage operation; and fusing the time sequence information and the space information of the embedded vector to obtain a target modeling vector, and determining a force application action recognition result of the target massage operation based on the target modeling vector. In this way, the force application action can be accurately quantified and recognized through the multi-dimensional data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an intelligent recognition method and device for force action based on AI and multiple sensors. BACKGROUND

[0002] Massage is a way that can effectively adjust the physical and mental state, and massage has many kinds of force actions.

[0003] In the prior art, when recognizing the force action, the current pressure-time curve data of the operator is usually used to determine the current force action category of the operator.

[0004] However, since the pressure characteristics corresponding to multiple force action categories are similar, the above method may cause incorrect recognition results, that is, the accuracy of force action recognition is low. SUMMARY

[0005] Embodiments of the present application provide an intelligent recognition method and device for force action based on AI and multiple sensors, to improve the accuracy of force action recognition.

[0006] In a first aspect, the embodiments of the present application provide an intelligent recognition method for force action based on AI and multiple sensors, and the method comprises the following steps:

[0007] Obtaining multiple-dimensional data corresponding to each array node in a contact surface array in a target massage operation, wherein the array nodes are arranged at equal intervals, and each array node has at least a preset number of neighbor nodes except for the boundary nodes;

[0008] Performing feature extraction on the multiple-dimensional data of each array node to obtain an initial feature vector of each array node, and updating the obtained initial feature vectors according to the spatial adjacency relationship of the array nodes to obtain an embedding vector of the target massage operation;

[0009] Fusing the embedding vector with time sequence information and spatial information to obtain a target modeling vector, and determining a force action recognition result of the target massage operation based on the target modeling vector.

[0010] In an optional embodiment, the multiple-dimensional data comprises force dimension data, posture dimension data and temperature dimension data.

[0011] In an optional embodiment, obtaining the multiple-dimensional data corresponding to each array node in the contact surface array in the target massage operation comprises:

[0012] Collecting first initial data of each array node in the target massage operation through the contact surface array arranged on the surface of the first object;

[0013] collecting second initial data through the hand multi-modal sensor deployed on the second object's hand;

[0014] determining the multi-dimensional data corresponding to each of the array nodes based on the second initial data and the first initial data of each of the array nodes.

[0015] In an optional embodiment, each of the array nodes comprises a pressure sensor, a force sensor and a temperature sensor, and the first initial data comprises normal force data, tangential force data and contact surface temperature data.

[0016] The hand multi-modal sensor comprises an inertial measurement unit, a depth camera and a temperature electrode, and the second initial data comprises inertial data, hand point cloud data and back of hand temperature data.

[0017] In an optional embodiment, determining the multi-dimensional data corresponding to each of the array nodes based on the second initial data and the first initial data of each of the array nodes comprises:

[0018] determining a pose time sequence based on the inertial data and the hand point cloud data, and taking the pose time sequence as the pose dimensional data corresponding to each of the array nodes;

[0019] determining force time sequence and torque time sequence of each hand joint based on the pose time sequence, and taking the normal force data, the tangential force data of each of the array nodes and the force time sequence and the torque time sequence of each hand joint as the force dimensional data corresponding to each of the array nodes;

[0020] taking the contact surface temperature data and the back of hand temperature data of each of the array nodes as the temperature dimensional data corresponding to each of the array nodes.

[0021] In an optional embodiment, determining the pose time sequence based on the inertial data and the hand point cloud data comprises:

[0022] performing adaptive extended Kalman filtering on the inertial data and the hand point cloud data to obtain the pose time sequence.

[0023] In an optional embodiment, determining the force time sequence and the torque time sequence of each hand joint based on the pose time sequence comprises:

[0024] determining end pose of each finger based on the pose time sequence and a DH parameter set of each hand joint;

[0025] performing inverse dynamics calculation based on the end pose to obtain the force time sequence and the torque time sequence of each hand joint.

[0026] In an optional embodiment, feature extraction is performed on the multi-dimensional data of each array node to obtain an initial feature vector of each array node, including:

[0027] Different domain feature extraction is performed on the multi-dimensional data of each array node to obtain a plurality of sub-initial feature vectors of each array node.

[0028] The plurality of sub-initial feature vectors of each array node are standardized and spliced to obtain an initial feature vector of each array node.

[0029] In an optional embodiment, different domain feature extraction is performed on the multi-dimensional data of each array node to obtain a plurality of sub-initial feature vectors of each array node, including:

[0030] Time domain feature extraction is performed on the force dimension data of each array node to obtain a first sub-initial feature vector of each array node.

[0031] Frequency domain feature extraction is performed on the posture dimension data of each array node to obtain a second sub-initial feature vector of each array node.

[0032] Topology domain feature extraction is performed on the temperature dimension data of each array node to obtain a third sub-initial feature vector of each array node.

[0033] In an optional embodiment, the obtained initial feature vectors are updated according to the spatial adjacency relationship of each array node to obtain an embedding vector of the target massage operation, including:

[0034] A graph is constructed based on the spatial adjacency relationship of each array node, wherein the vertices of the graph are the array nodes, and the edge set of the graph is the spatial adjacency relationship between the array nodes.

[0035] Based on the graph, graph convolution and attention aggregation are performed on the initial feature vectors to obtain an embedding vector.

[0036] In an optional embodiment, the embedding vector is fused with timing information and spatial information to obtain a modeling vector of the target massage operation, including:

[0037] Time domain dynamic feature extraction is performed on the embedding vector to obtain a timing feature vector, and spatial distribution feature extraction is performed on the embedding vector to obtain a spatial feature vector, wherein the timing feature vector represents the temporal dynamic semantics of the target massage operation, and the spatial feature vector represents the spatial distribution semantics of the target massage operation.

[0038] The timing feature vector and the spatial feature vector are fused to obtain a target modeling vector.

[0039] In one optional embodiment, the force application action recognition result of the target massage operation is determined based on the target modeling vector, including:

[0040] Determine the distance between the target modeling vector and the center vector of each force application action category, where the center vector is determined based on the sample modeling vectors of the corresponding force application action category;

[0041] Based on the obtained distances, the force application action recognition results of the target massage operation are determined.

[0042] In one optional embodiment, based on the obtained distances, the force application action recognition result of the target massage operation is determined, including:

[0043] When there is a target distance greater than the distance threshold among all distances, the force application action category of the target massage operation is determined to be the force application action category corresponding to the target distance;

[0044] When there is no target distance greater than the distance threshold among the various distances, the force application action recognition result of the target massage operation is determined based on the distance between the target modeling vector and the sample modeling vectors of each force application action category.

[0045] In one optional embodiment, the force application action recognition result is obtained by inputting the target modeling vector into the target force application action recognition model, wherein the target force application action recognition model is trained in the following manner:

[0046] Obtain a training sample set, wherein each training sample in the training sample set includes at least: the sample embedding vector of the sample massage operation;

[0047] Based on the training sample set, the force application action recognition model to be trained is iteratively trained to obtain the target force application action recognition model. During each iteration of training, the following operations are performed:

[0048] The temporal and spatial information of the sample embedding vectors of the selected target training samples are fused to obtain the sample modeling vectors of the target training samples.

[0049] Determine the sample distance between the sample modeling vector of the target training sample and the center vector of each force application action category;

[0050] Based on the obtained distances between samples, the sample force action recognition results of the target training sample are determined, and the parameters are tuned based on the loss value corresponding to the sample force action recognition results.

[0051] In an optional embodiment, after determining the force application action recognition result of the target training sample based on the obtained sample distances, and after parameter tuning based on the loss value corresponding to the sample force application action recognition result, the method further includes:

[0052] When the target training sample is a high-confidence sample, the sample modeling vector of the target training sample is used to update the center vector of the force action class to which the target training sample belongs.

[0053] In a second aspect, the embodiments of the present application also provide an AI and multi-sensor based intelligent recognition device for force action, which comprises:

[0054] The acquisition module is configured to acquire multi-dimensional data corresponding to each array node in the contact surface array in the target massage operation, wherein the array nodes are arranged at equal intervals, and each array node except the boundary nodes has at least a preset number of neighbor nodes.

[0055] The data fusion module is configured to extract features from the multi-dimensional data of each array node to obtain an initial feature vector of each array node, and update the obtained initial feature vectors according to the spatial adjacency relationship of the array nodes to obtain an embedding vector of the target massage operation.

[0056] The force action recognition module is configured to fuse the embedding vector with time sequence information and spatial information to obtain a target modeling vector, and determine a force action recognition result of the target massage operation based on the target modeling vector.

[0057] In an optional embodiment, the multi-dimensional data includes force dimension data, posture dimension data, and temperature dimension data.

[0058] In an optional embodiment, when acquiring the multi-dimensional data corresponding to each array node in the contact surface array in the target massage operation, the acquisition module is further configured to:

[0059] The first initial data of each array node in the target massage operation is collected through the contact surface array arranged on the surface of the first object.

[0060] The second initial data is collected through the hand multi-modal sensor arranged on the hand of the second object.

[0061] The multi-dimensional data corresponding to each array node is determined based on the second initial data and the first initial data of each array node.

[0062] In an optional embodiment, each array node includes a pressure sensor, a force sensor, and a temperature sensor, and the first initial data includes normal force data, tangential force data, and contact surface temperature data.

[0063] The hand multi-modal sensor includes an inertial measurement unit, a depth camera, and a temperature electrode, and the second initial data includes inertial data, hand point cloud data, and back of hand temperature data.

[0064] In an optional embodiment, when the multi-dimensional data corresponding to each array node is determined based on the second initial data and the first initial data of each array node, the obtaining module is further configured to:

[0065] determine a posture time sequence based on the inertial data and the hand point cloud data, and take the posture time sequence as the posture dimension data corresponding to each array node;

[0066] determine a force time sequence and a torque time sequence of each hand joint based on the posture time sequence, and take the normal pressure data, the tangential force data of each array node, and the force time sequence and the torque time sequence of each hand joint as the force dimension data corresponding to each array node;

[0067] take the contact surface temperature data and the back of hand temperature data of each array node as the temperature dimension data corresponding to each array node.

[0068] In an optional embodiment, when the posture time sequence is determined based on the inertial data and the hand point cloud data, the obtaining module is further configured to:

[0069] perform adaptive extended Kalman filtering on the inertial data and the hand point cloud data to obtain the posture time sequence.

[0070] In an optional embodiment, when the force time sequence and the torque time sequence of each hand joint are determined based on the posture time sequence, the obtaining module is further configured to:

[0071] determine an end pose of each finger based on the posture time sequence and a DH parameter set of each hand joint;

[0072] perform inverse dynamics calculation based on the end pose to obtain the force time sequence and the torque time sequence of each hand joint.

[0073] In an optional embodiment, when the initial feature vector of each array node is obtained by performing feature extraction on the multi-dimensional data of each array node, the data fusion module is further configured to:

[0074] perform feature extraction on the multi-dimensional data of each array node in different domains to obtain a plurality of sub-initial feature vectors of each array node;

[0075] standardize and splice the plurality of sub-initial feature vectors of each array node to obtain the initial feature vector of each array node.

[0076] In an optional embodiment, when the plurality of sub-initial feature vectors of each array node are obtained by performing feature extraction on the multi-dimensional data of each array node in different domains, the data fusion module is further configured to:

[0077] perform time domain feature extraction on the force dimension data of each array node to obtain a first sub-initial feature vector of each array node;

[0078] perform frequency domain feature extraction on the posture dimension data of each array node to obtain a second sub-initial feature vector of each array node;

[0079] perform topology domain feature extraction on the temperature dimension data of each array node to obtain a third sub-initial feature vector of each array node.

[0080] In an optional embodiment, when the obtained initial feature vectors are updated according to the spatial adjacency relationship of each array node to obtain an embedding vector of the target massage operation, the data fusion module is further configured to:

[0081] construct a graph based on the spatial adjacency relationship of each array node, wherein the vertices of the graph are the array nodes, and the edge set of the graph is the spatial adjacency relationship between the array nodes;

[0082] perform graph convolution and attention aggregation on the initial feature vectors based on the graph to obtain the embedding vector.

[0083] In an optional embodiment, when the embedding vector is fused with time sequence information and spatial information to obtain a modeling vector of the target massage operation, the force action recognition module is further configured to:

[0084] perform time domain dynamic feature extraction on the embedding vector to obtain a time sequence feature vector, and perform spatial distribution feature extraction on the embedding vector to obtain a spatial feature vector, wherein the time sequence feature vector represents the time dynamic semantics of the target massage operation, and the spatial feature vector represents the spatial distribution semantics of the target massage operation;

[0085] fuse the time sequence feature vector and the spatial feature vector to obtain the target modeling vector.

[0086] In an optional embodiment, when the force action recognition result of the target massage operation is determined based on the target modeling vector, the force action recognition module is further configured to:

[0087] determine the distance between the target modeling vector and a center vector of each force action category, wherein the center vector is determined based on the sample modeling vectors of the corresponding force action category;

[0088] determine the force action recognition result of the target massage operation based on the obtained distances.

[0089] In an optional embodiment, when the force action recognition result of the target massage operation is determined based on the obtained distances, the force action recognition module is further configured to:

[0090] When there is a target distance greater than the distance threshold among the distances, the force action category of the target massage operation is determined as the force action category corresponding to the target distance.

[0091] When there is no target distance greater than the distance threshold among the distances, the force action recognition result of the target massage operation is determined based on distances between the target modeling vector and each sample modeling vector of each force action category.

[0092] In an optional embodiment, the apparatus further includes a training module configured to:

[0093] obtain a training sample set, each training sample in the training sample set including at least a sample embedding vector of a sample massage operation;

[0094] based on the training sample set, iteratively train the force action recognition model to be trained to obtain a target force action recognition model, wherein in one iteration training process, the following operations are performed:

[0095] fuse the sample embedding vector of the selected target training sample with time sequence information and spatial information to obtain a sample modeling vector of the target training sample;

[0096] determine sample distances between the sample modeling vector of the target training sample and center vectors of each force action category;

[0097] based on the obtained sample distances, determine a sample force action recognition result of the target training sample, and perform parameter tuning based on a loss value corresponding to the sample force action recognition result.

[0098] In an optional embodiment, after the sample force action recognition result of the target training sample is determined based on the obtained sample distances, and the parameter tuning is performed based on the loss value corresponding to the sample force action recognition result, the training module is further configured to:

[0099] when the target training sample is a high-confidence sample, update the center vector of the force action category to which the target training sample belongs based on the sample modeling vector of the target training sample.

[0100] In a third aspect, the embodiments of the present application also provide an electronic device, including:

[0101] a processor; and

[0102] a memory storing a program,

[0103] wherein the program includes instructions that, when executed by the processor, cause the processor to perform the AI and multi-sensor based intelligent identification method of the force action as described in the first aspect.

[0104] In a fourth aspect, the embodiments of the present application further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the AI and multi-sensor based intelligent identification method of force action.

[0105] In a fifth aspect, the present application provides a computer program product, which, when invoked by a computer, causes the computer to execute the AI and multi-sensor based intelligent identification method steps of force action as described in the first aspect.

[0106] The present application has the following beneficial effects:

[0107] In the AI and multi-sensor based intelligent identification method of force action provided by the embodiments of the present application, the multi-dimensional data corresponding to each array node arranged at equal intervals is used to comprehensively quantify the target massage operation, so that a high-precision quantization result can be obtained, and the multi-dimensional data of each array node is processed for features, so that the discriminant feature space of different force actions can be maximized, and the embedding vector is fused with time sequence information and spatial information to capture the time and spatial information of the target massage operation. Therefore, the force action can be accurately quantified, the force action can be accurately identified, the accuracy of force action identification is improved, and a replicable and generalizable technical path is provided for traditional physiotherapy, rehabilitation training and intelligent teaching.

[0108] In addition, other features and advantages of the present application will be described in the following description, and some will become apparent from the description, or will be understood by those skilled in the art. The purpose and other advantages of the present application can be achieved and obtained by the structure specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0109] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings described here are used to provide further understanding of the present application, constitute a part of the present application, and do not constitute an undue limitation on the present application. In the drawings:

[0110] Figure 1 An optional system architecture schematic diagram applicable to the embodiments of the present application;

[0111] Figure 2 An implementation flowchart of the AI and multi-sensor based intelligent identification method of force action provided by the embodiments of the present application;

[0112] Figure 3 A first schematic diagram of a contact surface array provided by the embodiments of the present application;

[0113] Figure 4 A second schematic diagram of a contact array provided for an embodiment of the present application;

[0114] Figure 5 A logic schematic diagram of obtaining an embedding vector provided for an embodiment of the present application;

[0115] Figure 6 A logic schematic diagram of determining a force action recognition result provided for an embodiment of the present application;

[0116] Figure 7 A flow schematic diagram of target force action recognition model training provided for an embodiment of the present application;

[0117] Figure 8 A structural schematic diagram of an intelligent force action recognition device based on AI and multiple sensors provided for an embodiment of the present application;

[0118] Figure 9 A structural schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0119] Embodiments of the present application will be described in more detail by referring to the attached drawings. Although certain embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, but rather these embodiments are provided so as to more completely and thoroughly understand the present application. It is understood that the drawings and embodiments of the present application are for exemplary purposes only and are not intended to limit the scope of protection of the present application.

[0120] It should be understood that each of the steps recited in the method embodiments of the present application can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0121] The term “comprising” and variations thereof as used herein are open-ended, that is, “comprising but not limited to.” The term “based on” means “based, at least in part, on.” The term “one embodiment” means “at least one embodiment.” The term “another embodiment” means “at least one additional embodiment.” The term “some embodiments” means “at least some embodiments.” Related terms are defined as follows. It should be noted that the concepts mentioned in the present application are merely used to distinguish different devices, modules or units, and are not intended to limit the functions performed by these devices, modules or units or the order or interdependence of these functions.

[0122] It should be noted that the modification of "one" and "multiple" mentioned in the present application is illustrative but not restrictive, and those skilled in the art should understand that "one or more" should be understood unless otherwise explicitly indicated in the context.

[0123] The names of the messages or information exchanged between the devices in the embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0124] The following explains some terms in the embodiments of the present application to facilitate understanding by those skilled in the art.

[0125] (1) First object: the object to be massaged.

[0126] (2) Second object: the object to which the massage is performed.

[0127] (3) Nine-degree of freedom inertial measurement unit (9-DoF IMU): contains a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer, respectively, for detecting linear acceleration, angular velocity and magnetic field strength.

[0128] (4) Time of flight (TOF) depth camera: an advanced imaging device that uses light pulse sensing technology to measure the distance and depth between objects and cameras, enabling fast and accurate depth information acquisition.

[0129] (5) Adaptive extended Kalman filter (Adaptive EKF): an improved Kalman filter algorithm that can automatically adjust the filter parameters to adapt to changes in system noise and observation noise variance when the variance is unknown or changes over time.

[0130] (6) DH parameter. A standardized method is provided to describe the geometric relationship between joints and links, and the DH parameter model defines the relative position and attitude between two adjacent links through four key parameters: joint angle (θ), link length (a), link offset (d) and link offset angle (α).

[0131] (7) Graph convolutional network (GCN): extracts and learns features of nodes by defining convolution operations on graphs, which can process data with generalized topological graph structures and deeply mine data features and rules.

[0132] (8) Attention Mechanism: an algorithm that simulates human attention allocation, used to improve the model's ability to handle complex tasks, and can dynamically focus attention on the key parts of the input data, so that the network can efficiently handle tasks with large input length or structure changes.

[0133] (9) Objective Function: a function used in machine learning to measure and optimize the distance between the model's estimated value and the true sample label value.

[0134] (10) Embodied Intelligence: refers to the integration of artificial intelligence into physical entities such as robots, giving them the ability to perceive, learn, and dynamically interact with the environment like humans. Embodied intelligence emphasizes that the behavior of an intelligent agent is not only dependent on abstract computing and reasoning abilities, but is deeply rooted in interaction with its environment. Embodied intelligence is often associated with intelligent behavior of robots, animals, and humans, reflecting the ability to make decisions, learn, and reason through perception, action, and adaptation to the environment.

[0135] Based on the above explanations of the terms and related terms, the design idea of the embodiments of the present application is briefly introduced as follows:

[0136] Massage is a way to effectively adjust the physical and mental state, and there are many types of force-acting actions in massage.

[0137] In the prior art, when identifying force-acting actions, the current pressure-time curve data of the operator is usually used to determine the current force-acting action category of the operator.

[0138] However, since the pressure characteristics corresponding to multiple force-acting action categories are similar, the above method may result in incorrect identification results, i.e., low accuracy of force-acting action identification.

[0139] In an alternative implementation, the embodiments of the present application propose an intelligent identification method for force-acting actions based on AI and multiple sensors, which can specifically include: obtaining multi-dimensional data corresponding to each array node in the contact surface array in the target massage operation, wherein the array nodes are arranged at equal intervals, and each array node except the boundary nodes has at least a predetermined number of neighbor nodes; then performing feature extraction on the multi-dimensional data of each array node to obtain an initial feature vector of each array node; and updating the obtained initial feature vectors according to the spatial adjacency relationship of the array nodes to obtain an embedding vector of the target massage operation; finally, fusing the embedding vector with time sequence information and spatial information to obtain a target modeling vector, and determining the force-acting action identification result of the target massage operation based on the target modeling vector.

[0140] In the above manner, the target massage operation is comprehensively quantified based on the respective multi-dimensional data of the array nodes arranged at equal intervals, and a high-precision quantification result can be obtained. Furthermore, the multi-dimensional data of the array nodes are processed for features, so as to maximize the discriminant feature space of different force actions, and the embedding vector is fused with space-time information to capture the time and space information of the target massage operation. Therefore, the force action can be accurately quantified, the force action can be accurately recognized, the accuracy of force action recognition is improved, a replicable and generalizable technical path is provided for traditional physiotherapy, rehabilitation training and intelligent teaching. In addition, data and models are provided for future embodied intelligence and artificial intelligence automatic massage.

[0141] Particularly, the preferred embodiments of the present application are described below in conjunction with the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments and features in the embodiments can be combined with each other without conflict.

[0142] Referring to Figure 1 As shown in the figure, it is an optional system architecture diagram applicable to the embodiments of the present application. The system architecture can include: edge end 101 and cloud end 102. The edge end 101 and the cloud end 102 can interact information through a communication network, wherein the communication network adopts a communication mode including: wireless communication mode and wired communication mode. Exemplarily, the edge end 101 can access the network through Wi-Fi 6E or cellular mobile communication technology (such as 5G) to perform real-time data transmission with the cloud end 102; at the same time, it supports BLE5.3 short distance wireless communication for low power consumption interconnection of local sensor network.

[0143] The number of communication devices involved in the above system architecture is not limited by the embodiments of the present application. For example, the above system architecture can include more edge ends, or can include fewer edge ends, or can include other network devices. For example, Figure 1 As shown in the figure, only the edge end 101 and the cloud end 102 are described as an example, and the functions of the above communication devices and their respective functions are briefly introduced below.

[0144] The edge end 101 is a data acquisition and preprocessing end, configured to acquire multi-dimensional data corresponding to each array node in the contact surface array in a target massage operation, perform feature extraction on the multi-dimensional data of each array node to obtain an initial feature vector of each array node, update the obtained initial feature vectors according to the spatial adjacency relationship of the array nodes to obtain an embedding vector of the target massage operation, fuse time sequence information and spatial information into the embedding vector to obtain a target modeling vector, and determine a force action recognition result of the target massage operation based on the target modeling vector. In addition, the edge end 101 is also configured to acquire a training sample set and send the training sample set to the cloud end 102, wherein each training sample in the training sample set at least includes a sample embedding vector of a sample massage operation.

[0145] Optionally, a pre-trained force action recognition model can be deployed on the edge end 101. In this way, after the edge end 101 obtains the embedding vector of the target massage operation, the embedding vector can be input into the pre-trained force action recognition model to determine the force action recognition result of the target massage operation, thereby realizing offline force action recognition in an offline scenario.

[0146] The cloud end 102 is a model training and data management end, configured to receive the training sample set, perform iterative training on a to-be-trained force action recognition model based on the training sample set to obtain a target force action recognition model, wherein in one iteration training process, the following operations are performed: fusing time and space information into a sample embedding vector of a target training sample to obtain a sample modeling vector of the target training sample; determining a sample distance between the sample modeling vector of the target training sample and a center vector of each force action category; determining a sample force action recognition result of the target training sample based on the obtained sample distances, and adjusting parameters based on a loss value corresponding to the sample force action recognition result. In addition, the latest trained force action recognition model can be sent to the edge end 101, or the latest trained force action recognition model after being lightened can be sent to the edge end 101, and the present application does not limit this.

[0147] In addition, it is worth noting that the trained target force action recognition model can also be deployed on the cloud end 102. In this way, after the edge end 101 obtains the embedding vector of the target massage operation, the edge end 101 sends the embedding vector of the target massage operation to the cloud end 102, and the cloud end 102 inputs the embedding vector into the pre-trained target force action recognition model to determine the force action recognition result of the target massage operation.

[0148] In the embodiments of the present application, the part of the target force application action recognition model relates to artificial intelligence (AI) technology, and is designed based on voice technology, natural language processing technology and machine learning (ML) in artificial intelligence.

[0149] Artificial intelligence is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence.

[0150] Artificial intelligence is also the design principle and implementation method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology mainly includes computer vision technology, natural language processing technology, and machine learning / deep learning and several other directions. With the research and progress of artificial intelligence technology, artificial intelligence is researched and applied in many fields, such as common smart home, intelligent customer service, virtual assistant, smart speaker, intelligent marketing, unmanned vehicle, autonomous driving, robot, intelligent medical treatment and the like. It is believed that with the development of technology, artificial intelligence will be applied in more and more fields and play an increasingly important value.

[0151] Machine learning is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other subjects. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve the performance of the computer. Compared with data mining that finds mutual characteristics between big data, machine learning pays more attention to the design of algorithms, so that the computer can automatically "learn" the law from the data and use the law to predict unknown data.

[0152] Machine learning is the core of artificial intelligence and the fundamental approach to enabling computers to have intelligence, and its application is widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies. Reinforcement learning (RL), also known as relearning, evaluation learning or enhancement learning, is one of the paradigms and methodologies of machine learning, which is used to describe and solve the problem of learning strategy by an agent in the process of interaction with the environment to maximize the reward or achieve a specific goal.

[0153] The AI and multi-sensor based force action intelligent recognition method provided by the exemplary embodiments of the present application is described below in combination with the above system architecture and with reference to the accompanying drawings. It should be noted that the above system architecture is only shown for the purpose of facilitating the understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect.

[0154] Referring to Figure 2 As shown in the figure, it is an implementation flowchart of an AI and multi-sensor based force action intelligent recognition method provided by an embodiment of the present application. The specific implementation flow of the method is as follows:

[0155] S20: Obtain the multi-dimensional data corresponding to each array node in the contact surface array in the target massage operation.

[0156] In the embodiment of the present application, the array nodes are arranged at equal intervals, and each array node (except the boundary nodes) has at least a preset number of neighbor nodes. The multi-dimensional data includes force dimension data, posture dimension data, and temperature dimension data.

[0157] In the embodiment of the present application, when the second object performs the target massage operation on the first object, the multi-dimensional data corresponding to each array node is obtained.

[0158] Optionally, in the embodiment of the present application, a possible embodiment is provided for obtaining the multi-dimensional data corresponding to each array node, which specifically performs the following operations:

[0159] S200: Collect the first initial data of each array node in the target massage operation through the contact surface array deployed on the surface of the first object.

[0160] The contact surface array includes array nodes.

[0161] For example, referring to Figure 3 As shown in the figure, it is the first schematic diagram of the contact surface array in the embodiment of the present application. The contact surface array is a hexagonal array, which includes 27 array nodes arranged in an equal-interval "staggered row" manner (3 rows x 9 columns), so that each array node (except the boundary) has at least six neighbor nodes (up / down / left / right / diagonal), thereby achieving optimal spatial coverage and uniform reconstruction accuracy.

[0162] The interval can be 4.2 mm, which is not limited in the embodiment of the present application.

[0163] For example, referring to Figure 4As shown, it is a second schematic diagram of the contact surface array in the embodiment of the present application, the contact surface array is a hexagonal array, including 27 equidistant array nodes, so that each array node (except the boundary) has at least six neighbor nodes, wherein there is one array node in the center, 6 array nodes in the first adjacent layer, 12 array nodes in the second adjacent layer, and 8 array nodes attached on the periphery, so as to realize optimal spatial coverage and uniform reconstruction accuracy.

[0164] In the embodiment of the present application, each array node is a sensing unit, and each array node includes a pressure sensor, a force sensor and a temperature sensor.

[0165] The pressure sensor can be a flexible strain pressure sensor (based on MXene / PDMS micro-dome structure, real-time measurement of normal pressure and surface micro-strain), the force sensor can be a 6-dimensional force sensor (measurement of force and torque in X / Y / Z three directions), and the temperature sensor is a CNT-PDMS temperature sensor (32x32 pixel array). The normal pressure data includes normal pressure and bending strain, the tangential force data includes shear force along the X direction and shear force along the Y direction, and the contact surface temperature data includes temperature readings given by the temperature sensor, which is not limited in the embodiment of the present application. In this way, each array node simultaneously collects five physical quantities, and the contact surface array including 27 array nodes can realize 5-dimensional 27-channel synchronous acquisition.

[0166] S201: Collecting second initial data through hand multi-modal sensing deployed on the hand of the second object.

[0167] In the embodiment of the present application, the hand multi-modal sensing includes an inertial measurement unit, a depth camera and a temperature electrode, and the second initial data includes inertial data, hand point cloud data and back of hand temperature data.

[0168] The inertial measurement unit can be a wearable 9-DoF IMU, the depth camera can be a ToF depth camera, and the temperature electrode can be a hand temperature electrode (4x4 array, measuring hand back surface temperature mode).

[0169] In the embodiments of the present application, the hand temperature electrode is arranged on the back of the hand of the second object to detect the skin temperature of the back of the hand of the second object. The hand temperature electrode is different from the temperature sensor integrated with the pressure sensor and the force sensor set in each array node of the contact surface array. The temperature sensor is used to monitor the palm surface hot compress and the friction heat generation intensity. The hand temperature electrode does not interfere with the contact surface array and provides a "static reference" to make a difference with the palm temperature to separate the "external heat source" and the "internal temperature rise". At the same time, the hand temperature electrode cannot be integrated in each array node of the contact surface array because each array node needs to collect normal pressure data and tangential force data at the same time. Superimposing the hand temperature electrode will affect the softness and pressure sensitivity. In addition, if the hand temperature electrode is placed in the array node, the reading of the hand temperature electrode will be greatly disturbed by the hot compress, losing the meaning of "reference body temperature". In addition, the back of the hand temperature data collected by the hand temperature electrode can be used to estimate fatigue and direct current, and safety protection thresholds can be set during strong heat application. The contact surface temperature data collected by the temperature sensor integrated with the pressure sensor and the force sensor set in each array node can be used to adjust the heat therapy power in real time.

[0170] In addition, it is worth noting that in the embodiments of the present application, the sensor will be calibrated before collecting data to improve the accuracy of the collected data. Specifically, the 6-dimensional force sensor is matrix calibrated and temperature drift compensated, the CNT-PDMS temperature sensor is three-point water bath calibrated and online drift compensated, and the 9-DoF IMU is calibrated for internal parameters, external parameters and clock.

[0171] S202: Determine the multi-dimensional data corresponding to each array node based on the second initial data and the first initial data of each array node.

[0172] Optionally, the embodiments of the present application provide a possible embodiment for determining the multi-dimensional data corresponding to each array node based on the second initial data and the first initial data of each array node, which specifically performs the following operations:

[0173] S2020: Determine the attitude time sequence based on the inertial data and the hand point cloud data, and take the attitude time sequence as the attitude dimensional data corresponding to each array node.

[0174] Illustratively, the attitude dimensional data corresponding to one array node includes: the attitude time sequence.

[0175] In the embodiments of the present application, the inertial data and the hand point cloud data are adaptively extended Kalman filtered to obtain the attitude time sequence.

[0176] The attitude time sequence includes a quaternion time sequence and a position time sequence.

[0177] In this way, the problem of insufficient estimation accuracy in a traditional method under a fast dynamic or occlusion condition is avoided, and the posture data of the hand in the massage operation can be accurately and timely estimated.

[0178] Specifically, in the embodiment of the application, a state vector is constructed based on inertial data, and then the state vector is updated based on a Kalman gain and hand point cloud data to obtain a quaternion time sequence and a position time sequence.

[0179] The state vector includes a position, a velocity, a quaternion, an accelerometer bias, and a gyroscope bias, and the quaternion time sequence includes a posteriori quaternion at each time point.

[0180] In the embodiment of the application, one state vector update includes predicting a prior state vector at a current time based on a posteriori state vector at a previous time and a state local increment, and then determining a posteriori state vector at the current time based on a Kalman gain at the current time, an actual observation value, a predicted observation value, and the prior state vector at the current time.

[0181] The prior state vector at the current time is an initial estimation of the state vector at the current time, and the posteriori state vector at the current time is an optimized state vector obtained after correcting the error of the prior state vector at the current time, and is closer to the real state.

[0182] For example, the state vector described above can be specifically represented as follows: x = [p, v, q, b a , b g ] T wherein p is a position, v is a velocity, q is a quaternion, b a is an accelerometer bias, and b g is a gyroscope bias.

[0183] The calculation formula of the prior state vector at the current time described above can be specifically represented as follows:

[0184]

[0185] wherein x k-1 is the posteriori state vector at the k-1 time, u k-1 is the IMU measurement input at the k-1 time; u k-1 = [a k-1 , ω k-1 ] T , a k-1 is the “pure inertia” acceleration after removing gravity at the k-1 time, indicating the linear acceleration of the wrist reference point in the X, Y, and Z directions, and ω k-1is the rotation rate of the wrist reference point around the X, Y, Z axes at the k-1 moment; f(·) is a state transition model, which describes the dynamic change rule of the state over time, and short-time prediction is realized based on the IMU pre-integration principle, and a quaternion increment Δq = ∑ i exp((ω i -b g )Δt) is calculated through IMU pre-integration, a velocity increment Δv = ∫(a i -b a )dt, and a position increment Δp = ∫v i dt, ω i is the instantaneous angular velocity reading of the gyroscope, a i is the instantaneous linear acceleration of the accelerometer, w i is not measured by the IMU, is the intermediate estimation of the system state, and the previous key frame is used to update the estimation value of the wrist reference point velocity; w k-1 is a process noise coupling matrix, which describes the mapping matrix of random disturbances such as friction, mechanical clearance, and IMU quantization error projected to state quantities acting on each state dimension, and is subject to a zero-mean Gaussian distribution Q k-1 is a covariance matrix of the process noise, and the larger the value is, the lower the model's "trust degree" for the noise source is, which is used to adjust the uncertainty of EKF prediction.

[0186] The actual prediction value can be specifically represented as follows: where h(·) is an observation model function, which is a joint mapping function from the state to the observation, and is used to convert the predicted hand pose and position into observable joint coordinates; v k is an observation noise; R k is an observation noise covariance matrix. z k =[j1,…,j M ], which contains M joints, and M = 20, which is not limited in the embodiments of the present application.

[0187] The Kalman gain at the current moment can be specifically represented as follows:

[0188] K k =P k|k-1 H T (HP k|k-1 H T +R k ) -1

[0189] where P k|k-1 is the prior covariance matrix at the k moment, which is the uncertainty of the state vector X-1, and P k|k-1 =FP k-1 F T+ Q, F is the state transition Jacobian, mapping the small perturbation of the old state to the sensitivity matrix of the predicted state perturbation, Q is the process noise covariance matrix, describing the uncertainty of the one-step prediction of the model error plus the inherent noise injection of the sensor, P k-1 is the posterior covariance matrix at the k-1 time, the larger indicates that the prediction is less accurate, and the filter will rely more on the observation later, and the smaller indicates that the model is more trusted; H is an observation matrix, representing the partial derivative of the state vector to the observation vector, describing how the state variable is mapped to the observation space; P k is the observation noise covariance matrix, quantifying the size and mutual correlation of the sensor measurement error, is the basic observation noise variance, representing the inherent noise of the ToF depth camera under ideal conditions, λ is the attenuation factor, N p is the number of matched depth points, reflecting the number of detectable hand joint points in the current ToF frame, used for dynamically adjusting the observation confidence, I is the unit matrix.

[0190] The posterior state vector at the current time can be specifically represented as follows:

[0191] x k =x k|k-1 +K k (z k -h(x k|k-1 ))

[0192] Wherein, x k|k-1 is the prior state vector at time k, K k is the Kalman gain at time k, z k is the actual observation value at time k, and h(x k|k-1 ) is the predicted observation value at time k.

[0193] Further, after obtaining the posterior state vector at the current time, the posterior covariance matrix at the current time is determined, and the covariance matrix at the current time can be specifically represented as follows: P k =(I-K k H)P k|k-1 , I is the identity mapping, and the "residual projection" is constructed in the covariance update formula.

[0194] In the embodiments of the application, the quaternions in the state vectors at each time point are taken as a quaternion time sequence, and the positions in the state vectors at each time point are taken as a position time sequence, so as to obtain a pose time sequence.

[0195] Optionally, multiple depth cameras can be deployed, each with a corresponding quaternion time sequence, and a final quaternion time sequence can be obtained through fusion output of each quaternion time sequence. Similarly, a position time sequence can also be obtained through fusion of position time sequences corresponding to multiple depth cameras. In this way, single-view occlusion is compensated through multi-view, and the accuracy of the posture dimension data is improved.

[0196] The final quaternion time sequence described above can be specifically represented as follows:

[0197]

[0198] wherein L is the total number of depth cameras, q i is the quaternion time sequence corresponding to the i-th depth camera, w i is the weight of the i-th depth camera, is the covariance of the i-th depth camera posture estimation.

[0199] In addition, it is worth noting that in the embodiments of the present application, when |a k -g| < ε a and |ω k | < ε ω satisfy the preset number of frames continuously, zeroing is triggered and the bias of the accelerometer and the gyroscope is re-estimated. Wherein a k is the linear acceleration (including gravity) in the body coordinate system at the current time k, used to determine whether the linear motion is sufficient "static", g is the gravity vector, ω k is the IMU gyroscope reading (debiased / filtered) angular velocity vector of the body around the X / Y / Z axis at the current time, used to determine whether the rotation is almost stopped, ε a is the acceleration threshold, and ε ω is the angular velocity threshold.

[0200] In addition, it is worth noting that in the embodiments of the present application, the quaternion sequence can also be fitted with a cubic B-spline to alleviate high-frequency jumps.

[0201] S2021: Based on the posture time sequence, determine the force time sequence and the torque time sequence of each hand joint, and take the normal pressure data and the tangential force data of each array node and the force time sequence and the torque time sequence of each hand joint as the force dimension data corresponding to each array node.

[0202] Illustratively, the force dimension data corresponding to one array node includes: the force time sequence and the torque time sequence of each hand joint, and the normal pressure data and the tangential force data of the array node.

[0203] In addition, it is worth noting that in the embodiments of the present application, 4 joints of each finger are constructed, and a total of 20 joints of five fingers are constructed.

[0204] In the embodiments of the present application, based on the posture time sequence and the DH parameter set of each hand joint, the end posture of each finger is determined, and then based on the end posture, inverse dynamics calculation is performed to obtain the force time sequence and the torque time sequence of each hand joint.

[0205] Specifically, the DH parameter set of each hand joint is defined, then the transformation matrix is constructed to obtain the total transformation of the finger end including the end posture, then the mapping of the joint angular velocity and the end velocity is defined through the Jacobian matrix, and finally based on the end loading, the recursive inverse (k=4 to 1) is obtained to obtain the force time sequence and the torque time sequence of each joint of each finger.

[0206] For example, the DH parameter set of the jth joint of the ith finger can be represented as: {a i,j ,α i,j ,d i,j ,θ i,j}.

[0207] Wherein, a i,j is the length of the connecting rod, α i,j is the connecting rod offset angle, d i,j is the connecting rod offset, and θ i,j is the joint angle.

[0208] The transformation matrix can be represented as:

[0209]

[0210] The end velocity mapping can be represented as:

[0211] Wherein, the linear velocity column J v,i,k =z k-1 ×(p e,i -p k-1 ), the angular velocity column J ω,i,k =z k-1 , p e,i is the posture of the finger tip relative to the base, z k-1 is the joint rotation axis of the kth joint, p k-1 is the joint origin of the kth joint; v e,i is the instantaneous translational velocity of the end coordinate frame origin in 3D space, which describes how many millimeters per second the finger tip moves in X, Y, Z directions; ω e,i is the instantaneous rotation rate of the end coordinate frame around its X, Y, Z axes, which describes the instantaneous rotation of the finger tip, is the joint angular velocity.

[0212] The calculation formula of the total transformation of the finger end can be represented as:

[0213]

[0214] The force of the kth joint of the ith finger can be expressed as:

[0215]

[0216] where m k,i is the mass of the kth joint of the ith finger, a k,i is the link length of the kth joint of the ith finger, τ k,i is the external force / torque of the kth joint of the ith finger, Rk+1 is the attitude matrix of the link k+1 coordinate system relative to the inertial or base coordinate system, which converts any vector (force, torque, velocity, acceleration) expressed in the link local coordinates to the inertial / world system, or vice versa (take the transpose).

[0217] The torque of the kth joint of the ith finger can be expressed as:

[0218]

[0219] where I k,i is the inertia of the kth joint of the ith finger, a k,i is the link angle of the kth joint of the ith finger, ω k,i is the angular velocity of the kth joint of the ith finger, p k,i is the position vector of the kth joint of the ith finger, and p k+1,i is the position vector of the k+1th joint of the ith finger.

[0220] Further, τ k,i is taken modulo to obtain the torque values of each joint of each finger.

[0221] In this way, the fine motion and mechanical characteristics of the fingers during the massage process are accurately quantified, and the recognition accuracy of the force application action is improved.

[0222] S2022: The contact surface temperature data and the back of the hand temperature data of each array node are taken as the temperature dimension data corresponding to each array node respectively.

[0223] For example, the temperature dimension data corresponding to one array node includes the back of the hand temperature data and the contact surface temperature data of the array node.

[0224] In this way, high-dimensional discriminative features that can reflect dynamic changes of force action and depict spectral characteristics and spatial distribution are extracted from multi-source sensing signals, thereby providing input rich in force action semantics for subsequent force action recognition. The force dimension data, the posture dimension data, and the temperature dimension data each reflect different dimensions of the force action. The force dimension data reflects the strength of the contact force, the posture dimension data describes the hand movement trajectory, and the temperature dimension data reveals the touch temperature change of the force action. The joint of the features of time evolution (time domain), spectral components (frequency domain), and spatial topology (array arrangement) can completely reproduce the three elements of "when to force", "where to force", and "force change frequency" of the force action.

[0225] S21: performing feature extraction on the multi-dimensional data of each array node to obtain an initial feature vector of each array node, and updating the obtained initial feature vectors according to the spatial adjacency relationship of the array nodes to obtain an embedding vector of the target massage operation.

[0226] Optionally, in the embodiment of the present application, a possible embodiment is provided for performing feature extraction on the multi-dimensional data of each array node to obtain an initial feature vector of each array node, and the following operations are performed:

[0227] S210: performing feature extraction in different domains on the multi-dimensional data of each array node to obtain a plurality of sub-initial feature vectors of each array node.

[0228] The different domains include a time domain, a frequency domain, and a topology domain.

[0229] In the embodiment of the present application, time domain feature extraction is performed on the force dimension data of each array node to obtain a first sub-initial feature vector of each array node.

[0230] Specifically, the following operations are performed on each array node respectively: wavelet packet decomposition is performed on the force dimension data of an array node to obtain a quantized energy spectrum as the first sub-initial feature vector of the array node.

[0231] For example, Daubechies-5 wavelet packet decomposition is performed in each 128 ms sliding window to obtain a quantized energy spectrum, which can be expressed as:

[0232]

[0233] wherein W p,l is the lth level sub-band coefficient, l = 1, 2, …, L p , L p is the number of decomposition layers, and the dimension of the first sub-initial feature vector is L p .

[0234] In this embodiment of the application, frequency domain features are extracted from the attitude dimension data of each array node to obtain the second sub-initial feature vector of each array node.

[0235] Specifically, the following operations are performed on each array node: autoregressive modeling is performed on the attitude dimension data of an array node, and the autoregressive coefficients are extracted as the second sub-initial feature vector of that array node.

[0236] For example, in this embodiment of the application, the attitude dimension data (e.g., Euler angles θ(t), obtained through quaternion transformation) is fitted with an AR(p) model to obtain the coefficients {φ}. k}

[0237] in, With φ=[φ1,…,φ p ] is the second sub-initial feature vector, and the dimension of the second sub-initial feature vector is p.

[0238] In this embodiment of the application, topological domain features are extracted from the temperature dimension data of each array node to obtain the third sub-initial feature vector of each array node.

[0239] Specifically, the following operations are performed on each array node: the temperature dimension data of an array node is used to perform gradient calculation to obtain the temperature gradient spectrum, which is used as the third sub-initial feature vector of that array node.

[0240] For example, in this embodiment of the application, the gradient tensor is calculated for the temperature dimension data T(i,j), and the temperature gradient spectrum can be expressed as:

[0241]

[0242] S211: Standardize and concatenate the multiple sub-initial feature vectors of each array node to obtain the initial feature vector of each array node.

[0243] In this embodiment of the application, the mean and variance of the multiple sub-initial feature vectors of each array node are normalized, and then the normalized vectors are concatenated to obtain the initial feature vectors of each array node.

[0244] For example, the initial feature vector of an array node can be represented as: This is the first sub-initial eigenvector after mean and variance normalization. This is the second sub-initial eigenvector after mean and variance normalization. This is the third initial eigenvector after mean and variance normalization.

[0245] In this way, the first sub-initial feature vector represents time-varying force in different scales, the second sub-initial feature vector captures gesture rhythm and oscillation frequency, and the third sub-initial feature vector reveals spatial texture of heat distribution, so that the accuracy of force application action recognition can be improved.

[0246] Optionally, in the embodiments of the present application, a possible embodiment is provided for updating the obtained initial feature vectors according to the spatial adjacency relationship of the array nodes to obtain the embedding vector of the target massage operation, and the following operations are specifically performed:

[0247] S212: A graph is constructed based on the spatial adjacency relationship of the array nodes.

[0248] The vertices of the graph are the array nodes, and the edge set of the graph is the spatial adjacency relationship between the array nodes.

[0249] In the embodiments of the present application, a graph G=(V, E) is constructed based on the spatial adjacency relationship of the array nodes.

[0250] S213: The initial feature vectors are graph-convolved and attention-aggregated based on the graph to obtain the embedding vector.

[0251] In the embodiments of the present application, the initial feature vectors are taken as the attributes of the graph nodes, L-layer graph convolution is performed, an attention mechanism is introduced, the edge weight is calculated, and the embedding vector is obtained after weighted aggregation, full connection projection and activation.

[0252] Exemplarily, the graph convolution of the (l+1)th layer can be represented as:

[0253]

[0254] Wherein, A is an adjacency matrix generated from the spatial adjacency relationship of the array nodes: if the nodes v i and v j are adjacent, then A {ij} =1, otherwise A {ij} =0; I is an adjacency matrix, and the unit matrix adds a self-loop to each node to retain its own features; D is a degree matrix, i.e., the connection degree (including the self-loop) of the node i, which is used for normalizing the degree difference; H (l) is the input feature matrix of the lth layer, and W (l) is a learnable weight.

[0255] Exemplarily, the attention can be represented as:

[0256]

[0257] Exemplarily, the embedding vector can be represented as:

[0258]

[0259] wherein, H j (L) is the node representation of the Lth layer output of the graph convolution, that is, the high-order feature vector of node j after the Lth layer (the neighborhood information has been fused), when the graph is a 27-array node, there are 27 such vectors; is the neighborhood index, in the attention mechanism, the number of neighbor nodes aggregated for node i, if six-direction adjacency is adopted, then is at most 6; W o is an output projection weight matrix, which linearly maps the spliced whole graph feature to an embedding space.

[0260] In addition, it is worth noting that in the embodiments of the present application, the embedding vector is 512-dimensional.

[0261] Referring to Figure 5 , a logic diagram for obtaining an embedding vector provided in the embodiments of the present application is shown.

[0262] S22: Fusing the embedding vector with the timing information and the spatial information to obtain a target modeling vector, and determining the force action recognition result of the target massage operation based on the target modeling vector.

[0263] Optionally, in the embodiments of the present application, a possible embodiment is provided for fusing the embedding vector with the timing information and the spatial information to obtain a target modeling vector, and the following operations are performed:

[0264] S220: performing time-domain dynamic feature extraction on the embedding vector to obtain a timing feature vector, and performing spatial distribution feature extraction on the embedding vector to obtain a spatial feature vector.

[0265] The timing feature vector represents the time dynamic semantics of the target massage operation, and the spatial feature vector represents the spatial distribution semantics of the target massage operation.

[0266] In the embodiments of the present application, the embedding vector is added with the timing and spatial position encodings respectively, and is input into two parallel Transformer towers (time domain tower and space domain tower) to obtain the timing feature vector and the spatial feature vector.

[0267] For example, the input of the time domain tower is E t =f+P t , and the input of the space domain tower is E s =f+P s , wherein f is the embedding vector, P is the timing encoding, and P s is the spatial position encoding. Each tower contains L layers of the same structure, and the calculation of the lth layer is: H (l+1) =LN(FFN(MHA(H (l) ))+H (l)), where MHA is multi-head self-attention, FFN is a two-layer feedforward network, and LN is layer normalization.

[0268] S221: Fuse the time sequence feature vector and the space feature vector to obtain a target modeling vector.

[0269] In the embodiment of the application, after obtaining the time sequence feature vector and the space feature vector, the time sequence feature vector and the space feature vector are fused through a linear gate to obtain a target modeling vector.

[0270] For example, the target modeling vector can be represented as: z = σ(W t z t +W s z s ), where z t is the time sequence feature vector, z s is the space feature vector, σ is an activation function (such as ReLU), W t and W s are weights.

[0271] In addition, it is worth noting that in the embodiment of the application, the target modeling vector is 256-dimensional.

[0272] In this way, through space-time fusion, the target modeling vector can be obtained, the accuracy of force action recognition can be improved, and the target massage operation can also be accurately quantified.

[0273] Optionally, in the embodiment of the application, a possible embodiment is provided for determining the force action recognition result of the target massage operation based on the target modeling vector, and the following operations are performed:

[0274] S222: Determine the distance between the target modeling vector and the center vector of each force action category.

[0275] Each center vector of each force action category is determined based on the sample modeling vectors of each force action category, respectively.

[0276] In the embodiment of the application, the prototype library pre-stores the center vectors of each force action category, and the center vector of a force action category in the prototype library is obtained by clustering or calculating the center of mass of a large number of sample modeling vectors labeled with the force action category in the offline stage. The number of force action categories is 32 categories, a unified objective label is formulated, and large-scale intelligent training and feedback are supported.

[0277] For example, the formula for calculating the distance between the target modeling vector and the center vector of each force action category is: z is the target modeling vector, and p j is the center vector of the jth force action category.

[0278] S223: Determine the force action recognition result of the target massage operation based on the obtained distances.

[0279] In the embodiments of the present application, based on the obtained distances, the force action recognition result of the target massage operation is determined, including the following two cases:

[0280] Case 1: When there is a target distance greater than the distance threshold in the distances, the force action category of the target massage operation is determined as the force action category corresponding to the target distance.

[0281] In the embodiments of the present application, the probabilities corresponding to each force action category can be obtained by converting the distances into probability distributions, and when the maximum probability in the probabilities is greater than the probability threshold, the force action category of the target massage operation is determined as the force action category corresponding to the maximum probability.

[0282] The probability threshold can be 0.6, which is not limited in the embodiments of the present application.

[0283] For example, the probability of the jth force action category can be represented as:

[0284]

[0285] Case 2: When there is no target distance greater than the distance threshold in the distances, the force action recognition result of the massage operation is determined based on the distances between the target modeling vector and each sample modeling vector of each force action category.

[0286] In the embodiments of the present application, when the maximum probability in the probabilities is not greater than the probability threshold, the force action recognition result of the target massage operation is determined based on the distances between the target modeling vector and each sample modeling vector of each force action category.

[0287] Specifically, the first k sample modeling vectors closest to the target modeling vector are determined among the sample modeling vectors, and then the proportion of the target category with the highest frequency among the k sample modeling vectors, i.e. the consistency rate, is determined. If the consistency rate is greater than the consistency rate threshold, the force action category of the target massage operation is determined as the target category.

[0288] The consistency rate threshold can be 0.8, which is not limited in the embodiments of the present application.

[0289] In this way, the k-NN backup is performed for low confidence, and the robustness of classification is improved to ensure high reliability.

[0290] In summary, the embedding vector obtained after multi-modal fusion is mapped to a standard 32-class force action, enabling the system to automatically identify and label the force action category of the target massage operation, quickly locate the most matching category in the force action model library through prototype measurement, and enable k-NN backup when the confidence is insufficient, ensuring real-time and robustness, supporting continuous updating of the prototype library and model parameters in the cloud, and providing data support for personalized application requirements.

[0291] Further, in the embodiments of the present application, after determining the force action recognition result of the target massage operation, the control sequence of the target massage operation can be generated based on the standard trajectory template of the target force action category of the target massage operation, and the control sequence is used to drive the execution mechanism to execute the target massage operation, and / or the standard trajectory template of the target force action category of the target massage operation is used to determine whether the target massage operation is qualified.

[0292] Among them, the execution mechanism is a massage robot or other massage equipment.

[0293] In the embodiments of the present application, the standard trajectory template can be represented as:

[0294] Among them, u i is the control sequence to be optimized, g i is the target trajectory point, w i is the tracking weight, Δu is the control increment, and λ is the regularization coefficient.

[0295] Referring to Figure 6 , it is a logic diagram for determining a force action recognition result provided in the embodiments of the present application.

[0296] Next, the process of training the target force action recognition model is briefly introduced. Referring to Figure 7 , it is a flowchart of training the target force action recognition model.

[0297] S70: Obtain a training sample set, and each training sample in the training sample set at least includes a sample embedding vector of a sample massage operation.

[0298] In the embodiments of the present application, the sample embedding vector of the sample massage operation is obtained in the same way as the above-mentioned embedding vector acquisition method, and the training sample set is constructed.

[0299] In addition, it is worth noting that in the embodiments of the present application, the training sample set can be obtained by the edge side and then sent to the cloud side.

[0300] S71: Based on the training sample set, iteratively train the force action recognition model to be trained to obtain a target force action recognition model, wherein in one iteration training process, the following operations are performed:

[0301] The sample embedding vector of the selected target training sample is fused with the time sequence information and the spatial information to obtain a sample modeling vector of the target training sample; a sample distance between the sample modeling vector of the target training sample and the center vector of each force action category is determined; based on the obtained sample distance, a sample force action recognition result of the target training sample is determined, and parameter tuning is performed based on a loss value corresponding to the sample force action recognition result.

[0302] In the embodiments of the present application, the sample modeling vector of the target training sample is obtained by using the above-mentioned spatio-temporal information fusion manner, the sample distance between the sample modeling vector of the target training sample and the center vector of each force action category is determined, when there is a target sample distance greater than a sample distance threshold value in each sample distance, the sample force action category of the sample massage operation is determined as the force action category corresponding to the target sample distance, and when there is no target sample distance greater than the distance threshold value in each sample distance, the sample force action recognition result of the sample massage operation is determined based on the sample distance between the sample modeling vector and each sample modeling vector of each force action category.

[0303] Specifically, when the sample force action recognition result of the sample massage operation is determined based on the sample distance between the sample modeling vector and each sample modeling vector of each force action category, the first k sample modeling vectors with the closest sample distance to the sample modeling vector of the target training sample in each sample modeling vector are determined, and then the proportion of the target category with the highest occurrence frequency in the k sample modeling vectors, i.e. the sample consistency rate, is determined. If the sample consistency rate is greater than a sample consistency rate threshold value, the force action category of the sample massage operation is determined as the target category, otherwise, the force action category of the sample massage operation is determined by manual labeling, and the training sample is re-added to the training sample set for training.

[0304] Specifically, in the embodiments of the present application, when parameter tuning is performed based on the loss value corresponding to the sample force action recognition result, the objective function includes but is not limited to: cross-entropy loss, center loss and regularization term loss.

[0305] Further, in the embodiments of the present application, after parameter tuning is performed based on the loss value corresponding to the sample force action recognition result, the center vector of the force action category to which the target training sample belongs can be updated based on the sample modeling vector of the target training sample when the target training sample is a high-confidence sample.

[0306] The sample distance between the sample modeling vector of the target training sample and the center vector of each force action category is determined, and when there is a target sample distance greater than a sample distance threshold in each sample distance, the target training sample is a high-confidence sample. At this time, the center vector of the force action category to which the target training sample belongs is updated using momentum update.

[0307] For example, the updated center vector of the force action category to which the target training sample belongs can be represented as: c j ← mc j +(1-m)z, wherein c j is the center vector of the force action category to which the target training sample belongs, m is a momentum coefficient, which determines how much weight the old center vector has, and z is the sample modeling vector of the target training sample.

[0308] Further, based on the same technical concept, the embodiment of the present application provides an AI and multi-sensor based intelligent recognition device for force action, which is used to implement the above method flow of the embodiment of the present application. For example, as shown in Figure 8 The AI and multi-sensor based intelligent recognition device 800 for force action can include an acquisition module 801, a data fusion module 802, a force action recognition module 803, and a training module 804, wherein:

[0309] The acquisition module 801 is configured to acquire multi-dimensional data corresponding to each array node in the contact surface array in the target massage operation, wherein the array nodes are arranged at equal intervals, and each array node except the boundary nodes has at least a preset number of neighbor nodes.

[0310] The data fusion module 802 is configured to perform feature extraction on the multi-dimensional data of each array node to obtain an initial feature vector of each array node, and update each obtained initial feature vector according to the spatial adjacency relationship of the array nodes to obtain an embedding vector of the target massage operation.

[0311] The force action recognition module 803 is configured to fuse the embedding vector with time sequence information and spatial information to obtain a target modeling vector, and determine a force action recognition result of the target massage operation based on the target modeling vector.

[0312] In an optional embodiment, the multi-dimensional data includes force dimension data, posture dimension data, and temperature dimension data.

[0313] In an optional embodiment, when acquiring the multi-dimensional data corresponding to each array node in the contact surface array in the target massage operation, the acquisition module 801 is further configured to:

[0314] acquire first initial data of each array node in the target massage operation through the contact surface array deployed on the first object surface;

[0315] acquire second initial data through the hand multi-modal sensor deployed on the second object hand;

[0316] determine the multi-dimensional data corresponding to each array node based on the second initial data and the first initial data of each array node.

[0317] In an optional embodiment, each array node comprises a pressure sensor, a force sensor and a temperature sensor, and the first initial data comprises normal pressure data, tangential force data and contact surface temperature data.

[0318] The hand multi-modal sensor comprises an inertial measurement unit, a depth camera and a temperature electrode, and the second initial data comprises inertial data, hand point cloud data and back of hand temperature data.

[0319] In an optional embodiment, when determining the multi-dimensional data corresponding to each array node based on the second initial data and the first initial data of each array node, the acquisition module 801 is further configured to:

[0320] determine a posture time sequence based on the inertial data and the hand point cloud data, and take the posture time sequence as posture dimensional data corresponding to each array node;

[0321] determine force time sequence and torque time sequence of each hand joint based on the posture time sequence, and take the normal pressure data, the tangential force data of each array node and the force time sequence and the torque time sequence of each hand joint as force dimensional data corresponding to each array node;

[0322] take the contact surface temperature data of each array node and the back of hand temperature data as temperature dimensional data corresponding to each array node.

[0323] In an optional embodiment, when determining the posture time sequence based on the inertial data and the hand point cloud data, the acquisition module 801 is further configured to:

[0324] perform adaptive extended Kalman filtering on the inertial data and the hand point cloud data to obtain the posture time sequence.

[0325] In an optional embodiment, when determining the force time sequence and the torque time sequence of each hand joint based on the posture time sequence, the acquisition module 801 is further configured to:

[0326] determine end poses of each finger based on the posture time sequence and a DH parameter set of each hand joint;

[0327] Based on the end pose, inverse dynamics calculation is performed to obtain the force time sequence and the torque time sequence of each hand joint.

[0328] In an optional embodiment, when the feature extraction is performed on the multi-dimensional data of each array node to obtain the initial feature vector of each array node, the data fusion module 802 is further configured to:

[0329] The multi-dimensional data of each array node is extracted in different domains to obtain a plurality of initial feature vectors of each array node.

[0330] The plurality of initial feature vectors of each array node are standardized and spliced to obtain the initial feature vector of each array node.

[0331] In an optional embodiment, when the feature extraction is performed on the multi-dimensional data of each array node to obtain the initial feature vector of each array node, the data fusion module 802 is further configured to:

[0332] The force dimension data of each array node is extracted in the time domain to obtain a first initial feature vector of each array node.

[0333] The attitude dimension data of each array node is extracted in the frequency domain to obtain a second initial feature vector of each array node.

[0334] The temperature dimension data of each array node is extracted in the topological domain to obtain a third initial feature vector of each array node.

[0335] In an optional embodiment, when the initial feature vectors obtained are updated according to the spatial adjacency relationship of each array node to obtain the embedding vector of the target massage operation, the data fusion module 802 is further configured to:

[0336] Based on the spatial adjacency relationship of each array node, a graph is constructed, wherein the vertices of the graph are the array nodes, and the edge set of the graph is the spatial adjacency relationship between the array nodes.

[0337] Based on the graph, the initial feature vectors are graph-convolved and attention-aggregated to obtain the embedding vector.

[0338] In an optional embodiment, when the embedding vector is fused with the time sequence information and the spatial information to obtain the modeling vector of the target massage operation, the force action recognition module 802 is further configured to:

[0339] extracting time-domain dynamic features of the embedding vector to obtain a time sequence feature vector, and extracting spatial distribution features of the embedding vector to obtain a spatial feature vector, wherein the time sequence feature vector represents time dynamic semantics of the target massage operation, and the spatial feature vector represents spatial distribution semantics of the target massage operation;

[0340] fusing the time sequence feature vector and the spatial feature vector to obtain a target modeling vector.

[0341] In an optional embodiment, when determining the force action recognition result of the target massage operation based on the target modeling vector, the force action recognition module 803 is further configured to:

[0342] determine distances between the target modeling vector and respective center vectors of respective force action categories, wherein the center vectors are determined based on respective sample modeling vectors of the respective force action categories;

[0343] determine the force action recognition result of the target massage operation based on the obtained distances.

[0344] In an optional embodiment, when determining the force action recognition result of the target massage operation based on the obtained distances, the force action recognition module 803 is further configured to:

[0345] when there is a target distance greater than a distance threshold value in the distances, determining that the force action category of the target massage operation is the force action category corresponding to the target distance;

[0346] when there is no target distance greater than the distance threshold value in the distances, determining the force action recognition result of the target massage operation based on distances between the target modeling vector and respective sample modeling vectors of respective force action categories.

[0347] In an optional embodiment, the apparatus further comprises a training module 804, which is configured to:

[0348] obtain a training sample set, each training sample in the training sample set comprising at least a sample embedding vector of a sample massage operation;

[0349] based on the training sample set, iteratively train a force action recognition model to be trained to obtain a target force action recognition model, wherein in one iteration training process, the following operations are performed:

[0350] fuse the time sequence information and the spatial information of the selected sample embedding vector of the target training sample to obtain a sample modeling vector of the target training sample;

[0351] determine sample distances between the sample modeling vector of the target training sample and respective center vectors of respective force action categories;

[0352] Based on the obtained sample distances, a sample force action recognition result of the target training sample is determined, and parameter adjustment is performed based on a loss value corresponding to the sample force action recognition result.

[0353] In an optional embodiment, after determining the force action recognition result of the target training sample based on the obtained sample distances and performing parameter adjustment based on the loss value corresponding to the sample force action recognition result, the training module 804 is further configured to:

[0354] When the target training sample is a high-confidence sample, a center vector of a force action class to which the target training sample belongs is updated based on a sample modeling vector of the target training sample.

[0355] Based on the description of the method embodiments and the device embodiments, an electronic device is further provided in an exemplary embodiment of the present application, including at least one processor, and a memory in communication connection with the at least one processor. The memory stores a computer program capable of being executed by the at least one processor, and the computer program, when executed by the at least one processor, is configured to cause the electronic device to perform the method according to the embodiments of the present application.

[0356] The embodiments of the present application further provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is configured to cause the computer to perform the method according to the embodiments of the present application.

[0357] The embodiments of the present application further provide a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is configured to cause the computer to perform the method according to the embodiments of the present application.

[0358] Referring to Figure 9 As shown, a block diagram of an electronic device 900 that can serve as a server or client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent a variety of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent a variety of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown therein, their connections, and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present application described and / or claimed herein.

[0359] As Figure 9As shown, the electronic device 900 includes a computing unit 901 that can perform various appropriate actions and processes in accordance with a computer program stored in a readonly memory (ROM) 902 or a computer program loaded into a random access memory (RAM) 903 from a storage unit 908. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0360] A plurality of components in the electronic device 900 are connected to the I / O interface 905, including an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 can be any type of device that can input information to the electronic device 900, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 907 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a world wide interoperability for microwave access (WiMax) device, a cellular communication device, and / or the like.

[0361] The computing unit 901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processors, controllers, microcontrollers, and the like. The computing unit 901 performs various methods and processes described above. For example, in some embodiments, the above-described AI and multi-sensor based intelligent recognition method of force application actions can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 900 via the ROM 902 and / or the communication unit 909. In some embodiments, the computing unit 901 can be configured to perform the above-described AI and multi-sensor based intelligent recognition method of force application actions by any other appropriate means, such as by means of firmware.

[0362] Program code for carrying out the methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be embodied entirely on a machine, partially on a machine, as part of an independent software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0363] In the context of this application, a machine-readable medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine- readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0364] As used in this application, the terms "machine-readable medium" and "computer- readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0365] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0366] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0367] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0368] Moreover, it is to be understood that the above-described embodiments are only exemplary of the application, and that a person skilled in the art can make various changes and modifications without departing from the scope of the application. Therefore, the scope of the application should be determined by the following claims, rather than by the detailed description, and equivalents to the claims should be covered.

Claims

1. An AI and multi-sensor-based intelligent recognition method for force application actions, characterized by, The method comprises: obtaining multi-dimensional data corresponding to each array node in a contact surface array in a target massage operation, wherein the array nodes are arranged at equal intervals, and each array node has at least a preset number of neighbor nodes except for boundary nodes; performing feature extraction on the multi-dimensional data of each array node to obtain an initial feature vector of each array node, and updating each initial feature vector to obtain an embedding vector of the target massage operation according to the spatial adjacency relationship of the array nodes; fusing time sequence information and spatial information into the embedding vector to obtain a target modeling vector, and determining a force action recognition result of the target massage operation based on the target modeling vector.

2. The method of claim 1, wherein, The multi-dimensional data comprises force dimension data, posture dimension data, and temperature dimension data.

3. The method of claim 2, wherein, The method comprises: collecting first initial data of the array nodes in the target massage operation through the contact surface array arranged on a first object surface; collecting second initial data through a hand multi-modal sensor arranged on a second object hand; determining multi-dimensional data corresponding to each array node based on the second initial data and the first initial data of the array nodes.

4. The method of claim 3, wherein, Each array node comprises a pressure sensor, a force sensor, and a temperature sensor, and the first initial data comprises normal force data, tangential force data, and contact surface temperature data. The hand multi-modal sensor comprises an inertial measurement unit, a depth camera, and a temperature electrode, and the second initial data comprises inertial data, hand point cloud data, and back of hand temperature data.

5. The method of claim 4, wherein, The method comprises: determining a posture time sequence based on the inertial data and the hand point cloud data, and taking the posture time sequence as the posture dimension data corresponding to each array node; determining force time sequence and torque time sequence of each hand joint based on the posture time sequence, and taking the normal force data, the tangential force data of each array node, and the force time sequence and the torque time sequence of each hand joint as the force dimension data corresponding to each array node; taking the contact surface temperature data of each array node and the back of hand temperature data as the temperature dimension data corresponding to each array node.

6. The method of claim 5, wherein, The method comprises: performing adaptive extended Kalman filtering on the inertial data and the hand point cloud data to obtain the posture time sequence.

7. The method of claim 5, wherein, The method comprises: determining end pose of each finger based on the posture time sequence and a DH parameter set of each hand joint; performing inverse dynamics calculation based on the end pose to obtain the force time sequence and the torque time sequence of each hand joint.

8. The method of claim 2, wherein, The feature extraction is performed on the multi-dimensional data of each array node to obtain an initial feature vector of each array node, and the feature extraction includes: The feature extraction is performed on the multi-dimensional data of each array node to obtain an initial feature vector of each array node, and the feature extraction includes: The feature extraction is performed on the multi-dimensional data of each array node to obtain an initial feature vector of each array node, and the feature extraction includes:

9. The method of claim 8, wherein, The feature extraction is performed on the multi-dimensional data of each array node to obtain an initial feature vector of each array node, and the feature extraction includes: The time domain feature extraction is performed on the force dimension data of each array node to obtain a first sub-initial feature vector of each array node; The frequency domain feature extraction is performed on the posture dimension data of each array node to obtain a second sub-initial feature vector of each array node; The topology domain feature extraction is performed on the temperature dimension data of each array node to obtain a third sub-initial feature vector of each array node.

10. The method of claim 1, wherein, The obtained initial feature vectors are updated according to the spatial adjacency relationship of the array nodes to obtain an embedding vector of the target massage operation, and the updating includes: A graph is constructed based on the spatial adjacency relationship of the array nodes, wherein the vertices of the graph are the array nodes, and the edge set of the graph is the spatial adjacency relationship between the array nodes; The graph convolution and attention aggregation are performed on the initial feature vectors based on the graph to obtain the embedding vector.

11. The method of claim 1, wherein, The embedding vector is fused with time sequence information and spatial information to obtain a modeling vector of the target massage operation, and the fusing includes: The time domain dynamic feature extraction is performed on the embedding vector to obtain a time sequence feature vector, and the spatial distribution feature extraction is performed on the embedding vector to obtain a spatial feature vector, wherein the time sequence feature vector represents the time dynamic semantics of the target massage operation, and the spatial feature vector represents the spatial distribution semantics of the target massage operation; The time sequence feature vector and the spatial feature vector are fused to obtain the target modeling vector.

12. The method of claim 1, wherein, The force action recognition result of the target massage operation is determined based on the target modeling vector, and the determining includes: The distances between the target modeling vector and the center vectors of each force action category are determined, wherein the center vectors are determined based on the sample modeling vectors of the corresponding force action category; The force action recognition result of the target massage operation is determined based on the obtained distances.

13. The method of claim 12, wherein, The force action recognition result of the target massage operation is determined based on the obtained distances, and the determining includes: When there is a target distance greater than a distance threshold value in the distances, the force action category of the target massage operation is determined as the force action category corresponding to the target distance; When there is no target distance greater than the distance threshold value in the distances, the force action recognition result of the target massage operation is determined based on the distances between the target modeling vector and the sample modeling vectors of each force action category.

14. The method of any one of claims 12-13, wherein, The force action recognition result is obtained by inputting the target modeling vector into a target force action recognition model, wherein the target force action recognition model is obtained by training in the following manner: Obtain a training sample set, each training sample in the training sample set at least comprising a sample embedding vector of a sample massage operation; Based on the training sample set, iteratively train a force action recognition model to be trained to obtain the target force action recognition model, wherein in one iteration training process, the following operations are performed: Fuse the sample embedding vector of the selected target training sample with the time sequence information and the space information to obtain the sample modeling vector of the target training sample; Determine the sample distance between the sample modeling vector of the target training sample and the center vector of each force action category; Based on the obtained sample distance, determine the sample force action recognition result of the target training sample, and adjust the parameters based on the loss value corresponding to the sample force action recognition result.

15. The method of claim 14, wherein, After the above operations, the method further comprises: When the target training sample is a high confidence sample, update the center vector of the force action category to which the target training sample belongs based on the sample modeling vector of the target training sample.

16. An AI and multi-sensor based intelligent recognition device for force action, characterized in that, Comprise: An acquisition module is configured to acquire multi-dimensional data corresponding to each array node in a contact surface array in a target massage operation, wherein the array nodes are arranged at equal intervals, and each array node except the boundary nodes has at least a preset number of neighbor nodes; A data fusion module is configured to extract features from the multi-dimensional data of each array node to obtain an initial feature vector of each array node, and update each initial feature vector according to the spatial adjacency relationship of the array nodes to obtain an embedding vector of the target massage operation; A force action recognition module is configured to fuse the embedding vector with time sequence information and space information to obtain a target modeling vector, and determine a force action recognition result of the target massage operation based on the target modeling vector.

Images