Body environment cognition method and system based on multi-modal fusion and space-time correlation

Abstract

The invention belongs to the field of intelligent body robots, and particularly relates to a body environment cognition method and system based on multi-modal fusion and time-space correlation, and the method comprises the steps: collecting multi-modal data; constructing graph data by taking each mode as a node and taking the correlation between the nodes as an edge; the features of all the nodes jointly form a feature matrix H, the feature matrix H is weighted by using a modal attention matrix between different modals and a time attention matrix between different time points in sequence, and the weighted feature matrix H is input into a graph convolutional network to obtain a fused feature vector; and obtaining perceived physical entity nodes from the environment, decoding from the fused feature vectors to obtain a complete attribute set of each physical entity node, and predicting a label for describing the position relationship between the entity nodes according to the set. According to the method, the accuracy and robustness of robot environment cognition are remarkably improved, and a foundation is laid for subsequent complex operation and interaction of the robot.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligent robots, specifically relating to an embodied environment cognition method and system based on multimodal fusion and spatiotemporal correlation. Background Technology

[0002] With the widespread use of robots in various complex scenarios, the need for precise and in-depth environmental perception is becoming increasingly urgent. Giving robots environmental perception and interaction capabilities comparable to humans has become a core objective in this field. Currently, mainstream robot perception systems mainly rely on visual sensors. While these can achieve localization and recognition in specific environments, they fall short of meeting the higher requirements for safe, efficient, and autonomous interaction by intelligent robots in dynamic, unstructured environments. Furthermore, relying solely on vision cannot capture other key physical properties of objects, such as hardness and texture, resulting in a partial and incomplete understanding of the world by the robot.

[0003] Real-world physical scenarios are complex, and interactive tasks are highly variable. Existing perception systems cannot provide a sufficiently flexible and accurate cognitive model. Currently, most robots, when understanding their external environment, focus only on visual appearance while neglecting other physical attributes and their own situation. They also use simple feature splicing methods without deep fusion of multimodal information or consideration of the dynamic correlation between different sensory information in the spatiotemporal dimensions. Therefore, they fail to generate a structured environmental cognitive model that robots can directly use for intelligent decision-making, and the resulting cognitive output is insufficient to support higher-level complex task decision-making and planning. Therefore, how to integrate multimodal embodied information, including visual, tactile, and auditory information, and design an intelligent fusion mechanism that can dynamically assess the importance of different information sources, is a pressing technical problem to be solved in this field. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes an embodied environment cognition method based on multimodal fusion and spatiotemporal correlation, specifically including the following steps:

[0005] Multimodal data from vision, touch, hearing, and proprioception are collected, preprocessed, and feature extracted to obtain visual feature vectors. tactile feature vectors Auditory feature vectors and ontology-aware feature vectors ;

[0006] When constructing graph data, the features of each modality are used as nodes, and the correlation strength between different modalities at the same time is used as the edges between nodes. At different times, each node is connected to its previous time. The edge relationships between different times and different modalities constitute the adjacency matrix A.

[0007] Including Within the observation window of a frame, the features of all nodes together constitute a feature matrix. , express A real space of dimension N, where N is the number of modes and F is the dimension of the feature vector of each mode;

[0008] At the same time, the importance between different modalities is calculated to obtain the modal attention matrix, and the feature matrix H is weighted using the modal attention matrix;

[0009] The contribution of different time points within the observation window to the current cognitive task is calculated to obtain the temporal attention matrix. The feature matrix H is then weighted again using the temporal attention matrix to obtain the final weighted feature matrix. ;

[0010] feature matrix In the convolutional network of the input graph, which includes the adjacency matrix between nodes within the observation window, the features of the nodes are aggregated and propagated to obtain the fused feature vector. ;

[0011] Based on visual, tactile, auditory, and proprioceptive perception, perceived physical entity nodes are obtained from the environment. The edge between two physical entity nodes is labeled with a description of the positional relationship between the entity nodes.

[0012] Three parallel decoders are used to extract the fused feature vectors. The decoding process yields a complete set of attributes for each physical entity node, which includes the node's class label, three-dimensional spatial location, and physical attributes.

[0013] The classifier is used to determine the complete attribute sets of the two physical entity nodes and the fused feature vector. Predict the labels between two nodes to complete environmental cognition.

[0014] This invention also proposes an embodied environment cognition system based on multimodal fusion and spatiotemporal correlation, used to realize an embodied environment cognition method based on multimodal fusion and spatiotemporal correlation, including:

[0015] The multi-source information perception module is used to collect multimodal data, which includes at least visual information, tactile information, auditory information and proprioceptive perception information.

[0016] The information processing and feature extraction module is used to preprocess the data collected by the multi-source information perception module and extract features from the preprocessed data.

[0017] The graph data module uses the features of each modality as nodes, the correlation strength between different modalities at the same time as the edges between nodes, and the connection between each node at the current time and its previous time at different times. The edge relationships between different times and different modalities constitute an adjacency matrix A, and the features of all nodes together constitute a feature matrix H.

[0018] The attention module first calculates the importance between different modalities to obtain the intermodal attention matrix, and then weights the feature matrix H using the temporal attention matrix. Next, it calculates the contribution of different time points within the observation window to the current cognitive task to obtain the temporal attention matrix, and then weights the feature matrix H again using the temporal attention matrix to obtain the final weighted feature matrix. ;

[0019] Graph convolutional networks are used to aggregate and propagate node features based on the adjacency matrix between nodes within an observation window, resulting in a fused feature vector. ;

[0020] The scene graph construction module is used to obtain perceived physical entity nodes from the environment based on visual, tactile, auditory, and proprioceptive perception. The edge between two physical entity nodes is labeled with a label that describes the positional relationship between the entity nodes.

[0021] The decoder module employs three parallel decoders to decode the fused feature vectors. The decoding process yields a complete set of attributes for each physical entity node, which includes the node's class label, three-dimensional spatial location, and physical attributes.

[0022] The classifier is based on the complete attribute set of two physical entity nodes and the fused feature vector. Labels that predict the positional relationship between two nodes.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] 1. The modalities integrated in this method are truly embodied interaction-oriented. In addition to vision, it innovatively introduces tactile information, auditory information, and proprioceptive information to construct a four-modal spatiotemporal graph.

[0025] 2. To address the challenge of heterogeneous modal fusion, this invention, in addition to graph convolutional networks, introduces a spatiotemporal attention mechanism: First, through inter-modal attention, the importance of different modalities at the current moment is dynamically calculated; second, through temporal attention, the contribution of historical observation data to the current task is dynamically evaluated. Spatiotemporal graph modeling enables a more natural representation of the dynamic relationships between entities in the real physical world, improving the robot's environmental cognition accuracy and robustness; the cross-modal attention mechanism allows the robot to adaptively focus on key sensory information in the current scene, much like a human, enhancing perceptual intelligence in complex scenarios.

[0026] 3. The structured embodied scene diagram output by this invention overcomes the shortcomings of existing technologies, such as unstructured environment models that are difficult to use directly for decision-making. It provides a direct and semantically rich environment representation for high-level task planning and decision control, thereby simplifying the design difficulty of subsequent modules and laying a solid foundation for completing complex tasks. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating an embodiment of the embodied environment cognition method based on multimodal fusion and spatiotemporal correlation provided in one embodiment of the present invention.

[0028] Figure 2 This is a general framework diagram of an embodied environment cognition method based on multimodal fusion and spatiotemporal correlation;

[0029] Figure 3 This is a functional module structure diagram of an embodied environment cognition system based on multimodal fusion and spatiotemporal correlation;

[0030] Figure 4 This is a detailed structural diagram of the spatiotemporal graph neural network module of the present invention;

[0031] Figure 5 This is an example diagram illustrating the embodied scenario of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] This invention proposes an embodied environment cognition method based on multimodal fusion and spatiotemporal correlation, specifically including the following steps:

[0034] Multimodal data from vision, touch, hearing, and proprioception are collected, preprocessed, and feature extracted to obtain visual feature vectors. tactile feature vectors Auditory feature vectors and ontology-aware feature vectors ;

[0035] When constructing graph data, the features of each modality are used as nodes, and the correlation strength between different modalities at the same time is used as the edges between nodes. At different times, each node is connected to its previous time. The edge relationships between different times and different modalities constitute the adjacency matrix A.

[0036] Including Within the observation window of a frame, the features of all nodes together constitute a feature matrix. , express A real space of dimension N, where N is the number of modes and F is the dimension of the feature vector of each mode;

[0037] At the same time, the importance between different modalities is calculated to obtain the modal attention matrix, and the feature matrix H is weighted using the modal attention matrix;

[0038] The contribution of different time points within the observation window to the current cognitive task is calculated to obtain the temporal attention matrix. The feature matrix H is then weighted again using the temporal attention matrix to obtain the final weighted feature matrix. ;

[0039] feature matrix In the convolutional network of the input graph, which includes the adjacency matrix between nodes within the observation window, the features of the nodes are aggregated and propagated to obtain the fused feature vector. ;

[0040] Based on visual, tactile, auditory, and proprioceptive perception, perceived physical entity nodes are obtained from the environment. The edge between two physical entity nodes is labeled with a description of the positional relationship between the entity nodes.

[0041] Three parallel decoders are used to extract the fused feature vectors. The decoding process yields a complete set of attributes for each physical entity node, which includes the node's class label, three-dimensional spatial location, and physical attributes.

[0042] The classifier is used to determine the complete attribute sets of the two physical entity nodes and the fused feature vector. Predict the relationship between two nodes to complete environmental cognition.

[0043] In this embodiment, different types of sensors are used to acquire multimodal real-world physical information. At least visual, tactile, auditory, and proprioceptive information needs to be collected, wherein:

[0044] Use an RGB-D camera to acquire visual information, including at least depth images and RGB images;

[0045] Using a fingertip tactile array to acquire tactile information of an object's surface, including at least tactile images of the contact points, contact force vectors, and vibration signals;

[0046] Use a microphone array to acquire auditory information;

[0047] Joint encoders and IMUs are used to acquire robot body perception information.

[0048] The collected multimodal data is preprocessed, and features are extracted from the preprocessed data, specifically including:

[0049] Step 1: Visual information processing.

[0050] Data preprocessing and point cloud generation include: filtering the acquired depth image to fill data gaps and suppress noise; and using the camera intrinsic parameter matrix, back-projecting the aligned RGB image and depth image into a 3D point cloud with color information, the coordinate transformation relationship being:

[0051]

[0052] in, This represents a 3D point cloud; u and v are pixel coordinates, and d is the depth value. , Focal length , The coordinates of the main point.

[0053] The parallel feature extraction pathway involves semantic information extraction and geometric and pose information extraction. Semantic information extraction uses deep learning models based on RGB images for object detection, object recognition, and instance segmentation to obtain category labels and pixel-level masks for each object in the environment. Geometric and pose information extraction uses point clouds and image sequences to run a SLAM process, calculating the optimal camera pose transformation matrix by minimizing the reprojection error between adjacent frames. This allows for the acquisition of accurate robot pose and matrix. The calculation process is as follows:

[0054]

[0055] Where W is the camera pose transformation matrix; For the i-th 3D landmark point based on the coordinate system of the previous frame, For the current frame and The corresponding two-dimensional observation pixel coordinates, This is a camera projection model used to project 3D points. Projected onto a two-dimensional plane.

[0056] The method involves fusing multi-source information and structural representation, specifically by integrating the extracted 2D semantic information with 3D geometric and pose information. This is achieved by using a camera projection model to back-project the 2D mask obtained from instance segmentation into 3D space, assigning semantic labels to each object instance obtained from point cloud clustering, and calculating its 3D bounding box and centroid position. Simultaneously, the pixel-level mask is applied to an RGB image to extract image slices of the corresponding object instances, and these image slices are input into a pre-trained visual encoder to extract the visual appearance feature vector. These elements, together, constitute standardized feature nodes describing the object instance. Finally, the object labels identified from the image, the 3D bounding boxes and centroid positions corresponding to the point cloud data of each object, and the visual appearance feature vectors corresponding to each object are collectively used as visual feature vectors. .

[0057] Step 2: Tactile Information Processing.

[0058] Low-pass filtering is applied to the raw tactile signals (tactile image sequence, force vector sequence) to suppress high-frequency mechanical and electronic noise; based on the robot's kinematics chain, the measured values ​​in the sensor's local coordinate system are transformed to the robot's basic coordinate system to achieve spatial alignment with visual, auditory, and other information.

[0059] Feature extraction is performed, including static attribute extraction, dynamic attribute extraction, and interaction event monitoring. Static attribute extraction requires obtaining pressure distribution representation and contact force vectors. The pressure distribution representation is obtained from a single-frame visual image I (of size M). The two-dimensional centroid for calculating the pressure distribution in N) , The pressure distribution entropy H is used to quantify the concentration and disorder of the contact area, and the calculation formula is as follows:

[0060]

[0061]

[0062]

[0063] Where N and M are the number of pixels in a single frame visual image I along the X and Y axes, respectively; The pressure value located at pixel (m,n) in the tactile image; This represents the probability distribution of the normalized pressure values.

[0064] The contact force vector is directly read from the three-dimensional contact force vector provided by the sensor. The formula for calculating its scalar magnitude is as follows:

[0065]

[0066] The next step is the extraction of dynamic attributes, including texture recognition and hardness estimation. When the sensor slides relative to the object surface, it will collect the vibration signal s(t) generated by micro-deformation. This signal is then subjected to a Fast Fourier Transform to obtain its spectrum S(f), and the dominant frequency is extracted. Using specific frequency band energy as texture features, the calculation formula is as follows:

[0067]

[0068]

[0069] in, For Fast Fourier Transform; This indicates that the absolute value is being calculated.

[0070] Hardness estimation is achieved by fusing tactile and proprioceptive information, specifically by combining the normal contact force measured by the tactile sensor. and normal displacement To estimate the local equivalent stiffness k of an object:

[0071] .

[0072] Finally, there's the monitoring of interactive events, based on the magnitude of the contact force. The mutation is used to determine the occurrence of contact and separation events, and a precise timestamp is added to the occurrence of the events, ultimately resulting in a feature vector containing all tactile information. Specifically, this is determined by the centroid of the pressure distribution ( , Pressure distribution entropy H, contact force scalar F, and texture features The hardness estimate k and the timestamps of the interaction events are combined to form the tactile feature vector. .

[0073] Step 3: Auditory information processing.

[0074] Sound source localization: This refers to determining the spatial location of the sound source. In this embodiment, the sound source localization uses a location method based on Time Difference of Arrival (TDOA). Those skilled in the art can also choose other sound source localization algorithms. TDOA calculates the azimuth angle of the sound source by analyzing the small time differences between the arrival times of the same sound wave at different microphones in the microphone matrix. Assume that the signals collected by any pair of microphones (i and j) in the array are... and The Generalized Cross-Correlation with Phase Transform (GCC-PHAT) algorithm is used to calculate TDOA (denoted as ). ):

[0075]

[0076] in, This indicates the search for the time delay t that maximizes the value of the function within the parentheses. and They are and Fourier transform, yes The complex conjugate, It is angular frequency. In this context, j is the imaginary unit.

[0077] Calculate at least one pair of microphones Then, the azimuth angle of the sound source can be calculated based on the geometric relationship between the microphone arrays. :

[0078]

[0079] Where c is the speed of sound in air, and d is the distance between microphones i and j. By combining the localization results of multiple pairs of microphones, a more accurate location of the sound source can be obtained.

[0080] Auditory event recognition: To identify the semantic meaning of sound, audio signals need to be classified. Mel-Frequency Cepstral Coefficients (MFCCs) are used as the core features of the audio signal. The extracted MFCC feature sequence is then input into a pre-trained classification model (such as a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN), or other neural network structures with classification capabilities). This allows the model to identify the category labels of different auditory events, such as "knocking," "dropping," and "speech."

[0081] After the above processing, the original audio stream is transformed into a feature vector containing spatial and semantic information. Specifically, the azimuth angle of the sound source is... The auditory feature vector is a combination of the category label of the auditory event. .

[0082] Step 4: Processing of ontological perception information.

[0083] Joint encoder data processing: By measuring the rotation angle of each driven joint of the robot, the robot's motion state is determined. First, the joint state vectors are constructed, and then features are extracted from the joint velocities and accelerations. For example, at any time point t, the angle readings of all n joints are combined into a joint angle vector Q(t):

[0084]

[0085] in, Let be the angle value of the i-th joint at time t.

[0086] Joint angular velocity and joint angular acceleration The joint angle vector is then approximated by time difference:

[0087]

[0088]

[0089] in, This is the time interval for sampling sensor data.

[0090] IMU Data Processing: IMUs typically provide three-axis linear acceleration and angular velocity information to describe the robot's body pose. First, the raw IMU data is calibrated to eliminate inherent systematic errors. Then, a low-pass filter is used to process the data, resulting in smooth and accurate angular velocity and acceleration readings. Finally, the angular velocity signal is integrated over time to obtain the Euler angles at time t. This allows us to estimate the robot's posture.

[0091] Timing synchronization: Since different sensors have different sampling frequencies and clocks, they need to be synchronized in time. An alignment algorithm is used to align the features of different modes in time to a unified coordinate system.

[0092] Finally, all processed body perception information is integrated into a single feature vector, which combines joint angle vectors, joint angular velocities, joint angular accelerations, and body pose as the body perception feature vector. .

[0093] This invention innovatively employs a multimodal fusion module. This module uses a spatiotemporal graph neural network based on an attention mechanism to deeply fuse and correlate multi-source heterogeneous feature information in space and time. This simulates the process by which the human brain integrates multi-sensory information when perceiving the environment, enabling the robot to form a unified and comprehensive understanding of the environment, just like a human. The specific process includes:

[0094] Step 1: Construction of a Multimodal Spatiotemporal Graph: To effectively represent the complex relationships between various modal information, this invention constructs a dynamic spatiotemporal graph G=(V, E, A) from the multimodal feature sequences, where:

[0095] V represents the set of nodes. At any time point t, the nodes of the spatiotemporal graph are composed of the modal feature vectors obtained after preprocessing and feature extraction at that time. Specifically, the set of nodes includes visual feature vectors. tactile feature vectors Auditory feature vectors and ontology-aware feature vectors Including history Within the observation window of a frame, the features of all nodes collectively constitute a feature matrix H:

[0096]

[0097] Where N is the number of modalities, which is 4 in this embodiment, including four modalities: visual, tactile, auditory, and proprioceptive; F is the dimension of the feature vector.

[0098] In a spatiotemporal graph neural network, the edge set E represents the intrinsic connections between different nodes in the spatiotemporal dimension. The edges in the multimodal fusion module represent the intrinsic connections between different modal feature vectors at any time point t, embodied by the adjacency matrix A. As an optional implementation, the adjacency matrix is ​​preferably a learnable weighted matrix that integrates the correlation between modalities and temporal continuity, wherein:

[0099] The off-diagonal elements of matrix A represent the correlation strength between different modalities. For example, there is a natural strong correlation between vision and touch, and vision and hearing, so they have high weights.

[0100] The diagonal elements of the adjacency matrix A are set to 0, while the connections between nodes and themselves (i.e., self-connections) are added to the identity matrix I in subsequent graph convolutional network layers (i.e., ... This is achieved through a method that ensures nodes retain their own information during aggregation; the temporal continuity of information is handled by a temporal attention mechanism. In this way, the interrelationships between modalities can be effectively represented through the spatiotemporal graph G.

[0101] Step 2, Cross-modal Attention Mechanism: To enable the model to dynamically focus on the most critical sensory information based on the current task and scenario, thereby achieving embodied intelligence, this invention introduces a cross-modal attention mechanism to weight the input feature matrix H before graph convolutional fusion. This invention calculates attention weights from two dimensions: inter-modal and temporal, where:

[0102] Intermodal attention: This can simulate spatial attention and is used to dynamically calculate the importance of different modalities at the same time point. Modal attention matrix. The calculation formula is as follows:

[0103]

[0104] in, This is a feature slice extracted from the feature matrix H at a single time t, with dimension H. ; , , , and All of these are learnable parameter matrices in the network; It is a non-linear activation function; It is a normalized exponential function.

[0105] Temporal attention: Used to evaluate the contribution of information from different time points in a historical observation sequence to the current cognitive task. The temporal attention matrix is ​​obtained by acquiring the contribution of different time points within the observation window to the current cognitive task. The process includes:

[0106]

[0107] in, The time series feature matrix is ​​obtained by reconstructing the feature matrix H. Specifically, the reconstruction involves concatenating the F-dimensional feature vectors of the N modes at each time step t within the T time steps to form a... The combined feature vectors of dimension 1 ultimately yield a feature vector of dimension 2. The time series feature matrix; , , , and These are all learnable parameter matrices in the network.

[0108] The weighted feature matrix is ​​obtained by multiplying the calculated attention matrix with the original feature matrix H. This matrix can highlight the most critical modal and temporal information for the current task.

[0109] Step 3, Spatiotemporal Graph Convolution Fusion: The attention-weighted feature matrix is ​​then... The adjacency matrix A is input into the Graph Convolutional Network (GCN) layer to aggregate and propagate information between nodes. In the multimodal fusion module, the graph convolution operation follows the formula:

[0110]

[0111]

[0112] in, Let be the node feature matrix of the l-th layer GCN and ; This is the adjacency matrix with self-connections added; I is the identity matrix; for The degree matrix; Let be the trainable weight matrix of the l-th layer.

[0113] By stacking multiple layers of spatiotemporal graph convolutional networks, the model can repeatedly exchange and integrate information between different modalities and over time. Each convolutional layer fuses the features of a node with the features of its neighboring nodes and historical states, thereby achieving deep fusion from local correlation to global integration.

[0114] After being processed by this multimodal fusion core, a set of highly condensed fusion feature vectors containing rich spatiotemporal correlation information is output, which will be passed to the environmental cognition output layer for the final structured representation generation.

[0115] The final step in this embodiment of the invention is to decode and transform the spatiotemporal feature vectors from the multimodal fusion core into a structured environmental representation that can be directly used by the robot's high-level decision-making module, i.e., generating an embodied scene graph containing real physical semantics. This embodied scene graph not only includes the categories of objects and machines in the environment but also explicitly describes various physical attributes, spatiotemporal relationships, and dynamic interaction states between the robot's own limbs, between objects, and between objects and the robot. Its generation process mainly includes:

[0116] Step 1: Definition of the embodied scene diagram:

[0117] In this invention, a embodied scene diagram is used. At time t, it is defined as:

[0118]

[0119] in, It is a set of nodes in a graph, where each node... They all represent a physical entity, such as environmental objects (e.g., tables, chairs, etc.) or parts of one's own body (e.g., left arm, left hand, etc.). It is the set of edges in the graph, where each edge... Each is connected to a pair of nodes. It also includes a label describing the two nodes.

[0120] Step 2: Obtaining Node Attributes

[0121] For each perceived entity in the scene, i.e., a node This system requires the fused feature vectors to be used. The corresponding series of attributes are decoded from the global vector. To address the problem that specific entities cannot be distinguished solely from the global vector, this invention employs a decoder structure of "feature concatenation + multilayer perceptron (MLP)". Specifically, the input of each decoder includes:

[0122] 1. Local Feature Indexing: This involves indexing the visual appearance feature vectors of the object instance extracted in step one. As a unique identifier index for this instance;

[0123] 2. Global Multimodal Context: A fused feature vector from the output of a spatiotemporal graph neural network. .

[0124] The decoding process is as follows: First, and The features are concatenated to obtain the enhanced entity feature vector. Then, the entity feature vector The input is fed into three parallel, multilayer perceptron-based decoding branches, as shown in the following formula:

[0125]

[0126]

[0127]

[0128]

[0129] in, Represents a node A complete set of attributes; The category labels representing nodes (such as "table", "robotic arm end effector", etc.) are determined by the category branches. Output probability distribution; Representing the 3D spatial pose of a node, derived from the regression branch. It was predicted; The physical properties of a node are represented by a vector containing physical characteristics obtained from multimodal sensing, derived from the regression branch. It was predicted; , , All of them are neural network structures containing multiple fully connected layers and ReLU activation functions.

[0130] The multilayer perceptron learns an implicit attention mechanism through its multilayer nonlinear transformations. For example, when the MLP receives... When the "cup" feature is detected, the network will automatically pay attention. The MLP extracts visual components related to "cup position," tactile components related to "cup texture," and auditory components related to "cup tapping." Thus, guided by the visual feature vectors, the MLP retrieves and obtains the physical attributes of a specific object from the fused features.

[0131] By performing this concatenation and then decoding operation, the model can utilize visual appearance feature vectors. Different objects were distinguished, and then the fused feature vectors were used. By enabling the model to understand its different physical properties, the corresponding decoding of image entities and multimodal data was achieved.

[0132] Step 3: Classify edge relationships:

[0133] To build a complete scene understanding, in addition to obtaining the attributes of nodes, the system also needs to obtain the relationships between different nodes. For any pair of nodes... The system uses a classifier to predict whether and what kind of relationship exists between them:

[0134]

[0135] in, Represents a node Relationship tags between them; This represents the classifier used to classify edge relationships.

[0136] At the same time, the obtained relationship labels contain information from multiple dimensions, including:

[0137] Spatial dimension: It implies the spatial relationship between different nodes, such as "cup" on top of "table" and "cup" next to "left hand".

[0138] Dynamic interaction dimension: This establishes dynamic interaction relationships between nodes, such as the "left index finger" touching the "cup." The establishment of these dynamic relationships strongly relies on the spatiotemporal graph neural network's ability to process continuous time series information.

[0139] Through the steps outlined above, this method ultimately generates a complete and dynamically updatable embodied scene graph. This graph provides the robot with a common-sense, human-like understanding of its current environment, laying a solid foundation for subsequent decision-making processes such as grasping, obstacle avoidance, and human-computer interaction planning.

[0140] As an optional implementation, the spatiotemporal graph neural network based on the attention mechanism in this invention has the following specific structure: Figure 3 As shown, its main components include:

[0141] A multimodal data input layer is used to pass the preprocessed and feature-extracted visual, tactile, auditory, and robot-specific feature vectors to an attention-based spatiotemporal graph neural network for processing.

[0142] The spatiotemporal graph neural network layer based on the attention mechanism includes a multimodal spatiotemporal graph, a cross-modal attention mechanism, and stacked graph convolutional layers. The multimodal spatiotemporal graph is used to construct a graph neural network, which uses feature vectors from different modalities as nodes, the correlations and temporal continuity between feature vectors as edges, and outputs a feature matrix for subsequent processing. The cross-modal attention mechanism includes inter-modal attention and temporal attention. The inter-modal attention mechanism can assign different weights according to the correlation strength between different modalities, allowing the system to focus more on the most important sensory features at present. The temporal attention mechanism can pay attention to the contribution of information at different time points in the historical observation sequence to the current task. After processing by the cross-modal attention mechanism, a weighted feature matrix can be output. The stacked graph convolutional layers are the superposition of multiple graph convolutional layers. Each graph convolutional layer can improve the fusion degree between different nodes. After multiple layers of graph convolution processing, the information between different modalities is fused to achieve global correlation, and finally a highly unified feature matrix rich in intermodal spatiotemporal correlation information is formed.

[0143] The output layer outputs the results. In this layer, the system will output a processed spatiotemporal feature vector containing all modalities. This vector is highly structured and contains rich physical semantics, which is beneficial for the subsequent creation of embodied intelligent scene graphs by the system.

[0144] This embodiment also provides an embodied environment cognition system based on multimodal fusion and spatiotemporal correlation, which mainly has the following features: Figure 2 The modules shown specifically include:

[0145] The multi-source information perception module is used to collect multimodal data, which includes at least visual information, tactile information, auditory information and proprioceptive perception information.

[0146] The information processing and feature extraction module is used to preprocess the data collected by the multi-source information perception module and extract features from the preprocessed data.

[0147] The graph data module uses the features of each modality as nodes, the correlation strength between different modalities at the same time as the edges between nodes, and the connection between each node at the current time and its previous time at different times. The edge relationships between different times and different modalities constitute an adjacency matrix A, and the features of all nodes together constitute a feature matrix H.

[0148] The attention module first calculates the importance between different modalities to obtain the intermodal attention matrix, and then weights the feature matrix H using the temporal attention matrix. Next, it calculates the contribution of different time points within the observation window to the current cognitive task to obtain the temporal attention matrix, and then weights the feature matrix H again using the temporal attention matrix to obtain the final weighted feature matrix. ;

[0149] Graph convolutional networks are used to aggregate and propagate node features based on the adjacency matrix between nodes within an observation window, resulting in a fused feature vector. ;

[0150] The scene graph construction module is used to obtain perceived physical entity nodes from the environment based on visual, tactile, auditory, and proprioceptive perception. The edge between two physical entity nodes is labeled with a label that describes the positional relationship between the entity nodes.

[0151] The decoder module employs three parallel decoders to decode the fused feature vectors. The decoding process yields a complete set of attributes for each physical entity node, which includes the node's class label, three-dimensional spatial location, and physical attributes.

[0152] The classifier is based on the complete attribute set of two physical entity nodes and the fused feature vector. Labels that predict the positional relationship between two nodes.

[0153] The embodied scene diagram obtained by the method of this invention is as follows: Figure 4 As shown in the figure, the main components include:

[0154] At time t, the system primarily acquires information about the current environment through its visual sensors. Specifically, in the current scene, there is a table with a basket, a bitter melon, a plush duck, a lemon, and a slice of watermelon on it; the items on the table are close to each other; the plush duck is yellow, and the basket is gray; the robot's two arms are positioned above these items; and the robot's visual sensors are identifying all the items on the table to determine the target to grasp.

[0155] At time t+1, the machine has identified the target plush duck and executes the grasping command. The environmental information the robot receives at this point is: the plush duck is located in the lower left corner between the bitter melon and the lemon. Thanks to the information processing and feature extraction module, the robot perceives, through the grasping command and tactile sensors, that the plush duck's surface material is soft and fluffy.

[0156] At time t+2, the robot is controlling its left hand, which is holding the plush duck, to move towards the basket and eventually stop directly above the basket.

[0157] ...

[0158] At time t+n, the robot executes the release command and successfully puts the plush duck into the basket.

[0159] The above is the embodied intelligence scene diagram formed in this simple scenario, which includes the perception of vision, touch and the self. Through these perceptions, an embodied intelligence scene diagram containing external perception, self perception and external-self interaction information was successfully constructed.

[0160] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for embodied environment cognition based on multi-modal fusion and spatio-temporal association, characterized in that, Specifically comprising the following steps: Collect multi-modal data of vision, touch, hearing and proprioception, pre-process and extract features of the data to obtain a visual feature vector , a touch feature vector , a hearing feature vector and a proprioception feature vector ; When constructing the graph data, each modality is taken as a node, and the connection strength between different modalities at the same time is taken as the edge between the nodes, and at different times, each node is connected between the current time and the previous time, and the edge relationship between different modalities at different times constitutes an adjacency matrix A; In the embodiment, the feature matrix is constructed by using the features of all nodes at all time points in the observation window of the frame The features of all nodes at all time points in the observation window of the frame jointly constitute a feature matrix , The feature matrix is represented as a real number space with dimensions of N*F, where N is the number of modes, and F is the dimension of the feature vector of each mode At the same time, the importance between different modalities is calculated to obtain a modality attention matrix, and the modality attention matrix is used to weight the feature matrix H; The contribution degrees of different time points in the calculation observation window to the current cognitive task are obtained to obtain a time attention matrix, and the feature matrix H is weighted again by using the time attention matrix to obtain a final weighted feature matrix ; feature matrix In the convolutional network of the input graph, which includes the adjacency matrix between nodes within the observation window, the features of the nodes are aggregated and propagated to obtain the fused feature vector. ; According to vision, touch, hearing and proprioception, the perceived physical entity nodes are obtained from the environment, and the edge between two physical entity nodes is represented by a label describing the positional relationship between the entity nodes; Three parallel decoders are adopted, and for each perceived physical entity node, its corresponding visual appearance feature vector is spliced with the fused feature vector to construct an enhanced feature vector of the physical entity node , and the enhanced feature vector is input into the decoder to respectively decode the complete attribute set of each physical entity node, wherein the complete attribute set of the physical entity node includes the class label, three-dimensional spatial position and physical attribute of the node. Through the classifier, according to the complete attribute set of two physical entity nodes and the fused feature vector Predict the relationship between the two nodes, complete the environmental cognition.

2. The embodied environment cognition method based on multi-modal fusion and space-time correlation according to claim 1, characterized in that, At the same time, the attention coefficients between different modalities constitute an attention matrix, and the process of obtaining the attention matrix includes: wherein, denotes the modality attention matrix; is a feature slice at a single time instant t extracted from the feature matrix H with dimension ; , and are learnable parameter matrices for capturing inter-modal correlations; and are learnable parameter matrices in the network; is a nonlinear activation function; is a normalized exponential function.

3. The embodied environment cognition method based on multi-modal fusion and space-time correlation according to claim 1, characterized in that, The contribution of different time points in the observation window to the current cognitive task is obtained to obtain a time attention matrix, and the process includes: wherein, denotes the temporal attention matrix; is a time series feature matrix reconstructed from the feature matrix H with dimension , denotes the total feature dimension after concatenating the F-dimensional features of N modalities at each time step; , and are parameter matrices learnable in the network for capturing temporal dependencies; and are parameter matrices learnable in the network; is a non-linear activation function; is a normalized exponential function.

4. The embodied environment cognition method based on multi-modal fusion and space-time correlation according to claim 1, characterized in that, The graph convolution network includes a cascaded multi-layer graph convolution structure, and the processing of the lth layer graph convolution structure on the data includes: wherein, is the node feature matrix output by the l-th layer graph convolution layer, is the feature matrix input to the graph convolution network and ; is a non-linear activation function; is the adjacency matrix with self-loops; I is the identity matrix; is the degree matrix of ; is the trainable weight matrix of the l-th layer graph convolution layer.

5. The embodiment of claim 1, wherein, The pre-processing and feature extraction of the visual information include: The collected depth image and RGB image are processed, and the aligned RGB image and depth image are back-projected into a three-dimensional point cloud with color information by using a camera intrinsic matrix; A deep learning model is used for instance segmentation on the RGB image, so as to obtain the class label and pixel-level mask of each object in the environment; The pixel-level mask is back-projected into a three-dimensional space, and is used to assign a semantic label to the object instance point cloud obtained by clustering the three-dimensional point cloud, and calculate the three-dimensional bounding box and centroid position of the object instance point cloud; applying the pixel-level mask to the RGB image to obtain an image slice of the corresponding object, and inputting the image slice into a pre-trained visual encoder to extract a visual appearance feature vector ; The acquired category label, three-dimensional bounding box, centroid position, and visual appearance feature vector are combined together to obtain a unified visual feature vector 6. The embodied environment cognition method based on multi-modal fusion and space-time correlation according to claim 1, characterized in that, Tactile information includes at least tactile images of the contact points, contact force vectors, and vibration signals. Preprocessing and feature extraction of this tactile information are performed to determine the centroid of the pressure distribution. , Pressure distribution entropy H, contact force scalar F, and texture features The hardness estimate k and the timestamps of the interaction events are combined to form the tactile feature vector. ,in: calculating a two-dimensional center of mass of the pressure distribution (C , ) from the single frame of tactile image I; and calculating a pressure distribution entropy H from the two-dimensional center of mass. The scalar of the contact force vector is calculated according to the obtained contact force vector; The vibration signal s(t) collected when the sensor and the object surface slide relative to each other is subjected to fast Fourier transform to obtain a frequency spectrum , and a main frequency is extracted as a texture feature ; The ratio of the normal contact force and the normal displacement amount is taken as the hardness estimate k; The interaction event extraction is performed, that is, the occurrence of the contact event and the separation event is judged through the mutation of the contact force scalar detected by the sensor, and the time of the occurrence of the interaction event contact event and the separation event is taken as the time stamp of the interaction event.

7. The embodied environment cognition method based on multi-modal fusion and space-time correlation according to claim 1, characterized in that, The auditory information comprises at least speech signals received by two microphone arrays, the auditory information is pre-processed and feature extracted, and a sound source azimuth angle and a class label of the auditory event are jointly encoded into an auditory feature vector wherein: The generalized cross-correlation-phase transform algorithm is used to calculate the time difference of arrival between the voice signals collected by the two microphone arrays, and the azimuth of the sound source is determined by the time difference and the positional relationship between the two microphone arrays; The mel frequency cepstral coefficient is extracted from the voice signal collected by the microphone array as the audio feature, and the audio feature is input into a pre-trained classification model to identify the class label.

8. The embodied environment cognition method based on multi-modal fusion and space-time correlation according to claim 1, characterized in that, The body perception information at least includes a rotation angle of each driving joint of the embodied robot, data collected by an inertia unit, and the body perception information is preprocessed and feature extracted, and the joint angle vector, joint angular velocity, joint angular acceleration and body pose are collectively used as a body perception feature vector wherein: At time point t, the angle readings of all n joints are combined into a joint angle vector Q(t); based on the sampling interval of the sensor, a time differential approximation is calculated on the joint angle vector to obtain the joint angular velocity ; Based on the sampling interval of the sensor, the joint angular velocity is calculated approximately in time difference to obtain the joint angular acceleration ; The angular velocity signal is time-integrated to obtain the Euler angle at time t after data processing by the inertial unit, i.e. calibration and filtering of the IMU data The Euler angle is taken as the body pose of the embodied robot.

9. A system for embodied environment cognition based on multi-modal fusion and spatio-temporal association, characterized in that, A somatosensory environment cognitive method based on multi-modal fusion and spatio-temporal correlation is used to realize the method of claim 1, comprising: A multi-source information perception module is used to collect multi-modal data, and the multi-modal data at least includes visual information, tactile information, auditory information and proprioceptive information; An information processing and feature extraction module is used to pre-process the data collected by the multi-source information perception module, and extract features from the pre-processed data; A graph data module, which takes the features of each modality as nodes, and the connection strength between different modalities at the same time as the edge between the nodes, and at different times, each node is connected between the current time and the previous time, and the edge relationship between different modalities at different times constitutes an adjacency matrix A, and all the features of the nodes constitute a feature matrix H; The attention module firstly calculates the importance between different modalities to obtain an inter-modality attention matrix, and weights the feature matrix H by using the time attention matrix; then calculates the contribution of different time points in the observation window to the current cognitive task to obtain a time attention matrix, and weights the feature matrix H again by using the time attention matrix to obtain a final weighted feature matrix ; A graph convolutional network is used to aggregate and propagate features of nodes based on an adjacency matrix between nodes within an observation window to obtain a fused feature vector ; a scene graph construction module for obtaining perceived physical entity nodes from the environment according to vision, touch, hearing and proprioception, an edge between two physical entity nodes is labeled with a label describing the positional relationship between the entity nodes; The decoder module employs three parallel decoders to decode the fused feature vectors. The decoding process yields a complete set of attributes for each physical entity node, which includes the node's class label, three-dimensional spatial location, and physical attributes. a classifier, according to the complete attribute set of two physical entity nodes and the fused feature vector a label of a location relationship between two nodes.

Images