Cross-modal space-time perception tensor generation method, device, equipment, medium and product

Abstract

The embodiment of the invention provides a cross-modal space-time perception tensor generation method and device, equipment, a medium and a product, and relates to the field of artificial intelligence. According to the method, through the synergistic effect of an endogenous space-time diagram neural network and a cross-modal attention mechanism, the cross-modal space-time perception tensor is generated by means of the structured modeling capability of the diagram neural network; according to the method, heterogeneous modal data are mapped to a unified representation space, modal interaction strength is dynamically regulated and controlled based on a cross-modal attention mechanism, causal consistency of cross-modal information on a spatial-temporal scale is ensured, in addition, an endogenous spatial-temporal diagram neural network can model an interaction relationship between modals through diagram edges, and the spatial-temporal diagram neural network is more accurate. The adaptability to dynamic environment changes is effectively enhanced, the robustness and generalization ability of multi-modal fusion are further improved, and finally the problem that in the prior art, the sensing tensor generation efficiency is low is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, medium and product for generating cross-modal spatiotemporal sensing tensors. Background Technology

[0002] The perceptual tensor is a mathematical structure that uses a high-dimensional matrix to uniformly represent the multimodal perceptual information acquired by an embodied intelligent agent from the environment. Traditional single-modal data representation or simple data superposition methods are difficult to fully preserve the inherent correlation of data within a unified framework. The perceptual tensor was proposed to address this challenge, aiming to provide a structured carrier for the integrated integration and deep correlation mining of multimodal perceptual information, and to support the intelligent agent's comprehensive perception of complex scenes.

[0003] In existing technologies, the raw data from each sensor are typically extracted and encoded independently first. Then, through a specific fusion strategy, these heterogeneous features are integrated into a multidimensional array structure to obtain the perceptual tensor.

[0004] However, existing technologies suffer from low efficiency in generating perceptual tensors. Multimodal inputs exhibit heterogeneity in format and differences in spatiotemporal dimensions, requiring additional data format adaptation and cross-modal spatiotemporal calibration operations. Furthermore, the complex computations of cross-modal feature fusion and the redundant processes of tensor structured representation construction result in time-consuming perceptual tensor generation, making it difficult to meet the real-time perception and decision-making needs in dynamic scenarios. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and product for generating cross-modal spatiotemporal sensing tensors, in order to solve the problem of low efficiency in sensing tensor generation in the prior art.

[0006] In a first aspect, embodiments of this application provide a method for generating cross-modal spatiotemporal sensing tensors, including:

[0007] Acquire multimodal input data; wherein, the multimodal input data includes visual features, language features, tactile features and ontological state features, the multimodal input data is data collected by multiple sensors and multiple interfaces on a preset intelligent agent, and the multimodal input data is used to represent the environmental state, posture and received task instructions of the intelligent agent;

[0008] The multimodal input data is encoded using a pre-defined endogenous spatiotemporal graph neural network to obtain a graph structure representation. The endogenous spatiotemporal graph neural network includes multiple graph nodes and multiple graph edges. The multiple graph nodes represent the state of the agent, and the multiple graph edges represent the interaction relationships between the multimodal input data. The graph structure representation is a feature carrier obtained by encoding the multimodal input data using the endogenous spatiotemporal graph neural network.

[0009] The graph structure representation is weighted and fused using a pre-defined cross-modal attention mechanism to obtain a cross-modal spatiotemporal perception tensor; wherein the cross-modal spatiotemporal perception tensor is used to provide input for the agent to make endogenous decisions and control motion.

[0010] Secondly, embodiments of this application provide a device for generating cross-modal spatiotemporal sensing tensors, comprising:

[0011] The first acquisition module is used to acquire multimodal input data; wherein, the multimodal input data includes visual features, language features, tactile features and ontological state features, the multimodal input data is data collected by multiple sensors and multiple interfaces on a preset intelligent agent, and the multimodal input data is used to represent the environmental state, posture and received task instructions of the intelligent agent;

[0012] The first encoding module is used to encode the multimodal input data through a preset endogenous spatiotemporal graph neural network to obtain a graph structure representation; wherein, the endogenous spatiotemporal graph neural network includes multiple graph nodes and multiple graph edges, the multiple graph nodes are used to represent the state of the agent, the multiple graph edges are used to represent the interaction relationship between the multimodal input data, and the graph structure representation is a feature carrier obtained by the endogenous spatiotemporal graph neural network encoding the multimodal input data;

[0013] The first fusion module is used to perform weighted fusion of the graph structure representation using a preset cross-modal attention mechanism to obtain a cross-modal spatiotemporal perception tensor; wherein the cross-modal spatiotemporal perception tensor is used to provide input for the agent to make endogenous decisions and control motion.

[0014] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0015] The memory stores computer-executed instructions;

[0016] When the processor executes the computer execution instructions stored in the memory, it is used to implement the method for generating cross-modal spatiotemporal sensing tensors as described in any of the first aspects.

[0017] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method for generating cross-modal spatiotemporal sensing tensors as described in any of the first aspects.

[0018] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, is used to implement the method for generating cross-modal spatiotemporal sensing tensors as described in any of the first aspects.

[0019] This application provides a method, apparatus, device, medium, and product for generating cross-modal spatiotemporal sensing tensors. Through the synergistic effect of an endogenous spatiotemporal graph neural network and a cross-modal attention mechanism, and leveraging the structured modeling capabilities of the graph neural network, heterogeneous modal data are mapped to a unified representation space. Simultaneously, the cross-modal attention mechanism dynamically adjusts the intensity of modal interactions to ensure the causal consistency of cross-modal information across spatiotemporal scales. The endogenous spatiotemporal graph neural network can model the interaction relationships between modalities through graph edges, effectively enhancing its adaptability to dynamic environmental changes, thereby improving the robustness and generalization ability of multimodal fusion. Ultimately, this solves the problem of low generation efficiency of sensing tensors in existing technologies. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] Figure 1 A schematic diagram illustrating an application scenario of the method for generating cross-modal spatiotemporal sensing tensors provided in this application embodiment;

[0022] Figure 2 One of the flowcharts illustrating the method for generating cross-modal spatiotemporal sensing tensors provided in this application embodiment;

[0023] Figure 3 One of the flowcharts illustrating the method for generating cross-modal spatiotemporal sensing tensors provided in this application embodiment;

[0024] Figure 4 One of the flowcharts illustrating the method for generating cross-modal spatiotemporal sensing tensors provided in this application embodiment;

[0025] Figure 5 System architecture diagram provided for embodiments of this application;

[0026] Figure 6 A flowchart of the agent decision-making process provided in the embodiments of this application;

[0027] Figure 7 This is a schematic diagram of the structure of the data query device provided in the embodiments of this application;

[0028] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0029] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply difference. It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more.

[0032] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the method, apparatus, device, medium, and product for generating a cross-modal spatiotemporal sensing tensor provided in the embodiments of this application are merely examples; a method, apparatus, device, medium, and product for generating a cross-modal spatiotemporal sensing tensor may include more or fewer elements.

[0033] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0034] Endogenous spatiotemporal graph neural network: This is a neural network structure that simulates the perception and interaction process of an intelligent agent in a spatiotemporal environment. The network is based on a graph structure, where nodes represent different states or perceptual elements of the agent, and edges are used to characterize the dynamic relationships and mutual influences between these nodes in the temporal and spatial dimensions.

[0035] Cross-modal attention mechanism: This is a computational framework for dynamically integrating multiple perceptual information. It analyzes the correlations between features from different modalities and automatically assigns different weights to each modality or intermodal interaction to highlight information more important for understanding the current task or environment. This mechanism enables the system to adaptively adjust its information fusion strategy based on context, strengthening modal interactions with strong semantic or causal relationships, and suppressing redundant or interfering information, thereby improving the discriminative ability and task-oriented adaptability of cross-modal representations in complex scenarios.

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0037] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0038] To clearly understand the technical solution of this application, the solutions of existing technologies will first be described in detail. Traditional single-modal data representation or simple data overlay methods are difficult to fully preserve the inherent correlation of data within a unified framework. Perceptual tensors are proposed to address this challenge, aiming to provide a structured carrier for the integrated integration and deep correlation mining of multimodal perceptual information.

[0039] In existing technologies, features are typically extracted and encoded independently from the raw data of each sensor. Then, using a specific fusion strategy, these heterogeneous features are integrated into a multidimensional array structure to obtain a perceptual tensor. Multimodal inputs exhibit format heterogeneity and spatiotemporal differences, requiring additional data format adaptation and cross-modal spatiotemporal calibration operations. This, coupled with the complex computations of cross-modal feature fusion and the redundant process of tensor structure representation construction, results in time-consuming perceptual tensor generation, making it difficult to meet the real-time perception and decision-making requirements of dynamic scenarios. Therefore, existing technologies suffer from low perceptual tensor generation efficiency.

[0040] Therefore, addressing the low efficiency of perceptual tensor generation in existing technologies, this research found that to solve this problem, a unified structured feature modeling approach can be adopted to encode multimodal heterogeneous input data in an integrated manner. Combined with dynamic cross-modal information adaptation and weighted fusion strategies, redundant cross-modal data format conversion and spatiotemporal calibration steps can be omitted, achieving efficient perceptual tensor generation: ① A unified multimodal input preprocessing and feature encoding integrated framework can be built, completing the standardized mapping of heterogeneous formats during the data access stage. Simultaneously, an adaptive calibration module for spatiotemporal dimension alignment is built-in, integrating format conversion, spatiotemporal calibration, and initial feature extraction into a single process, avoiding interface redundancy and repeated data calls caused by independent processing of each step. ② A pre-built modal correlation determination module can identify the effective correlation of different modal data in the current scenario, performing deep fusion operations only on strongly correlated modal data, and using lightweight feature splicing or weighted summarization methods for weakly correlated or uncorrelated modal data. Simultaneously, the computational granularity of the fusion operation can be dynamically adjusted to reduce meaningless complex computational overhead. ③ To address the redundancy in the process of constructing perceptual tensor structured representations, an adaptive multimodal structured representation generation mechanism can be introduced. This mechanism directly maps heterogeneous multimodal data to a preset structured feature space. Through an adaptive learning module, the spatiotemporal correlation logic of different modal data is automatically mined to generate a structured feature carrier with spatiotemporal consistency, thus omitting redundant steps in the later tensor structured reconstruction.

[0041] Specifically, an integrated multimodal data collaborative processing architecture can be built. First, a unified heterogeneous format adaptation and spatiotemporal dimension adaptive calibration module is deployed in the data access stage to synchronously complete the standardization conversion and spatiotemporal correlation alignment of multimodal input data from different sources and in different formats. Then, a dynamic lightweight fusion strategy based on modal correlation strength is adopted to perform deep feature interaction only on strongly correlated modal data and simple feature aggregation on weakly correlated modal data. Finally, a structured perception tensor with spatiotemporal consistency is directly generated from the fused feature set.

[0042] This application discloses a method, apparatus, device, medium, and product for generating cross-modal spatiotemporal sensing tensors. Through the synergistic effect of an endogenous spatiotemporal graph neural network and a cross-modal attention mechanism, and leveraging the structured modeling capabilities of the graph neural network, heterogeneous modal data is mapped to a unified representation space. Simultaneously, the cross-modal attention mechanism dynamically adjusts the intensity of modal interactions to ensure the causal consistency of cross-modal information across spatiotemporal scales. The endogenous spatiotemporal graph neural network can model the interaction relationships between modalities through graph edges, effectively enhancing its adaptability to dynamic environmental changes, thereby improving the robustness and generalization ability of multimodal fusion. Ultimately, this solves the problem of low generation efficiency of sensing tensors in the prior art.

[0043] Based on the above-mentioned inventive discovery, the technical solution of this application is proposed.

[0044] The following describes the application scenarios of the cross-modal spatiotemporal sensing tensor generation method provided in the embodiments of the present invention. Figure 1 This is a schematic diagram illustrating an application scenario of the method for generating cross-modal spatiotemporal sensing tensors provided in this application embodiment. For example... Figure 1 As shown, this application scenario includes a mobile device 101 and a server 102. The mobile device 101 collects multimodal input data and sends it to the server 102. The server 102 encodes the multimodal input data using a pre-defined endogenous spatiotemporal graph neural network to obtain a graph structure representation. The server 102 then uses a pre-defined cross-modal attention mechanism to perform weighted fusion of the graph structure representation to obtain a cross-modal spatiotemporal perception tensor.

[0045] The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0046] Figure 2 This is one of the flowcharts illustrating the method for generating cross-modal spatiotemporal sensing tensors provided in this application. Figure 2 As shown, in this embodiment, the execution entity of this invention is a server. The method for generating cross-modal spatiotemporal sensing tensors provided in this embodiment includes the following steps:

[0047] S201. Acquire multimodal input data; wherein, the multimodal input data includes visual features, language features, tactile features and ontological state features. The multimodal input data is data collected by multiple sensors and multiple interfaces on the pre-set intelligent agent. The multimodal input data is used to represent the environmental state, posture and received task instructions of the intelligent agent.

[0048] Specifically, multimodal input data can be acquired by synchronously collecting raw signal streams from multiple sensors and data interfaces mounted on the intelligent agent. Sensors can include optical imaging devices, sound acquisition units, force sensing units, and inertial measurement modules, which respectively capture visual images, auditory information, physical contact forces from the external environment, and the intelligent agent's own posture and motion state. Simultaneously, the data interface can receive task commands or high-level semantic information from external sources. This step is used to aggregate the multi-dimensional physical state of the intelligent agent's environment, its own ontological state, and the received task commands into a structured or semi-structured raw data set, providing complete data for subsequent unified encoding and fusion processing.

[0049] S202. The multimodal input data is encoded by a pre-defined endogenous spatiotemporal graph neural network to obtain a graph structure representation. The endogenous spatiotemporal graph neural network includes multiple graph nodes and multiple graph edges. The multiple graph nodes are used to represent the state of the agent, and the multiple graph edges are used to represent the interaction relationship between the multimodal input data. The graph structure representation is the feature carrier obtained by the endogenous spatiotemporal graph neural network encoding the multimodal input data.

[0050] Specifically, by leveraging the structured modeling capabilities of endogenous spatiotemporal graph neural networks, various multimodal input data can be first associated with multiple graph nodes and edges in the network. The graph nodes in the network represent the relevant states of the agent, and the graph edges depict the intrinsic interaction relationships between different types of multimodal input data. Then, through the hierarchical operation logic of the network, the node and edge information associated with these modal data are integrated and deeply encoded layer by layer. This process gradually transforms the multimodal input data, which originally had different formats and spatiotemporal dimensional differences, into a graph structure representation with spatiotemporal correlation characteristics. This step is used to map the originally scattered and heterogeneous multimodal input data to a unified structured representation space, thus constructing a feature carrier with spatiotemporal causal correlation for the subsequent weighted fusion of cross-modal information.

[0051] S203. The graph structure representation is weighted and fused using a pre-defined cross-modal attention mechanism to obtain a cross-modal spatiotemporal perception tensor. The cross-modal spatiotemporal perception tensor is used to provide input for the agent to make endogenous decisions and control motion.

[0052] Specifically, the dynamic weight adjustment capability of cross-modal attention mechanisms can be used to first perform hierarchical analysis on the various modal correlation features carried in the graph structure representation, identifying the core value and correlation degree of different modal information in the current scene. Then, based on the actual needs of the agent's environmental state, its own posture, and the received task instructions, differentiated attention weights are assigned to the modal features corresponding to different nodes and edges in the graph structure. For example, the weights of features strongly related to environmental perception are strengthened, and the priority of features matching task instructions is increased. Subsequently, the graph structure features with differentiated weights are integrated layer by layer, fusing the scattered structured features into a multi-dimensional representation with unified spatiotemporal correlation attributes, resulting in a cross-modal spatiotemporal perception tensor. This step is used to provide accurate input for the agent to make endogenous decisions and control motion. At the same time, the efficiency and relevance of cross-modal information fusion are optimized through dynamic weight adjustment, enhancing the robustness and generalization ability of multimodal fusion, and solving the problems of long generation time and difficulty in meeting the real-time needs of dynamic scenes in existing technologies.

[0053] This embodiment provides a method for generating cross-modal spatiotemporal sensing tensors. Through the synergistic effect of an endogenous spatiotemporal graph neural network and a cross-modal attention mechanism, and leveraging the structured modeling capabilities of the graph neural network, heterogeneous modal data are mapped to a unified representation space. Simultaneously, the cross-modal attention mechanism dynamically adjusts the intensity of modal interactions to ensure the causal consistency of cross-modal information across spatiotemporal scales. Furthermore, the endogenous spatiotemporal graph neural network can model the interaction relationships between modalities through graph edges, effectively enhancing adaptability to dynamic environmental changes, thereby improving the robustness and generalization ability of multimodal fusion. Ultimately, this solves the problem of low generation efficiency of sensing tensors in existing technologies.

[0054] In one possible design, S202 encodes the multimodal input data using a pre-defined endogenous spatiotemporal graph neural network to obtain a graph structure representation, including:

[0055] S2021. Map the multimodal input data to multiple graph nodes and multiple graph edges, and perform information transfer and feature updates on multiple graph nodes and multiple graph edges through the pre-defined aggregation operation of the endogenous spatiotemporal graph neural network to obtain the graph structure representation.

[0056] Specifically, raw data from different sensors and interfaces can be directly assigned to nodes representing the agent's state, as well as to connecting nodes and edges representing data interaction relationships. Then, utilizing the inherent information transmission mechanism of graph neural networks, nodes and edges repeatedly exchange and integrate received neighborhood information, dynamically updating their respective feature representations through multiple iterations, ultimately obtaining a graph structure representation. This step is used to complete spatiotemporal correlation modeling and feature encoding of multimodal data within a unified graph model framework. It avoids the redundant process of independent encoding followed by forced alignment in traditional methods, directly achieving the co-evolution and refinement of features during information transmission.

[0057] For example, based on the type and representational meaning of the multimodal input data, the features of different modalities can be mapped to multiple graph nodes and edges of the endogenous spatiotemporal graph neural network. For instance, visual features reflecting the agent's environmental state, ontological state features reflecting its own posture, and tactile features perceiving external contact can be mapped to different graph nodes, making each graph node a carrier of the corresponding modal features to represent the relevant state of the agent. Then, the interaction relationships between different modal data, such as the correlation between visual and tactile features in environmental perception, and the matching relationship between language features and ontological state features in task execution, can be mapped to the corresponding graph edges, so that the graph edges can characterize the intrinsic connections between modalities. After the mapping is completed, the pre-defined aggregation operation of the endogenous spatiotemporal graph neural network is initiated. This operation first realizes the information transmission between graph nodes and graph edges, allowing each graph node to receive modal interaction information carried by adjacent graph edges. At the same time, each graph edge also receives modal state information of associated graph nodes. Then, based on the pre-defined operation logic, the transmitted node information and edge information are updated in terms of features. For example, the multi-source interaction information received by the node is weighted and integrated to optimize the node features, and the node state information received by the edge is fused and adjusted to improve the associated features of the edge. After multiple rounds of information transmission and feature updates, a graph structure representation is formed.

[0058] The technical effect of this scheme in this embodiment is that by directly mapping multimodal input data to nodes and edges of the graph network and performing aggregation operations, native fusion and collaborative evolution of cross-modal information in a unified graph structure are achieved. This endogenous encoding process avoids the information loss and computational delay caused by first extracting modal isolation features and then performing post-concatenation in traditional methods. It enables the spatiotemporal correlation and interaction constraints between different modalities to be dynamically and consistently modeled through information transmission and feature updates of graph nodes and edges, improving the semantic consistency and structuring degree of the information carried by the perceptual tensor, and providing accurate cross-modal environment representation for subsequent decision-making and control.

[0059] In one possible design, S203 employs a pre-defined cross-modal attention mechanism to perform weighted fusion of the graph structure representation, obtaining a cross-modal spatiotemporal perception tensor, including:

[0060] S2031. Obtain multiple interaction relationship representations from the graph structure representation and calculate the attention weights corresponding to each interaction relationship representation; wherein, each interaction relationship representation is a feature vector generated based on the attributes of each graph edge, and each interaction relationship representation is used to represent the interaction relationship between the graph nodes connected to each graph edge.

[0061] Specifically, feature vectors representing specific interaction patterns between the two connected nodes can be extracted from the constructed graph structure. Based on the global correlation between these feature vectors, an attention mechanism is used to dynamically calculate the importance of the interaction represented by each edge in the overall environmental understanding and task execution, i.e., assigning a learnable weight to each interaction. This step is used to selectively focus on intermodal interactions that are crucial to the current decision from the complex relationship network contained in the graph structure. By evaluating and quantifying the influence of different interaction relationships, it provides a differentiated contribution basis for subsequent fusion, thereby ensuring that the final generated perceptual tensor can highlight key spatiotemporal correlations and suppress irrelevant noise.

[0062] S2032. Based on multiple interaction relationship representations and the attention weights corresponding to each interaction relationship representation, a weighted fusion is performed to obtain a cross-modal spatiotemporal perception tensor.

[0063] Specifically, the feature vector representing the intermodal interaction relationship carried by each edge can be multiplied by its weight coefficient, calculated through an attention mechanism, reflecting the importance of that relationship. All weighted interaction relationship representations are then integrated into a unified multidimensional data structure, forming the final cross-modal spatiotemporal perception tensor. This step is used to selectively fuse dynamic interaction information between all modalities based on their criticality to the current environment and task. Through weighted summation, it transforms the relational network in the graph structure into a compact tensor representation rich in priority information, thereby providing the agent's decision-making and control with perceptual input highlighting core spatiotemporal correlations.

[0064] The technical effect of this scheme in this embodiment is as follows: by extracting feature vectors representing the interaction relationships between nodes from the graph structure representation, and dynamically calculating the attention weights of each interaction relationship based on these vectors for weighted fusion, the scheme achieves refined and adaptive adjustment of the intensity of cross-modal information interaction. This ensures that when generating the perceptual tensor, information with strong causal or constraint relationships between different modalities can obtain higher fusion weights, overcoming the shortcomings of traditional uniform or static fusion methods that ignore the dynamic dependencies between modalities, and improving the accuracy of the perceptual tensor in complex decision-making tasks.

[0065] Figure 3 This is one of the flowcharts illustrating a method for generating cross-modal spatiotemporal sensing tensors provided in an embodiment of this application. In this embodiment, in Figure 2 Based on the provided embodiments, the method for generating cross-modal spatiotemporal sensing tensors is further explained. The method for generating cross-modal spatiotemporal sensing tensors includes:

[0066] S301. Acquire multimodal input data; wherein, the multimodal input data includes visual features, language features, tactile features and ontological state features. The multimodal input data is data collected by multiple sensors and multiple interfaces on the pre-set intelligent agent. The multimodal input data is used to represent the environmental state, posture and received task instructions of the intelligent agent.

[0067] S302. The multimodal input data is encoded by a pre-defined endogenous spatiotemporal graph neural network to obtain a graph structure representation. The endogenous spatiotemporal graph neural network includes multiple graph nodes and multiple graph edges. The multiple graph nodes are used to represent the state of the agent, and the multiple graph edges are used to represent the interaction relationship between the multimodal input data. The graph structure representation is the feature carrier obtained by the endogenous spatiotemporal graph neural network encoding the multimodal input data.

[0068] S303. A pre-defined cross-modal attention mechanism is used to perform weighted fusion on the graph structure representation to obtain a cross-modal spatiotemporal perception tensor; wherein, the cross-modal spatiotemporal perception tensor is used to provide input for the agent to make endogenous decisions and control motion.

[0069] S301-S303 are similar to S201-S203, and will not be described again in this embodiment.

[0070] S304. Embed the cross-modal spatiotemporal perception tensor into a preset embodied cognitive manifold to obtain the cognitive state; wherein, the embodied cognitive manifold is a geometric space.

[0071] Specifically, the feature dimensions of the cross-modal spatiotemporal perception tensor can be adjusted first to match the dimensional parameters of the pre-defined embodied cognitive manifold, eliminating the differences between the two at the dimensional level. Then, based on the geometric properties of the embodied cognitive manifold, such as the topological structure in space, the coordinate distribution pattern, and the cognitive meaning corresponding to different geometric positions, the feature values ​​of each dimension in the cross-modal spatiotemporal perception tensor are precisely mapped to the geometric coordinates in the manifold. This allows the cross-modal spatiotemporal information carried in the tensor to be reflected in the form of geometric positions and distance relationships in the manifold. During the mapping process, it is also necessary to ensure that the spatiotemporal causal relationships in the cross-modal spatiotemporal perception tensor remain consistent in the embodied cognitive manifold, avoiding the information association breakage caused by the embedding process. Finally, through such dimensional adaptation and feature-coordinate mapping operations, the cross-modal spatiotemporal perception tensor is transformed into a cognitive state. This step transforms the abstract cross-modal spatiotemporal perception tensor into a geometrical cognitive state that conforms to the embodied cognitive logic of the agent. This allows the agent's perceptual information to exist in a more intuitive form that better fits its cognitive pattern. At the same time, it provides input data with geometrical operational properties for subsequent multi-objective joint optimization of the cognitive state, ensuring that the subsequent optimization process can be carried out based on a clear spatial representation. This provides reliable cognitive support for generating accurate control input trajectories, making the agent's decision-making and control more aligned with its actual needs for interacting with the environment.

[0072] S305. Perform multi-objective joint optimization on the cognitive state to obtain the control input trajectory; where the control input trajectory refers to the trajectory that the agent needs to move.

[0073] Specifically, multiple constraints such as task completion efficiency, action smoothness, energy consumption, and distance to environmental obstacles can be considered simultaneously within the cognitive manifold space. An optimization algorithm searches for an optimal or feasible path from the current cognitive state to the target state within this geometric space. The continuous evolution of this path on the manifold corresponds to the agent's control input sequence in physical space, i.e., the control input trajectory. This step transforms the agent's high-level task understanding and internal cognitive state into a series of specific, executable action instructions that accommodate multiple performance requirements. By weighing and searching among multi-dimensional objectives, it ensures that the final allocated motion trajectory is not only mathematically optimal but also conforms to the dynamic and safety requirements of the actual physical world.

[0074] S306. Perform operational safety verification on the control input trajectory and obtain the verification results; the verification results include verification passed and verification failed.

[0075] Specifically, before sending the allocated action commands to the actuator, a pre-check can be performed on the command sequence using a set of preset rules or models. This primarily verifies whether the command will collide with known environmental obstacles, whether it exceeds the physical limits of the agent's own movement mechanisms, and whether it complies with specific operational safety regulations. Based on whether the check passes, a corresponding verification result is generated. This step provides a crucial safety assurance for the agent's autonomous actions. By conducting simulation or logical-level safety screening of control commands before final execution, it proactively prevents inappropriate or dangerous actions that may result from perception errors or allocation defects.

[0076] S307. In response to the verification result being successful, the control input trajectory is input into the agent to obtain task feedback and acquire environmental change data; wherein, the environmental change data is used to represent the changes in the environment in which the agent is located before and after the control input trajectory is input into the agent.

[0077] Specifically, after security verification confirms the accuracy of the commands, the sequence of control instructions can be issued to the agent's drive system to execute the corresponding movements and operations. After the agent completes its actions, its internal sensors capture changes in its own state and the alterations in the external environment caused by the operation. Simultaneously, the system receives indication signals regarding the achievement of the task objective. This information collectively constitutes task feedback and environmental change data. This step completes the closed loop from internal decision-making to external action, translating the calculated strategy into real-world physical effects. By collecting information on changes in the environment and task state after the action is executed, data is provided for evaluating the effectiveness of decisions and continuously optimizing perceptual decision-making.

[0078] S308. Update the endogenous spatiotemporal graph neural network, cross-modal attention mechanism, and embodied cognitive manifold based on task feedback and environmental change data to obtain the updated endogenous spatiotemporal graph neural network, the updated cross-modal attention mechanism, and the updated embodied cognitive manifold.

[0079] Specifically, by utilizing information about environmental state changes and task completion status obtained after the agent actually performs an action, an optimization objective reflecting the difference between the action's consequences and expectations can be constructed. Through backpropagation algorithms or related optimization techniques, the feature extraction rules for nodes and edges in the graph neural network, the weight allocation preferences in the attention mechanism, and the geometric parameters of the cognitive manifold space can be adjusted simultaneously, making the entire model more adaptable to the dynamics of the real environment and the demands of the task. This step enables the entire system to adapt and continuously learn. By using empirical data from actual interactions as supervisory signals, it guides iterative optimization of parameters in multiple key aspects of the model, such as perception encoding, information fusion, and state representation, thereby improving the agent's ability to accurately perceive and make efficient decisions in complex dynamic environments.

[0080] The technical effect of this scheme in this embodiment is as follows: by embedding the generated cross-modal spatiotemporal perception tensor into the embodied cognitive manifold to obtain a geometrical cognitive state, and based on this, multi-objective joint optimization and security verification are performed to form an executable control input trajectory, thus constructing a verifiable closed loop from perception to decision-making to execution. This not only ensures the semantic consistency between the decision-making process and multimodal perception, but also provides reliable protection for the physical actions of the agent through the security verification mechanism. Finally, the online update mechanism introduced for task feedback and environmental changes enables the perception, fusion, and decision-making models of the system to continuously evolve with experience, enhancing the environmental adaptability of the agent in long-term operation.

[0081] In one possible design, S304 embeds the cross-modal spatiotemporal perception tensor into a predefined embodied cognitive manifold to obtain cognitive states, including:

[0082] S3041. Obtain multiple constraints; among which, the multiple constraints include geodesic distance constraints, metric tensor learning constraints, and smoothing regularization constraints.

[0083] Specifically, rules for constructing and constraining the geometric properties of the cognitive manifold can be derived from prior knowledge or design specifications such as the agent's physical structure and kinematic model, the safety and efficiency requirements of task execution, and the desired continuity of cognitive states. These rules include, but are not limited to, ensuring the rationality of the shortest path between two points on the manifold, learning an intrinsic metric that accurately reflects the similarity between states, and ensuring that the evolution of states on the manifold is sufficiently smooth. This step provides rigorous geometric and physical guidance for the embedding process from the perceptual tensor to the cognitive state. By introducing these constraints derived from the agent's physical characteristics and the nature of the task, it ensures that the generated cognitive state is not merely a mathematical point, but an effective representation that is compatible with real physical motion and task logic, interpretable, and easily manipulated in subsequent operations.

[0084] Among them, the geodesic distance constraint is a constraint set based on the geometric topological properties of the embodied cognitive manifold. It is used to regulate the spatial distance relationship of the cognitive states formed after cross-modal spatiotemporal perception tensor embedding in the manifold, so that the geodesic distance between different cognitive states matches the degree of correlation between their corresponding agent's actual perceived information. For example, the geodesic distance of cognitive representations related to the same environmental state in the manifold will be constrained to a smaller value, while the distance between cognitive representations with weaker correlation will be relatively larger. This maintains the spatiotemporal causal consistency of perceived information in the manifold and avoids the situation of information correlation misalignment during the embedding process.

[0085] The metric tensor learning constraints are constraints centered around the metric tensor of the embodied cognitive manifold. They require that the learning process of the metric tensor conforms to the geometric operational rules of the manifold itself, while also adapting to the matching requirements between the feature dimensions of the cross-modal spatiotemporal perception tensor and the dimensions of the manifold. Simultaneously, it constrains the metric tensor to maintain a certain consistency with the identity matrix, ensuring that when the perception tensor is embedded, its multimodal features can achieve reasonable dimensional transformation and attribute adaptation. This allows the metric attributes of the manifold to possess both standardized geometric characteristics and conform to the embodied cognitive patterns of the agent, providing a suitable metric foundation for the effective embedding of the perception tensor.

[0086] Smooth regularization constraints are used to regulate the characteristics of cognitive state changes after cross-modal spatiotemporal perception tensor embedding. It requires that the distribution and changes of cognitive states in the embodied cognitive manifold be smooth and continuous, avoiding abrupt changes in the representation of cognitive states in the manifold due to small local fluctuations in multimodal perception information. For example, the transition of cognitive representations in adjacent time periods or similar scenarios in the geometric position of the manifold will be constrained to a smooth form, thereby ensuring stable correlation between cognitive states and adapting to the needs of agents for continuous perception and decision-making in dynamic scenarios.

[0087] S3042. Based on multiple constraints, the cross-modal spatiotemporal perception tensor is embedded into the embodied cognitive manifold to obtain the cognitive state.

[0088] Specifically, during the embedding mapping process, various pre-defined requirements regarding the geometric structure of the cognitive manifold can be considered and satisfied simultaneously. For example, ensuring that the geodesic distance between embedded state points on the manifold reflects the semantic proximity of the task, accurately characterizing the intrinsic differences between states through adaptive learning metrics, and forcing state change paths to have smooth properties to avoid abrupt non-physical actions, thereby mapping the high-dimensional perceptual tensor into a concrete coordinate system on the manifold that conforms to all these geometric and physical constraints. This step transforms the fused perceptual information into an intrinsic state that combines mathematical rigor with physical realizability. Through a multi-constraint-guided embedding process, the cognitive state can compactly represent the environment and task.

[0089] The technical effect of this scheme in this embodiment is as follows: By introducing multiple geometric constraints such as geodesic distance constraints, metric tensor learning constraints, and smoothness regularization constraints, the cross-modal spatiotemporal perception tensor is embedded into the embodied cognitive manifold to obtain the cognitive state. This ensures that the agent's internal cognitive representation not only reflects the complex structure of external multimodal perception but also conforms to the geometric and smoothness priors inherent in physical motion and task logic. This geometric embedding process enables subsequent decision optimization to be carried out in the manifold space that follows these essential constraints, improving the physical feasibility, task adaptability, and behavioral consistency and stability of the generated control trajectory in dynamic environments.

[0090] In one possible design, S305 performs multi-objective joint optimization on the cognitive state to obtain the control input trajectory, including:

[0091] S3051. Construct a target optimization function based on the embodied cognitive manifold; wherein the target optimization function includes a cognitive consistency term, an endogenous energy term, a stability term, and a security constraint term.

[0092] Specifically, based on the geometric characteristics and intrinsic metrics of the cognitive manifold, a comprehensive mathematical function can be designed. This function will consist of multiple sub-terms, which respectively measure the consistency between the allocated trajectory and the cognitive task objective on the manifold, the intrinsic energy consumption required to execute the trajectory, the state stability during the motion process, and whether all known physical safety boundaries are satisfied. This step transforms the agent's high-level cognition into a complex multi-criteria decision problem of specific motion trajectories, formalizing it as a mathematical optimization problem computable within a specific geometric space. The objective function unifies the requirements of task semantics, physical constraints, and motion quality into a single framework, providing clear mathematical standards and search directions for subsequently solving a comprehensively optimal control trajectory.

[0093] The cognitive consistency term is constructed based on the cognitive state association logic of the embodied cognitive manifold. It constrains the agent's current cognitive state to match the reference cognitive state in the corresponding scenario, while ensuring that the cognitive state conforms to structured cognitive rules such as graph Laplace and adjacency consistency. This term is mainly used to ensure that the cognitive state after the transformation of cross-modal perception information conforms to the agent's embodied cognitive logic, avoiding deviations between cognitive representation and actual perception and task requirements, and providing a reliable cognitive foundation for subsequent decision-making and control.

[0094] The endogenous energy term is an energy functional term constructed by combining the system state and control input of the agent. It typically includes state inertia, control consumption, and potential terms, quantifying the energy cost incurred by the agent in maintaining the system state and performing control operations during operation. This term is mainly used to balance the energy consumption during the agent's control process, avoiding excessive energy loss from control operations, while ensuring that control behavior matches the system's own state characteristics, thus achieving a unified characterization of energy, control, and system constraints.

[0095] The stability term is constructed around the changing characteristics of cognitive states within an embodied cognitive manifold. It constrains the distribution and changes of cognitive states within the manifold to remain smooth and continuous, preventing significant jumps in cognitive states due to fluctuations in multimodal perceptual information or adjustments in control operations. This term primarily ensures that the agent's cognitive states maintain stable correlations in dynamic scenarios, enabling continuous and smooth transitions in the perception and decision-making processes, thus meeting the agent's need for continuous interaction with the environment.

[0096] Safety constraints are constraint terms constructed by combining the agent's cognitive state and control input. They are typically quantified through parameterization such as safety zone covariance, target safety state, and gradient terms to quantify the safety compliance of the combination of control operations and the current cognitive state. This term is mainly used to constrain the agent's control behavior within a safe range, avoid generating potentially dangerous control inputs, and ensure that the control trajectory conforms to the safety specifications of the environment and task, thereby guaranteeing the safety of the agent's operation.

[0097] S3052. Using the cognitive state as input, solve the objective optimization function to obtain the control input trajectory.

[0098] Specifically, the agent's current coordinates on the cognitive manifold can be used as the initial state. Guided by a comprehensive objective function that considers cognitive consistency, energy consumption, motion stability, and safety boundaries, a numerical optimization algorithm is used to explore and search for an optimal or feasible path from the current state to the target state within the manifold space. This continuous path on the manifold corresponds to the control input sequence that the agent needs to execute in the physical space. This step is used to find the best action plan in the constrained cognitive geometry space through mathematical optimization, thereby ensuring that the final output control command not only logically conforms to the task intent but also physically meets multiple practical requirements such as efficiency, smoothness, and safety.

[0099] The technical effect of this solution in this embodiment is as follows: a multi-objective optimization function that integrates cognitive consistency, endogenous energy, stability and security constraints is constructed in the embodied cognitive manifold space, and the control input trajectory is generated by solving the solution with the cognitive state as input. This ensures that the generated control trajectory is not only feasible and stable at the kinematic and dynamic levels, but more importantly, it is highly consistent with the agent's cross-modal understanding of the environment in terms of semantics, and strictly embeds verifiable security boundaries, thus overcoming the problems of intention distortion and softening of security constraints caused by module fragmentation in traditional decision-making.

[0100] In one possible design, S308 updates the endogenous spatiotemporal graph neural network, the cross-modal attention mechanism, and the embodied cognitive manifold based on task feedback and environmental change data, resulting in updated endogenous spatiotemporal graph neural network, updated cross-modal attention mechanism, and updated embodied cognitive manifold, including:

[0101] S3081. Using task feedback and environmental change data as online learning samples, the posterior distribution of parameters of the endogenous spatiotemporal graph neural network, cross-modal attention mechanism and embodied cognitive manifold is calculated through a variational derivation framework.

[0102] Specifically, effective learning signals can be extracted from the results obtained after each action of the agent. This real-time data is considered valuable experience guiding model evolution. Using a probabilistic learning framework capable of handling uncertainty, the possible state ranges of various parameters within the model after observing these new facts are analyzed, thereby calculating the updated probability distribution of these parameters. This step enables the entire system to autonomously learn and self-improve from continuous interaction. By transforming actual execution consequences into a basis for correcting the model's internal structure, the agent can dynamically adjust its understanding of the environment, information fusion, and state representation, thus becoming more adaptable to complex and ever-changing task scenarios.

[0103] The posterior distribution of parameters describes the updated probabilistic perception of the likelihood of unknown parameters in a model after some actual data or evidence has been observed. Essentially, it combines prior guesses about the parameters with currently observed facts using Bayes' theorem to arrive at a more accurate and realistic probabilistic judgment about the parameters.

[0104] For example, suppose we are studying the relationship between the growth rate of a plant and the duration of light exposure. Before starting the experiment, based on common sense, we have a preliminary conjecture: the longer the light exposure, the faster the growth may be, but the specific coefficient of the relationship is uncertain. This preliminary conjecture is the prior distribution.

[0105] Then, an experiment was conducted for a week, recording the changes in plant height under different light durations each day, obtaining a set of real observational data. Based on this week's experimental data, the parameter of the influence coefficient of light duration on growth rate was recalculated and evaluated.

[0106] The calculated new probability distribution, which incorporates the experimental data, is the parametric posterior distribution. It is more reliable than pre-experimental guesses, not only telling us the most likely value of the influence coefficient but also providing a confidence range for this estimate; for example, the coefficient is likely to be between 0.5 and 0.8. Subsequent analysis and predictions will be based on this updated and more reliable posterior distribution.

[0107] S3082. Based on the posterior distribution of parameters, update the graph convolution weights of the endogenous spatiotemporal graph neural network, the attention weights of the cross-modal attention mechanism, and the metric tensor parameters of the embodied cognitive manifold to obtain the updated endogenous spatiotemporal graph neural network, the updated cross-modal attention mechanism, and the updated embodied cognitive manifold.

[0108] Specifically, based on the parameter distribution guidelines provided by the probabilistic framework, we can adjust the weights that control the transmission and integration of information between nodes in the control graph, adjust the attention values ​​that determine the degree of attention given to different modalities, and modify the metric parameters that shape the geometry of the cognitive space, thereby completing the actual numerical updates of the core components of the model. This step is used to transform the theoretical learning results into real improvements in model performance. By systematically adjusting the key settings within the model based on new experiences, the agent will perceive more acutely, make more rational decisions, and act more effectively when faced with similar situations in the future, thus achieving incremental optimization and continuous adaptation of the entire system.

[0109] For example, in a continuous learning scenario for a home service robot, after the robot completes a task of delivering a water cup, it collects information on scene changes observed by the camera, force feedback from the arm sensors, and signals indicating whether the task was successful. This data is then fed into an online learning module. Based on the specific interaction results, this module calculates the adjustment direction for key parameters in the model. For instance, it slightly increases the weight of the connection edges between the visual and tactile sensor nodes in the graph neural network, because the successful grasp confirms the strong correlation between the visually recognized position of the water cup and the tactile feedback of the grip strength. Simultaneously, it increases the attention given to the proprioceptive posture modality in its cross-modal attention mechanism, as body balance during movement is crucial for task completion. Furthermore, it fine-tunes the measurement of the handheld object's movement state region in the cognitive manifold parameters, making this state closer to the stable standing state on the manifold, reflecting that the robot has mastered the skill. After these subtle adjustments based on single-experience, the model is updated, and the robot's perception fusion and decision-making planning will be more accurate and efficient the next time it performs a similar task.

[0110] The technical effect of this solution in this embodiment is as follows: by using task execution feedback and real-time environmental change data as online learning samples, and using a variational derivation framework to calculate the posterior distribution of the core model parameters of the system for updating, the endogenous spatiotemporal graph neural network, cross-modal attention mechanism, and embodied cognitive manifold can dynamically adjust their internal parameters in a way that quantifies uncertainty based on the real interaction experience between the agent and the environment. This realizes the transformation of the system from a static pre-trained model to an adaptive online learning agent, and enhances the robustness and learning ability of the agent in dealing with non-stationary environments, unknown disturbances, and task scenario migrations during long-term operation.

[0111] Figure 4 This is one of the flowcharts illustrating a method for generating cross-modal spatiotemporal sensing tensors provided in an embodiment of this application. In this embodiment, in Figure 3 Based on the provided embodiments, the method for generating cross-modal spatiotemporal sensing tensors is further explained. The method for generating cross-modal spatiotemporal sensing tensors includes:

[0112] S401. Embed the cross-modal spatiotemporal perception tensor into a preset embodied cognitive manifold to obtain the cognitive state; wherein, the embodied cognitive manifold is a geometric space.

[0113] S402. Perform multi-objective joint optimization on the cognitive state to obtain the control input trajectory; where the control input trajectory refers to the trajectory that the agent needs to move.

[0114] S403. Perform operational safety verification on the control input trajectory and obtain the verification results; the verification results include verification passed and verification failed.

[0115] S401-S403 are similar to S304-S306, and will not be described again in this embodiment.

[0116] S404. In response to the verification result being that the verification failed, the multimodal input data is re-encoded according to the endogenous spatiotemporal graph neural network to obtain the re-encoded graph structure representation.

[0117] Specifically, after identifying potential collisions or violations during the security verification process, an adjustment process can be initiated. This process guides the graph neural network to pay different attention to or interpret specific environmental factors or modal interactions that led to the problem, based on clues implied by the current verification failure. This allows for a new round of feature encoding and relational modeling of the original multimodal input data, forming a new graph structure representation. This step provides the system with a self-correction and re-decision-making mechanism when faced with security verification failures. It dynamically adjusts the focus and method of perceptual encoding using verification feedback information to ensure that a safer action plan is subsequently generated.

[0118] S405. The re-encoded graph structure representation is weighted and fused through a cross-modal attention mechanism to obtain a re-fused cross-modal spatiotemporal perception tensor. The re-fused cross-modal spatiotemporal perception tensor is used to regenerate the control input trajectory and perform security verification until the verification result is successful or the number of verifications reaches a preset threshold.

[0119] Specifically, based on the newly generated and adjusted graph structure, the attention mechanism can be used again to calculate the importance weights of the interaction relationships represented by each edge in the graph. These new weights are then used to weight and integrate the relationship representations, resulting in a revised cross-modal spatiotemporal perception tensor. This step is used to perform a new round of information focusing and fusion on the adjusted perception understanding. Based on the experience of previous failed adjustments, it selectively strengthens or weakens the contribution of certain intermodal interaction information to the final perception representation, thus providing a targeted and optimized perception input at the information fusion level for regenerating the control trajectory.

[0120] S406. In response to the verification result being successful, the control input trajectory is input into the agent to obtain task feedback and acquire environmental change data; wherein, the environmental change data is used to represent the changes in the environment in which the agent is located before and after the control input trajectory is input into the agent.

[0121] S407. Update the endogenous spatiotemporal graph neural network, cross-modal attention mechanism, and embodied cognitive manifold based on task feedback and environmental change data to obtain the updated endogenous spatiotemporal graph neural network, the updated cross-modal attention mechanism, and the updated embodied cognitive manifold.

[0122] S406-S407 are similar to S307-S308, and will not be described again in this embodiment.

[0123] The technical effect of this solution in this embodiment is as follows: by triggering a re-encoding and fusion process of multimodal input data when the control input trajectory security verification fails, and iteratively generating new perception tensors and candidate control trajectories, an online security self-correction mechanism based on a perception-decision closed loop is provided for the agent. This mechanism ensures that when the system faces dynamic environmental disturbances, uncertainties, or initial decision conflicts, it can proactively trace back to the perception source, use updated environmental information and internal models to adjust its strategy, thereby dynamically exploring action plans that meet strict security constraints and enhancing the system's fault tolerance in complex and unstructured scenarios.

[0124] It should be noted that:

[0125] The existing technology still has the following problems:

[0126] 1. Insufficient decision-making autonomy: Many solutions rely on preset rules or policy networks trained offline, which are difficult to maintain stable online endogenous decision-making capabilities when faced with external disturbances in the environment and changes in tasks.

[0127] 2. Shallow cross-modal fusion: Common practices include feature splicing or post-fusion, which makes it difficult to establish deep endogenous relationships between modalities with causal and constraint consistency on a spatiotemporal scale, resulting in limited accuracy in understanding complex semantic instructions and operational scenarios.

[0128] 3. Weak environmental adaptability and evolution: When the environment and task change, there is a lack of variational and Bayesian evolutionary mechanisms that can update historical knowledge and online experience in a unified manner, and the strategy is prone to degradation.

[0129] 4. Insufficient security guarantees: Human-machine collaboration and obstacle avoidance are mostly implemented through constraints, penalties, or heuristic rules, making it difficult to provide provable security boundaries and real-time verification mechanisms.

[0130] The purpose of this application is:

[0131] 1. Establish cross-modal consistent representations and constraint targets for multi-source information such as vision, language, and touch to improve the accuracy and consistency of understanding instruction semantics and operational scenarios.

[0132] 2. A decision-making mechanism is proposed that can adaptively adjust according to changes in environment and task at runtime, thereby improving online stability and robustness to distributed external disturbances.

[0133] 3. Construct quantifiable and verifiable safety constraints and online detection and intervention mechanisms to provide clear safety boundaries and real-time verification capabilities for human-machine collaboration and obstacle avoidance.

[0134] 4. Form an integrated closed-loop modeling objective of perception-decision-control, reduce the accumulation of cascaded errors through global consistency constraints, and improve the overall consistency and controllability of the system.

[0135] 5. Addressing common issues (weak adaptability and evolutionary capabilities): Achieve a unified goal of updating historical knowledge and online experience to ensure performance and stability during long-term operation and scenario migration.

[0136] This application also provides a cross-modal cognitive-driven embodied intelligent endogenous decision-making system, which is divided into five layers from top to bottom according to function and interconnected through closed-loop feedback: perception layer - fusion layer - cognition layer - decision-making layer - control layer. Figure 5 The system architecture diagram provided for the embodiments of this application is as follows: Figure 5 As shown, the relationships and functions of each layer are as follows:

[0137] Perception layer: Acquires multimodal inputs such as vision (V), language (L), action, ontology (A), and touch (H), and outputs basic features for fusion.

[0138] Fusion Layer: Multimodal processing is encoded using an endogenous spatiotemporal graph neural network (where spatiotemporal graph nodes represent agent states and edges represent endogenous associations), and intermodal interactions are enhanced using a cross-modal attention mechanism to obtain a unified cross-modal spatiotemporal perception tensor. The formula is:

[0139]

[0140] in, For cross-modal spatiotemporal sensing tensors, For visual frames / visual features, Features of the subject / action As a linguistic feature, Let F be the tactile feature, C be the joint mapping function, H be the height in the spatial (or spatiotemporal) dimension, and W be the width in the spatial (or spatiotemporal) dimension.

[0141] Cognitive layer: through embodied cognitive manifold mapping : Embedding the perceptual tensor into an optimizable Riemannian manifold, Embedded onto the cognitive manifold, we obtain the cognitive state. .

[0142] Decision-making level: Establish a comprehensive optimization objective on the cognitive manifold, and jointly consider the loss of cognitive consistency. Endogenous energy functional Stability regularization With safety constraints To optimize control input .

[0143] Control layer: Includes real-time security verification and action execution modules, which realize runtime verification, execution and adaptive evolution update of generated decisions, forming a feedback loop from the control layer back to the perception layer.

[0144] The purpose of the closed-loop system design is to maintain the robustness and generalization ability of endogenous decisions in dynamic environments and to provide provable security guarantees.

[0145] The implementation flow of the method:

[0146] Composed of five steps, S1-S5, it can run in real time on a conventional industrial computing platform. Figure 6 The agent decision-making flowchart provided in the embodiments of this application is as follows: Figure 6 As shown, the process includes:

[0147] S1 Cross-Modal Sensing and Dynamic Mapping (Input Conditions and Output): Input is visual frame / visual features. Ontology / Motion Features linguistic features tactile characteristics Adaptive multimodal fusion is employed on these inputs to construct a cross-modal spatiotemporal sensing tensor. Its spatiotemporal graph structure.

[0148] S2 Embedded Cognitive Manifold Embedding (Mapping and Target): Utilizing Deep Manifold Learning to... Embedded into cognitive manifold Minimize the endogenous manifold learning objective This includes constraints on geodesic distance, consistency and smoothness regularization, and the metric tensor G:

[0149]

[0150] in, For the parameters of the embedding mapping function, G is the metric tensor corresponding to the embodied cognitive manifold. For the endogenous manifold learning objective, For the squared term of the geodesic distance on the embodied cognitive manifold, For the metric properties corresponding to the manifold itself, The cognitive state obtained after embedding the i-th sample is... This represents the reference cognitive state corresponding to the sample. It is the regularization coefficient corresponding to the tensor consistency constraint. represents the regularization coefficient corresponding to the smoothness constraint of the cognitive state. For embedding mapping functions Regarding the cognitive state of the i-th sample The gradient.

[0151] S3 Endogenous Decision Making and Global Optimization (Technical Issues and Roles): In By combining the upper and lower equations, the control input trajectory is obtained. :

[0152]

[0153] in, To control the input trajectory, To determine the starting time corresponding to the optimization process, To optimize the termination time corresponding to the process, The cognitive loss term, which includes graph Laplace, adjacency consistency, and sparsity regularization terms, is used to constrain the structured representation characteristics of cognitive states, ensuring that they conform to the agent's embodied cognitive logic. It is the energy functional of the control energy system, including state inertia terms, control consumption and potential terms, etc. It is a safety regularization term used to constrain the combination of the cognitive state and the corresponding control input at time t to comply with safety specifications and avoid generating dangerous control behaviors.

[0154] S4 Real-time Security Verification and Decision Execution (Relationships and Conditions): Based on the S3 generation... Perform runtime safety verification. If the safety barrier and dynamic constraints are met, the program is executed; otherwise, it enters the reconfiguration branch, providing verifiable safety boundaries and real-time intervention for human-machine collaboration / obstacle avoidance.

[0155] S5 Adaptive Adjustment and Evolutionary Optimization (Feedback and Update): Based on task feedback and environmental changes, parameters and strategies are updated online using variational derivation; a closed-loop feedback path from execution to cognition / fusion is formed to ensure performance maintenance and knowledge evolution during long-term operation.

[0156] For example, in an intelligent assembly system for an industrial robotic arm, a 6-axis industrial robotic arm is configured, equipped with vision, force, touch, and body sensors. It accepts natural language work orders / rule-based instructions. The implementation environment is an industrial assembly line; environmental materials present obstructions and pose disturbances. The control steps are:

[0157] S1: Acquire RGB-D visual and end-effector force sensations, and combine them with joint status and verbal descriptions to form... .

[0158] S2: Through get It also adaptively learns the metric tensor G.

[0159] S3: In The combined goal is minimized to generate desired force and pose control. And it imposes timing constraints on skills such as grabbing / plugging.

[0160] S4: Real-time safety verification (including speed / torque / human-machine distance thresholds and barriers), which is executed by the controller if it passes; otherwise, it will revert to readjustment.

[0161] S5: Based on task success rate and process deviation, variational updates are performed, and the next batch of jobs automatically adapts.

[0162] This application constructs a unified high-dimensional cross-modal spatiotemporal perception tensor based on vision, language, action, and touch, and encodes it using an endogenous spatiotemporal graph neural network. Graph nodes represent embodied states, and edges represent endogenous associations. Cross-modal attention is used to enhance modal interaction, thereby achieving deep semantic fusion and structurally consistent expression. This design improves the accuracy of understanding complex instructions and scene perception.

[0163] Embodied cognitive manifold mapping and endogenous manifold learning objectives: Mapped to an optimizable embodied cognitive manifold The system is trained using an endogenous manifold learning objective consisting of geodesic distance, metric tensor G, and smoothness regularization, resulting in a stable and geometrically interpretable cognitive state. This key point provides a rigorous geometric foundation, enhancing the robustness and consistency of the representation.

[0164] Unified optimization of cognitive consistency, endogenous energy, and stability / security: establishing a single comprehensive objective on the manifold. It explicitly defines the graph Laplace / adjacency consistency term and the endogenous energy functional (including the state-dependent inertia matrix M(x), control weights R(x), and obstacle term). This unified optimization combines perception, cognition, and control, suppresses cascade errors, and obtains globally consistent control decisions.

[0165] Runtime verification of higher-order endogenous security barrier functions: Proposing higher-order endogenous barrier functions and their costs The integral expression enables real-time verifiable security verification before decision issuance. Compared with common heuristic penalties, it can provide measurable security boundaries, making it suitable for security-critical scenarios such as human-machine collaboration.

[0166] Variational derivation-driven online adaptation and memory-preserving evolution: During execution, parameters are updated a posteriori based on variational derivation, and parameter importance is introduced to achieve robust migration and continuous performance improvement under long-term operation.

[0167] This application covers the entire process from cross-modal spatiotemporal tensor construction, cognitive manifold mapping, unified objective optimization, runtime security verification to variational online evolution. It also clarifies that cognitive consistency, intrinsic energy, stability, and intrinsic security are four types of constraints that are parallel constraints or regularization relationships within the unified objective function. The detailed fusion graph structure section defines the semantics of nodes and edges in the intrinsic spatiotemporal graph encoding, as well as the calculation method and projection matrix of intrinsic attention weights, while emphasizing the role of attention vectors to highlight the strong constraints of cross-modal interactions. The detailed manifold learning objective specifies... The parallel weight structure of geodesic distance, metric tensor learning, and smoothness regularization is used to obtain a stable and geometrically interpretable cognitive representation by combining these three terms. The refinement of the endogenous energy functional gives the position and state dependencies of the inertia matrix and control weights, and clarifies their roles in the optimal control integral, thereby reflecting the unified characterization of energy, control, and constraints. The refinement of the safety barrier and criterion section limits the integral form of the safety term and the parameterized forms of safety zone covariance, target safety state, gradient term, etc., highlighting the verifiable safety realization mechanism.

[0168] Figure 7 This is a schematic diagram of the structure of the data query device provided in an embodiment of this application. Figure 7 As shown, the data query device includes:

[0169] The first acquisition module 701 is used to acquire multimodal input data; wherein, the multimodal input data includes visual features, language features, tactile features and ontological state features, and the multimodal input data is data collected by multiple sensors and multiple interfaces on a preset intelligent agent, and the multimodal input data is used to represent the environmental state, posture and received task instructions of the intelligent agent.

[0170] The first encoding module 702 is used to encode multimodal input data through a preset endogenous spatiotemporal graph neural network to obtain a graph structure representation. The endogenous spatiotemporal graph neural network includes multiple graph nodes and multiple graph edges. The multiple graph nodes are used to represent the state of the agent, and the multiple graph edges are used to represent the interaction relationship between the multimodal input data. The graph structure representation is a feature carrier obtained by encoding the multimodal input data by the endogenous spatiotemporal graph neural network.

[0171] The first fusion module 703 is used to perform weighted fusion of the graph structure representation using a preset cross-modal attention mechanism to obtain a cross-modal spatiotemporal perception tensor; wherein, the cross-modal spatiotemporal perception tensor is used to provide input for the agent to make endogenous decisions and motion control.

[0172] In one possible design, the first encoding module 702 includes:

[0173] The first input unit is used to map multimodal input data to multiple graph nodes and multiple graph edges, and to perform information transmission and feature updates on multiple graph nodes and multiple graph edges through the pre-defined aggregation operation of the endogenous spatiotemporal graph neural network to obtain a graph structure representation.

[0174] In one possible design, the first fusion module 703 includes:

[0175] The first acquisition unit is used to acquire multiple interaction relationship representations from the graph structure representation and calculate the attention weights corresponding to each interaction relationship representation; wherein, each interaction relationship representation is a feature vector generated based on the attributes of each graph edge, and each interaction relationship representation is used to represent the interaction relationship between the graph nodes connected to each graph edge.

[0176] The first fusion unit is used to perform weighted fusion based on multiple interaction relationship representations and the attention weights corresponding to each interaction relationship representation to obtain a cross-modal spatiotemporal perception tensor.

[0177] In one possible design, the device for generating cross-modal spatiotemporal sensing tensors also includes:

[0178] The embedding module is used to embed cross-modal spatiotemporal perception tensors into a preset embodied cognitive manifold to obtain cognitive states; wherein, the embodied cognitive manifold is a geometric space.

[0179] The optimization module is used to perform multi-objective joint optimization of the cognitive state to obtain the control input trajectory; where the control input trajectory refers to the trajectory that the agent needs to move.

[0180] The verification module is used to perform operational safety verification on the control input trajectory and obtain verification results; the verification results include verification passed and verification failed.

[0181] The second input module is used to input the control input trajectory into the agent in response to the verification result being successful, obtain task feedback, and acquire environmental change data; wherein, the environmental change data is used to represent the changes in the environment in which the agent is located before and after the control input trajectory is input into the agent.

[0182] The update module is used to update the endogenous spatiotemporal graph neural network, cross-modal attention mechanism, and embodied cognitive manifold based on task feedback and environmental change data, resulting in updated endogenous spatiotemporal graph neural network, updated cross-modal attention mechanism, and updated embodied cognitive manifold.

[0183] In one possible design, the embedded module includes:

[0184] The second acquisition unit is used to acquire multiple constraints, including geodesic distance constraints, metric tensor learning constraints, and smoothing regularization constraints.

[0185] The embedding unit is used to embed cross-modal spatiotemporal perception tensors into embodied cognitive manifolds based on multiple constraints to obtain cognitive states.

[0186] In one possible design, the device for generating cross-modal spatiotemporal sensing tensors also includes:

[0187] The re-encoding module is used to re-encode the multimodal input data according to the endogenous spatiotemporal graph neural network in response to the verification result being that the verification failed, so as to obtain the re-encoded graph structure representation.

[0188] The re-fusion module is used to perform weighted fusion of the re-encoded graph structure representation through a cross-modal attention mechanism to obtain a re-fused cross-modal spatiotemporal perception tensor. The re-fused cross-modal spatiotemporal perception tensor is used to regenerate the control input trajectory and perform security verification until the verification result is successful or the number of verifications reaches a preset threshold.

[0189] In one possible design, the optimization module includes:

[0190] The function construction unit is used to construct a target optimization function based on the embodied cognitive manifold; wherein the target optimization function includes a cognitive consistency term, an endogenous energy term, a stability term, and a security constraint term.

[0191] The solver unit is used to solve the objective optimization function with the cognitive state as input to obtain the control input trajectory.

[0192] In one possible design, the update module includes:

[0193] The computational unit is used to use task feedback and environmental change data as online learning samples to compute the posterior distribution of parameters of endogenous spatiotemporal graph neural networks, cross-modal attention mechanisms, and embodied cognitive manifolds through a variational derivation framework.

[0194] The update unit is used to update the graph convolution weights of the endogenous spatiotemporal graph neural network, the attention weights of the cross-modal attention mechanism, and the metric tensor parameters of the embodied cognitive manifold based on the posterior distribution of the parameters, so as to obtain the updated endogenous spatiotemporal graph neural network, the updated cross-modal attention mechanism, and the updated embodied cognitive manifold.

[0195] The cross-modal spatiotemporal sensing tensor generation device provided in this embodiment can execute... Figure 2 , Figure 3 and Figure 4 The technical solution of the embodiment of the method for generating cross-modal spatiotemporal sensing tensors is shown, and its implementation principle and technical effects are similar to those of the present invention. Figure 2 , Figure 3 and Figure 4 The embodiment of the method for generating cross-modal spatiotemporal sensing tensors shown is similar and will not be described in detail here.

[0196] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device includes at least one processor 810 and a memory 820. The electronic device also includes a communication component 830. The processor 810, memory 820, and communication component 830 are connected via a bus 840.

[0197] In the specific implementation process, at least one processor 810 executes computer execution instructions stored in memory 820, so that at least one processor 810 is used to implement a method for generating cross-modal spatiotemporal sensing tensors according to the above embodiment.

[0198] The specific implementation process of processor 810 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0199] In the above embodiments, it should be understood that the processor 810 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0200] The memory 820 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage.

[0201] Bus 840 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 840 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 840 in the accompanying drawings of this application is not limited to only one bus or one type of bus.

[0202] The above description of the functions implemented by electronic devices and main control devices has introduced the solutions provided by the embodiments of the present invention. It is understood that, in order to implement the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0203] This application also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these instructions are used to implement a method for generating a cross-modal spatiotemporal sensing tensor as described in the above embodiments. In the specific implementation of the aforementioned method for generating a cross-modal spatiotemporal sensing tensor, each module can be implemented as a processor.

[0204] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0205] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0206] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement a method for generating a cross-modal spatiotemporal sensing tensor as described in the above embodiments.

[0207] The computer program is stored in a readable storage medium, and at least one processor can read the computer program from the readable storage medium and execute the computer program to perform the scheme provided in any of the above embodiments.

[0208] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0209] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating cross-modal spatiotemporal sensing tensors, characterized in that, include: Acquire multimodal input data; wherein, the multimodal input data includes visual features, language features, tactile features and ontological state features, the multimodal input data is data collected by multiple sensors and multiple interfaces on a preset intelligent agent, and the multimodal input data is used to represent the environmental state, posture and received task instructions of the intelligent agent; The multimodal input data is encoded using a pre-defined endogenous spatiotemporal graph neural network to obtain a graph structure representation. The endogenous spatiotemporal graph neural network includes multiple graph nodes and multiple graph edges. The multiple graph nodes represent the state of the agent, and the multiple graph edges represent the interaction relationships between the multimodal input data. The graph structure representation is a feature carrier obtained by encoding the multimodal input data using the endogenous spatiotemporal graph neural network. The graph structure representation is weighted and fused using a pre-defined cross-modal attention mechanism to obtain a cross-modal spatiotemporal perception tensor; wherein the cross-modal spatiotemporal perception tensor is used to provide input for the agent to make endogenous decisions and control motion.

2. The method for generating cross-modal spatiotemporal sensing tensors according to claim 1, characterized in that, The process of encoding the multimodal input data using a pre-defined endogenous spatiotemporal graph neural network to obtain a graph structure representation includes: The multimodal input data is mapped to the multiple graph nodes and multiple graph edges, and information is transferred and features are updated for the multiple graph nodes and multiple graph edges through the preset aggregation operation of the endogenous spatiotemporal graph neural network to obtain the graph structure representation.

3. The method for generating cross-modal spatiotemporal sensing tensors according to claim 1, characterized in that, The step of employing a pre-defined cross-modal attention mechanism to perform weighted fusion of the graph structure representation to obtain a cross-modal spatiotemporal perception tensor includes: Multiple interaction relationship representations are obtained from the graph structure representation, and attention weights corresponding to each interaction relationship representation are calculated; wherein, each interaction relationship representation is a feature vector generated based on the attributes of each graph edge, and each interaction relationship representation is used to represent the interaction relationship between graph nodes connected to each graph edge; The cross-modal spatiotemporal perception tensor is obtained by weighted fusion of the multiple interaction relationship representations and the attention weights corresponding to each interaction relationship representation.

4. The method for generating cross-modal spatiotemporal sensing tensors according to claim 1, characterized in that, After weighted fusion of the graph structure representation using a preset cross-modal attention mechanism to obtain the cross-modal spatiotemporal perception tensor, the method further includes: The cross-modal spatiotemporal perception tensor is embedded into a preset embodied cognitive manifold to obtain a cognitive state; wherein, the embodied cognitive manifold is a geometric space; The cognitive state is subjected to multi-objective joint optimization to obtain the control input trajectory; wherein, the control input trajectory refers to the trajectory that the agent needs to move along. The control input trajectory is subjected to operational safety verification to obtain verification results; wherein, the verification results include verification passed and verification failed. In response to the verification result being that the verification is passed, the control input trajectory is input to the agent to obtain task feedback and acquire environmental change data; wherein, the environmental change data is used to represent the changes in the environment in which the agent is located before and after the control input trajectory is input to the agent; The endogenous spatiotemporal graph neural network, the cross-modal attention mechanism, and the embodied cognitive manifold are updated based on the task feedback and the environmental change data to obtain the updated endogenous spatiotemporal graph neural network, the updated cross-modal attention mechanism, and the updated embodied cognitive manifold.

5. The method for generating cross-modal spatiotemporal sensing tensors according to claim 4, characterized in that, The step of embedding the cross-modal spatiotemporal perception tensor into a preset embodied cognitive manifold to obtain a cognitive state includes: Multiple constraints are obtained; wherein, the multiple constraints include geodesic distance constraints, metric tensor learning constraints, and smoothing regularization constraints. Based on the aforementioned multiple constraints, the cross-modal spatiotemporal perception tensor is embedded into the embodied cognitive manifold to obtain the cognitive state.

6. The method for generating cross-modal spatiotemporal sensing tensors according to claim 4, characterized in that, After performing operational safety verification on the control input trajectory and obtaining the verification result, the process further includes: In response to the verification result being that the verification failed, the multimodal input data is re-encoded according to the endogenous spatiotemporal graph neural network to obtain a re-encoded graph structure representation; The re-encoded graph structure representation is weighted and fused using the cross-modal attention mechanism to obtain a re-fused cross-modal spatiotemporal perception tensor. The re-fused cross-modal spatiotemporal perception tensor is used to regenerate the control input trajectory and perform security verification until the verification result is successful or the number of verifications reaches a preset threshold.

7. The method for generating cross-modal spatiotemporal sensing tensors according to claim 4, characterized in that, The multi-objective joint optimization of the cognitive state to obtain the control input trajectory includes: Based on the embodied cognitive manifold, a target optimization function is constructed; wherein the target optimization function includes a cognitive consistency term, an endogenous energy term, a stability term, and a security constraint term; Using the cognitive state as input, the objective optimization function is solved to obtain the control input trajectory.

8. The method for generating cross-modal spatiotemporal sensing tensors according to claim 4, characterized in that, The step of updating the endogenous spatiotemporal graph neural network, the cross-modal attention mechanism, and the embodied cognitive manifold based on the task feedback and the environmental change data, to obtain updated endogenous spatiotemporal graph neural network, updated cross-modal attention mechanism, and updated embodied cognitive manifold, includes: Using the task feedback and the environmental change data as online learning samples, the posterior distributions of the parameters of the endogenous spatiotemporal graph neural network, the cross-modal attention mechanism, and the embodied cognitive manifold are calculated through a variational derivation framework. Based on the posterior distribution of the parameters, the graph convolution weights of the endogenous spatiotemporal graph neural network, the attention weights of the cross-modal attention mechanism, and the metric tensor parameters of the embodied cognitive manifold are updated to obtain the updated endogenous spatiotemporal graph neural network, the updated cross-modal attention mechanism, and the updated embodied cognitive manifold.

9. A device for generating cross-modal spatiotemporal sensing tensors, characterized in that, include: The first acquisition module is used to acquire multimodal input data; wherein, the multimodal input data includes visual features, language features, tactile features and ontological state features, the multimodal input data is data collected by multiple sensors and multiple interfaces on a preset intelligent agent, and the multimodal input data is used to represent the environmental state, posture and received task instructions of the intelligent agent; The first encoding module is used to encode the multimodal input data through a preset endogenous spatiotemporal graph neural network to obtain a graph structure representation; wherein, the endogenous spatiotemporal graph neural network includes multiple graph nodes and multiple graph edges, the multiple graph nodes are used to represent the state of the agent, the multiple graph edges are used to represent the interaction relationship between the multimodal input data, and the graph structure representation is a feature carrier obtained by the endogenous spatiotemporal graph neural network encoding the multimodal input data; The first fusion module is used to perform weighted fusion of the graph structure representation using a preset cross-modal attention mechanism to obtain a cross-modal spatiotemporal perception tensor; wherein the cross-modal spatiotemporal perception tensor is used to provide input for the agent to make endogenous decisions and control motion.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; When the processor executes the computer execution instructions stored in the memory, it is used to implement the method for generating cross-modal spatiotemporal sensing tensors as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for generating cross-modal spatiotemporal sensing tensors as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The method includes a computer program, which, when executed by a processor, is used to implement the method for generating cross-modal spatiotemporal sensing tensors as described in any one of claims 1 to 8.

Images