Training method, inference method and system of perception-cognition-control multi-modal large model based on riemannian manifold

Abstract

The application provides a training method, an inference method and a system of a perception-cognition-control multi-modal large model based on a Riemannian manifold. The method comprises the following steps: obtaining Euclidean features corresponding to multi-modal training samples, and inputting the Euclidean features into a perception-cognition-control multi-modal large model to be trained; obtaining feature point representations corresponding to the multi-modal training samples through a multi-modal mapping alignment module in the large model, and determining a multi-modal alignment loss based on the similarity between the feature point representations; taking the feature point representations corresponding to first modal training samples and second modal training positive samples as inputs of a cognition module, and taking the output of the cognition module as an input of a control module; determining a spatial consistency loss based on the input and output of the cognition module and the output of the control module; and iteratively updating based on multiple loss terms to obtain a logically integrated multi-modal large model. The application can overcome the geometric mismatch defect of the Euclidean space.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method, inference method, and system for a large multimodal model of perception-cognition-control based on Riemannian manifolds. Background Technology

[0002] With the deep integration of artificial intelligence and the real economy, fields such as autonomous driving, smart cockpits, and embodied robotics are at a critical stage of transitioning from "weak artificial intelligence" to "general embodied intelligence." In these complex physical interaction scenarios, intelligent agents need to act as complete intelligent behavioral units, utilizing input environmental perception data and reasoning through internal models according to the "Sensation-Cognition-Control" paradigm to derive physical actions. Therefore, intelligent agents face extremely challenging task requirements: they must process massive amounts of unstructured perception data collected by data sensing devices such as LiDAR, cameras, and inertial measurement units (IMUs), understand natural language commands from human users for cognitive reasoning, and ultimately output high-precision, high-stability control signals to drive the movement of vehicles or robotic arms.

[0003] To ensure that intelligent agents can fulfill the aforementioned task requirements, the Vision-Language-Action (VLA) model has emerged as the core carrier of the "perception-cognition-control" paradigm. Representative approaches of current VLA models executing the "perception-cognition-control" paradigm include: RT-1 (Reference 1), which focuses on multi-task sequence modeling; RT-2 (Reference 2), which transfers internet-scale visual language knowledge to robot control; PaLM-E (Reference 3), which injects multimodal observations into a large language model to achieve embodied reasoning; VIMA (Reference 4), which uses multimodal prompts to uniformly express tasks and train operational strategies; PerAct (Reference 6), which focuses on voxelized observations and discretized 6-DoF (Reference 5) and actions; and Diffusion Policy, which models continuous action distributions as a diffusion process.

[0004] However, the current mainstream approach to applying VLA models to the "perception-cognition-control" paradigm has significant drawbacks: ① Although it follows the "vision + language → action" process of the VLA task and has made progress in end-to-end learning, its cross-modal alignment and state representation are mostly based on Euclidean space (or unit spherical normalized space). For example, Reference 7 obtains shared Euclidean embeddings through image-text contrast learning, and Flamingo bridges vision and language through cross-attention and uses RT-2 to discretize actions into tokens and train them isomorphically with text tokens. ② Traditional solutions typically employ a modular pipeline architecture, where perception, prediction, planning, and control modules are developed independently. The unified representation of "default Euclidean geometry" leads to the following systematic mismatches among these three types of modules: 1) The hierarchical relationship structure commonly found in language semantics and environmental perception data is difficult to represent in Euclidean vector space; 2) The data structure of Euclidean space does not match the manifold structure of pose / geometry and action increments; 3) Reasoning strategies and cognitive states often require explicit expression of uncertainty and multimodality, which cannot correspond to the global flatness of Euclidean space. This results in problems such as information loss and error accumulation between modules and weak generalization ability in existing fragmented architectures.

[0005] Therefore, there is an urgent need for a multimodal large model that can break down modal barriers and unify the modeling of the entire "perception-cognition-control" chain to solve the information bias problem caused by Euclidean space geometric mismatch.

[0006] Existing technical documents: Document 1: Brohan A, Brown N, Carbajal J, et al. Rt-1: Robotics transformer for real-world control at scale[J]. arXiv preprint arXiv:2212.06817, 2022. Document 2: Zitkovich B, Yu T, Xu S, et al. Rt-2: Vision-language-actionmodels transfer web knowledge to robotic control[C]. Conference on RobotLearning. PMLR, 2023: 2165-2183. Document 3: Driess D, Xia F, Sajjadi MSM, et al. Palm-e: An embodied multimodal language model[J]. 2023. Document 4: Jiang Y, Gupta A, Zhang Z, et al. Vima: Robot manipulation with multimodal prompts[J]. 2023. Document 5: Shridhar M, Manuelli L, Fox D. Perceiver-actor: A multi-tasktransformer for robotic manipulation[C]. Conference on Robot Learning. PMLR, 2023: 785-799. Document 6: Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]. International conference on machine learning. PmLR, 2021: 8748-8763. Document 7: Nickel M, Kiela D. Poincaré embeddings for learninghierarchical representations[J]. Advances in neural information processing systems, 2017, 30. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a training method, inference method, and system for a large multimodal model of perception-cognition-control based on Riemannian manifolds. By realizing cross-modal alignment, cognitive decision-making, and control mapping on Riemannian manifolds, the problem of Euclidean space ignoring the inherent geometric differences in the spaces where different modal variables reside can be solved. It can also reduce the accumulation of representation bias caused by geometric mismatch between modules, thereby improving the cross-scene generalization ability and robustness of the large multimodal model.

[0008] One aspect of the present invention provides a training method for a large-scale perception-cognition-control multimodal model based on a Riemannian manifold. The large-scale perception-cognition-control multimodal model includes a multimodal mapping and alignment module, a cognition module, and a control module. The multimodal mapping and alignment module maps the Euclidean features corresponding to multiple perceived modal samples to feature point representations on a Riemannian manifold. The cognition module obtains cognitive prediction point representations on the Riemannian manifold based on the input feature point representations. The control module obtains control point representations on the Riemannian manifold based on the cognitive prediction point representations output by the cognition module. The method includes the following steps: The Euclidean features corresponding to the first modality training sample, the Euclidean features corresponding to the second modality training sample corresponding to the first modality training sample, and the action label are obtained. The obtained Euclidean features are then input into the perception-cognition-control multimodal large model to be trained. The first modality or the second modality includes the text modality, and the second modality training samples include the second modality training positive samples and the second modality training negative samples. The feature point representations corresponding to each modality training sample are obtained through the multimodal mapping alignment module, and the multimodal alignment loss is determined based on the similarity between the obtained feature point representations. The feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the second modality training positive samples are used as the input data of the cognitive module. The spatial consistency loss is determined based on the feature point representations input to the cognitive module, the cognitive prediction point representations output by the cognitive module, and the control point representations output by the control module. The action loss is determined based on the control actions and action labels obtained by mapping the control point representations output by the control module; The large perception-cognition-control multimodal model to be trained is iteratively updated based on action loss, multimodal alignment loss, and spatial consistency loss.

[0009] In some embodiments of the present invention, a multimodal mapping alignment module is used to obtain feature point representations corresponding to each modality training sample, and a multimodal alignment loss is determined based on the similarity between the obtained feature point representations, including: The feature point representations corresponding to the first modality training samples, the second modality training positive samples, and the second modality training negative samples are determined by the multimodal mapping alignment module. The multimodal alignment loss is determined based on the similarity between the feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the positive and negative samples of the second modality training. The similarity between feature point representations is characterized by the geodesic distance between them, and the multimodal alignment loss is the InfoNCE loss.

[0010] In some embodiments of the present invention, the feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the second modality training positive samples are used as input data for the cognitive module, including: fusing the feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the second modality training positive samples to obtain a multimodal fused feature point representation, and using the multimodal fused feature point representation as input data for the cognitive module.

[0011] In some embodiments of the present invention, the loss term used for iteratively updating the perception-cognition-control multimodal large model further includes a control direction loss, which is determined in the following manner: The cognitive prediction point representation output by the cognitive module is mapped to the tangent space of the multimodal fusion feature point representation to obtain the fusion point tangent vector; The control point representation output by the control module is mapped to the tangent space of the preset base point to obtain the first tangent vector of the base point; The control direction loss is determined based on the angular difference between the fusion point tangent vector and the base point tangent vector.

[0012] In some embodiments of the present invention, when the Riemannian manifold is a hyperbolic Riemannian manifold, the spatial uniformity loss is determined in the following manner: By using logarithmic mapping, the control point representation output by the control module, the cognitive prediction point representation output by the cognitive module, and the multimodal fusion feature point representation are respectively mapped to the tangent space of the preset base point to obtain the first tangent vector, the second tangent vector, and the third tangent vector of the base point; The spatial consistency loss is determined based on the distance between the first tangent vector and the third tangent vector of the base point, as well as the distance between the second tangent vector and the third tangent vector of the base point.

[0013] In some embodiments of the present invention, obtaining the Euclidean features corresponding to the first modality training sample, the Euclidean features corresponding to the second modality training sample corresponding to the first modality training sample, and the action label includes: Obtain a multimodal training sample set; the multimodal training sample set includes a first modality training sample, a second modality training sample corresponding to the first modality training sample, and action labels; wherein, the action labels correspond to the first modality training sample and the second modality training sample; The first modality training sample and the second modality training sample are respectively input into the preset feature extractor of the corresponding modality, and the Euclidean features corresponding to the first modality training sample and the Euclidean features corresponding to the second modality training sample are output.

[0014] Another aspect of the present invention provides an inference method based on a pre-trained perception-cognition-control multimodal large model, wherein the pre-trained perception-cognition-control multimodal large model is trained by the method described in any of the above embodiments, and the method includes the following steps: Obtain the Euclidean features corresponding to the first modality sample and the Euclidean features corresponding to the second modality sample; wherein, the first modality or the second modality includes the text modality; The Euclidean features corresponding to the acquired multimodal samples are input into the multimodal mapping and alignment module, and the output is the feature point representation corresponding to the multimodal samples; Input the feature point representations corresponding to the multimodal samples into the cognitive module, and output the cognitive prediction point representations corresponding to the multimodal samples; The cognitive prediction point representation corresponding to the multimodal sample is input into the control module, and the control point representation corresponding to the multimodal sample is output, so that the controller performs the corresponding control action based on the control point representation; wherein, the feature point representation, the cognitive prediction point representation and the control point representation are located on the same Riemannian manifold.

[0015] Another aspect of the present invention provides a training system for a large multimodal perception-cognition-control model based on Riemannian manifolds, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method described in any of the above embodiments.

[0016] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0017] Another aspect of the present invention provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0018] This invention proposes a training method, inference method, and system for a multimodal large-scale model of perception-cognition-control based on Riemannian manifolds. By introducing Riemannian manifolds into the "perception-cognition-control" paradigm to construct a multimodal large-scale model, it can align a high-dimensional semantic space using manifold representations. Furthermore, it fixes the implementation processes of the multimodal mapping alignment module, cognitive module, and control module, as well as the information transfer between modules, within the same manifold space. The method proposed in this application upgrades the alignment metric, cognitive decision, and control output from empirical engineering design to a geometrically consistent computational framework through Riemannian manifolds. This solves the problems of structural destruction and geometric mismatch in Euclidean space, greatly improving the robustness of embodied intelligence tasks and enhancing cross-scenario generalization performance.

[0019] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0020] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a flowchart illustrating a multimodal large model training method for perception-cognition-control in one embodiment of the present invention.

[0022] Figure 2 This is a flowchart illustrating the training method for a multimodal large model of perception-cognition-control in another embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of the reasoning method based on a perception-cognition-control multimodal large model in one embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0025] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0026] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0027] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0028] The "Perception-Cognition-Control" paradigm is an embodied intelligence closed-loop paradigm that organically integrates environmental perception, semantic reasoning, and action execution. It is commonly used to build intelligent robot systems with autonomous decision-making capabilities. This paradigm can be based on a large model or agent obtained through end-to-end policy learning. In the perception phase, multimodal data is input and aligned across modalities in a unified feature space. In the cognition phase, the aligned cross-modal features are mapped to task intentions. In the control phase, the abstract task intentions are transformed into concrete motion control signals. In other words, the "Perception-Cognition-Control" paradigm aims to map multimodal perception data (which may include language instructions) to executable actions (such as discrete tokens, continuous control variables, or action trajectories), thereby supporting the closed-loop execution of embodied tasks.

[0029] However, the current application of the "perception-cognition-control" paradigm has the following significant drawbacks: Current multimodal alignment methods in the perception stage usually assume that all modal data can be uniformly encoded into the same Euclidean latent space, and use the addition, dot product, cosine similarity or concatenation of Euclidean vectors as the basic operations for cross-modal alignment and fusion. Thus, in implementation, the inherent geometric differences in the space where different modal perception data are located are ignored: ① In the embodied task scenario, variables such as pose and orientation (which can fall on spherical or rotation group related manifolds), rigid body pose (which can fall on Lie group / quotient space such as SE(3), uncertainty and covariance (which can fall on symmetric positive definite matrix manifolds) and probability distribution and confidence representation (which can fall on statistical manifolds or simplex related spaces) are essentially specific Riemannian measures with non-Euclidean geometric structures. Directly "flattening" such variables into Euclidean vectors and using Euclidean distance for similarity measurement naturally introduces inconsistencies in measurement. This leads to discrepancies between the proximity relationships between modal data learned by the multimodal large model and the true semantic similarity, resulting in alignment errors, retrieval errors, and fusion biases. Secondly, gradient updates in Euclidean space often disrupt the original geometric constraints of modality-aware data (such as orthogonality, positive definiteness, or group structure closure). If the features encoded in Euclidean space are then corrected through post-processing methods such as normalization or projection back to the constraint set, problems such as numerical discontinuity, iteration drift, slow convergence, and sudden performance drops near singular parameterization can easily occur. As a result, the systematic error of "structural destruction" cannot be compensated for by simple training.

[0030] Furthermore, existing end-to-end pipelines for embodied intelligence lack a consistent mapping mechanism across manifolds and interpretable geometric alignment constraints between the perception, cognition, and control modules. Since the alignment features obtained by the perception module, the task intent determined by the cognition module, and the control signals generated by the control module often fall on different non-Euclidean geometric objects, continuing to use Euclidean vector space mapping methods will lead to a mismatch between the semantic representations and geometric structures of the modules: the geometric quantities processed by the perception module are distorted when entering the cognition module, and the reasoning results of the cognition module are difficult to maintain continuity and composability when mapped to the control space. This causes the error in the perception-cognition-control closed loop to accumulate in the time dimension and be amplified in the backprojection / normalization stage, typically manifesting as motion jitter, uneven trajectory, unstable planning, or even control instability. Furthermore, existing methods based on Euclidean space typically compress uncertainty into Euclidean confidence levels or scalar weights, making it difficult to propagate and integrate multi-source uncertainties in a consistent geometric manner throughout the "perception-cognition-control" chain. This results in a lack of reliable basis for the outputs of the cognitive module used for risk-sensitive decision-making and the control module used for safety constraint control, thereby reducing the robustness and cross-scenario generalization ability of embodied intelligence tasks. This also makes large models need to rely on more data to compensate for the representation bias caused by geometric mismatch.

[0031] To overcome the limitations of Euclidean space, and considering that pose / geometry, action increments, uncertainties, and covariance naturally belong to manifold structures, this application combines Riemannian manifolds with the perception-cognition-control paradigm, proposing a training and inference method for a large-scale multimodal model based on the Riemannian manifold. A Riemannian manifold is a smooth manifold geometry with Riemannian metrics. This structure defines an inner product in the tangent space at each point, enabling the measurement of lengths, angles, and geodesic distances in curved surfaces / spaces, and supporting optimization and learning in non-Euclidean spaces. It is commonly used to construct representation spaces with curvature. The method proposed in this application, based on the Riemannian manifold, allows multimodal representation learning, model iterative updates, and variable fusion and transfer to be completed under Riemannian metrics that match the intrinsic structure of the modules. Utilizing a logically integrated large-scale multimodal model, it fundamentally avoids metric inconsistencies and structural damage caused by Euclidean processing.

[0032] As an example, this application can use a hyperbolic Riemannian manifold with constant negative curvature (a type of Riemannian manifold) as the geometric structure of the latent space, thereby completing representation learning of multimodal sensing data, cross-modal feature fusion, and obtaining corresponding state representations on the hyperbolic manifold, and better embedding data with multi-layered semantics. For example, this application can define a hyperbolic Riemannian manifold space shared by different modal data and modules. It can be represented as , The curvature of the cross section is ( )of Hyperbolic manifold.

[0033] Specifically, this invention introduces a manifold-aware alignment mechanism, allowing representations of different modalities to remain on their natural manifolds or reasonable product manifolds. Alignment targets are constructed based on similarity metrics (belonging to Riemannian metrics) of representations on the manifolds, thereby achieving geometrically consistent cross-modal semantic alignment. Simultaneously, this invention establishes geometrically consistent interfaces and closed-loop alignment frameworks between the perception, cognition, and control stages. This enables perception outputs to be inferred and predicted by the cognitive model within a consistent geometry, and cognitive results to generate continuous control in a compatible tangent space and stably map back to the control manifold, thereby improving temporal continuity, numerical stability, and control safety. The perception, cognition, and control processes based on this geometrically consistent framework support the expression, propagation, and fusion of multi-source uncertainties within a unified geometric framework, providing a consistent metric basis for risk-sensitive planning and safety control, ultimately achieving higher data utilization efficiency, stronger robustness, and better cross-scenario generalization performance.

[0034] The "Perception-Cognition-Control" paradigm aims to construct a universal "embodied brain" capable of understanding multimodal inputs through a unified feature space, just like humans, and directly generating action strategies that conform to physical constraints. This application proposes a large-scale multimodal model (hereinafter referred to as the Perception-Cognition-Control Multimodal Model or the Multimodal Model) that can realize the "Perception-Cognition-Control" paradigm. This large-scale multimodal model includes a multimodal mapping alignment module, a cognition module, and a control module. The data flow within the large-scale multimodal model is sequential: multimodal mapping alignment module, cognition module, and control module. The multimodal mapping alignment module maps multimodal perceived data to representations on Riemannian manifolds during the perception phase (i.e., mapping the perceived multimodal data to a high-dimensional latent space). The cognition module maps the representations output by the multimodal mapping alignment module to obtain representations indicating task intent. The control module obtains representations indicating control actions based on the representations output by the cognition module. The representations output by the multimodal mapping alignment module, the cognitive module, and the control module are all point representations on the same Riemannian manifold (i.e., the multimodal mapping alignment module, the cognitive module, and the control module all output features located in a unified Riemannian manifold space). In order to distinguish the representations obtained by different modules, this application may refer to the point representation output by the multimodal mapping alignment module as the feature point representation, the point representation output by the cognitive module as the cognitive prediction point representation, and the point representation output by the control module as the control point representation.

[0035] The perception-cognition-control multimodal large model proposed in this application can be constructed based on a VLA model or a large language model that supports multimodal data inputs such as vision, text, and point clouds, and this invention is not limited thereto. Moreover, since the multimodal mapping alignment module in the multimodal large model can align different modal features, this application does not specifically limit the modal types applicable to this multimodal large model, and it can be extended to any combination of modalities.

[0036] Figure 1 This is a training method for a large-scale multimodal model of perception-cognition-control based on Riemannian manifolds in one embodiment of the present invention. For example... Figure 1 As shown, the method includes steps S110 to S140, as detailed below: Step S110: Obtain the multimodal features corresponding to the multimodal training sample set. The multimodal training sample set includes first-modal training samples, corresponding second-modal training positive samples, and corresponding second-modal training negative samples. The multimodal features can be features in Euclidean space, thus allowing them to be viewed as features in the tangent space of a preset base point, and subsequently converted into representations on a Riemannian manifold. Step S110 is: obtaining the Euclidean features corresponding to the first-modal training samples and the Euclidean features corresponding to the second-modal training samples (which include second-modal training positive samples and second-modal training negative samples). The second-modal training positive samples are training samples that match the target semantics of the first-modal training samples, and the second-modal training negative samples are training samples that do not match the target semantics of the first-modal training samples.

[0037] The first modality can be a single modality, and the second modality can be any other modality besides the first modality or a combination of other modalities. This application does not specifically limit the modality type (visual image, text, or point cloud, etc.). For example, to facilitate user input of natural language commands, the first or second modality may include a text modality. Furthermore, in the case of supervised training, step S110 may also obtain action labels corresponding to the first modality training samples and the second modality training samples.

[0038] In some embodiments of the present invention, the multimodal large model proposed in this application may further include an encoding module for extracting multimodal features from multimodal training samples. The encoding module may include a preset extractor corresponding to each modality. In this case, step S110 may include: acquiring a multimodal training sample set; inputting a first modality training sample into the preset feature extractor of the corresponding modality, outputting Euclidean features corresponding to the first modality training sample, and inputting a second modality training positive sample and a second modality training negative sample into the preset feature extractor of the corresponding modality, outputting Euclidean features corresponding to the second modality training positive sample and the second modality training negative sample (i.e., inputting each modality training sample into the preset feature extractor of the corresponding modality, outputting Euclidean features corresponding to each modality training sample).

[0039] As an example, taking the first modality as the visual modality and the second modality as the text modality, the multimodal training sample set... It may include visual modal data Text modal data and corresponding action tags , ; Pre-built visual encoder Training samples can be obtained from perceived visual modalities Extract the corresponding Euclidean features Pre-built text encoder Training samples can be obtained from perceived text modalities Extract Euclidean features from (including positive and negative training samples of text modality) The implementation process of the encoding module can be represented as follows: , .

[0040] The acquired Euclidean features are input into the large-scale perceptual-cognitive-control model to be trained (firstly, the multimodal mapping and alignment module). A loss function for iteratively updating the large-scale multimodal model can be constructed based on the input and output data of each module.

[0041] Step S120: Obtain the feature point representations corresponding to each modality training sample through the multimodal mapping alignment module in the multimodal large model to be trained, and determine the multimodal alignment loss based on the similarity between the feature point representations corresponding to different modal training samples.

[0042] The multimodal mapping alignment module proposed in this application is used to achieve semantic interoperability between different modal data and perform cross-modal semantic alignment by learning the joint probability distribution between different modal data in a shared, semantically consistent unified Riemannian manifold space. Specifically, the function of the multimodal mapping alignment module in step S120 is as follows: It takes into account the Euclidean features corresponding to the first modality training sample, the Euclidean features corresponding to the second modality training positive sample, and the Euclidean features corresponding to the second modality training negative sample, and outputs the feature point representations corresponding to the first modality training sample (i.e., the feature point representations corresponding to the Euclidean features of the first modality training sample), the second modality training positive sample (i.e., the feature point representations corresponding to the Euclidean features of the second modality training positive sample), and the second modality training negative sample (i.e., the feature point representations corresponding to the Euclidean features of the second modality training negative sample).

[0043] This application does not specify the method by which the multimodal mapping alignment module maps Euclidean features to point representations on the Riemannian manifold. Existing projection mapping, contraction mapping, or exponential mapping can be used, or a self-designed nonlinear mapping method can be employed. The following example uses the first modality as the visual modality and the second modality as the text modality to demonstrate how the multimodal mapping alignment module maps the Euclidean features corresponding to the visual modality. Euclidean features corresponding to text modalities Mapped to point representations on hyperbolic Riemannian manifolds respectively and Furthermore, to ensure the differentiability and numerical stability of the mapping, Euclidean features are considered. and European features This application may employ a manifold mapping mechanism based on a "tangent space-exponential mapping". For example... Figure 2 As shown, the implementation process of the multimodal mapping alignment module can be as follows: By using a linear transformation, the Euclidean eigenvalues ​​are mapped to tangent vectors, which can be viewed as hyperbolic Riemannian manifolds. Vectors in the tangent space of the upper base point (the base point can be preset): , ; The vectors in the tangent space are mapped to a hyperbolic Riemannian manifold through the exponential mapping. Points on the top represent: , ; in, and Representing visual features respectively and text features The corresponding linear transformation matrix, This represents the exponential mapping at the base point o. and These represent the tangent vectors corresponding to the visual modality training samples and the text modality training samples, respectively. and ( ( ) represent the visual modality training samples and the text modality training samples in the hyperbolic Riemannian manifold, respectively. Points on the surface are represented.

[0044] Furthermore, during the training phase of the multimodal large model, the feature point representations corresponding to different modal training samples mapped by the multimodal mapping alignment module are not necessarily semantically aligned. Therefore, it is necessary to construct a multimodal alignment loss and perform iterative training. During the inference phase of the multimodal large model, the feature point representations corresponding to different modal samples output by the trained multimodal mapping alignment module are semantically aligned. Therefore, this application designs a multimodal mapping alignment module that, when performing cross-modal alignment (i.e., cross-modal semantic alignment), needs to determine the multimodal alignment loss based on the input and output data of the multimodal mapping alignment module. This ensures that the trained multimodal mapping alignment module can make the feature point representation pairs corresponding to semantically consistent multimodal samples closer in the Riemannian manifold space, while the semantically inconsistent modal samples are farther apart (promoting different modal data to form a consistent semantic structure in the same Riemannian manifold space). The purpose of the multimodal mapping alignment module is to ensure that, when the semantics of the first and second modal samples are consistent, the feature point representations corresponding to the first and second modal samples are close in the Riemannian manifold space (e.g., the aligned feature point representations have similar geodesic distances on the manifold), and when the semantics of the first and second modal samples are inconsistent, the feature point representations corresponding to the first and second modal samples are far apart in the Riemannian manifold space. Unlike traditional Euclidean space alignment, the feature alignment process in this application occurs directly in the Riemannian manifold geometry, thus naturally supporting the organization and separation of multi-level semantics at the latent space level, and providing a structured semantic foundation for subsequent cognition and control.

[0045] In some embodiments of the present invention, the multimodal alignment loss can be determined based on the similarity between the feature point representation corresponding to the first modality training sample and the feature point representation corresponding to the positive training sample and the negative training sample of the second modality. The present invention does not specifically limit the calculation method of the multimodal alignment loss; it is sufficient to ensure that the similarity between the feature point representation corresponding to the first modality training sample and the feature point representation corresponding to the positive training sample of the second modality is high, and the similarity between the feature point representation corresponding to the first modality training sample and the feature point representation corresponding to the negative training sample of the second modality is low. This application also does not specifically limit the method for determining the similarity between feature point representations corresponding to different modalities. For example, the similarity can be determined based on the geodesic distance between feature point representations (geodesic distance is proportional to similarity), or the similarity between feature point representations can be determined based on the Euclidean distance between tangent vectors after mapping the feature point representations to tangent vectors in the tangent space of the base points (Euclidean distance is proportional to similarity).

[0046] To ensure the transferability of aligned feature point representations from abstract semantics to concrete execution, this invention uses geodesic distance between point representations as a geometric characterization of semantic abstraction. During training, by imposing geodesic distance constraints on the abstract / concrete relationships of task semantics (e.g., the global intent of instructions is relatively more abstract, while object / fine-grained semantics are relatively more concrete), the alignment space can possess a more stable hierarchical organization, providing a more separable and generalizable semantic structure foundation for subsequent cognitive planning and control mapping.

[0047] The following text will use the multimodal alignment loss as the InfoNCE (Information Noise Contrastive Estimation) loss, and will be based on the hyperbolic Riemannian manifold. Taking the geodesic distance between feature point representations as an example to measure the similarity between feature point representations (the geodesic distance can be used subsequently for text-image alignment, target approximation, and control consistency constraints, thus enabling the training target and inference execution to share the same geometric basis), the process of determining the multimodal alignment loss is described as follows: Assuming the first modality is the visual modality and the second modality is the text modality, represented in hyperbolic Riemannian manifold space as a metric point. and The hyperbolic geodesic distance between them can be expressed by the formula: ; in, For the Minkowski inner product under the Lorentz model, Represents the inverse hyperbolic cosine function. Represents a hyperbolic Riemannian manifold curvature, Point representation Sum of dots The hyperbolic geodesic distance between them.

[0048] Define temperature parameters ( (where is a hyperparameter), then the multimodal alignment loss is the InfoNCE loss of the hyperbolic geodesic distance, and the calculation formula can be expressed as: ; in, This represents the point representation corresponding to the training sample of the first modality. This represents the point representation corresponding to the positive training samples of the second modality. This represents the point representation corresponding to the positive training samples of the second modality. and Together they form the text modality training samples on the hyperbolic Riemannian manifold Points on , This represents the number of positive training samples in the second modality within the multimodal training sample set. Based on the above... The trained multimodal large model can utilize the geometric properties of hyperbolic Riemannian space and geodesic distance metrics to achieve structured alignment of vision and language within the Riemannian manifold.

[0049] In some embodiments of the present invention, after obtaining the feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the second modality training positive samples through the multimodal mapping alignment module (the feature point representations corresponding to the second modality training negative samples do not participate in the subsequent training process), the training method further includes: fusing the feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the second modality training positive samples to obtain a multimodal fused feature point representation (corresponding to the multimodal training sample set).

[0050] As an example, the perception-cognition-control multimodal large model proposed in this application may also include a feature fusion module, used to fuse the feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the second modality training positive samples, that is, to achieve deep semantic fusion of different modal data within the Riemannian manifold space. Taking the first modality as the visual modality, the second modality as the text modality, and the Riemannian manifold as a hyperbolic Riemannian manifold as an example, the model obtains data located on the same Riemannian manifold... and Subsequently, a joint representation for characterizing the unified state can be constructed. This joint representation can be obtained through a feature fusion module on the Riemannian manifold, as shown in the following formula: ; in, This represents the multimodal fusion feature point representation on a hyperbolic Riemannian manifold. This refers to the feature fusion function built into the feature fusion module. This represents the feature point representation corresponding to the positive training samples of the second modality. This application does not specifically limit the type of the feature fusion function; it only needs to be able to fuse the feature point representations corresponding to different modalities on a Riemannian manifold. For example, a multilayer linear network can be used as the fusion function. To construct a feature fusion module.

[0051] Step S130: Use the feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the second modality training positive samples as the input data of the cognitive module (i.e., use the multimodal fusion feature point representations as the input data of the cognitive module) and construct the spatial consistency loss.

[0052] More specifically, the multimodal fusion feature point representation is input into the cognitive module of the multimodal large model to be trained. The cognitive module can output the cognitive prediction point representation corresponding to the multimodal training sample set. The cognitive prediction point representation output by the cognitive module is input into the control module of the multimodal large model to be trained. The control module can output the control point representation corresponding to the multimodal training sample set.

[0053] In addition to utilizing the Riemannian manifold as the semantic representation space for aligning multimodal perceptual data, this application further designs to extend the Riemannian manifold space into a unified latent space shared by perception, cognition, and control. That is, all representations (i.e., feature point representations, cognitive prediction point representations, and control point representations) upon which the perceptual state, cognitive goal, and control state depend are constrained within the same Riemannian manifold space, and the mapping relationship between the three is kept consistent in a metrical sense through spatial consistency loss. This avoids the error accumulation and generalization degradation caused by the fragmented spatial inconsistency between "perceptual representation - task intention (i.e., cognitive goal) - control state" in the traditional architecture.

[0054] In some embodiments of the present invention, spatial consistency loss can be determined based on the feature point representation of the input cognitive module (i.e., the multimodal fusion feature point representation), the cognitive prediction point representation output by the cognitive module (i.e., the cognitive prediction point representation corresponding to the multimodal training sample set), and the control point representation output by the control module (i.e., the control point representation corresponding to the multimodal training sample set). The multimodal large model trained based on the spatial consistency loss can ensure that the cognitive target and the control representation are aligned, eliminating the spatial mismatch between perception and control.

[0055] Specifically, at least two of the following can be used to determine the spatial consistency loss: the linear difference between the feature point representation of the input cognitive module and the cognitive prediction point representation output by the cognitive module, the linear difference between the feature point representation of the input cognitive module and the control point representation output by the control module, and the linear difference between the cognitive prediction point representation output by the cognitive module and the control point representation output by the control module.

[0056] As an example, when the Riemannian manifold is a hyperbolic Riemannian manifold, the spatial consistency loss can be determined as follows: by using a logarithmic mapping, the control point representation output by the control module, the cognitive prediction point representation output by the cognitive module, and the multimodal fusion feature point representation are mapped to the tangent space of a preset base point, respectively, to obtain the first tangent vector, the second tangent vector, and the third tangent vector of the base point; the spatial consistency loss is determined based on the distance between the first tangent vector and the third tangent vector of the base point, and the distance between the second tangent vector and the third tangent vector of the base point.

[0057] Assuming the first modality is the visual modality and the second modality is the text modality, the multimodal fusion feature point representation in the hyperbolic Riemannian manifold space can be expressed as follows: , It serves both as a "multimodal perception data understanding" module and as an input to the subsequent cognition-control module, structurally ensuring that perception and control reside in the same latent space. The cognition module can output: ; in, This represents a cognitive model on a shared hyperbolic Riemannian manifold. The control module is based on... Control-related representations on hyperbolic Riemannian manifolds can be obtained. .

[0058] To ensure that the feature point representations input to the cognitive module, the cognitive prediction point representations output by the cognitive module, and the control point representations output by the control module are all under the same geometric logic, this invention employs a tangent space consistency constraint: the point representations on the hyperbolic Riemannian manifold are pulled back to be compared with all spaces through a logarithmic mapping, aligning the perceptual representation, cognitive representation, and control representation on a local linear structure. The formula for the space consistency loss can then be expressed as: ; in, Represents the third tangent vector at the base point. Represents the second tangent vector of the base point. Represents the first tangent vector of the base point. This represents the L2 norm in Euclidean space. , and All belong to , Represents a hyperbolic Riemannian manifold At the base point The tangent space at that point.

[0059] Spatial consistency loss ensures that the "cognitive target" and "control representation" are no longer feature representations belonging to different spaces, but rather geometric objects on the same Riemannian manifold as the perceptual state. This guarantees that subsequent control modules can approximate the cognitive target using a geometric structure (geodesic distance, geodesic direction) consistent with the perceptual features. Therefore, multimodal alignment loss and spatial consistency loss can guarantee that "similarity measurement, representation fusion, and control state" are valid within the same geometric system, avoiding the geometric mismatch problem caused by the separation of "semantic alignment space" and "control state space" in traditional Euclidean space.

[0060] Step S140: Iteratively update the large-scale perception-cognition-control multimodal model to be trained based on multimodal alignment loss and spatial consistency loss to obtain a pre-trained large-scale perception-cognition-control multimodal model. For example, the multimodal alignment loss can be... Consistent loss with space The total loss function is combined and optimized end-to-end for the multimodal large model to be trained. The total loss function can then be expressed as: .

[0061] In some embodiments of the present invention, the loss term used for iteratively updating the perception-cognition-control multimodal large model may further include a control orientation loss. The purpose of introducing control direction loss is to define the geodesic direction on the Riemannian manifold, which is the direction from the multimodal fused feature point representation to the cognitive prediction point representation, as the optimal control direction. This serves as the geometric guide for control learning, thereby transforming the output representation of the cognitive module into an executable control signal through the control module.

[0062] This application designs a control direction loss to measure the difference between the optimal control direction and the actual control direction (the control point representation obtained by the control module based on the cognitive prediction point representation). To reflect directionality, this application determines the magnitude of the control direction loss by directly comparing the differences in the directions of the tangent vectors in the tangent space. For example, the control direction loss can be determined as follows: mapping the cognitive prediction point representation output by the cognitive module to the tangent space of the multimodal fusion feature point representation to obtain the fusion point tangent vector; mapping the control point representation output by the control module to the tangent space of a preset base point to obtain the base point first tangent vector; and determining the control direction loss based on the angle difference between the fusion point tangent vector and the base point tangent vector (direction similarity is equivalent to control direction loss). This application does not specifically limit the method of mapping the point representation to the tangent space, such as using nonlinear mappings like projection mapping or logarithmic mapping; nor does it specifically limit the method of calculating the angle difference between the tangent vectors, such as using cosine similarity, chordal distance, or Gram matrix. Based on the multimodal large model trained using the control direction loss, the direction output by the control module can be forced to approach the target along the geometrically optimal path, thereby achieving high-precision closed-loop control.

[0063] As an example, when the Riemannian manifold is a hyperbolic Riemannian manifold, it can be represented by a logarithmic mapping at multimodal fusion feature points. Point to fusion point tangent vector The formula is expressed as: ; in, , Represents a hyperbolic Riemannian manifold In point representation The tangent space at that point.

[0064] Point representation based on multimodal sensing data This allows the control module to output control point representations. Therefore, the first tangent vector of the base point can be used. This represents the actual control direction. Therefore, cosine similarity is used to determine the actual control direction. With optimal control direction The difference between them yields the control direction loss: ; in, express and The inner product, and They represent and The Euclidean norm.

[0065] By controlling the directional loss, this invention enables the control representation to approximate the cognitive target in a geodesic consistent manner in the tangent space, based on the fact that the perceived state, cognitive target, and control representation are all represented at points on the same Riemannian manifold, thereby forming a geometrically consistent closed-loop execution logic.

[0066] After introducing the control orientation loss, the perception-cognition-control multimodal large model to be trained can be iteratively updated based on the multimodal alignment loss, spatial consistency loss, and control orientation loss to obtain the pre-trained perception-cognition-control multimodal large model. At this time, the total loss function can be expressed as: .

[0067] In some embodiments of the present invention, in order to convert the control signal output by the control module into accurate control actions, the present application can employ a supervised training method for the multimodal large model. Therefore, the loss term designed in this application for iteratively updating the perception-cognition-control multimodal large model may further include action loss. Furthermore, action loss can be determined by comparing control actions and action labels.

[0068] The control actions are obtained by mapping the control point representations output by the control module. Therefore, this application can design a multimodal large model that also includes a controller. The controller executes the step of mapping the control point representations output by the control module (control point representations corresponding to the multimodal sample set) to obtain the control actions. Moreover, the controller can be constructed based on a learnable policy network. This application does not specifically limit the architecture of the policy network; for example, a diffusion policy, a Gaussian policy network, or a deterministic policy network with a continuous action space can be selected.

[0069] As an example, in a shared hyperbolic Riemannian manifold space, the output of the control module... As input data to the controller, a policy network that maps uniform geometric states to action outputs is utilized. Get control action And utilize action labels from the multimodal training sample set. Supervised training enables large multimodal models to acquire executable control capabilities while fulfilling multimodal alignment and spatial consistency constraints.

[0070] The process by which the controller obtains control actions using the policy network can be represented as: ; Behavioral cloning-based supervised loss (continuous control): .

[0071] This invention can combine the above four training objectives into a total loss function and perform end-to-end optimization (minimizing the total loss function), such as... Figure 2 As shown, the total loss function can be expressed as: ; in Multimodal alignment loss Spatial consistency loss , control direction loss and motion loss The balancing weight coefficient.

[0072] The multimodal large model trained through the above steps in this application is logically integrated. This logical integration is reflected in the fact that multiple processes, including cross-modal feature fusion, cognitive decision-making, control output, and inter-module information transmission, occur within the same Riemannian manifold space. It should be noted that to train the logically integrated multimodal large model, the total loss function can be used for large model training. If each module is trained separately using different loss terms (e.g., training the multimodal mapping alignment module based on multimodal alignment loss, and training the cognitive and control modules based on spatial consistency loss), geometric information bias due to module interfaces may occur.

[0073] In some embodiments of the present invention, this application also proposes a reasoning method for implementing action control based on a perception-cognition-control multimodal large model trained according to any of the above embodiments, such as... Figure 3 As shown, the method flow is as follows: Step S210: Obtain the first modality sample and the corresponding second modality sample (without distinguishing between positive and negative samples), and input them into the encoding module in the pre-trained multimodal large model to obtain the Euclidean features corresponding to the first modality sample and the Euclidean features corresponding to the second modality sample, respectively.

[0074] Step S220: Input the Euclidean features corresponding to the multimodal samples into the multimodal mapping alignment module in the pre-trained multimodal large model to obtain the feature point representations corresponding to the multimodal samples. The trained multimodal mapping alignment module enables the input different modal perception data to no longer reside in two independent Euclidean representation spaces, but to form comparable, composable, and semantically aligned geometric objects on the same hyperbolic manifold. By transforming traditional linear feature vectors into point representations on the manifold, this multimodal mapping alignment module can provide a unified non-Euclidean geometric basis for subsequent multimodal processing, avoiding structural distortions caused by Euclidean space for data with multi-layered semantics.

[0075] Step S220 can be divided into: Step S221, inputting the Euclidean features corresponding to the first modality sample and the Euclidean features corresponding to the second modality sample into the multimodal mapping alignment module in the pre-trained multimodal large model, and outputting the aligned feature point representations corresponding to the first modality sample and the second modality sample; Step S222, inputting the feature point representations corresponding to the first modality sample (i.e., the feature point representations corresponding to the Euclidean features of the first modality sample) and the feature point representations corresponding to the second modality sample (i.e., the feature point representations corresponding to the Euclidean features of the second modality sample) into the feature fusion module in the pre-trained multimodal large model, and outputting the multimodal fused feature point representations.

[0076] Step S230: Input the feature point representations corresponding to the multimodal samples (i.e., the multimodal fusion feature point representations) into the cognitive module of the pre-trained multimodal large model, and output the cognitive prediction point representations corresponding to the multimodal samples.

[0077] Step S240: Input the cognitive prediction point representation corresponding to the multimodal sample into the control module, and output the control point representation corresponding to the multimodal sample, so that the controller outputs the corresponding control action based on the control point representation and executes the control action; wherein, the feature point representation, the cognitive prediction point representation and the control point representation are located on the same Riemannian manifold.

[0078] The multimodal large model trained using the training method proposed in this application has the following significant advantages: ① The “perception-cognition-control” multimodal big model architecture proposed in this application can unify the modeling of environmental perception, semantic reasoning and action execution, thereby completing the entire closed loop from multimodal input understanding to physical action generation within the logically integrated model, and realizing autonomous decision-making and execution of embodied intelligent tasks.

[0079] ② Unlike existing technologies that forcibly "flatten" hierarchical semantics and non-Euclidean physical states into Euclidean space, this invention overcomes the limitations of traditional VLA models that rely on Euclidean space. It uses the Riemannian manifold as a shared latent space for perception, cognition, and control, naturally aligning with the hierarchical tree structure of semantics and the manifold characteristics of actions. Moreover, by replacing Euclidean distance with a similarity metric based on manifold space, it effectively avoids structural distortions caused by Euclidean embedding, thereby significantly improving the alignment accuracy and generalization ability of multimodal data in the deep semantic latent space.

[0080] ③ To address the spatial separation of multimodal features and control signals, unlike existing technologies that rely solely on splicing or dot product fusion, this invention utilizes the mutual transformation between the manifold space of the Riemannian manifold and the tangent space represented by points on the manifold during training. This not only introduces spatial geometric consistency constraints to solve the problem of the geometric separation between perceptual representation and control signals, but also forces the control module to generate control actions along the geodesic direction pointing to the cognitive target through control direction loss and action loss. This eliminates policy drift and trajectory jitter caused by inconsistent geometric measurements, significantly improving the numerical stability, trajectory smoothness, and closed-loop accuracy of embodied task execution.

[0081] Corresponding to the above method, the present invention also provides a training system for a large multimodal perception-cognition-control model based on Riemannian manifolds. The system includes a computer device, which includes a processor and a memory. The memory stores computer programs / instructions, and the processor is used to execute the computer programs / instructions stored in the memory. When the computer programs / instructions are executed by the processor, the system implements the steps of the method described above.

[0082] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0083] This invention also provides a computer program product storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer program product can be a tangible product, such as random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of product known in the art.

[0084] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0085] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0086] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training method for a large-scale multimodal model of perception-cognition-control based on Riemannian manifolds, characterized in that, The perception-cognition-control multimodal large model includes a multimodal mapping and alignment module, a cognition module, and a control module. The multimodal mapping and alignment module is used to map the Euclidean features corresponding to multiple perceived modal samples to feature point representations on a Riemannian manifold. The cognition module is used to obtain a cognitive prediction point representation on the Riemannian manifold based on the input feature point representations. The control module is used to obtain a control point representation on the Riemannian manifold based on the cognitive prediction point representations output by the cognition module. The method includes the following steps: The Euclidean features corresponding to the first modality training sample, the Euclidean features corresponding to the second modality training sample corresponding to the first modality training sample, and the action label are obtained, and the obtained Euclidean features are input into the perception-cognition-control multimodal large model to be trained; wherein, the first modality or the second modality includes the text modality, and the second modality training samples include the second modality training positive samples and the second modality training negative samples. The feature point representations corresponding to each modality training sample are obtained through the multimodal mapping alignment module, and the multimodal alignment loss is determined based on the similarity between the obtained feature point representations. The feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the second modality training positive samples are used as the input data of the cognitive module. The spatial consistency loss is determined based on the feature point representations input to the cognitive module, the cognitive prediction point representations output by the cognitive module, and the control point representations output by the control module. The action loss is determined based on the control action obtained by mapping the control point representation output by the control module and the action label; The perception-cognition-control multimodal large model to be trained is iteratively updated based on action loss, multimodal alignment loss, and spatial consistency loss.

2. The method according to claim 1, characterized in that, The step of obtaining feature point representations corresponding to each modality training sample through the multimodal mapping alignment module, and determining the multimodal alignment loss based on the similarity between the obtained feature point representations, includes: The feature point representations corresponding to the first modality training samples, the second modality training positive samples, and the second modality training negative samples are determined by the multimodal mapping alignment module. The multimodal alignment loss is determined based on the similarity between the feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the positive and negative samples of the second modality training. The similarity between feature point representations is characterized by the geodesic distance between them, and the multimodal alignment loss is the InfoNCE loss.

3. The method according to claim 1, characterized in that, The step of using the feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the second modality training positive samples as input data for the cognitive module includes: fusing the feature point representations corresponding to the first modality training samples and the feature point representations corresponding to the second modality training positive samples to obtain a multimodal fused feature point representation, and using the multimodal fused feature point representation as input data for the cognitive module.

4. The method according to claim 3, characterized in that, The loss term used for iteratively updating the perception-cognition-control multimodal large model also includes a control orientation loss, which is determined in the following way: The cognitive prediction point representation output by the cognitive module is mapped to the tangent space of the multimodal fusion feature point representation to obtain the fusion point tangent vector; The control point representation output by the control module is mapped to the tangent space of the preset base point to obtain the first tangent vector of the base point; The control direction loss is determined based on the angular difference between the fusion point tangent vector and the base point tangent vector.

5. The method according to claim 3, characterized in that, When the Riemannian manifold is a hyperbolic Riemannian manifold, the spatial uniformity loss is determined as follows: By using logarithmic mapping, the control point representation output by the control module, the cognitive prediction point representation output by the cognitive module, and the multimodal fusion feature point representation are respectively mapped to the tangent space of the preset base point to obtain the first tangent vector, the second tangent vector, and the third tangent vector of the base point; The spatial consistency loss is determined based on the distance between the first tangent vector and the third tangent vector of the base point, as well as the distance between the second tangent vector and the third tangent vector of the base point.

6. The method according to claim 1, characterized in that, The step of obtaining the Euclidean features corresponding to the first modality training sample, the Euclidean features corresponding to the second modality training sample corresponding to the first modality training sample, and the action label includes: Obtain a multimodal training sample set; the multimodal training sample set includes a first modality training sample, a second modality training sample corresponding to the first modality training sample, and action labels; wherein, the action labels correspond to the first modality training sample and the second modality training sample; The first modality training sample and the second modality training sample are respectively input into the preset feature extractor of the corresponding modality, and the Euclidean features corresponding to the first modality training sample and the Euclidean features corresponding to the second modality training sample are output.

7. An inference method based on a pre-trained perception-cognition-control multimodal large model, characterized in that, The pre-trained perception-cognition-control multimodal large model is obtained by training the method described in any one of claims 1 to 6, the method comprising the following steps: Obtain the Euclidean features corresponding to the first modality sample and the Euclidean features corresponding to the second modality sample; wherein, the first modality or the second modality includes the text modality; The Euclidean features corresponding to the acquired multimodal samples are input into the multimodal mapping and alignment module, and the output is the feature point representation corresponding to the multimodal samples; Input the feature point representations corresponding to the multimodal samples into the cognitive module, and output the cognitive prediction point representations corresponding to the multimodal samples; The cognitive prediction point representation corresponding to the multimodal sample is input into the control module, and the control point representation corresponding to the multimodal sample is output, so that the controller performs corresponding control actions based on the control point representation; wherein, the feature point representation, the cognitive prediction point representation and the control point representation are located on the same Riemannian manifold.

8. A training system for a large-scale multimodal perception-cognition-control model based on Riemannian manifolds, comprising a processor, a memory, and computer programs / instructions stored in the memory, characterized in that, The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 6.

Images