Action method and device based on pulse body model

Abstract

The invention provides an action method and device based on a pulse body model, and relates to the technical field of body intelligence related to robot control, and the method comprises the steps: obtaining a language instruction, a visual image and ontology sensing data received by a body intelligent agent; inputting the language instruction, the visual image and the ontology perception data into the trained pulse body model to obtain an action instruction; wherein the pulse body size model is constructed based on a first pulse neural network block and a second pulse neural network block, the first pulse neural network block is a pulse neural network block based on convolution, and the second pulse neural network block is a pulse neural network block based on Transform; and controlling the body agent to act based on the action instruction. According to the invention, high-energy-consumption calculation in the network is converted into a low-energy-consumption pulse-driven calculation form, and the reasoning energy consumption is greatly reduced by means of the neuromorphic chip, so that the endurance time and the capability of the self-service agent are greatly improved.

Description

Technical Field

[0001] This application relates to the field of embodied intelligence technology related to robot control, and in particular to a motion method and apparatus based on a pulse embodied large model. Background Technology

[0002] In recent years, with the rapid development of large model technology and the robotics industry, embodied large models have received widespread attention and application. By integrating multimodal information (such as vision, language, and touch), embodied large models enable intelligent agents to perceive the environment more comprehensively and, based on this, perform more complex reasoning and planning, thereby achieving full-body control of the robot.

[0003] However, existing robot products typically employ reinforcement learning and imitation learning methods to control motors, resulting in a limited range of capabilities for single tasks. Furthermore, they lack the ability to generalize to dynamic environmental changes, requiring retraining to perform new tasks. In addition, the limited battery capacity of robots struggles to meet the massive energy demands of AI-powered control systems, and cloud computing solutions introduce high latency, posing challenges to the robot's endurance and real-time performance. Therefore, an effective solution is urgently needed to address at least one of these issues. Summary of the Invention

[0004] To address the aforementioned issues, this application provides an action method based on a pulse embodied large model.

[0005] This application provides a motion method based on a pulse-embodied large model, including: Acquire language commands, visual images, and ontological perception data received by the embodied intelligent agent; The language instructions, the visual images, and the ontology perception data are input into the trained pulse embodied large model to obtain action instructions; wherein, the pulse embodied large model is constructed based on a first pulse neural network block and a second pulse neural network block, the first pulse neural network block being a convolution-based pulse neural network block, and the second pulse neural network block being a Transformer-based pulse neural network block; The embodied intelligent agent is controlled to perform actions based on the action commands.

[0006] According to the action method based on a pulse embodied large model provided in this application, before inputting the language instruction, the visual image, and the ontology perception data into the trained pulse embodied large model to obtain the action instruction, the method further includes: Based on the first spiking neural network block and the second spiking neural network block, a large spiking model is constructed. A training sample set is constructed based on heterogeneous open-source embodied datasets and / or synthetic datasets. The training sample set includes multiple training samples, including language instruction samples, state sequence samples, and action sequence labels. The state sequence samples include image samples from different perspectives and ontology-aware data samples. Based on the training sample set, the pulse large model is trained to obtain the pulse embodied large model.

[0007] According to the action method based on the pulse embodied large model provided in this application, the step of training the pulse large model based on the training sample set to obtain the pulse embodied large model includes: Obtain a trained spiking multimodal large model and a trained spiking large language model; wherein the spiking multimodal large model is constructed based on the first spiking neural network block, the text tagger, and the second spiking neural network block, and the spiking large language model is constructed based on the second spiking neural network block; Based on the training sample set, the pulse multimodal large model, and the pulse large language model, the pulse large model is trained to obtain the pulse embodied large model.

[0008] According to the action method based on a pulse embodied large model provided in this application, the pulse large model is trained based on the training sample set, the pulse multimodal large model, and the pulse large language model to obtain the pulse embodied large model, including: For any training sample in the training sample set, the image sample in the training sample is input into the pulse multimodal large model to obtain image encoding information, and the language instruction sample in the training sample is input into the pulse large language model to obtain text encoding information; The image encoding information is input into the first adapter for conversion to obtain visual modality information; the text encoding information is input into the second adapter for conversion to obtain text modality information; and the ontology perception data samples in the training samples are input into the third adapter for conversion to obtain perception modality information. The visual modal information, the text modal information, and the perceptual modal information are input into the pulse model to predict actions and obtain a predicted action sequence. Based on the predicted action sequence and the action sequence labels in the training samples, the model parameters of the pulse large model are adjusted; Repeat the steps of inputting image samples from the training sample set into the pulse multimodal large model and the following steps to continue training the pulse large model until the training stop condition is met, and obtain the pulse embodied large model.

[0009] According to the action method based on the pulse embodied large model provided in this application, after training the pulse large model based on the training sample set to obtain the pulse embodied large model, the method further includes: Obtain the initial fine-tuning dataset; Based on data augmentation techniques, the initial fine-tuning dataset is expanded to obtain the target fine-tuning dataset; Based on the target fine-tuning dataset, the model parameters of the pulse embodiment large model are fine-tuned.

[0010] According to the action method based on the pulse embodiment large model provided in this application, the step of fine-tuning the model parameters of the pulse embodiment large model based on the target fine-tuning dataset includes: Based on the target fine-tuning dataset, the model parameters of the pulse embodiment large model are fine-tuned using a multi-view observation task.

[0011] According to the action method based on a pulse embodiment large model provided in this application, the construction of a training sample set based on heterogeneous open-source embodiment datasets and / or synthetic datasets includes: Collect heterogeneous open-source embodied datasets, preprocess the open-source embodied datasets to obtain preprocessed open-source embodied datasets, and / or generate the synthetic dataset through a simulation environment; wherein, the preprocessing includes format conversion and action space conversion; The preprocessed open-source embodied dataset and / or the synthetic dataset are augmented to obtain the training sample set.

[0012] According to the action method based on the pulse embodied large model provided in this application, the first pulse neural network block includes a pulse neuron layer, a convolutional layer and a batch normalization layer connected in sequence. The second spiking neural network block includes a spiking-driven self-attention layer, a spiking neuron layer, a fully connected layer, and a batch normalization layer connected in sequence.

[0013] This application also provides a motion device based on a pulse-based avatar large model, including: The acquisition module is configured to acquire language instructions, visual images, and ontological perception data received by the embodied intelligent agent. The input module is configured to input the language instruction, the visual image, and the ontology perception data into the trained pulse embodied large model to obtain the action instruction; wherein the pulse embodied large model is constructed based on a first pulse neural network block and a second pulse neural network block, the first pulse neural network block being a convolution-based pulse neural network block, and the second pulse neural network block being a Transformer-based pulse neural network block; The action module is configured to control the embodied intelligent agent to perform actions based on the action instructions.

[0014] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the action method based on the pulse embodied large model as described above.

[0015] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the action method based on the pulse embodiment large model as described above.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the action method based on the pulse embodied large model as described above.

[0017] The action method and apparatus based on a pulsed embodied large model provided in this application acquires language commands, visual images, and proprioceptive data received by the embodied intelligent agent; inputs the language commands, visual images, and proprioceptive data into a trained pulsed embodied large model to obtain action commands; wherein the pulsed embodied large model is constructed based on a first spiking neural network block and a second spiking neural network block, the first spiking neural network block being a convolution-based spiking neural network block, and the second spiking neural network block being a Transformer-based spiking neural network block; and controls the embodied intelligent agent to perform actions based on the action commands. Addressing the problem of high inference energy consumption in multimodal large models leading to insufficient endurance of humanoid robots, this application replaces the multimodal large model with a pulsed architecture, converting the high-energy-consuming computation in the network into a low-energy-consuming pulsed-driven computation form. Specifically, by introducing spiking neural networks, the original large model structure is converted into a pulsed architecture, including convolution-based SNN blocks and Transformer-based SNN blocks. With the help of neuromorphic chips, inference energy consumption is greatly reduced, thereby significantly improving the endurance and capability of the embodied intelligent agent. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the motion method based on the pulse embodied large model provided in this application.

[0020] Figure 2 This is a schematic diagram of the structure of the first spiking neural network block provided in this application.

[0021] Figure 3 This is a schematic diagram of the structure of the second spiking neural network block provided in this application.

[0022] Figure 4 This is a schematic diagram of the structure of the spiking neuron layer provided in this application.

[0023] Figure 5 This is a flowchart illustrating the training process of the large pulse embodied model provided in this application.

[0024] Figure 6 This is a schematic diagram of the motion device based on the pulse embodied large model provided in this application.

[0025] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] First, a brief explanation of the relevant content involved in this application will be given.

[0028] Among the related technologies, three types of solutions for controlling robots based on embodied large models are provided.

[0029] Option 1 involves controlling embodied agents based on a multimodal embodied large model. This approach creates an instruction-adjusted dataset according to the task scenario of the embodied agent, then uses this dataset to validate and select a model base for the pre-trained model. Finally, the selected model base is trained using retrieval-enhanced generation and supervised fine-tuning methods. However, this approach requires a large amount of high-quality embodied data and a long training period, making it prone to overfitting and loss of the generalization ability of the original model base.

[0030] Scheme 2 uses a simple spiking neural network and a support vector machine as the mobile robot's movement state recognition model and the robotic arm's motion state recognition model, respectively, and employs a reinforcement learning training policy network and a value network as the composite robot collaborative scheduling module. However, this scheme has a relatively simple action space, the reward function for reinforcement learning is difficult to design, and the interaction cost in real-world environments is high, making it only suitable for simple, coarse-grained tasks.

[0031] Option 3 inputs the joint angles of the robotic arm and the position of the target object into the prediction model to obtain the change values ​​of the joint angles, and then adjusts the model parameters of the prediction model based on the strategy optimization algorithm. However, this option is only applicable to the grasping task of the robotic arm, has a limited application scenario, and has a low overall intelligence level.

[0032] Furthermore, in recent hardware implementations, spiking-based computation has proven to be orders of magnitude faster and more energy-efficient. Embodied large models based on spiking neural networks (SNNs) could be a low-energy solution for robot control.

[0033] Therefore, in order to solve the above problems, this application provides a motion method and apparatus based on a pulse-embodied large model.

[0034] The following is combined Figures 1-7 This application describes the motion method and apparatus based on a pulse embodied large model.

[0035] Figure 1 This is a flowchart illustrating the motion method based on a pulse embodied large model provided in this application, as shown below. Figure 1 As shown, the method includes the following: Step 101: Acquire the language instructions, visual images, and proprioceptive perception data received by the embodied intelligent agent; Step 102: Input the language instruction, the visual image, and the ontology perception data into the trained pulse embodied large model to obtain the action instruction; wherein, the pulse embodied large model is constructed based on a first pulse neural network block and a second pulse neural network block, the first pulse neural network block is a convolution-based pulse neural network block, and the second pulse neural network block is a Transformer-based pulse neural network block; Step 103: Control the embodied intelligent agent to perform actions based on the action instructions.

[0036] It should be noted that the action method based on the pulse embodied large model provided in this application can be applied to the control unit of the embodied intelligent agent, that is, the execution subject is the control unit.

[0037] Specifically, an embodied intelligent agent can be an embodied robot. Proprioceptive perception data can be low-dimensional proprioceptive perception data.

[0038] Specifically, the pulse embodied large model includes a first pulse neural network block and a second pulse neural network block. In addition, the pulse embodied large model also includes a decision head. The input of the pulse embodied large model can be different modal information, and the output is an action sequence.

[0039] Specifically, the pulse embodied model can be deployed on the control unit of the embodied intelligent agent, such as a neuromorphic chip, to maximize the energy efficiency advantages of neuromorphic computing.

[0040] Specifically, see Figure 2 , Figure 2 This is a schematic diagram of the structure of the first spiking neural network block provided in this application: The first spiking neural network block is a convolution-based SNN block, including a spiking neural layer (SN), a convolutional layer (Conv), and a batch normalization layer (BN) connected in sequence; the first spiking neural network block can be represented as BN(Conv(SN(X1)))+X1, to ensure that the input to the convolutional layer is a binary pulse signal. Here, X1 represents the information input to the first spiking neural network block.

[0041] Specifically, see Figure 3 , Figure 3 This is a schematic diagram of the structure of the second spiking neural network block provided in this application: The second spiking neural network block is a Transformer-based SNN block, including a spike-driven self-attention (SDSA), a spiking neuron (SN) layer, a fully connected layer (FC), and a batch normalization (BN) layer connected in sequence. The first spiking neural network block first converts the query Q, key K, and value V into spike form, then integrates them through spike-driven self-attention, and finally outputs the result through a multilayer perceptron (MLP) of the form BN(FC(SN(X2)))+X2.

[0042] Where X2 represents the query, key, and value in pulse form after pulse-driven self-attention integration. The inputs to the pulse-driven self-attention module are all pulse tensors, calculated using the formula SDSA(Q, K, V) = SN((QK)). T For residual learning in SNNs, a shortcut can be established between the membrane potentials of neurons within a block using the membrane shortcut method, and identity mapping can be achieved through impulse-driven mechanisms.

[0043] It's important to note that SNNs represent a low-power alternative to traditional machine intelligence. The key to realizing this ambitious concept is the unique spiking mechanism: when an SNN runs on a neuromorphic chip, only a subset of spiking neurons are activated to perform sparse synaptic computation. Neuromorphic computing is essentially an algorithm-hardware co-design paradigm where neurons are first modeled as spiking neurons, then an SNN is built at the algorithmic level based on these spiking neurons, and finally, the spiking-driven SNN is deployed on the neuromorphic chip.

[0044] The spiking neuron layer integrates spatiotemporal information into the membrane potential, then converts it into a binary pulse for pulse-driven computation in the next layer. See [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of the spiking neuron layer provided in this application: The spiking neuron layer can adopt the standard leaky integration-and-fire model, and its dynamics can be expressed by the following three formulas: H t =V t-1 +X t S t =Heaviside(H t -v th ) V t =v reset S t +βH t (1) t -S t ) Among them, X t It is a time step The input current at the point is obtained from the pulse signal of the previous layer through pulse-driven operators such as convolutional layers, fully connected layers, and self-attention layers. During charging, the membrane voltage H... t Integrating the membrane voltage V from the previous moment t-1 With the current input current X t Then, the Heaviside step function is used to determine whether to fire a pulse. When H t Reaching the membrane voltage threshold vth At that time, S t =1, representing the spiking neuron firing a pulse, and the final membrane voltage V at the current moment. t It will be reset to the resting potential v reset Conversely, S t =0 indicates that the spiking neuron does not fire a pulse, H t It will decay to V based on the decay factor β. t This represents leakage current in neurons. Figure 4 In the diagram, yellow lines represent charging, blue lines represent discharging, and red lines represent resetting.

[0045] In practical applications, user commands can be acquired through input devices. For example, voice input can be acquired through a voice input device and then converted into spoken commands using a speech recognizer. Alternatively, text input, i.e., spoken commands, can be acquired through a touchscreen and / or text editing device. Furthermore, visual images can be obtained through whole-body visual sensors installed on the specific agent, and proprioceptive data can be acquired through other modal sensors installed on the specific agent.

[0046] Furthermore, visual images acquired by the whole-body visual sensor and proprioceptive data acquired by other modal sensors are asynchronously input to the pulse embodied large model at a higher frequency. These data, along with recently obtained language commands, serve as the basis for decision-making, outputting action commands. Then, based on the action commands, specific intelligent agents are controlled to perform corresponding actions.

[0047] The action method based on the spiking embodied large model provided in this application addresses the problem of high inference energy consumption in multimodal large models, which leads to insufficient endurance of humanoid robots. It replaces the multimodal large model with a spiking architecture, transforming the high-energy-consuming computation in the network into a low-energy-consuming spiking-driven computation form. That is, by introducing a spiking neural network, the original large model structure is transformed into a spiking architecture, which includes convolution-based SNN blocks and Transformer-based SNN blocks. With the help of neuromorphic chips, the inference energy consumption is greatly reduced, thereby significantly improving the endurance and capability of the embodied intelligent agent.

[0048] Optionally, before inputting the language instruction, the visual image, and the ontology perception data into the trained pulse embodied large model to obtain the action instruction, the method further includes: Based on the first spiking neural network block and the second spiking neural network block, a large spiking model is constructed. A training sample set is constructed based on heterogeneous open-source embodied datasets and / or synthetic datasets. The training sample set includes multiple training samples, including language instruction samples, state sequence samples, and action sequence labels. The state sequence samples include image samples from different perspectives and ontology-aware data samples. Based on the training sample set, the pulse large model is trained to obtain the pulse embodied large model.

[0049] In practical applications, an initial pulse embodied large model, i.e., a pulse large model, can be constructed by using convolution-based SNN blocks and Transformer-based SNN blocks.

[0050] Furthermore, a pre-training dataset, or training sample set, is constructed based on heterogeneous open-source embodied datasets and / or synthetic datasets. The pre-training data (training samples) includes language instructions (language instruction samples), action sequences (action sequence labels), and state sequences acquired from different sensors (state sequence samples), where the states are images from different perspectives (image samples) and low-dimensional ontology-aware data (ontology-aware data samples). Based on the obtained training sample set, the pulse large model can be trained to obtain a trained pulse embodied large model.

[0051] In this embodiment of the application, the pulsation embodied large model is trained using heterogeneous open-source embodied datasets and / or synthetic datasets, which can ensure the robustness of the pulsation embodied large model.

[0052] Optionally, training the pulse big model based on the training sample set to obtain the pulse embodied big model includes: Obtain a trained spiking multimodal large model and a trained spiking large language model; wherein the spiking multimodal large model is constructed based on the first spiking neural network block, the text tagger, and the second spiking neural network block, and the spiking large language model is constructed based on the second spiking neural network block; Based on the training sample set, the pulse multimodal large model, and the pulse large language model, the pulse large model is trained to obtain the pulse embodied large model.

[0053] Specifically, the pulse multimodal large model is constructed using convolution-based SNN blocks, text tokenizers, and Transformer-based SNN blocks. Image data is converted into visual modal information through convolution-based SNN blocks, then aligned with the text modal information obtained by the text tokenizer, and then passed through the Transformer-based SNN blocks to obtain embedding vectors. The model is pre-trained on a large-scale image and text training dataset to obtain the trained pulse multimodal large model.

[0054] Specifically, the pulse large language model is constructed using Transformer-based SNN blocks and pre-trained on a large-scale text training dataset to obtain a trained pulse large language model.

[0055] In practical applications, pre-trained pulse multimodal large models and pulse large language models can be used to convert language instruction samples and image samples in the training sample set into information that the pulse embodied large model can understand, thereby training the pulse embodied large model. This achieves cross-robot pre-trained embodied large models.

[0056] Optionally, the step of training the pulse big model based on the training sample set, the pulse multimodal big model, and the pulse big language model to obtain the pulse embodied big model includes: For any training sample in the training sample set, the image sample in the training sample is input into the pulse multimodal large model to obtain image encoding information, and the language instruction sample in the training sample is input into the pulse large language model to obtain text encoding information; The image encoding information is input into the first adapter for conversion to obtain visual modality information; the text encoding information is input into the second adapter for conversion to obtain text modality information; and the ontology perception data samples in the training samples are input into the third adapter for conversion to obtain perception modality information. The visual modal information, the text modal information, and the perceptual modal information are input into the pulse model to predict actions and obtain a predicted action sequence. Based on the predicted action sequence and the action sequence labels in the training samples, the model parameters of the pulse large model are adjusted; Repeat the above steps to continue training the pulse large model until the training stop condition is met, thus obtaining the pulse embodied large model.

[0057] The above steps refer to inputting the image samples from the training samples into the pulse multimodal large model for any training sample in the training sample set and the subsequent steps.

[0058] Specifically, the training stopping condition can be at least one of the following: accuracy reaches an accuracy threshold, loss value is less than a loss threshold, loss value change rate is less than a change rate threshold, and the number of iterations reaches an iteration threshold.

[0059] In practical applications, images and language commands are encoded into corresponding pulse sequences through pre-trained pulse multimodal large models and pulse large language models. The pulse sequences (image encoding information and text encoding information) obtained from the encoding of ontology-aware data (ontology-aware data samples) and images and language commands (language command samples and image samples) are aligned with the input dimensions of each modality via an adapter. Finally, the pulse embodied large model outputs the action commands that the robot needs to execute. The last layer of the pulse embodied large model is the readout layer, which converts the pulse sequences into corresponding action commands.

[0060] For example, see Figure 5 , Figure 5 This is a flowchart illustrating the training process of the pulse embodied large model provided in this application: During training, the parameters of the pulse multimodal large model are frozen and used as a visual encoder, then converted into visual modal information via adapter 1 (first adapter). Additionally, the parameters of the pulse large language model are frozen and used as a text encoder, then converted into text modal information via adapter 2 (second adapter). Low-dimensional proprioceptive data, representing low-dimensional vectors of robot physical quantities, including proprioception, action blocks, and control frequencies, are converted into perceptual modal information via adapter 3 (third adapter). All three adapters employ an MLP architecture and are trained together with the parameters of the pulse embodied large model to align different modal information. This is achieved by predicting a series of actions (predicting action sequences), i.e., action blocks, in a single inference. This encourages time consistency and mitigates error accumulation over time by reducing the number of decisions in the task, where T a Indicates the length of the action block, a t Indicates the action at time t. Indicates time The action at that time Represents time t and T thereafter a A set of actions. Action instructions can be used for robot control.

[0061] In this embodiment, a pre-trained pulse multimodal large model and a pulse large language model are used to convert visual and linguistic information into information that the pulse embodied large model can understand. Decisions are made in conjunction with ontology perception data, maximizing the use of network-scale training data to fully leverage the generalization and rapid learning capabilities of the large model.

[0062] Optionally, after training the pulse big model based on the training sample set to obtain the pulse embodied big model, the method further includes: Obtain the initial fine-tuning dataset; Based on data augmentation techniques, the initial fine-tuning dataset is expanded to obtain the target fine-tuning dataset; Based on the target fine-tuning dataset, the model parameters of the pulse embodiment large model are fine-tuned.

[0063] Specifically, after training, the pulsation embodied large model can be fine-tuned on high-quality, specified real-machine robot data to enhance its precise operational capabilities within that robot configuration. During fine-tuning, data augmentation techniques will be used to address overfitting issues caused by the relatively small size of the fine-tuning dataset. Thus, utilizing a high-quality fine-tuning dataset facilitates the generalization capabilities of the pulsation embodied large model.

[0064] Optionally, the step of fine-tuning the model parameters of the pulse embodiment large model based on the target fine-tuning dataset includes: Based on the target fine-tuning dataset, the model parameters of the pulse embodiment large model are fine-tuned using a multi-view observation task.

[0065] Specifically, during the fine-tuning process, based on the target fine-tuning dataset, single-view and multi-view (image acquisition devices set for head, wrist and other parts, etc.) observations can be selected according to the task requirements. Multi-view images are processed using a shared visual encoder (visual encoders with the same weight, i.e., pulse multimodal large model). Then, visual features are connected along the token dimension to fine-tune the model parameters of the pulse embodied large model.

[0066] In this embodiment, by combining data-augmented target fine-tuning datasets with multi-view observations, the generalized operational capabilities of a specified robot are achieved using high-quality fine-tuning datasets. Addressing the issues of reward function design and environment interaction in reinforcement learning schemes, supervised training and fine-tuning are performed using data collected from real machines and simulation data. The generalization capability of embodied large models is leveraged to solve multi-task problems that are difficult to accomplish with reinforcement learning.

[0067] Optionally, the construction of the training sample set based on heterogeneous open-source embodied datasets and / or synthetic datasets includes: Collect heterogeneous open-source embodied datasets, preprocess the open-source embodied datasets to obtain preprocessed open-source embodied datasets, and / or generate the synthetic dataset through a simulation environment; wherein, the preprocessing includes format conversion and action space conversion; The preprocessed open-source embodied dataset and / or the synthetic dataset are augmented to obtain the training sample set.

[0068] Specifically, heterogeneous open-source embodied datasets can be processed to unify them into a specified data format, and the actions of various robots can be embedded into a unified space. Furthermore, users can generate synthetic data through a simulation environment. Further, the preprocessed open-source embodied dataset and / or synthetic dataset can be augmented to obtain a training dataset.

[0069] The action method based on a spiking embodied large model provided in this application firstly transforms the original large model structure into a spiking architecture by introducing a spiking neural network, thus constructing a spiking embodied large model. Secondly, by collecting heterogeneous open-source embodied datasets and synthetic datasets, and combining pre-trained spiking multimodal large models and spiking large language models, a cross-robot pre-trained spiking embodied large model is realized. Then, by collecting a large amount of high-quality fine-tuning data from specific robots, the pre-trained model is fine-tuned for the required scenarios, tasks, and objectives, enabling precise robot operation under complex tasks. Finally, by deploying the spiking embodied large model onto a neuromorphic chip for action control of the embodied agent, the reliability and low energy consumption of real-world operation are ensured.

[0070] Furthermore, the motion method based on the impulse embodied large model provided in this application can be used in mobile robot products and is suitable for R&D personnel engaged in embodied intelligent dexterity operations. When the robot's task generalization and inference energy efficiency requirements are high, the embodied intelligence method based on the impulse large model can be used to control the robot. While leveraging the task generalization of the embodied large model, the inference energy consumption of the embodied large model is greatly reduced, which has high commercial value.

[0071] The following describes the motion device based on the pulse embodied large model provided in this application. The motion device based on the pulse embodied large model described below and the motion method based on the pulse embodied large model described above can be referred to in correspondence.

[0072] Figure 6 This is a schematic diagram of the motion device based on the pulse avatar large model provided in this application, such as... Figure 6 As shown, the device includes: The acquisition module 601 is configured to acquire language instructions, visual images and ontological perception data received by the embodied intelligent agent; The input module 602 is configured to input the language instruction, the visual image, and the ontology perception data into the trained pulse embodied large model to obtain the action instruction; wherein the pulse embodied large model is constructed based on a first pulse neural network block and a second pulse neural network block, the first pulse neural network block being a convolution-based pulse neural network block, and the second pulse neural network block being a Transformer-based pulse neural network block; Action module 603 is configured to control the embodied intelligent agent to perform actions based on the action instructions.

[0073] The motion device based on the spiking embodied large model provided in this application addresses the problem of high inference energy consumption in multimodal large models, which leads to insufficient endurance of humanoid robots. It replaces the multimodal large model with a spiking architecture, transforming the high-energy-consuming computation in the network into a low-energy-consuming spiking-driven computation form. That is, by introducing a spiking neural network, the original large model structure is transformed into a spiking architecture, which includes convolution-based SNN blocks and Transformer-based SNN blocks. With the help of a neuromorphic chip, the inference energy consumption is greatly reduced, thereby significantly improving the endurance and capability of the embodied intelligent agent.

[0074] Optionally, the device further includes a training module configured to: Based on the first spiking neural network block and the second spiking neural network block, a large spiking model is constructed. A training sample set is constructed based on heterogeneous open-source embodied datasets and / or synthetic datasets. The training sample set includes multiple training samples, including language instruction samples, state sequence samples, and action sequence labels. The state sequence samples include image samples from different perspectives and ontology-aware data samples. Based on the training sample set, the pulse large model is trained to obtain the pulse embodied large model.

[0075] Optionally, the training module is specifically configured as follows: Obtain a trained spiking multimodal large model and a trained spiking large language model; wherein the spiking multimodal large model is constructed based on the first spiking neural network block, the text tagger, and the second spiking neural network block, and the spiking large language model is constructed based on the second spiking neural network block; Based on the training sample set, the pulse multimodal large model, and the pulse large language model, the pulse large model is trained to obtain the pulse embodied large model.

[0076] Optionally, the training module is specifically configured as follows: For any training sample in the training sample set, the image sample in the training sample is input into the pulse multimodal large model to obtain image encoding information, and the language instruction sample in the training sample is input into the pulse large language model to obtain text encoding information; The image encoding information is input into the first adapter for conversion to obtain visual modality information; the text encoding information is input into the second adapter for conversion to obtain text modality information; and the ontology perception data samples in the training samples are input into the third adapter for conversion to obtain perception modality information. The visual modal information, the text modal information, and the perceptual modal information are input into the pulse model to predict actions and obtain a predicted action sequence. Based on the predicted action sequence and the action sequence labels in the training samples, the model parameters of the pulse large model are adjusted; Repeat the above steps to continue training the pulse large model until the training stop condition is met, thus obtaining the pulse embodied large model.

[0077] Optionally, the device further includes a fine-tuning module configured to: Obtain the initial fine-tuning dataset; Based on data augmentation techniques, the initial fine-tuning dataset is expanded to obtain the target fine-tuning dataset; Based on the target fine-tuning dataset, the model parameters of the pulse embodiment large model are fine-tuned.

[0078] Optionally, the fine-tuning module is specifically configured as follows: Based on the target fine-tuning dataset, the model parameters of the pulse embodiment large model are fine-tuned using a multi-view observation task.

[0079] Optionally, the training module is specifically configured as follows: Collect heterogeneous open-source embodied datasets, preprocess the open-source embodied datasets to obtain preprocessed open-source embodied datasets, and / or generate the synthetic dataset through a simulation environment; wherein, the preprocessing includes format conversion and action space conversion; The preprocessed open-source embodied dataset and / or the synthetic dataset are augmented to obtain the training sample set.

[0080] Optionally, the first spiking neural network block includes a spiking neuron layer, a convolutional layer, and a batch normalization layer connected in sequence; The second spiking neural network block includes a spiking-driven self-attention layer, a spiking neuron layer, a fully connected layer, and a batch normalization layer connected in sequence.

[0081] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute an action method based on a pulse embodied model. The method includes: acquiring language instructions, visual images, and proprioceptive perception data received by the embodied agent; inputting the language instructions, the visual images, and the proprioceptive perception data into a trained pulse embodied model to obtain action instructions; wherein the pulse embodied model is constructed based on a first spiking neural network block and a second spiking neural network block, the first spiking neural network block being a convolution-based spiking neural network block and the second spiking neural network block being a Transformer-based spiking neural network block; and controlling the embodied agent to perform actions based on the action instructions.

[0082] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the action method based on the spiking embodied large model provided by the above methods. The method includes: acquiring language instructions, visual images, and ontology perception data received by the embodied agent; inputting the language instructions, the visual images, and the ontology perception data into a trained spiking embodied large model to obtain action instructions; wherein the spiking embodied large model is constructed based on a first spiking neural network block and a second spiking neural network block, the first spiking neural network block being a convolution-based spiking neural network block and the second spiking neural network block being a Transformer-based spiking neural network block; and controlling the embodied agent to perform actions based on the action instructions.

[0084] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an action method based on a spiking embodied large model provided by the above methods. The method includes: acquiring language instructions, visual images, and ontology perception data received by the embodied agent; inputting the language instructions, the visual images, and the ontology perception data into a trained spiking embodied large model to obtain action instructions; wherein the spiking embodied large model is constructed based on a first spiking neural network block and a second spiking neural network block, the first spiking neural network block being a convolution-based spiking neural network block and the second spiking neural network block being a Transformer-based spiking neural network block; and controlling the embodied agent to perform actions based on the action instructions.

[0085] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A motion method based on a pulse embodied large model, characterized in that, include: Acquire language commands, visual images, and ontological perception data received by the embodied intelligent agent; The language instructions, the visual images, and the ontology perception data are input into the trained pulse embodied large model to obtain action instructions; wherein, the pulse embodied large model is constructed based on a first pulse neural network block and a second pulse neural network block, the first pulse neural network block being a convolution-based pulse neural network block, and the second pulse neural network block being a Transformer-based pulse neural network block; The embodied intelligent agent is controlled to perform actions based on the action commands.

2. The motion method based on a pulse-embodied large model according to claim 1, characterized in that, Before inputting the language instructions, the visual images, and the proprioceptive data into the trained pulse embodied large model to obtain action instructions, the method further includes: Based on the first spiking neural network block and the second spiking neural network block, a large spiking model is constructed. A training sample set is constructed based on heterogeneous open-source embodied datasets and / or synthetic datasets. The training sample set includes multiple training samples, including language instruction samples, state sequence samples, and action sequence labels. The state sequence samples include image samples from different perspectives and ontology-aware data samples. Based on the training sample set, the pulse large model is trained to obtain the pulse embodied large model.

3. The motion method based on a pulse-embodied large model according to claim 2, characterized in that, The step of training the pulse big model based on the training sample set to obtain the pulse embodied big model includes: Obtain a trained spiking multimodal large model and a trained spiking large language model; wherein the spiking multimodal large model is constructed based on the first spiking neural network block, the text tagger, and the second spiking neural network block, and the spiking large language model is constructed based on the second spiking neural network block; Based on the training sample set, the pulse multimodal large model, and the pulse large language model, the pulse large model is trained to obtain the pulse embodied large model.

4. The motion method based on a pulse-embodied large model according to claim 3, characterized in that, The step of training the pulse big model based on the training sample set, the pulse multimodal big model, and the pulse big language model to obtain the pulse embodied big model includes: For any training sample in the training sample set, the image sample in the training sample is input into the pulse multimodal large model to obtain image encoding information, and the language instruction sample in the training sample is input into the pulse large language model to obtain text encoding information; The image encoding information is input into the first adapter for conversion to obtain visual modality information; the text encoding information is input into the second adapter for conversion to obtain text modality information; and the ontology perception data samples in the training samples are input into the third adapter for conversion to obtain perception modality information. The visual modal information, the text modal information, and the perceptual modal information are input into the pulse model to predict actions and obtain a predicted action sequence. Based on the predicted action sequence and the action sequence labels in the training samples, the model parameters of the pulse large model are adjusted; Repeat the steps of inputting image samples from the training sample set into the pulse multimodal large model and the following steps to continue training the pulse large model until the training stop condition is met, and obtain the pulse embodied large model.

5. The motion method based on a pulse-embodied large model according to any one of claims 2-4, characterized in that, After training the pulse big model based on the training sample set to obtain the pulse embodied big model, the process further includes: Obtain the initial fine-tuning dataset; Based on data augmentation techniques, the initial fine-tuning dataset is expanded to obtain the target fine-tuning dataset; Based on the target fine-tuning dataset, the model parameters of the pulse embodiment large model are fine-tuned.

6. The motion method based on a pulse-embodied large model according to claim 5, characterized in that, The fine-tuning of the model parameters of the pulse embodiment large model based on the target fine-tuning dataset includes: Based on the target fine-tuning dataset, the model parameters of the pulse embodiment large model are fine-tuned using a multi-view observation task.

7. The motion method based on a pulse-embodied large model according to any one of claims 2-4, characterized in that, The training sample set, constructed based on heterogeneous open-source embodied datasets and / or synthetic datasets, includes: Collect heterogeneous open-source embodied datasets, preprocess the open-source embodied datasets to obtain preprocessed open-source embodied datasets, and / or generate the synthetic dataset through a simulation environment; wherein, the preprocessing includes format conversion and action space conversion; The preprocessed open-source embodied dataset and / or the synthetic dataset are augmented to obtain the training sample set.

8. The motion method based on a pulse-embodied large model according to any one of claims 1-4, characterized in that, The first spiking neural network block includes a spiking neuron layer, a convolutional layer, and a batch normalization layer connected in sequence; The second spiking neural network block includes a spiking-driven self-attention layer, a spiking neuron layer, a fully connected layer, and a batch normalization layer connected in sequence.

9. A motion device based on a pulse-embodied large model, characterized in that, include: The acquisition module is configured to acquire language instructions, visual images, and ontological perception data received by the embodied intelligent agent. The input module is configured to input the language instruction, the visual image, and the ontology perception data into the trained pulse embodied large model to obtain the action instruction; wherein the pulse embodied large model is constructed based on a first pulse neural network block and a second pulse neural network block, the first pulse neural network block being a convolution-based pulse neural network block, and the second pulse neural network block being a Transformer-based pulse neural network block; The action module is configured to control the embodied intelligent agent to perform actions based on the action instructions.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the action method based on the pulse embodied large model as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the action method based on the pulse embodied large model as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the action method based on the pulse embodied large model as described in any one of claims 1 to 8.

Images