Brain-like decision-making method, device and equipment for body intelligence and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a brain-like decision-making method, device and equipment for body intelligence and a storage medium, and is applied to a robot. The method comprises the following steps: receiving a human language instruction, and carrying out semantic understanding on the human language instruction to obtain a task description; collecting multi-modal data, and generating an environment model based on the multi-modal data; and generating a pulse event sequence by adopting a pulse neural network model based on the task description and the environment model, and generating a motion strategy of the robot based on the pulse event sequence. Through application of the bionic neuromorphic computing architecture, the decision-making speed of the robot in a complex scene is increased by 50 times, the power consumption is reduced to 1 / 10, the real-time response capability of the robot is greatly improved, the robot can quickly cope with various emergencies, for example, when encountering a sudden obstacle, the robot can quickly do an avoidance action, and the robot can be prevented from being damaged. And collision accidents are avoided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to brain-like decision-making methods, devices, equipment and storage media for embodied intelligence. Background Art

[0002] Embodied intelligence, short for embodied artificial intelligence, is a key research direction in the field of artificial intelligence. It emphasizes the emergence of intelligent behavior through real-time interaction between an agent and its environment. In recent years, robotics hardware technology has advanced rapidly, with significant breakthroughs achieved in the diversity of physical forms and the sophistication of robotic movements. Humanoid robots, for example, are capable of not only simulating basic human movements like walking and running, but also performing more complex movements such as dancing and gymnastics. Mobile robots are also increasingly capable of navigating complex terrain, now capable of traversing rugged mountain paths and crossing ravines.

[0003] However, the development of robotic decision-making and control technology has lagged behind, becoming a bottleneck that restricts their widespread application in complex scenarios. In complex, unstructured environments, robots struggle to quickly and accurately perceive environmental changes and make informed decisions, severely limiting their scope of application and effectiveness.

[0004] (2) Deficiencies of existing robot decision-making technology:

[0005] 1. Limitations of pre-programmed logic:

[0006] Currently, the vast majority of robots rely on pre-programmed logic to make decisions. In structured, relatively stable environments, this approach allows robots to successfully complete their pre-set tasks. However, once they enter dynamic, uncertain, and unstructured environments, the drawbacks of pre-programmed logic become apparent. Take the Boston Dynamics Atlas robot, for example: its fall rate exceeds 15% when operating in complex terrain. This is primarily because its pre-set decision-making process struggles to cope with unexpected situations, such as sudden changes in terrain and the random appearance of obstacles. This highlights the serious lack of adaptability of traditional pre-programmed decision-making models in complex and dynamic environments, and their inability to guarantee stable robot operation and task execution.

[0007] 2. Decision-making speed and power consumption issues:

[0008] Traditional robotic decision-making systems often suffer from slow decision-making speeds and excessive power consumption when handling complex scenarios. From a hardware perspective, traditional computing chips require extensive data transmission and calculations to execute complex algorithms, resulting in a lengthy decision-making process.

[0009] 3. Lack of multimodal perception fusion and learning capabilities:

[0010] In practical applications, robots need to integrate multimodal sensory information, such as vision, force, and hearing, to fully and accurately understand their surroundings. However, existing technologies have significant limitations in the accuracy and speed of integrating multimodal sensory data, resulting in large errors in environmental modeling. Summary of the Invention

[0011] In view of this, the present invention provides a brain-like decision-making method, device, equipment and storage medium for embodied intelligence to solve the problems of lack of multimodal perception fusion and unintelligent decision-making strategies of robots.

[0012] In a first aspect, the present invention provides a brain-inspired decision-making method for embodied intelligence, applied to a robot, comprising:

[0013] Receive human language instructions, perform semantic understanding on the human language instructions, and obtain task descriptions;

[0014] Collect multimodal data and generate an environmental model based on the multimodal data;

[0015] Based on the task description and environment model, a pulse neural network model is used to generate a pulse event sequence, and the robot's motion strategy is generated based on the pulse event sequence.

[0016] The present invention provides a brain-like decision-making method for embodied intelligence, which receives human language instructions and performs semantic understanding, and can accurately convert natural language instructions into abstract semantic representations. This accurate semantic understanding allows the robot to clearly define the task objectives and avoid task execution deviations due to misunderstanding of instructions. Collecting multimodal data and generating an environmental model enables the robot to obtain more comprehensive and accurate environmental information. Whether it is complex terrain recognition or delicate object manipulation, it can make reasonable decisions based on accurate environmental perception, significantly enhancing the robot's adaptability in complex environments. Based on the task description and environmental model, a pulse neural network model is used to generate a pulse event sequence and generate a motion strategy, so that the robot can quickly and intelligently plan the action path and operation mode when facing dynamically changing environments and task requirements, achieve efficient decision-making and precise action execution, and solve the problem of the robot's lack of multimodal perception fusion and inaccurate intelligence in the decision-making strategy.

[0017] In an optional embodiment, performing semantic understanding on human language instructions to obtain a task description includes:

[0018] A large language model is used to perform lexical analysis, syntactic analysis, and semantic analysis on human language instructions, converting them into abstract semantic representations for robots, and then obtaining a robot task description based on the semantic representation.

[0019] The present invention provides a brain-like decision-making method for embodied intelligence, which uses a large language model to perform lexical, syntactic and semantic analysis on human language instructions. It can deeply analyze the instruction structure and meaning, and clearly present the logical relationship and hierarchical structure of the task, converting natural language into an abstract semantic representation that can be understood by the robot. The task description obtained based on this is accurate and provides accurate guidance for the robot to perform subsequent tasks.

[0020] In an optional embodiment, collecting multimodal data and generating an environment model based on the multimodal data includes:

[0021] Multimodal sensors are used to collect multimodal data, and data preprocessing is performed to remove noise and redundant data; multimodal data includes visual data, force data, and auditory data;

[0022] The spatiotemporal alignment algorithm is used to time-align the multimodal data after data preprocessing based on the timestamp;

[0023] Based on the spatial position, a spatiotemporal alignment algorithm is used to map the multimodal data after data preprocessing into a unified spatial coordinate system for spatial alignment;

[0024] A fusion algorithm is used to fuse the time-aligned and space-aligned multimodal data to generate an environmental model.

[0025] The present invention provides a brain-inspired decision-making method for embodied intelligence. This method effectively improves data quality by preprocessing multimodal data, including denoising and removing redundant data. Time alignment based on timestamps and spatial alignment based on spatial position eliminate time delays and spatial deviations between multimodal data. Precise alignment enables robots to obtain more accurate environmental information. In tasks such as navigation and obstacle avoidance, the robot can more accurately determine the actual position and motion state of obstacles, avoiding misjudgments caused by temporal and spatial inconsistencies in the data, significantly enhancing the accuracy of the robot's environmental perception. A fusion algorithm is used to fuse the temporally and spatially aligned multimodal data. This method integrates the advantages of various types of data, such as vision, force perception, and hearing, to generate a more comprehensive and accurate environmental model. The complementary fusion of multimodal data enables the environmental model to include richer environmental features and details, effectively improving the accuracy of environmental modeling and providing a more reliable basis for robot decision-making and task execution in complex environments.

[0026] In an optional embodiment, the spiking neural network model includes a spiking event conversion layer and a spiking neural network decision layer;

[0027] Based on the task description and environment model, a pulse neural network model is used to generate a pulse event sequence, and the robot's motion strategy is generated based on the pulse event sequence, including:

[0028] The task description and the environment model are mapped into a pulse firing rate by adopting a pulse event conversion layer according to a preset time window slicing, and the pulse firing rate is pulse thinned to generate a space-time pulse matrix.

[0029] The motion parameter control signal is generated by adopting a pulse neural network decision layer to decode the space-time pulse matrix, and the motion parameter control signal is used as the motion strategy of the robot.

[0030] The brain-like decision method for embodied intelligence provided by the application has the pulse event conversion layer mapping the task description and the environment model into a pulse firing rate according to a preset time window slicing and performing pulse thinning to generate a space-time pulse matrix. This processing mode effectively reduces the redundancy of data, and the complex information is represented by sparse pulse signals. Compared with the traditional continuous data processing mode, unnecessary data transmission and calculation are reduced, and the data processing is more efficient. The pulse neural network decision layer decodes the space-time pulse matrix to generate a motion parameter control signal as the motion strategy of the robot. The pulse neural network has strong pattern recognition and information processing capability, can accurately extract key information related to the task and the environment according to the input pulse matrix, and convert the key information into specific motion parameters. When the robot performs a grasping task, the object position, shape in the environment model and the grasping requirements in the task description can be used to accurately calculate the motion trajectory, grasping force and other parameters of the robot, so that the robot can accurately perform the task. Through the bionic neural morphological computing architecture, the decision speed of the robot in a complex scene is greatly improved, and the decision time is shortened from the traditional long period to instantaneous response. At the same time, the power consumption is significantly reduced, the energy consumption is reduced, and the endurance time of the robot is prolonged, so that the robot can stably and continuously operate in more scenes.

[0031] In an optional embodiment, the brain-like decision method for embodied intelligence further comprises:

[0032] In the process of generating a pulse event sequence by adopting a pulse neural network model based on the task description and the environment model, and generating the motion strategy of the robot based on the pulse event sequence, the pulse neural network model is trained by adopting a combination of reinforcement learning and unsupervised learning;

[0033] In the training process, large-scale simulation environment and actual scene data are collected as training data of the pulse neural network model.

[0034] The brain-like decision method for embodied intelligence provided by the application, reinforcement learning guides the pulse neural network model to continuously adjust the decision strategy in the process of task execution by setting clear task objectives and reward mechanisms, and unsupervised learning analyzes large-scale multi-modal perception data to mine potential patterns and rules in the data. In the training process, in the face of complex and variable environmental data, unsupervised learning can automatically discover the internal relationship between environmental characteristics, task descriptions and motion strategies in different scenes. When the robot encounters a completely new environment or a slightly changed task form, based on the rules learned by unsupervised learning, the architecture can quickly understand the new situation and generate a reasonable pulse event sequence and motion strategy, avoiding the "overfitting" problem, significantly enhancing the generalization ability of the model in different scenes, and enabling the robot to flexibly cope with diversified tasks.

[0035] In an optional implementation, the robot comprises a motion executor, and the brain-like decision method for embodied intelligence further comprises:

[0036] The motion strategy of the robot is published to the motion executor of the robot, so that the robot executes the task according to the motion strategy, and the motion strategy is adjusted by the pulse neural network model according to the environmental changes and the task execution situation.

[0037] The brain-like decision method for embodied intelligence provided by the application, after the motion strategy is published to the motion executor, the robot can quickly execute the task according to the strategy, and the strategy can be adjusted in real time according to the environmental changes, the pulse neural network model can quickly perceive the environmental changes, reanalyze the task description and the new environmental model, generate a new pulse event sequence and motion strategy, and timely publish to the motion executor, so that the robot quickly changes the motion trajectory to avoid obstacles and avoid collision. This real-time dynamic response mechanism greatly improves the adaptability of the robot in a dynamic environment, ensures the continuous and efficient execution of the task, and solves the problem that the existing robot relies on pre-programmed logic and has poor adaptability in a dynamic environment. The robot is endowed with the ability to perceive environmental changes in real time, so that it can quickly and accurately respond, effectively reducing the occurrence of accidents such as falling and collision, and improving the stability and reliability of the robot in a complex environment.

[0038] In an optional implementation, the robot further comprises an online meta-learning module, and the brain-like decision method for embodied intelligence further comprises:

[0039] In the process of executing the task by the robot, the online meta-learning module extracts new skill key features, and the new skill key features are learned online, so that the robot learns new skills; the new skill key features include motion trajectory, angle, force and speed.

[0040] This invention provides a brain-inspired decision-making method for embodied intelligence. Its online meta-learning module can rapidly extract key features of new skills, such as movement trajectory, angle, force, and speed, while a robot performs a task. This embeds efficient online meta-learning capabilities in the robot, enabling it to rapidly learn new skills with a small number of demonstrations. This enhances the robot's intelligence and application flexibility, enabling it to meet diverse task requirements. Compared to traditional robots that learn new skills through extensive repetitive training, this significantly shortens the learning cycle.

[0041] In a second aspect, the present invention provides a brain-like decision-making device for embodied intelligence, applied to a robot, comprising:

[0042] The instruction receiving and semantic analysis module is used to receive human language instructions, perform semantic understanding on human language instructions, and obtain task descriptions;

[0043] Multimodal data acquisition and fusion module, used to collect multimodal data and generate an environmental model based on the multimodal data;

[0044] The decision module is used to generate a pulse event sequence using a pulse neural network model based on the task description and the environment model, and to generate a robot motion strategy based on the pulse event sequence.

[0045] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to thereby execute the brain-like decision-making method for embodied intelligence of the above-mentioned first aspect or any corresponding embodiment thereof.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the brain-like decision-making method for embodied intelligence according to the first aspect or any corresponding embodiment thereof.

[0047] In a fifth aspect, the present invention provides a computer program product comprising computer instructions, which are used to enable a computer to execute the brain-like decision-making method for embodied intelligence according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is a flowchart of a brain-like decision-making method for embodied intelligence according to an embodiment of the present invention;

[0050] Figure 2 is a flowchart of another brain-like decision-making method for embodied intelligence according to an embodiment of the present invention;

[0051] Figure 3 is a flowchart of another brain-like decision-making method for embodied intelligence according to an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of the structure of a bionic neuromorphic computing architecture according to an embodiment of the present invention;

[0053] Figure 5 is a schematic diagram of an implementation of a pulse neural network model according to an embodiment of the present invention;

[0054] Figure 6 is a schematic diagram of a flow chart of a spatiotemporal alignment algorithm according to an embodiment of the present invention;

[0055] Figure 7 is a structural block diagram of a brain-like decision-making device for embodied intelligence according to an embodiment of the present invention;

[0056] Figure 8 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0058] Traditional robotic decision-making systems often suffer from slow decision-making and excessive power consumption when handling complex scenarios. For example, when faced with sudden danger, robots may be unable to make timely evasive maneuvers due to slow decision-making, leading to safety accidents. From an energy consumption perspective, the computing model of traditional chips consumes a large amount of power during robot operation, shortening their battery life. In long-duration operations such as field exploration and emergency rescue, robots may be unable to complete their tasks due to insufficient power, severely limiting their application.

[0059] In practical applications, robots need to integrate multimodal sensory information, such as vision, force, and hearing, to fully and accurately understand their surroundings. For example, when a robot grasps an object, inaccurate fusion of visual and force data may lead to deviations in the grasping position, making it impossible to successfully grasp the object. In addition, learning new skills usually requires a large amount of training data and complex programming operations, and lacks efficient online learning capabilities. When robots are required to master new operational skills, professionals often need to spend a lot of time on programming and training, which not only increases costs but also reduces the robot's application flexibility.

[0060] An embodiment of the present invention provides a brain-like decision-making method for embodied intelligence. Based on multimodal data, it adopts a bionic neuromorphic computing architecture to generate the robot's motion strategy, realizes millisecond-level fusion of multi-sensor data and efficient decision-making of the robot in complex environments.

[0061] According to an embodiment of the present invention, an embodiment of a brain-like decision-making method for embodied intelligence is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0062] In this embodiment, a brain-like decision-making method for embodied intelligence is provided, which can be used in a robot. A bionic neuromorphic computing architecture (Neuromorphic-LLM) is provided in the robot, such as Figure 4 As shown in the figure, the bionic neuromorphic computing architecture includes a cross-modal perception enhancement module (MSA-Align algorithm module), an LLM-driven pulse decision core module (including a large language model and a pulse neural network model, referred to as the LLM-SNN module), Figure 1 is a flow chart of a brain-like decision-making method for embodied intelligence according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0063] Step S101: receiving a human language instruction, and performing semantic understanding on the human language instruction to obtain a task description.

[0064] Specifically, the robot receives human language commands through devices such as microphones and text input interfaces. These commands can be in either voice or text format. Voice commands are first converted into text using speech recognition technology to prepare for subsequent processing. The microphone must be a highly sensitive, directional microphone capable of accurately capturing sound information for voice interaction and ambient sound monitoring.

[0065] A large language model (LLM) is used to perform semantic understanding of human language instructions to obtain task descriptions, where semantic understanding includes lexical analysis, syntactic analysis, and semantic analysis.

[0066] Step S102: Collect multimodal data and generate an environment model based on the multimodal data.

[0067] Specifically, data is collected using multimodal sensors such as visual sensors (such as cameras), force sensors, and auditory sensors. Visual sensors acquire environmental image information, force sensors detect the magnitude and direction of force when in contact with objects, and auditory sensors collect environmental sound signals.

[0068] The environment model is a digital representation of the robot's environment. It integrates multimodal data and provides a key basis for robot decision-making. The environment model is the final product of a series of operations, including the collection, preprocessing, temporal alignment, spatial alignment, and fusion of multimodal data such as vision, force, and hearing. The environment model provides accurate environmental information to support the robot's decision-making.

[0069] Step S103 : generating a pulse event sequence using a pulse neural network model based on the task description and the environment model, and generating a motion strategy of the robot based on the pulse event sequence.

[0070] Specifically, a spike event sequence is a transformed representation of the task description and environmental model information. It slices the action requirements in the task description and the positions of objects in the environmental model according to preset time windows and maps them into spike rates. For example, when a robot receives the task of "grasping a red cup on a table," the environmental information related to the cup's location and the task description of the grasping action are converted into the spike firing rate of specific neurons. This chronological arrangement of spike firing information constitutes a spike event sequence, which carries key information about the task and environment in the form of pulse signals. The spike event sequence is the direct input for generating the robot's motion strategy.

[0071] A motion strategy consists of a series of motion parameter control signals output by the decision layer of a spiking neural network. These signals include key parameters such as joint angles, movement speed, and force. For example, when a robot grasps an object, the motion strategy specifies the specific angles to which each joint of the robotic arm should rotate to accurately reach the object. It also specifies the speed at which the robotic arm approaches and grasps the object to avoid damage from excessive speed. It also determines the amount of force applied during grasping to ensure a stable grasp without causing damage due to excessive force.

[0072] The brain-like decision-making method for embodied intelligence provided in this embodiment receives human language instructions and performs semantic understanding, and can accurately convert natural language instructions into abstract semantic representations. This accurate semantic understanding allows the robot to clearly define the task objectives and avoid task execution deviations due to misunderstanding of instructions. Collecting multimodal data and generating an environmental model enables the robot to obtain more comprehensive and accurate environmental information. Whether it is complex terrain recognition or delicate object manipulation, it can make reasonable decisions based on accurate environmental perception, significantly enhancing the robot's adaptability in complex environments. Based on the task description and environmental model, a pulse neural network model is used to generate a pulse event sequence and generate a motion strategy, so that the robot can quickly and intelligently plan the path of action and operation mode when facing dynamically changing environments and task requirements, achieve efficient decision-making and precise action execution, and solve the problem of the robot's lack of multimodal perception fusion and inaccurate intelligence in the decision-making strategy.

[0073] In this embodiment, a brain-like decision-making method for embodied intelligence is provided, which can be used in a robot. A bionic neuromorphic computing architecture (Neuromorphic-LLM) is provided in the robot, such as Figure 4 As shown in the figure, the bionic neuromorphic computing architecture includes a cross-modal perception enhancement module (also called the MSA-Align algorithm module, abbreviated as the LLM-SNN module), an LLM-driven pulse decision core module (including a large language model and a pulse neural network model), Figure 2 is a flow chart of a brain-like decision-making method for embodied intelligence according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0074] Step S201: receiving a human language instruction, and performing semantic understanding on the human language instruction to obtain a task description.

[0075] Specifically, the robot receives human language commands through a speech recognition device, which are then fed into a large language model for semantic understanding. The large language model analyzes the commands lexically, syntactically, and semantically, converting them into task descriptions that the robot can understand. For example, when receiving the command "take the cup from the table to the kitchen," the large language model can analyze the target object, location, and action requirements of the task and pass this information on to subsequent modules.

[0076] like Figure 4 As shown, the above step S201 includes:

[0077] In step a, a large language model is used to perform lexical analysis, syntactic analysis, and semantic analysis on human language instructions, converting them into an abstract semantic representation of the robot, and obtaining a robot task description based on the semantic representation.

[0078] The large language model (LLM) includes a language instruction input layer and a semantic analysis layer. The semantic analysis layer is stacked with a Transformer encoder (12 layers). The Transformer encoder introduces an 8-head attention mechanism (Multi-Head Attention) for weight sharing cross-modal semantic analysis and positional encoding for enhancing the spatiotemporal correlation of instruction sequences.

[0079] Human language instructions are input into the language instruction input layer and analyzed by the semantic analysis layer. The semantic representation of the robot is output in a 768-dimensional dense vector (Contextual Embedding).

[0080] Lexical analysis: A large language model based on the Transformer architecture is used for lexical analysis of instruction text. The model learns language knowledge through pre-training and identifies words, phrases, and parts of speech in instructions, such as nouns (e.g., "cup", "table"), verbs (e.g., "take", "put"), and adjectives (e.g., "red", "clean"). Key vocabulary in the instruction is extracted.

[0081] Syntactic analysis: Based on lexical analysis, the large language model further analyzes the syntactic structure of the instruction and constructs a syntax tree. It determines the subject, predicate, and object components of a sentence, as well as the modification and domination relationships between components. For example, for the instruction "put the blue cup in the kitchen on the coffee table in the living room", it is clear that the sentence structure is "put... on...". The blue cup is the object, the kitchen is the starting position, and the coffee table in the living room is the target position.

[0082] Semantic analysis: The large language model understands the semantic information of words and sentences in the instruction by combining semantic knowledge trained from large-scale corpora. Through semantic role labeling and coreference resolution techniques, it clarifies the true intent of the instruction and converts natural language instructions into abstract semantic representations that robots can understand. Finally, accurate task descriptions are generated based on the semantic representation.

[0083] It should be noted that in order to improve the accuracy of the large language model's understanding of robot commands, a domain-specific knowledge graph is introduced during the model training process. The knowledge graph contains knowledge such as common tasks, object categories, and action types in the robotics field. When understanding commands, the large language model can use the knowledge graph to perform semantic reasoning and better understand the intent of the command. For example, when receiving the command "Put the red cup on the coffee table in the living room," the large language model can quickly identify objects such as "red cup" and "coffee table in the living room" through the knowledge graph and determine the corresponding actions, improving the accuracy and efficiency of command understanding.

[0084] Step S202: Collect multimodal data and generate an environment model based on the multimodal data.

[0085] Specifically, if Figure 4 As shown, the robot collects visual, force, and auditory data of the surrounding environment in real time through multimodal sensors such as high-resolution cameras, force sensors, and microphones. These data are transmitted to the cross-modal perception enhancement module (MSA-Align algorithm module) for spatiotemporal alignment and fusion processing to build an accurate environmental model. During the collection process, the sensor collects data at a high frequency to ensure that it can capture real-time changes in the environment. The above step S202 includes:

[0086] Step S2021: Use a multimodal sensor to collect multimodal data, and perform data preprocessing on the multimodal data to denoise and remove redundant data; the multimodal data includes visual data, force data, and auditory data.

[0087] Specifically, multimodal sensors include high-resolution cameras, force sensors, and microphones. Equipping the robot with a high-resolution camera, such as an industrial-grade camera with a resolution of 4K, can clearly capture the details and motion of objects, ensuring the accuracy and reliability of the collected visual data. The force sensor uses a high-precision six-axis force sensor, which can accurately sense the forces acting between the robot and the object, generating force data and providing data support for precise control. The microphone uses a highly sensitive directional microphone, which can accurately collect sound information, enabling voice interaction and environmental sound monitoring. The rational combination of these sensors can comprehensively and accurately collect environmental information, providing a solid foundation for subsequent data processing and decision-making.

[0088] The collected multimodal data is processed for denoising and redundancy removal. For visual data, algorithms such as median filtering and Gaussian filtering are used to remove image noise. Force data is preprocessed by calibrating sensor parameters and removing outliers. Auditory data is processed using noise reduction algorithms to filter out ambient noise.

[0089] Step S2022: Time-align the multimodal data after data preprocessing using a spatiotemporal alignment algorithm based on the timestamp.

[0090] Specifically, the flow chart of the spatiotemporal alignment algorithm is as follows: Figure 6 As shown, the spatiotemporal alignment algorithm, namely the MSA-Align algorithm, is a key technology for ensuring the consistency of multimodal data in the brain-like decision-making method for embodied intelligence. It is mainly used to process the differences in data in time and space dimensions. In this embodiment, the spatiotemporal alignment algorithm is embedded in the cross-modal perception enhancement module (also called the MSA-Align algorithm module). All multimodal data carry timestamp information.

[0091] In the MSA-Align algorithm module, multimodal data collected by multimodal sensors is spatiotemporally aligned using timestamps and spatial location information, eliminating time delays and spatial deviations between sensors. After alignment, the data is fused to generate an accurate environmental model. When the robot observes a new skill demonstration, the Online Meta-Learning (OML) module applies advanced feature extraction algorithms to extract key features of the skill. This learning algorithm updates model parameters, enabling rapid learning and application of new skills.

[0092] A global timestamp mapping table is constructed to uniformly convert the timestamps of data collected by each sensor to a time coordinate system based on the robot system clock. For sensor data with slight timestamp deviations, linear interpolation is used to compensate. For example, if the visual data timestamp is t1 and the force data timestamp is t1 + Δt (Δt < sampling interval), the force data estimate at time t1 is calculated using linear interpolation.

[0093] For sensors with asynchronous sampling (such as 30fps visual sensors and 1000Hz force sensors), a dynamic time warping algorithm is used to calculate the optimal alignment path between two time series, find the correspondence between data with different sampling rates, and achieve precise alignment of time series. Through time alignment, a unified time base is established, and sensor data with different sampling frequencies are adjusted to the same time dimension through methods such as time interpolation or offset, ensuring the temporal consistency of multimodal data. In the time alignment process, a combination of hardware synchronization mechanism, software-level dynamic compensation, and key optimization strategies is adopted, among which:

[0094] 1. Hardware synchronization mechanisms include: a unified clock reference using PPS (pulse per second) signals and GPRMC (global positioning time code) as the master clock source, providing nanosecond-level synchronization accuracy through GPS / Beidou modules. For example, the lidar and camera receive the same PPS (pulse per second) pulse signal through a hardware trigger line, forcing the sensors to sample at the same physical moment, eliminating the cumulative error caused by crystal oscillator drift. Sensor data is embedded with an atomic clock-level timestamp (such as based on the IEEE1588PTP protocol) when it is collected, and cross-device synchronization is achieved in Ethernet through the gPTP protocol (a time synchronization protocol based on the IEEE 802.1AS standard), with transmission delay compensation to the microsecond level.

[0095] 2. Software-level dynamic compensation includes:

[0096] Delay Modeling and Online Estimation: Build a sensor transmission delay model, including fixed delays (e.g., 5ms from camera exposure to data transmission) and random delays (network jitter). Use Kalman filtering to estimate and compensate for time deviations in each channel in real time.

[0097] Motion-constrained interpolation: For asynchronous data (such as a 30Hz camera and a 100Hz IMU), B-spline interpolation or Gaussian process regression (GPR) is used to predict the sensor state at the moment of misalignment using the IMU (Inertial Measuring Unit) kinematic model and interpolate to generate a synchronized data stream.

[0098] 3. Key optimization strategies, including:

[0099] Priority scheduling: In dynamic environments, high-frequency obstacle detection signals (such as LiDAR burst point clouds) are given higher processing priority, and key event data is prioritized through a time window sliding mechanism.

[0100] Fault-tolerant mechanism introduction: redundant clock verification, switching to a backup clock (such as the IMU internal clock) when the main clock fails, and reconstructing the time series through historical data backtracking to ensure robustness in extreme scenarios.

[0101] Step S2023 , based on the spatial position, a spatiotemporal alignment algorithm is used to map the multimodal data after data preprocessing into a unified spatial coordinate system for spatial alignment.

[0102] Specifically, due to the different installation positions and angles of each sensor, the data collected by them are not uniform in spatial coordinates. The spatiotemporal alignment algorithm uses known parameters such as the sensor installation position and angle to determine a global coordinate system as a reference. Next, the coordinate transformation relationship of each sensor data in the global coordinate system is calculated, and the pixel coordinates in the visual data, the point coordinates of the force data, the sound source coordinates of the auditory data, etc. are all converted to this unified global coordinate system. For example, the position information of the object in the image captured by the camera is converted from image pixel coordinates to three-dimensional space coordinates, thereby realizing the unified spatial expression of multi-source data, laying the foundation for subsequent data fusion and environmental model construction.

[0103] In spatial alignment, a calibration plate (such as a checkerboard or AprilTag) is used to collect multimodal data. The rotation matrix and translation vector between sensors are solved through the iterative closest point (ICP) algorithm or feature matching optimization (such as SIFT / SURF) for offline high-precision calibration, and the reprojection error is controlled at the sub-millimeter level.

[0104] In step S2024, a fusion algorithm is used to fuse the time-aligned and space-aligned multimodal data to generate an environment model.

[0105] Specifically, the fused data is fed into a 3D reconstruction algorithm (such as point cloud-based Poisson reconstruction and octree modeling) to construct a high-precision environmental model that includes object geometry, position, physical properties (hardness, weight, etc.), and semantic information about the environment (room layout, object category). This model is stored as a mesh, point cloud, or voxel, providing an intuitive and accurate representation of the environment for the robot's path planning and task decision-making.

[0106] It should be noted that in this embodiment, a cache mechanism is used to temporarily store the multimodal data collected by the sensor to solve the problem of inconsistent data collection frequencies of different sensors. When sensor data collected at a high frequency arrives, it is first stored in a cache queue, waiting for synchronization of low-frequency sensor data. In addition, the Kalman filter algorithm is used to denoise the sensor data to reduce the interference of environmental noise on data fusion and further improve the accuracy of the data. In a multi-robot collaborative operation scenario, the sensor data of different robots can also be aligned in time and space across robots based on the MSA-Align algorithm provided in this embodiment, providing a unified data basis for collaborative decision-making.

[0107] To ensure the reliability of multimodal data fusion collected by multiple sensors, a redundant design and data verification mechanism are employed. When selecting sensors, multiple sensors of the same type are selected to collect the same environmental information. If a sensor fails, the remaining sensors can continue to provide data, ensuring the normal operation of the system. During the data fusion process, sensor data is verified. By calculating the checksum of the data and comparing the consistency of data from different sensors, erroneous data can be promptly detected and eliminated, thereby improving the reliability of data fusion.

[0108] Step S203: Generate a pulse event sequence using a pulse neural network model based on the task description and the environment model, and generate a robot motion strategy based on the pulse event sequence. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0109] The brain-inspired decision-making method for embodied intelligence provided in this embodiment uses a large language model to perform lexical, syntactic, and semantic analysis on human language instructions. This method can deeply analyze the structure and meaning of instructions, clearly present the logical relationships and hierarchical structure of tasks, and convert natural language into abstract semantic representations understandable to robots. The resulting task descriptions are accurate and provide precise guidance for the robot's subsequent task execution. The MSA-Align algorithm achieves millisecond-level fusion of multi-sensor data, controlling environmental modeling errors to less than 2 cm, thereby improving the robot's perception of the environment.

[0110] In this embodiment, a brain-like decision-making method for embodied intelligence is provided, which can be used in a robot. A bionic neuromorphic computing architecture (Neuromorphic-LLM) is provided in the robot, such as Figure 4 As shown in the figure, the bionic neuromorphic computing architecture includes a cross-modal perception enhancement module (also called the MSA-Align algorithm module), an LLM-driven pulse decision core module (including a large language model and a pulse neural network model, referred to as the LLM-SNN module), Figure 3 is a flow chart of a brain-like decision-making method for embodied intelligence according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0111] Step S301: Receive human language instructions and perform semantic understanding on the human language instructions to obtain a task description. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0112] Step S302: Collect multimodal data and generate an environment model based on the multimodal data. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0113] Step S303: Generate a pulse event sequence using a pulse neural network model based on the task description and the environment model, and generate a motion strategy for the robot based on the pulse event sequence.

[0114] Specifically, the schematic diagram of the pulse neural network model is as follows Figure 5 As shown, the spiking neural network model is a spiking neural network model (SNN) driven by a large language model, including a data input layer, a pulse event conversion layer, a spiking neural network decision layer and a data output layer; the above step S303 includes:

[0115] Step S3031: Use the pulse event conversion layer to map the task description and environment model output by the data input layer into pulse firing rates according to preset time window slices, and perform pulse sparsification on the pulse firing rates to generate a spatiotemporal pulse matrix.

[0116] Specifically, the spike event conversion layer consists of a temporal encoder (also known as a time window slicing module) and a spike train optimizer. The spike train optimizer includes a spike sparsification module and a spatiotemporal spike matrix output module. The temporal encoder's encoding strategy is to map the task description and environment model into spike rates according to time window slices (e.g., 10ms / window) and transmit them to the spike train optimizer. The spike sparsification module of the spike train optimizer performs spike sparsification (i.e., suppressing low-weight spikes (firing rate <20Hz) to reduce noise) and time alignment (synchronizing multimodal spike streams (error <1ms)) on the spike rates. The spatiotemporal spike matrix output module outputs a spatiotemporal spike matrix of dimension [N_neurons × T_timesteps], where N = 256 and T = 100ms.

[0117] Step S3032: Use the pulse neural network decision layer to decode the spatiotemporal pulse matrix, generate a motion parameter control signal, and use the motion parameter control signal as the robot's motion strategy.

[0118] Specifically, the spiking neural network decision layer includes a neuron model, which includes a network input layer, a network hidden layer, and a network output layer. The network input layer consists of 256 neurons, the network hidden layer consists of 128 LIF (Leaky Integrate-and-Fire) neurons, and the network output layer consists of 64 Motor Neurons. The neuron model uses a pulse routing table to dynamically configure synaptic connections (based on task priority), and the synaptic delay is 0.5ms to 5ms (simulating biological nerve conduction delay).

[0119] The motion parameter decoding module of the data output layer is used to decode the spatiotemporal pulse matrix to generate motion parameter control signals including speed, angle, strength, etc. The motion parameter control signals are finally output by the three-dimensional control signal output module. The decoding formula is:

[0120] Among them, θ joint The robot's joint angle output is a key parameter for controlling robot motion. k represents the gain coefficient, which is used to scale the calculation result to conform to the physical range of the joint angle or the control strategy requirements. Sum the time series, w i is the weight coefficient of the i-th motor neuron signal, which reflects the importance of different sensor data, S i (t) is the pulse state of the i-th motor neuron at time t. The decoding process is PID control based on the pulse firing rate to adjust the smoothness of the motion trajectory.

[0121] like Figure 4 As shown in the figure, the LLM-SNN module generates a sequence of pulse events based on the task description and environment model. These pulse event sequences are converted into motion parameter control signals and transmitted to the robot's motion actuators, controlling the robot's motion and achieving the task objectives. During the control process, the LLM-SNN module continuously adjusts the motion strategy based on environmental changes and task execution to ensure successful task completion.

[0122] Step S304, in the process of generating a pulse event sequence using a pulse neural network model based on the task description and the environment model, and generating a robot's motion strategy based on the pulse event sequence, the pulse neural network model is trained by combining reinforcement learning and unsupervised learning; during the training process, large-scale simulation environment and actual scene data are collected as training data for the pulse neural network model.

[0123] Specifically, the spiking neural network model is embedded in Intel's Loihi neuromorphic computing chip, which is designed specifically for neuromorphic computing and features efficient spiking neural network computing capabilities, providing strong hardware support for biomimetic neuromorphic computing architectures. The Loihi neuromorphic computing chip utilizes an event-driven computing model, enabling high-speed computing with low power consumption, meeting the robot's requirements for decision-making speed and power consumption. Furthermore, it is paired with high-performance servers for deploying large language models, ensuring efficient model operation and rapid instruction processing.

[0124] The LLM-SNN module was designed and trained using the open-source deep learning framework PyTorch. During the design process, the spiking neural network architecture was meticulously constructed, with the number of neurons and their connections carefully configured. During training, the parameters for reinforcement learning and unsupervised learning were carefully set, and extensive experimentation and optimization were used to improve model performance. Distributed training techniques were employed to accelerate model training and shorten training time. During task execution, the robot continuously optimized its decision-making strategy and skill model through reinforcement learning and unsupervised learning. Reinforcement learning uses feedback from task performance, such as task completion quality and time, to adjust decision-making strategies and improve task execution efficiency.

[0125] To improve the performance of the LLM-SNN module, a training method combining reinforcement learning and unsupervised learning is employed. In reinforcement learning, the robot is given clear mission objectives and reward mechanisms. During the mission, the robot continuously adjusts its decision-making strategy based on environmental feedback to maximize reward. Unsupervised learning analyzes multimodal sensory data to uncover underlying patterns and regularities, helping the model better understand environmental information and improve its generalization capabilities. During training, the model is repeatedly trained and optimized using large-scale simulation environments and real-world scenario data to ensure robust performance in a variety of scenarios.

[0126] Among them, reinforcement learning training includes:

[0127] Multi-dimensional reward signals: Construct a reward function that includes multiple dimensions such as task completion, energy consumption, and safety. For example, in a grasping task, a reward of +100 is given for successfully grasping the target object, a reward of -50 is given for colliding with an obstacle, and a reward of -0.1 is given for energy consumption per action.

[0128] Hierarchical reward mechanism: Using a hierarchical reinforcement learning architecture, the bottom-level control joint motion strategy rewards +0.5 steps, the middle-level task planning rewards +20 for successfully reaching the target position, and the high-level decision-making system rewards +200 for completing complex task sequences.

[0129] Dynamic adjustment coefficient: Introduce a time-varying reward coefficient k(t). Set a larger exploration reward coefficient k_explore = 0.8 at the beginning of training, and gradually decay it to k_explore = 0.2 in the later stage to balance exploration and utilization.

[0130] Unsupervised learning training includes:

[0131] 1. Multimodal feature extraction:

[0132] Visual feature extraction: A 3-layer spike convolutional neural network with a 5×5 convolution kernel and a time window of T = 10ms is used to extract the spatiotemporal features of the visual spike data and output a 256-dimensional feature vector.

[0133] Force sense feature processing: process the force pulse sequence through the LIF neuron network.

[0134] Cross-modal fusion: use attention mechanism to fuse visual and force sense features, and calculate cross-modal attention weights.

[0135] 2. Unsupervised learning:

[0136] Contrastive learning: construct positive and negative sample pairs, positive samples are pulse sequences of different perspectives of the same object, and negative samples are pulse sequences of different objects.

[0137] Reconstruction learning: use pulse autoencoder (SAE), the encoder compresses the 10000-dimensional pulse matrix into 256-dimensional features, the decoder reconstructs the pulse matrix, and the reconstruction error uses binary cross-entropy.

[0138] K-means clustering is performed on the extracted features, K=50, and the cluster centers are used as semantic representations of the environment state.

[0139] Step S305, the motion strategy of the robot is published to the motion executor of the robot, so that the robot executes the task according to the motion strategy, and adjusts the motion strategy according to the change of the environment and the execution of the task using the pulse neural network model.

[0140] Specifically, according to the application scenario of the robot, select the appropriate motion executor. In the home service scenario, you can choose a motor as the motion executor, such as a brushless DC motor, which has the characteristics of small size, high efficiency and high control precision, and can realize the flexible movement and precise operation of the robot. In the industrial manufacturing scenario, hydraulic drive can be used to meet the high load and high precision motion control requirements, to ensure that the robot can accurately execute the decision instruction and complete various complex industrial tasks.

[0141] The motion strategy output by the pulse neural network model usually exists in the form of parameter signals, such as joint angle, motion speed, force size, etc. Before publishing, these parameter signals need to be encoded according to the communication protocol of the motion executor. For example, for a robot arm executor using CAN bus communication, the joint angle parameter needs to be converted into a specific binary encoding format, and a check bit is added to ensure the accuracy of data transmission.

[0142] The encoded motion strategy is sent to the motion executor through a high-speed, low-latency communication network. In the industrial robot scenario, real-time Ethernet (such as EtherCAT) is often used for data transmission to ensure that the strategy can be delivered to the executor within a few milliseconds. For example, in an AGV (Automated Guided Vehicle), the controller sends the path planning and speed control strategy to the drive wheel motor controller through the EtherCAT network to achieve precise movement. After the motion executor receives the strategy data, it first performs a checksum and decoding to restore the binary data to the actual control parameters. For example, after the servo motor driver receives the joint angle target value, it drives the motor to rotate to the specified angle through the internal closed-loop control system, driving the robot arm to perform the corresponding action.

[0143] The motion executor initializes the execution parameters based on the received motion strategy. In a robot grasping task, the robot arm first moves the end effector to a pre-set position above the target object, adjusts the pose to make it parallel to the object's grasping surface, and sets the initial value of the grasping force. During execution, the motion executor collects motion state data in real time through built-in sensors (such as encoders, force sensors) and feeds it back to the control system. For example, when the robot arm is grasping an object, the force sensor monitors the grasping force in real time and compares it with the target force value in the motion strategy. If the actual grasping force is less than the target value, the control system will increase the motor driving force until the appropriate grasping force is reached.

[0144] The robot perceives environmental changes in real time through multi-modal sensors (vision, force sense, hearing, etc.). In a warehouse transportation scenario, when the AGV is driving, the laser radar sensor detects a sudden obstacle in front, and the vision camera further identifies the shape and size of the obstacle, transmitting this information to the pulse neural network model. The control system evaluates the task execution based on the state data feedback from the motion executor. In a welding robot operation, by monitoring parameters such as the position of the welding torch, welding current and voltage, it can determine whether the welding is normal. If there is a weld offset or abnormal welding current, it indicates that the task execution has deviated, and the motion strategy needs to be adjusted.

[0145] After receiving environmental change information and task execution status data, the spiking neural network model performs preprocessing. A convolutional neural network is used to extract obstacle features from the visual image data. Force data is filtered to remove noise. The processed data is then combined with the current task description as the model input. Based on the input data, the model regenerates the pulse event sequence through internal neuron computation and pulse transmission mechanisms, ultimately generating an adjusted motion strategy. In AGV obstacle avoidance scenarios, the model plans a new obstacle avoidance path based on the obstacle location and the AGV's current state, and generates corresponding speed and steering control parameters. Simultaneously, a combination of reinforcement learning and unsupervised learning is used to optimize the new strategy to ensure its feasibility and efficiency. The adjusted motion strategy is then sent to the motion actuator according to the aforementioned publishing process, and the robot switches to the new motion strategy to continue the task. For example, the AGV adjusts its driving direction according to the new strategy, avoids the obstacle, and then returns to the originally planned path to continue the task.

[0146] In step S306, the robot also includes an online meta-learning module. During the robot's task execution, the online meta-learning module extracts key features of the new skill and performs online learning on the key features of the new skill so that the robot can learn the new skill. The key features of the new skill include motion trajectory, angle, strength and speed.

[0147] Specifically, the Online Meta-Learning module (OML module) enables the robot to efficiently learn new skills. When the robot observes humans or other intelligent agents performing new skills, the OML module uses advanced feature extraction algorithms to extract the key features of the skill, such as the trajectory, strength, speed, etc. of the action, and converts them into learnable parameters. Through online learning, the robot can master new skills, such as the "opening a bottle cap" action, with only three demonstrations, greatly improving the robot's learning efficiency. During the learning process, the OML module uses a combination of transfer learning and reinforcement learning to transfer the learned skill knowledge to the learning of new skills, accelerating the learning process of new skills.

[0148] To improve the generalization capabilities of the OML module, a meta-learning strategy is introduced during the learning process. Meta-learning enables the robot to quickly adapt to new tasks by learning common knowledge across multiple tasks. For example, after learning rotational actions such as "opening a bottle cap" and "tightening a screw," the OML module can extract the common features of these actions. When encountering a new rotational task, it can quickly transfer the learned knowledge to the new task, improving learning efficiency and generalization capabilities.

[0149] The embodiment provided in the method for brain-like decision-making facing embodied intelligence adopts a bionic neural morphological computing architecture, and asynchronous communication mechanism is adopted between modules in the architecture to avoid system delay caused by data transmission waiting. After receiving the instruction of the large language model, the LLM-SNN module triggers the task processing procedure immediately, and feeds back the processing result to the motion control module asynchronously. After completing the multi-sensor data fusion, the MSA-Align algorithm module actively pushes the fused data to the LLM-SNN module and the OML module, so that the modules can obtain the latest environmental information in time. This asynchronous communication mechanism greatly improves the concurrent processing capability of the system, so that the robot can quickly respond in a complex environment. The architecture has good scalability, and new sensors, algorithms and functional modules can be easily integrated. When a new sensor type needs to be added, only the corresponding sensor data processing interface needs to be added in the MSA-Align algorithm module, so that the fusion of new sensor data can be realized. For new algorithms or functional modules, such as a target detection module based on deep learning, they can be integrated into the system through a standard interface and work cooperatively with the existing modules to endow the robot with more functions.

[0150] As one or more specific application embodiments of the embodiment of the present application, the method for brain-like decision-making facing embodied intelligence provided by the present application is further described in detail as follows:

[0151] (I) Hardware platform building:

[0152] 1. Sensor selection: A high-resolution camera, such as an industrial-grade camera with a resolution of 4K, is provided for the robot, which can clearly capture the details and motion trajectories of objects, ensuring that the visual information collected is accurate and reliable. A high-precision six-axis force sensor is used, which can accurately perceive the force between the robot and the object, providing data support for accurate control. A high-sensitivity directional microphone is selected, which can accurately collect sound information and realize voice interaction and environmental sound monitoring. The reasonable collocation of these sensors can comprehensively and accurately collect environmental information, providing a solid foundation for subsequent data processing and decision-making.

[0153] 2. Selection of computing hardware: Intel's Loihi neuromorphic computing chip is used, which is designed specifically for neuromorphic computing and has high-efficiency spiking neural network computing capability, providing strong hardware support for the bionic neuromorphic computing architecture. The Loihi chip adopts an event-driven computing mode, which can realize high-speed computing at low power consumption, meeting the requirements of the robot for decision-making speed and power consumption. At the same time, a high-performance server is matched to deploy the large language model, ensuring the efficient operation of the model and the rapid processing of instructions.

[0154] 3. Motion Actuator Configuration: Select the appropriate motion actuator based on the robot's application scenario. In home service scenarios, motors, such as brushless DC motors, can be used as motion actuators. These motors are compact, efficient, and offer high control accuracy, enabling flexible movement and precise operation of the robot. In industrial manufacturing scenarios, hydraulic drives can be used to meet high-load, high-precision motion control requirements, ensuring the robot can accurately execute decision-making instructions and complete various complex industrial tasks.

[0155] (2) Software system development:

[0156] 1. Large Language Model Deployment: Select an open-source large language model, such as GPT-Neo, and perform targeted fine-tuning to adapt it to robotic application scenarios. During fine-tuning, the model is trained using a large corpus relevant to robotic tasks to improve its understanding of robotic commands. The large language model is deployed on a server and communicates with the robot over the network, enabling rapid transmission and processing of commands. Load balancing technology is also employed to ensure stable model operation under high concurrency conditions.

[0157] 2. Spiking Neural Network Design and Training: We designed and trained the LLM-SNN module using the deep learning framework PyTorch. During the design process, we meticulously constructed the spiking neural network structure, rationally setting the number of neurons and their connections. During training, we carefully set parameters for reinforcement learning and unsupervised learning, and through extensive experimentation and optimization, we improved model performance. We also employed distributed training techniques to accelerate model training and shorten training time.

[0158] 3. Implementation of a multi-sensor spatiotemporal alignment algorithm: A caching mechanism temporarily stores sensor data, resolving the issue of inconsistent data collection frequencies across different sensors. When high-frequency sensor data arrives, it is first stored in a cache queue, awaiting synchronization with lower-frequency sensor data. Furthermore, a Kalman filter algorithm is used to denoise the sensor data, reducing the interference of environmental noise on data fusion and further improving data accuracy. In multi-robot collaborative operation scenarios, sensor data from different robots can also be aligned in spatiotemporal across robots using this algorithm, providing a unified data foundation for collaborative decision-making.

[0159] 4. Development of online meta-learning modules: In the feature extraction phase, a combination of Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) is adopted. CNN is responsible for extracting the spatial features of the demonstration action, while LSTM captures the time series features of the action. The two work together to comprehensively and accurately extract the key features of the skill. In terms of learning algorithms, the Proximal Policy Optimization (PPO) algorithm is used. This algorithm accelerates the learning speed while ensuring the stability of policy updates. Taking the robot learning complex assembly skills as an example, the OML module can quickly extract the key steps and techniques of the assembly action through three high-quality demonstrations, convert this knowledge into executable strategies, and significantly shorten the robot's learning cycle.

[0160] (3) System integration and testing:

[0161] 1. System Integration: At the hardware level, standardized interfaces and communication protocols are used to achieve seamless connectivity between sensors, computing hardware, and motion actuators. The electrical characteristics of different devices are matched to ensure stable and accurate signal transmission. At the software level, unified drivers and middleware are developed to provide consistent access interfaces for all software modules, achieving a high degree of software system integration. Comprehensive compatibility testing is performed on the system to promptly identify and resolve potential compatibility issues.

[0162] 2. Functional Testing: Build test environments simulating a variety of complex scenarios, including homes, industrial workshops, and disaster sites. In the simulated home scenario, the robot's ability to perform tasks such as cleaning and object handling will be tested; in the industrial workshop scenario, the robot's performance in high-precision assembly and material handling will be tested; and in the disaster site scenario, the robot's decision-making and execution capabilities in complex terrain and hazardous environments will be tested. By simulating various emergencies, such as the sudden appearance of obstacles and temporary changes in mission objectives, the robot's real-time response capabilities and decision-making accuracy will be verified.

[0163] 3. Performance Optimization: Using performance monitoring tools, we monitor each system module in real time and analyze system bottlenecks. For modules that consume large amounts of computing resources, we employ techniques such as model compression and pruning to reduce model complexity and computational effort. We dynamically adjust algorithm parameters to optimize performance based on different scenarios and task requirements. For example, in scenarios with low environmental complexity, we reduce the algorithm's computational accuracy to increase execution speed; in tasks requiring high precision, we appropriately increase computing resources to ensure high-quality completion.

[0164] The embodiment of the application provides a brain-like decision-making method for embodied intelligence, which is realized based on a neuromorphic-LLM architecture, as shown in a structural diagram Figure 4 The neuromorphic-LLM architecture includes a cross-modal perception enhancement module (MSA-Align algorithm module), an LLM-driven pulse decision core module (including a large language model and a pulse neural network model, referred to as an LLM-SNN module), and an online meta-learning module (OML module).

[0165] I. Neuromorphic-LLM architecture:

[0166] 1. Inter-module interaction mechanism: Asynchronous communication mechanism is adopted between modules to avoid system delay caused by data transmission waiting. After receiving the instruction of the large language model, the LLM-SNN module immediately triggers the task processing flow and asynchronously feeds back the processing result to the motion control module. After completing the multi-sensor data fusion, the MSA-Align algorithm module actively pushes the fused data to the LLM-SNN module and the OML module, ensuring that each module can timely obtain the latest environmental information. This asynchronous communication mechanism greatly improves the concurrent processing capability of the system, enabling the robot to quickly respond in a complex environment.

[0167] 2. Scalability of the architecture: The architecture has good scalability and can easily integrate new sensors, algorithms and functional modules. When a new sensor type needs to be added, only the corresponding sensor data processing interface needs to be added in the MSA-Align algorithm module, so that the fusion of new sensor data can be realized. For new algorithms or functional modules, such as a target detection module based on deep learning, they can be integrated into the system through a standard interface and work collaboratively with existing modules to give the robot more functions.

[0168] II. LLM-driven pulse decision core (LLM-SNN):

[0169] 1. Optimization of instruction understanding: In order to improve the accuracy of the large language model in understanding the instructions of the robot, a domain-specific knowledge graph is introduced in the model training process. The knowledge graph contains common tasks, object categories, action types and other knowledge in the robot field. When understanding the instructions, the large language model can use the knowledge graph for semantic reasoning to better understand the intent of the instructions. For example, when receiving the instruction "put the red cup on the living room coffee table", the large language model can quickly identify objects such as "red cup" and "living room coffee table" through the knowledge graph and determine the corresponding action, improving the accuracy and efficiency of instruction understanding.

[0170] 2. Dynamic Adjustment of Spiking Neural Networks: Spiking neural networks can dynamically adjust their structure and parameters based on task complexity and environmental changes. For simple tasks, spiking neural networks simplify their structure, reducing computational effort and improving decision-making speed. For complex tasks, they increase the number of neurons and connection density, improving the model's expressive power. Furthermore, through online learning mechanisms, spiking neural networks can adjust synaptic weights in real time based on environmental feedback, optimizing decision-making strategies.

[0171] 3. Cross-modal perception enhancement module:

[0172] 1. Ensuring the reliability of multi-sensor data fusion: To ensure the reliability of multi-sensor data fusion, redundant design and data verification mechanisms are employed. When selecting sensors, multiple sensors of the same type are selected to collect information about the same environment. If a sensor fails, the remaining sensors can continue to provide data, ensuring the normal operation of the system. During the data fusion process, sensor data is verified. By calculating the checksum of the data and comparing the consistency of data from different sensors, erroneous data can be promptly detected and eliminated, thereby improving the reliability of data fusion.

[0173] 2. Improving the generalization capabilities of the online meta-learning module: To enhance the generalization capabilities of the OML module, a meta-learning strategy is introduced during the learning process. Meta-learning enables the robot to quickly adapt to new tasks by learning common knowledge across multiple tasks. For example, after learning rotational actions such as "opening a bottle cap" and "tightening a screw," the OML module can extract the common features of these actions. When encountering new rotational tasks, it can quickly transfer the learned knowledge to the new task, improving learning efficiency and generalization capabilities.

[0174] In actual application scenarios, the brain-like decision-making large model architecture for embodied intelligence of the present invention will provide strong technical support for the intelligent development of robots, promote the widespread application of robots in various fields, and create huge economic and social value.

[0175] The workflow of the brain-inspired decision-making method for embodied intelligence provided by an embodiment of the present invention is as follows:

[0176] 1. Command Input and Understanding: The robot receives human language commands through a speech recognition device. These commands are then transmitted to a large language model for semantic understanding. The large language model analyzes the commands lexically, syntactically, and semantically, converting them into task descriptions that the robot can understand. For example, when receiving the command "Take the cup from the table to the kitchen," the large language model can analyze the target object, location, and action requirements of the task and pass this information on to subsequent modules.

[0177] 2. Multimodal Perception Data Acquisition and Fusion: The robot uses multimodal sensors, including high-resolution cameras, force sensors, and microphones, to collect real-time visual, force, and auditory data from its surroundings. This data is transmitted to the MSA-Align algorithm module for spatiotemporal alignment and fusion processing to construct an accurate environmental model. During this acquisition process, the sensors collect data at a high frequency to ensure that real-time changes in the environment are captured.

[0178] 3. Decision-making and Control: The LLM-SNN module generates a sequence of pulse events based on the task description and environmental model. These pulse event sequences are converted into control signals and transmitted to the robot's motion actuators, controlling the robot's motion and achieving the task objectives. During the control process, the LLM-SNN module continuously adjusts the motion strategy based on environmental changes and task performance to ensure successful task completion.

[0179] 4. Learning and Optimization: During task execution, the robot continuously optimizes its decision-making strategies and skill models through reinforcement learning and online meta-learning. Reinforcement learning adjusts decision-making strategies based on task performance feedback, such as completion quality and time, to improve task execution efficiency. Online meta-learning continuously enriches the robot's skill base and enhances its performance by learning new skills.

[0180] The brain-inspired decision-making method for embodied intelligence provided by the embodiments of the present invention has the following beneficial effects:

[0181] (1) Significantly Improved Decision-Making Speed ​​and Reduced Power Consumption: Through the application of the bionic neuromorphic computing architecture (Neuromorphic-LLM), the robot's decision-making speed in complex scenarios has increased by 50 times, while power consumption has been reduced to 1 / 10. This not only greatly improves the robot's real-time response capabilities, enabling it to quickly respond to various emergencies, such as when encountering sudden obstacles, it can quickly take evasive action to avoid collisions. At the same time, it significantly extends the robot's battery life, enabling it to operate continuously and stably in a wider range of scenarios, such as playing a greater role in long-term field exploration and emergency rescue scenarios.

[0182] (2) Enhanced Adaptability to Dynamic Environments: The collaborative work of the LLM-SNN module and the MSA-Align algorithm enables the robot to quickly and accurately perceive and respond to environmental changes. In sudden obstacle avoidance tests, the response delay was only 8ms, effectively reducing the robot's fall rate in dynamic environments and improving its stability and reliability. For example, when the robot navigated complex terrain, it was able to avoid falls and successfully complete the task by sensing terrain changes in real time and adjusting its walking strategy.

[0183] (3) Optimizing multimodal perception fusion and learning capabilities: The MSA-Align algorithm achieves millisecond-level fusion of multi-sensor data, controlling the environmental modeling error to less than 2 cm, greatly improving the robot's perception accuracy of the environment. The online meta-learning module (OML) enables the robot to learn new skills with only three demonstrations, significantly improving the robot's learning efficiency and flexibility. This enables the robot to better adapt to diverse task requirements. For example, in home service scenarios, it can quickly learn new housework skills and provide users with more convenient services.

[0184] In this embodiment, a brain-like decision-making device for embodied intelligence is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the details that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0185] This embodiment provides a brain-like decision-making device for embodied intelligence, which is applied to robots, such as Figure 7 Shown, including:

[0186] The instruction receiving and semantic analysis module 701 is used to receive human language instructions, perform semantic understanding on the human language instructions, and obtain a task description.

[0187] The multimodal data collection and fusion module 702 is used to collect multimodal data and generate an environment model based on the multimodal data.

[0188] The decision module 703 is used to generate a pulse event sequence using a pulse neural network model based on the task description and the environment model, and generate a motion strategy of the robot based on the pulse event sequence.

[0189] In some optional implementations, the instruction receiving and semantic analysis module 701 includes:

[0190] The instruction analysis unit is used to use a large language model to perform lexical analysis, syntactic analysis and semantic analysis on human language instructions, convert them into abstract semantic representations of the robot, and obtain the robot task description based on the semantic representation.

[0191] In some optional implementations, the multimodal data acquisition and fusion module 702 includes:

[0192] The multimodal data acquisition and preprocessing unit is used to acquire multimodal data using a multimodal sensor, and to perform data preprocessing on the multimodal data to denoise and remove redundant data; the multimodal data includes visual data, force data, and auditory data.

[0193] The time alignment unit is used to perform time alignment on the multimodal data after data preprocessing using a spatiotemporal alignment algorithm based on the timestamp.

[0194] The spatial alignment unit is used to map the multimodal data after data preprocessing into a unified spatial coordinate system for spatial alignment using a spatiotemporal alignment algorithm based on spatial position.

[0195] The environment model generation unit is used to fuse the multimodal data after time alignment and space alignment using a fusion algorithm to generate an environment model.

[0196] In some optional embodiments, the spiking neural network model includes a spiking event conversion layer and a spiking neural network decision layer; the decision module 703 includes:

[0197] The spatiotemporal pulse matrix generation unit is used to map the task description and environment model into pulse firing rates according to preset time window slices using the pulse event conversion layer, and to perform pulse sparsification on the pulse firing rates to generate a spatiotemporal pulse matrix.

[0198] The motion strategy generation unit is used to decode the spatiotemporal pulse matrix using the pulse neural network decision layer, generate motion parameter control signals, and use the motion parameter control signals as the robot's motion strategy.

[0199] In some optional embodiments, the brain-inspired decision-making device for embodied intelligence further includes:

[0200] The learning and optimization module is used to train the spiking neural network model by combining reinforcement learning and unsupervised learning in the process of generating a sequence of pulse events using the spiking neural network model based on the task description and environment model, and generating the robot's motion strategy based on the pulse event sequence; during the training process, large-scale simulation environment and actual scene data are collected as training data for the spiking neural network model.

[0201] In some optional embodiments, the brain-inspired decision-making device for embodied intelligence further includes:

[0202] The motion strategy execution and adjustment module is used to publish the robot's motion strategy to the robot's motion actuator, so that the robot can perform tasks according to the motion strategy, and adjust the motion strategy using the pulse neural network model according to environmental changes and task execution status.

[0203] In some optional embodiments, the robot further includes an online meta-learning module, and the brain-inspired decision-making device for embodied intelligence further includes:

[0204] The online meta-learning module is used to extract key features of new skills during the robot's task execution and conduct online learning of these key features, enabling the robot to learn new skills. Key features of new skills include movement trajectory, angle, force, and speed. This module is consistent with the online meta-learning module in the bionic neuromorphic computing architecture.

[0205] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0206] The brain-like decision-making device for embodied intelligence in this embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0207] The embodiment of the present invention also provides a computer device having the above Figure 7 The brain-like decision-making device for embodied intelligence is shown.

[0208] See also Figure 8 , Figure 8 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0209] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0210] The memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform the methods implemented by the above-described embodiments.

[0211] The memory 20 can include a program region and a data region. The program region can store an operating system and application programs required by at least one function. The data region can store data created according to use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory such as at least one of a magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory disposed remotely with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0212] The memory 20 can include a volatile memory such as a random access memory, and can further include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk. The memory 20 can also include a combination of the above-mentioned types of memories.

[0213] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30, and the output device 40 can be connected by a bus or other means, Figure 8 For example, by way of a bus connection.

[0214] The input device 30 can receive input digital or character information, and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, and the like. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), a tactile feedback device (e.g., a vibration motor), and the like. The display device includes, but is not limited to, a liquid crystal display, a light emitting diode, a display, and a plasma display. In some alternative embodiments, the display device can be a touch screen.

[0215] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0216] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0217] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A brain-like decision-making method for embodied intelligence, characterized by: Applied to a robot, the method comprises: receiving human language instructions, and performing semantic understanding on the human language instructions to obtain a task description; Collecting multimodal data and generating an environment model based on the multimodal data; A pulse event sequence is generated using a pulse neural network model based on a task description and an environment model, and a motion strategy of the robot is generated based on the pulse event sequence.

2. The method according to claim 1, characterized in that Performing semantic understanding on the human language instruction to obtain a task description includes: A large language model is used to perform lexical analysis, syntactic analysis and semantic analysis on the human language instructions, and convert them into an abstract semantic representation of the robot, and a robot task description is obtained based on the semantic representation.

3. The method according to claim 1, characterized in that The collecting of multimodal data and generating an environment model based on the multimodal data includes: Using a multimodal sensor to collect multimodal data, and performing data preprocessing on the multimodal data to remove noise and redundant data; the multimodal data includes visual data, force data, and auditory data; The spatiotemporal alignment algorithm is used to time-align the multimodal data after data preprocessing based on the timestamp; Based on the spatial position, a spatiotemporal alignment algorithm is used to map the multimodal data after data preprocessing into a unified spatial coordinate system for spatial alignment; A fusion algorithm is used to fuse the time-aligned and space-aligned multimodal data to generate an environmental model.

4. The method according to claim 1, wherein The spiking neural network model includes a spiking event conversion layer and a spiking neural network decision layer; The method of generating a pulse event sequence using a pulse neural network model based on a task description and an environment model, and generating a robot motion strategy based on the pulse event sequence, includes: The task description and environment model are mapped into pulse firing rates according to preset time window slices using a pulse event conversion layer, and the pulse firing rates are pulse-sparsed to generate a spatiotemporal pulse matrix. A pulse neural network decision layer is used to decode the spatiotemporal pulse matrix to generate a motion parameter control signal, which is used as the motion strategy of the robot.

5. The method according to claim 1, wherein The method further comprises: In the process of generating a pulse event sequence using a pulse neural network model based on a task description and an environment model, and generating a robot motion strategy based on the pulse event sequence, the pulse neural network model is trained by combining reinforcement learning and unsupervised learning; During the training process, large-scale simulation environment and actual scene data are collected as training data for the bionic neuromorphic computing architecture.

6. The method according to claim 1, characterized in that The robot includes a motion actuator, and the method further includes: The motion strategy of the robot is published to the motion actuator of the robot so that the robot performs the task according to the motion strategy, and the motion strategy is adjusted using a pulse neural network model according to environmental changes and task execution status.

7. The method according to claim 1, characterized in that The robot further includes an online meta-learning module, and the method further includes: During the robot's task execution, the key features of the new skill are extracted through the online meta-learning module, and the key features of the new skill are learned online so that the robot can learn the new skill; the key features of the new skill include movement trajectory, angle, strength and speed.

8. A brain-like decision-making device for embodied intelligence, characterized by: Applied to a robot, the device comprises: An instruction receiving and semantic analysis module is used to receive human language instructions and perform semantic understanding on the human language instructions to obtain a task description; A multimodal data acquisition and fusion module, configured to acquire multimodal data and generate an environment model based on the multimodal data; The decision module is used to generate a pulse event sequence using a pulse neural network model based on the task description and the environment model, and generate a robot motion strategy based on the pulse event sequence.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the brain-like decision-making method for embodied intelligence according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the brain-like decision-making method for embodied intelligence according to any one of claims 1 to 7.

Images