Leg-foot robot autonomous behavior control method and system based on multi-modal large model

Abstract

The invention discloses an autonomous behavior control method and system for a leg-foot robot based on a multi-modal large model, the method is applied to a controller of the robot, and the method comprises the steps: obtaining a rescue instruction, carrying out the instruction analysis of the rescue instruction, and obtaining a rescue task and a rescue destination; the method comprises the following steps: firstly, carrying out terrain recognition; then, positioning the current position of the robot by adopting a mapping algorithm; and finally, according to the current position of the robot and the rescue destination, navigation path planning is carried out. The robot advances according to the planned navigation path, and gait adjustment, speed adjustment and path optimization are carried out on the robot according to an environment self-adaptive strategy in the advancing process of the robot; error detection is carried out on the robot, and if an error is found, the error is corrected; and the robot executes a rescue task after arriving at the rescue destination.

Description

Technical Field

[0001] The present invention relates to the technical field of rescue robots, and in particular to an autonomous behavior control method and system of a leg-foot robot based on a multi-modal large model. Background Art

[0002] In the field of intelligent robots, legged robots (bipeds, quadrupeds, hexapods, humanoid robots, etc.) have attracted much attention due to their excellent movement ability and adaptability. As an important branch of the field of intelligent robots, legged robots have shown great application potential in search and rescue, home services, industrial inspection and other scenarios due to their unique movement ability and adaptability. The design and control technology of legged robots has always been a hot issue in the field of robotics research.

[0003] Although legged robots have broad application prospects in many fields, there are some problems in the existing technology that need to be solved urgently.

[0004] Especially in terms of environmental perception, legged robots have limited target recognition capabilities under conditions of drastic changes in light, making it difficult to accurately capture and analyze environmental information.

[0005] In addition, the current leg-foot robot technology is still at a rudimentary level in terms of natural language interaction, mainly manifested in the lack of ability to understand and reason about complex human grammatical structures, context dependencies, new vocabulary expressions, and rhetorical techniques such as implicit semantics. This limits the robot's ability to accurately grasp user intentions during the interaction process, greatly reducing the efficiency of human-computer interaction and user experience.

[0006] In terms of autonomous behavior control, existing legged robot technology performs poorly in dealing with complex dynamic environments that require real-time decision-making and strategy adjustments. When the robot encounters unknown obstacles or changes in the environmental layout, it is often difficult for the robot to make optimal decisions, resulting in inefficient navigation and task execution. Summary of the invention

[0007] In order to address the deficiencies in the prior art, the present invention provides a method and system for autonomous behavior control of a legged robot based on a multimodal large model; utilizing the powerful semantic understanding capability of a large language model to more deeply parse complex semantic instructions of users and improve the accuracy and efficiency of human-computer interaction; through deep fusion and intelligent analysis of multimodal information such as visual images and voices by a large language model, the robot is endowed with a comprehensive understanding and precise perception capability of complex environments; based on the continuous learning and self-optimization capabilities of a large language model, the robot is able to quickly generate and execute optimal behavior strategies based on the current environmental status and expected goals in a complex dynamic environment.

[0008] On the one hand, a method for autonomous behavior control of a legged robot based on a multimodal large model is provided, which is applied to the robot controller and includes:

[0009] Obtaining a rescue instruction, performing instruction analysis on the rescue instruction, and obtaining a rescue mission and a rescue destination;

[0010] Perceive the robot's current environment: First, identify the terrain; then use a mapping algorithm to locate the robot's current location; finally, plan a navigation path based on the robot's current location and the rescue destination;

[0011] The robot moves along the planned navigation path. During the movement, the robot adjusts its gait, speed and path according to the environment adaptation strategy.

[0012] Perform error detection on the robot and correct the error if any;

[0013] After the robot arrives at the rescue destination, it performs the rescue mission.

[0014] On the other hand, an autonomous behavior control system for legged robots based on a multimodal large model is provided, including:

[0015] An acquisition module is configured to: acquire a rescue instruction, perform instruction analysis on the rescue instruction, and obtain a rescue mission and a rescue destination;

[0016] The perception module is configured to: perceive the current environment of the robot: first, perform terrain recognition; then, use a mapping algorithm to locate the current position of the robot; finally, perform navigation path planning based on the current position of the robot and the rescue destination;

[0017] The optimization module is configured to: the robot moves according to the planned navigation path, and in the process of the robot moving, according to the environment adaptation strategy, the robot gait adjustment, speed adjustment and path optimization;

[0018] A detection module is configured to: perform error detection on the robot and correct the error if an error is found;

[0019] The output module is configured to: after the robot arrives at the rescue destination, perform the rescue mission.

[0020] The above technical solution has the following advantages or beneficial effects:

[0021] In terms of environmental perception, a multimodal perception and command-based navigation method for legged robots that integrates a large language model is proposed. By integrating multiple devices such as radar, camera, and voice sensor, comprehensive collection of environmental information is achieved. The large language model is used to efficiently integrate and analyze multimodal perception data, which improves the robot's ability to perceive, identify, and understand key features and dynamically changing target objects in the environment. The command-based navigation technology achieves deep perception and understanding of the environment, as well as parsing and path planning of target instructions through the combination of deep learning and a large language model. The optimal trajectory optimizer and back-propagation mechanism are used to minimize the cost function to optimize the motion trajectory, allowing the robot to make real-time decisions in an unknown environment to achieve safe and efficient navigation.

[0022] In terms of natural language interaction, a method for legged robots to move, learn, and interact with emotions under natural language interaction was invented. By integrating the large language model technology, the legged robots are provided with logical reasoning capabilities, enabling them to conduct in-depth semantic analysis of human natural language instructions and image information, and thus more accurately understand the user's intentions and perform corresponding actions or tasks accordingly. The large language model also gives the robot the ability to learn autonomously and interact with emotions, providing a more personalized interactive experience, greatly improving the user experience.

[0023] In terms of autonomous behavior control, a method for intelligent behavior scheduling of legged robots based on a large language model is designed. The input information is converted into semantic information executable by the robot, and high-level behavior planning and dynamic adjustment are achieved by generating task sub-functions. The integrated high-level feedback mechanism ensures the accuracy and execution efficiency of task planning, while the integration of motion state machine and motion planning technology concretizes the abstract planning into the precise movement of the robot. Real-time evaluation and adaptive control further improve the smoothness and safety of movement, and the underlying feedback mechanism provides strong support for coping with environmental changes and task requirements. In addition, the continuous monitoring and optimization mechanism ensures the continuous improvement of the robot's behavior, and the cross-module coordination capability ensures the overall consistency and coordination of the system. The intelligent behavior scheduling method improves the autonomy and adaptability of the legged robot and supports its application in a variety of task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 is a flow chart of the method of embodiment 1;

[0026] Figure 2 This is a flow chart of a semantic parsing method for natural human-computer interaction according to Embodiment 1;

[0027] Figure 3 A flowchart of constructing environmental semantics based on a large language model according to Embodiment 1;

[0028] Figure 4 This is a flowchart of intelligent behavior scheduling based on a large language model according to Embodiment 1.

[0029] Figure 5 This is a network structure diagram of the imperative path planning of embodiment 1.

[0030] Figure 6 This is a hardware electrical connection diagram of Example 1. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0032] Embodiment 1

[0033] This embodiment provides a method for controlling autonomous behavior of a legged robot based on a multimodal large model;

[0034] like Figure 1 As shown, the autonomous behavior control method of the leg-foot robot based on the multi-modal large model is applied to the controller of the robot, including:

[0035] S101: Obtain a rescue instruction, analyze the rescue instruction, and obtain a rescue mission and a rescue destination;

[0036] S102: Perceiving the current environment of the robot: first, performing terrain recognition; then, using a mapping algorithm to locate the current position of the robot; finally, performing navigation path planning based on the current position of the robot and the rescue destination;

[0037] S103: The robot moves according to the planned navigation path. During the movement of the robot, the robot adjusts its gait, speed and path according to the environment adaptation strategy;

[0038] S104: Perform error detection on the robot. If an error is found, correct the error and return to S101;

[0039] S105: After the robot arrives at the rescue destination, it performs the rescue mission.

[0040] Furthermore, the S101: obtain rescue instructions, wherein the rescue instructions include: searching a set area, transporting survival supplies, avoiding potential dangers, and detecting signs of life.

[0041] Furthermore, if Figure 2 As shown, the S101: analyzing the rescue instruction to obtain a rescue mission and a rescue destination, and the specific analysis process includes:

[0042] S101-1: Convert the rescue instruction from voice form to text form;

[0043] S101-2: Preprocessing the rescue instruction in text form;

[0044] S101-3: parsing the text rescue instruction after the preprocessing operation using a linear temporal logic method to obtain a number of subtasks;

[0045] S101-4: Convert each subtask into subtask text through a large language model, and convert all subtask texts into speech output.

[0046] The large language models mentioned in this invention refer to Baidu Smart Cloud Qianfan large model platform, and the robot system integration is performed by calling Baidu Qianfan large language model online.

[0047] Furthermore, the S101-1: converts the rescue instruction from voice form to text form, uses Baidu Qianfan Voice Recognition Express (ASR, Automatic Speech Recognition) to process the rescue instruction, obtains the ASR's AppID, API Key, Secret Key and exchanges tokens to authenticate and create a voice recognition request, ASR recognizes the text result, and then uses the text result recognized by ASR for natural language processing.

[0048] Furthermore, the S101-2: performs preprocessing operations on the rescue instructions in text form, including text cleaning to remove noise in the text, such as extra spaces, special characters, etc.; splitting the long text into separate words or phrases and extracting keywords such as location, "transporting supplies", etc., which are the intent of the rescue instructions, so as to improve the accuracy and efficiency of subsequent natural language processing tasks.

[0049] Furthermore, the S101-3: the text rescue instruction after the preprocessing operation is parsed in a linear temporal logic manner to obtain a number of subtasks, specifically including:

[0050]

[0051] in, represents an atomic proposition, p i represents the i-th subtask, N AP represents the number of subtasks, ::= represents a production rule, represents logical negation, ∧ and ∨ represent logical AND and logical OR respectively, for example represents the atomic proposition itself, Represents the logical AND of two linear temporal logic (LTL) formulas. express. Indicates that a condition is always true throughout the time range. Indicates that a condition will eventually be met. Represents a relationship between two conditions, where one condition remains true until the other condition is true. For example, if a rescue instruction contains multiple steps or conditions, you can think of "reach location A" as a proposition. A , "maintain communication" is regarded as proposition p comm , then “maintain communication before reaching location A” can be expressed as This approach provides a paradigm foundation that conforms to linear temporal logic for large models to generate subtask texts.

[0052] Furthermore, the S101-4: converting each subtask into a subtask text by using a large language model is converting each subtask into a subtask text conforming to a linear temporal logic formula by using a large language model.

[0053] Furthermore, the S101-4: converts all subtask texts into voice output, using Baidu Qianfan's short text to speech synthesis (Text To Speech, TTS) technology to convert text information into voice output.

[0054] It should be understood that in the process of enhancing the human-computer interaction capabilities of the legged robot, the present invention uses LTL formulas to accurately define and standardize the robot's task behavior. LTL is a formal method for describing the temporal properties of a system, which enhances our ability to describe the robot's behavior through grammatical structures. Using LTL formulas, complex tasks can be decomposed into a series of logically verifiable subtasks to ensure that the robot's behavior meets user expectations. This logical expression capability is crucial to improving user experience because it allows us to define tasks in a precise and consistent way.

[0055] The complexity of natural language processing (NLP) places high standards on large language models, requiring them to not only be able to parse the polysemy, complex syntax, and contextual dependencies of language, but also to be able to adapt to the continuous evolution of language, including understanding and adapting to emerging vocabulary and expressions. In order to meet these requirements, large language models must have the ability to continuously learn and update, as well as handle the complexity of rhetoric such as implicit meaning, irony, and humor in language.

[0056] In this context, the introduction of prompt language models has become a key strategy to improve the quality and efficiency of natural human-computer interaction. The prompt language model can be visualized as a large model prompt template. First, the robot receives the rescue command, and then the large model parses the command and analyzes the command type, including single dialogue, motion control, autonomous navigation, etc. According to the parsed rescue command, it enters different control modes and returns to the controller for execution. For example, when the robot receives the rescue command, if the large model does not parse an obvious rescue command when parsing the command, it enters the dialogue mode; if the large model parses the keyword "go to the destination", the system will directly enter the autonomous navigation mode and plan the path according to the command path planning network; if the large model parses "cross the high step", it will enter the motion control mode, increase the robot step height and reduce the step frequency.

[0057] In this way, the prompt language model not only improves the response speed of AI and optimizes the semantic parsing ability of AI, but also provides strong support for building a more intelligent and adaptable interactive system. This system can adapt to the personalized needs of users and the continuous changes in language, making human-computer interaction smoother and more natural, and providing users with a richer and more satisfying interactive experience.

[0058] The prompt language model adopted by the present invention consists of five modules: input, prompt generation, large language model, output and feedback. The input module first receives and pre-processes the original text, performs word segmentation, removes stop words and part-of-speech tagging, and enhances text comprehension. Then, the prompt generation module creates task prompts according to user needs, guides the model to understand the text and generate the expected output. The large language model, that is, the large language model, uses its rich language knowledge base to conduct in-depth analysis of the text and generate the internal representation required for the task. The output module converts these internal representations into texts that conform to grammatical rules and accurately convey core information. Finally, the feedback module allows users to provide feedback to ensure the accuracy and relevance of the output results.

[0059] When integrating large models into robot systems, ensuring accurate understanding of non-standardized user voice commands is a major challenge. The solution proposed in this paper is a prompt engineering method based on a large language model, which converts the user's natural language requests into control commands for the robot. The interaction process is divided into two steps: first, the speech recognition module ASR converts the voice signal into text and identifies key operation words, such as "jogging". Secondly, the keywords are converted into standardized control commands by the prompt engineering module of the large language model, sent to the robot controller for execution, and the speech synthesis module TTS provides user feedback to optimize the accuracy of the command and achieve more precise robot control.

[0060] Furthermore, the S102: sensing the current environment of the robot: first, performing terrain recognition; wherein the specific process of terrain recognition includes:

[0061] The RGB image and depth image captured by the depth camera are transmitted to the large language model for analysis. The large language model extracts feature information in the image to perform terrain recognition and infer the environment directly in front of the robot.

[0062] For example, when the robot camera captures a grassy terrain in an outdoor environment and there are many trees around, the large language model analyzes the color and texture information of the RGB image and the spatial information of the depth image to accurately identify that the terrain in front is grass and that the robot is probably in a park environment. This method can effectively avoid the problem of inaccurate terrain recognition in strong light conditions and can quickly and accurately give terrain recognition results.

[0063] It should be understood that the S102: using a mapping algorithm to locate the current position of the robot, the lidar odometer and mapping algorithm are based on the LOAM (Lida Odometry and Mapping) framework, which uses a point cloud model to predict the dynamic characteristics and environmental changes of the robot in the future in each processing cycle, and then seek the optimal feature point detection and matching strategy in the current processing cycle in real time.

[0064] Furthermore, the S102: using a mapping algorithm to locate the current position of the robot specifically includes:

[0065] S102-1: Receive and synchronize point cloud data, perform data preprocessing, and remove invalid nan value points;

[0066] S102-2: timestamp calibration, calculating the smoothness of the points, and identifying feature points, the feature points including plane points and edge points;

[0067] S102-3: Match the feature points of the current frame with the feature points of the adjacent frames, estimate the motion of the robot, and obtain the position of the robot at the current moment.

[0068] Furthermore, the S102-1: receiving and synchronizing point cloud data, performing data preprocessing, and removing invalid nan value points includes:

[0069] The point cloud data collected by the lidar is received and synchronized through network communication, and data preprocessing is performed, that is, invalid nan value points (not a number, not a digital data point) are removed.

[0070] Furthermore, the S102-2: calibrating by time stamp includes:

[0071] During the point cloud data acquisition stage, each point cloud data is assigned a timestamp to record the exact time it was collected; the timestamps of all point cloud data frames are analyzed to ensure that they are synchronized with a unified time reference (such as the time when the system started); then the timestamps are used to sort the data frames to ensure that they are arranged in chronological order.

[0072] Furthermore, the calculating the smoothness of the points and identifying the feature points include:

[0073]

[0074] Among them, S represents the i-th point in the k-th frame point cloud data A set of continuous points near, |S| represents the number of midpoints, L represents the laser radar coordinate system, represents the coordinates of the i-th point in the k-th frame of point cloud data in the laser radar coordinate system, and ||·|| represents the Euclidean norm. If the smoothness is less than the set threshold, it indicates that the current point is a plane feature; if the smoothness is greater than the set threshold, it indicates that the current point is an edge feature.

[0075] Furthermore, the S102-3: matching the feature points of the current frame with the feature points of the adjacent frames to estimate the motion of the robot and obtain the position of the robot at the current moment, includes:

[0076] Select the feature points of the same type with the closest distance between the two frames as the feature points to be matched, and calculate the relative distance d between the two. ε , the calculation formula is defined as:

[0077]

[0078] in, represents the predicted coordinates of the i-th point in the k+1-th frame in the laser radar coordinate system L, and They represent the reference coordinates of the j-th point and the l-th point in the k-th frame respectively, × represents the vector cross product, and |·| represents the modulus of the vector.

[0079] Assuming the robot moves at a constant speed, a nonlinear optimization method is used to make the distance d ε The minimum value can be obtained by combining the movement of the robot between time k and time k+1 with the movement of the previous k moments to obtain the positioning of the robot at the current moment.

[0080] Furthermore, in S102: performing imperative navigation path planning according to the current position of the robot and the rescue destination, in order to ensure that the robot system can have high real-time performance and accuracy when operating in an unknown environment, an imperative learning (IL) end-to-end path planning method is proposed, which specifically includes:

[0081] S102-4: The coordinates of the rescue destination are given in the world coordinate system, and the coordinates of the destination are converted into the robot coordinate system through the conversion relationship between the world coordinate system and the robot coordinate system;

[0082] S102-5: pre-processing the depth image captured by the camera, and inputting the processed image into the trained command path planning network to obtain a planned path;

[0083] The trained imperative path planning network includes: a linear layer, a perception layer, a serial splicing layer and a fully connected layer; the perception layer is implemented by Resnet-18; the input end of the linear layer is used to input the coordinates of the starting point and the coordinates of the target position point; the input end of the perception layer is used to input the depth image; the output end of the linear layer and the output end of the perception layer are both connected to the input end of the serial splicing layer; the output end of the serial splicing layer is connected to the input end of the fully connected layer; the output end of the fully connected layer outputs the planned path.

[0084] The training process of the trained imperative path planning network includes: constructing a training set, wherein the training set is the starting point and the target point of the known planned path; inputting the training set into the imperative path planning network, training the imperative path planning network, and stopping the training when the loss function value of the imperative path planning network no longer decreases, thereby obtaining the trained imperative path planning network.

[0085] The preprocessing of S102-5 includes: using median filtering to reduce image noise, adjusting pixel values ​​to ensure that their conversion relationship with actual depth values ​​is correct and remains within a reasonable range, and truncating the depth range as needed to retain appropriate depth information. The range is determined according to demand. A range that is too small will result in insufficient collection of information, while a range that is too large will result in overly complex spatial information, both of which will reduce the effect of path planning.

[0086] Furthermore, the design of the imperative path planning network covers multiple key links such as linear layer, perception layer, fully connected layer, loss generation and parameter update, forming a complete system. The network structure is as follows Figure 5 shown.

[0087] The perception layer is based on the convolutional neural network (CNN) technology, which uses its powerful feature extraction capabilities to process the input depth image. The core of the perception layer is the Resnet (Residual Network) network structure, which effectively solves the gradient vanishing and gradient exploding problems in deep network training. The efficient variant of Resnet-18 is used to reduce the computational burden while maintaining performance. The features extracted by the perception layer are combined with the target point features extracted by the linear layer to provide input for the planning layer.

[0088] The fully connected layer predicts a path consisting of multiple key points that point to the target location together, forming an optimized path K t . The generation of the path not only relies on the features of the depth image, but also involves the precise understanding and prediction of the target point. The generated path is then mapped into a pre-constructed Euclidean Signed Distance Field (ESDF) cost map, a step that converts the path into a quantifiable loss C. The loss C is combined with the other task losses L in the network and used to update the network parameters θ through the network's back-propagation mechanism.

[0089] The linear layer is constructed by using a fully connected layer to achieve the fusion of the spatial information of the target point and the output features of the perception network. The three-dimensional channel information of the target point is expanded to match the number of channels output by the perception network to ensure the consistency of the feature dimension, and then the composite structure of the convolutional neural network (CNN) and the multi-layer perceptron (MLP) is combined to process high-dimensional feature information. The CNN part uses its local connection and parameter sharing characteristics to effectively capture local spatial features, while the MLP part further abstracts and integrates the features through its fully connected layer, and finally predicts a path K containing n key points. t .

[0090] The cost C obtained by mapping the key point path to the cost map and other task-level losses L are used to construct the network loss function. The cost map is constructed by extending the traditional binary occupancy grid map using the signed distance field (ESDF) based on the Euclidean distance. Each grid point in the ESDF identifies the existence of an obstacle and records the exact Euclidean distance to the nearest obstacle. Gaussian filtering technology is applied to improve the smoothness and differentiability of the ESDF. While maintaining the distance information, the cost map is locally smoothed to ensure the continuity and differentiability of the cost map locally. i Mapped into the cost map, the collision loss C is obtained o for:

[0091]

[0092] Where H(p i ) is p in the cost graph iThe value of the corresponding position, m is the number of key points in the path. Target loss C G By path K t The distance from the end point to the target point is calculated:

[0093]

[0094] Function E calculates the Euclidean distance between two given points, and path smoothness loss C M The calculation formula is:

[0095]

[0096] is the coordinate of the current position, is the coordinate of the target point, function represents the path interval length between two given points, n is the number of path points, and the total loss C of the path can be expressed as C o , C G and C M Combination of:

[0097]

[0098] Among them, α, β, and γ are hyperparameters used to balance the costs between different items.

[0099] The planning network outputs a collision probability to assess the risk of a collision. The collision probability can also constitute a fear loss, which is L(K t ) is obtained through binary cross entropy (BCE):

[0100]

[0101] The BCELoss(p, y) loss function is:

[0102] BCELoss(p,y)=-y·log(p)+(1-y)·log(1-p) (9) In the above formula, p is the probability predicted by the model, and y is the actual label, usually 0 or 1, indicating whether the sample belongs to the positive class. Then formula (8) shows two cases: when y = 1.0, the loss function measures the probability of the model prediction u t When y = 0.0, the loss function measures the model prediction probability u t In the case of , the actual label is the loss of the negative class (i.e., a collision occurs). The final training loss F is:

[0103] F=C(K t )+L(K t ) (10)

[0104] Where C(K t ) is the cost of planning the path, L(K t ) is the fear loss of collision. Back-propagate the training loss F together, use gradient descent to optimize the network parameter θ, and iterate to find the optimal path:

[0105]

[0106] Where θ is the current network parameter, θ i+1 is the updated network parameter, ω is a hyperparameter about the learning rate, controlling the step size of the gradient descent, K t For the path, is the loss function F relative to the network parameters θ i The gradient of the training loss F on the network parameter θ i The gradient is:

[0107]

[0108] Furthermore, the S103: the robot moves according to the planned navigation path, and in the process of the robot moving, the robot performs gait adjustment, speed adjustment and path optimization according to the environment adaptation strategy, which specifically includes:

[0109] After registering the RGB image with the depth image, the position information of the object in the image is obtained at the same time;

[0110] The RGB image and position information are input into the large language model to obtain information about objects in the robot's surrounding environment, identify objects or terrain in the environment, and adjust the robot's gait and speed according to the environmental adaptation strategy: if it is a snowy environment, increase the robot's gait frequency; if it is a continuous step, reduce the robot's gait frequency, speed and stride height;

[0111] A large language model is used to parse the position of objects in the image, and the trained imperative path planning network is used to generate an optimized motion path based on the object position information.

[0112] For example, Figure 3 The flowchart of building environment semantics based on a large language model is shown. The environment information is collected using the depth camera Intel Realsense D435i to obtain RGB images and depth images respectively. In order to obtain the semantic information and location information of the object at the same time, it is necessary to add the corresponding color to each pixel in the depth image. The RGB image is used as the input of the GML Calibaration Toolbox (camera calibration software) and the internal parameters are calculated to obtain the internal parameter matrix H of the RGB image. rgb And the intrinsic parameter matrix H of the depth map ir. Construct the depth image point coordinates P ir =(x, y, z), p ir is the projection coordinate of the point on the image plane. Where (x, y) is the coordinate of the pixel in the depth image, and z is the depth value of the pixel. From the pinhole imaging model, we can see that the relationship is as follows:

[0113] P ir =H ir p ir (13)

[0114] Let P rgb is the spatial coordinate of the same point in the RGB camera coordinate system, P rgb is the projection coordinate of the point on the image plane. Since the coordinates of the depth map and the RGB map may be different, they need to be connected using rotation and translation transformation, that is,

[0115] P rgb =RP ir +T (14)

[0116] Where R is the rotation matrix, T is the translation vector, and H rgb P rgb Projection, calculate the coordinates corresponding to the point:

[0117] p rgb =H rgb P rgb (15)

[0118] In constructing p ir When calculating p, the original pixel coordinates (x, y) need to be multiplied by the depth value z. rgb After that, you need to rgb Divide by its z component to get the actual pixel coordinate.

[0119] The external parameter matrix is ​​composed of the rotation matrix R ir (or R rgb ) and the translation vector T ir (or T rgb ), which represents the transformation from the global coordinate system to the depth image or RGB image coordinate system:

[0120] P ir =R ir P+T ir (16)

[0121] P rgb =R rgb P+T rgb (17) In formula (16), P is replaced by P ir , R ir and Tir Expressed as, and substituted into formula (17) to obtain:

[0122]

[0123] Compare P rgb =RP ir +T formula can be obtained:

[0124]

[0125] in is the rotation matrix R ir The inverse of R and T can be obtained from the coordinate P of the depth image. ir After coordinate transformation, it is converted to the coordinate P under the RGB image rgb , in the internal parameter matrix H rgb Multiplying it by p gives rgb . And p rgb It is also a three-dimensional vector, whose x and y coordinates are the pixel coordinates in the RGB image. The color of the RGB pixel coordinate can be used as the color of the corresponding coordinate in the depth image. After the RGB image and the depth image are registered, the semantic information and position information of the image can be obtained at the same time.

[0126] On this basis, by calling the token of the large language model, the RGB image and location information are analyzed online to accurately identify different objects and terrains, such as grass, sand, continuous steps, etc., and by analyzing the information of objects in the robot's surrounding environment, it is inferred that the environment may be an office, campus, park or road. According to the identified terrain environment, the corresponding control strategy is set, and the robot will autonomously switch to a motion state that adapts to different terrains. For example, in the snow, the robot will adopt more frequent gaits to avoid slipping; on the grass, the robot will reduce the stride height to adapt to the uneven ground; on the continuous steps, the robot will reduce the step frequency and increase the step height to achieve smooth up and down step movement, etc., and return the text and control instructions after the large model analysis to the controller to execute the control instructions.

[0127] A large language model is used to analyze the positions of surrounding objects, and a motion path is generated based on the object position information using an imperative path planning network. This is to plan the robot's future movement, adjust the autonomous navigation strategy in real time, and enhance the robot's perception of unknown environments.

[0128] Finally, the Baidu Qianfan short text synthesis module (TTS, Text-To-Speech) is called to perform speech synthesis on the parsed text. After obtaining the wav voice file, it is output through the output device Newman DP01 Bluetooth speaker to further improve the ability of human-computer interaction.

[0129] Furthermore, the S104: error detection is performed on the robot, and if an error is found, the error is corrected. The specific process of the error detection includes:

[0130] When the robot is moving along the path generated by the command path planning network, if the actual path is found to be deviated, a path deviation error will occur;

[0131] The large language model re-parses the destination coordinates in the rescue instructions and triggers the trained path planning network to re-plan the path;

[0132] If the robot encounters an irregular structure terrain, the irregular structure terrain refers to a step whose height is higher than a set threshold, then the control instruction of the large language model is to increase the step height so that the robot can cross the step;

[0133] If the increased stride height is not enough to cross the high step, it means that a motion command error has occurred. The large model will re-parse the information of the RGB image and the depth image to determine whether the step height is greater than the physical limit of the robot joint. If the stride height increases to the physical limit of the robot joint, the robot is not enough to cross the high step, and the large model will output an instruction that the current step cannot be crossed.

[0134] Furthermore, S105: after the robot arrives at the rescue destination, it performs rescue tasks, specifically including: placing rescue supplies such as water, medicines, etc. on the robot for material transportation; communicating with the injured to provide psychological construction and emotional support; collecting on-site data such as images, videos and other information that is conducive to rescue, and providing information support for the rescue operation.

[0135] Furthermore, if Figure 6 As shown, the robot's controller is connected to the switch through the first network port, the switch is connected to the robot's leg and foot drivers through the second network port, the controller is also connected to the lidar through the third network port, the controller is also connected to the depth camera through the USB interface, and the controller is also connected to the speaker through Bluetooth.

[0136] Figure 4 The intelligent behavior scheduling flow chart based on the large language model is presented, showing the relationship between the robot's environmental perception, path planning, and error detection. This method of using large models for intelligent behavior scheduling allows legged robots to achieve efficient task planning under physical and environmental constraints, and still have adaptability and robustness when dealing with multi-task and multi-objective optimization problems.

[0137] After an earthquake or other natural disaster, search and rescue missions are crucial but often challenging. By integrating the above key technologies, quadruped robots can be used to perform search and rescue missions in such extreme environments. First, through the powerful natural human-computer interaction technology of the large language model, the rescue team can issue rescue instructions to the robot, such as "search a specific area or transport supplies and avoid potential dangers", etc., and use the efficient and accurate semantic parsing ability of the large language model to convert complex search and rescue instructions into action sequences that the robot can execute, and complete the execution of rescue operations according to linear temporal logic. This natural human-computer interaction capability greatly improves the flexibility and response speed of rescue missions. Secondly, the robot is equipped with sensors such as lidar and cameras. Using the lidar odometer and mapping algorithm positioning strategy, it can create a three-dimensional map of the surrounding environment in real time at the disaster site and accurately locate its own position. Combined with the end-to-end path planning method of command-based learning, the robot can automatically plan the optimal path to avoid obstacles and quickly reach the area where survivors may exist. At the same time, through the environmental adaptation strategy based on the large language model, the quadruped robot can accurately identify and understand the terrain environment around the disaster site, and autonomously adjust the movement behavior (such as switching gait, adjusting speed, etc.) according to the terrain environment information to adapt to different search and rescue scenarios. In addition, during the search process, the intelligent behavior scheduling method based on the large language model can perform real-time analysis of the sensor data and body information such as radar, camera and voice collected by the robot. This information is used by the large language model to detect planning errors or execution errors generated by the robot when performing search and rescue tasks in real time. When an error occurs, the feedback mechanism of the large language model will be triggered for re-planning, allowing the robot to maintain efficient and stable movement in complex terrain environments. Through this integrated technical solution, the quadruped robot shows excellent adaptability and robustness in search and rescue missions, providing a powerful auxiliary tool for the rescue team and significantly improving the success rate of rescue in complex disaster environments.

[0138] The intelligent behavior scheduling strategy based on the large language model can realize the semantic parsing and normalization of voice or text input by calling the large language model online, convert natural language instructions into semantic instructions that can be understood by the robot, and realize natural human-robot interaction between humans and robots. Based on the semantic instructions generated by the large language model, task sub-functions containing robot motion constraints are generated through sub-task planning to realize high-level planning and scheduling of robot behavior. The high-level feedback mechanism is used to correct errors generated in the task planning process in a timely manner to ensure the accuracy and efficiency of task execution. At the same time, the dynamic adjustment and optimization of robot behavior are realized by importing task sub-functions and robot motion constraints. In the task execution stage, the high-level planning is converted into specific motion sequences and trajectories through motion state machines and motion planning. The motion controller evaluates and adjusts the motion execution in real time according to the robot attributes and environmental feedback to ensure the smoothness and safety of the motion. When encountering motion errors, the system can trigger re-planning through the underlying feedback mechanism to adapt to environmental changes and task requirements. In addition, text input and natural language processing technology are used to continuously monitor and optimize robot behavior. This method realizes the intelligent behavior scheduling and autonomous decision-making capabilities of legged robots in complex environments, providing strong technical support for the intelligence and automation of robots. This not only improves the autonomy and adaptability of robots, but also provides more possibilities for the application of robots in diverse task scenarios.

[0139] The present invention integrates a large language model into the leg-foot robot system to give the robot natural language processing capabilities, which can accurately parse the user's instructions and intentions, achieve smooth communication and dialogue with the user, quickly respond to and execute the user's corresponding action instructions, and enhance the user's interactive experience.

[0140] The present invention uses a large language model to perform deep information fusion and analysis of visual and language sensor data, giving the robot a comprehensive understanding and precise perception of complex environments. It can accurately identify a variety of different terrains and objects, such as sand, snow, grass, and continuous steps. Based on the identification of objects and terrain, it can infer and judge that the environment may be an office, campus, park, or road, and thus switch between different motion states to adapt to movement on different terrains.

[0141] Through the large language model, the user's natural language instructions such as "avoid obstacles and go to the specified location" are converted into semantic instructions executable by the robot, and the corresponding action sequence is generated to guide the robot's action execution. At the same time, the large language model is used to monitor the robot's motion state and environmental changes in real time. When encountering dynamic obstacles or environmental changes, the behavior strategy can be quickly adjusted, and the gait and speed can be automatically adjusted to ensure that the robot maintains stable and efficient movement in complex terrain, effectively improving the robot's task execution efficiency and expanding the application scenarios of leg-footed robots.

[0142] Embodiment 2

[0143] This embodiment provides an autonomous behavior control system for a legged robot based on a multimodal large model, including:

[0144] An acquisition module is configured to: acquire a rescue instruction, perform instruction analysis on the rescue instruction, and obtain a rescue mission and a rescue destination;

[0145] The perception module is configured to: perceive the current environment of the robot: first, perform terrain recognition; then, use a mapping algorithm to locate the current position of the robot; finally, perform navigation path planning based on the current position of the robot and the rescue destination;

[0146] The optimization module is configured to: the robot moves according to the planned navigation path, and in the process of the robot moving, according to the environment adaptation strategy, the robot gait adjustment, speed adjustment and path optimization;

[0147] A detection module is configured to: perform error detection on the robot and correct the error if an error is found;

[0148] The output module is configured to: after the robot arrives at the rescue destination, perform the rescue mission.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The autonomous behavior control method of legged robot based on multi-modal large model is characterized by: Controllers for robots, including: Obtaining a rescue instruction, performing instruction analysis on the rescue instruction, and obtaining a rescue mission and a rescue destination; Perceive the robot's current environment: First, identify the terrain; then use a mapping algorithm to locate the robot's current location; finally, plan a navigation path based on the robot's current location and the rescue destination; The robot moves according to the planned navigation path. During the movement of the robot, the robot adjusts its gait, speed and path according to the environment adaptation strategy. Perform error detection on the robot and correct the error if any; After the robot arrives at the rescue destination, it performs the rescue mission.

2. The method for controlling the autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, characterized in that: The rescue command is analyzed to obtain a rescue mission and a rescue destination. The specific analysis process includes: Convert rescue instructions from voice to text; Perform preprocessing operations on rescue instructions in text form; The text rescue instructions after preprocessing are parsed using linear temporal logic to obtain several subtasks; Each subtask is converted into subtask text through a large language model, and all subtask texts are converted into speech output.

3. The method for controlling the autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 2, characterized in that: The text rescue instructions after preprocessing are parsed using linear temporal logic to obtain several subtasks, including: in, represents an atomic proposition, p i represents the i-th subtask, N AP Indicates the number of subtasks, ∷= indicates a production rule, Indicates logical negation, ∧ and ∨ are logical AND and logical OR respectively, represents the atomic proposition itself, Represents the logical AND of two linear sequential logic formulas; rescue instructions are used express; Indicates that a condition is always true throughout the time range. Indicates that a condition will eventually be met. Expresses a relationship between two conditions where one condition remains true until the other condition is true.

4. The method for controlling the autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, characterized in that: Perceive the robot's current environment: First, perform terrain recognition; the specific process of terrain recognition includes: transmitting the RGB image and depth image captured by the depth camera to the large language model for analysis, and the large language model extracts feature information in the image to perform terrain recognition and infer the environment directly in front of the robot.

5. The method for controlling the autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, characterized in that: Use a mapping algorithm to locate the robot's current position, including: Receive and synchronize point cloud data, perform data preprocessing, and remove invalid nan value points; Timestamp calibration, calculating the smoothness of points, identifying feature points, the feature points including plane points and edge points; The feature points of the current frame are matched with the feature points of the adjacent frames to estimate the robot's motion and obtain the robot's position at the current moment.

6. The method for controlling the autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 5 is characterized in that: The calculating the smoothness of the points and identifying the feature points include: Among them, S represents the i-th point in the k-th frame point cloud data A set of continuous points near, |S| represents the number of midpoints, L represents the laser radar coordinate system, represents the coordinates of the i-th point in the k-th frame point cloud data in the laser radar coordinate system, and ||·|| represents the Euclidean norm; if the smoothness is less than the set threshold, it indicates that the current point is a plane feature; if the smoothness is greater than the set threshold, it indicates that the current point is an edge feature; Match the feature points of the current frame with the feature points of the adjacent frames to estimate the robot's motion and obtain the robot's position at the current moment, including: Select the feature points of the same type with the closest distance between the two frames as the feature points to be matched, and calculate the relative distance d between the two. ε , the calculation formula is defined as: in, represents the predicted coordinates of the i-th point in the k+1-th frame in the laser radar coordinate system L, and denote the reference coordinates of the jth point and the lth point of the kth frame, respectively, × denotes the vector cross product, and |·| denotes the vector modulus; under the assumption that the robot moves at a constant speed, a nonlinear optimization method is used to make the distance d ε The minimum value is obtained, and the movement of the robot between time k and time k+1 can be obtained. By associating it with the movement of the previous k moments, the positioning of the robot at the current moment can be obtained.

7. The method for controlling the autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, characterized in that: According to the robot's current location and rescue destination, navigation path planning is carried out, including: The coordinates of the rescue destination are given in the world coordinate system, and the coordinates of the destination are converted to the robot coordinate system through the conversion relationship between the world coordinate system and the robot coordinate system; The depth image captured by the camera is preprocessed and the processed image is input into the trained command path planning network to obtain the planned path; The trained command path planning network includes: a linear layer, a perception layer, a series connection layer and a fully connected layer; the perception layer is implemented by Resnet-18; the input end of the linear layer is used to input the coordinates of the starting point and the coordinates of the target position point; the input end of the perception layer is used to input the depth image; the output end of the linear layer and the output end of the perception layer are both connected to the input end of the series connection layer; the output end of the series connection layer is connected to the input end of the fully connected layer; the output end of the fully connected layer outputs the planned path; The training process of the trained imperative path planning network includes: constructing a training set, wherein the training set is the starting point and the target point of the known planned path; inputting the training set into the imperative path planning network, training the imperative path planning network, and stopping the training when the loss function value of the imperative path planning network no longer decreases, thereby obtaining the trained imperative path planning network.

8. The method for controlling the autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, characterized in that: The robot moves along the planned navigation path. During the movement, the robot adjusts its gait, speed and path according to the environment adaptation strategy, including: After registering the RGB image with the depth image, the position information of the object in the image is obtained at the same time; The RGB image and position information are input into the large language model to obtain information about objects in the robot's surrounding environment, identify objects or terrain in the environment, and adjust the robot's gait and speed according to the environmental adaptation strategy: if it is a snowy environment, increase the robot's gait frequency; if it is a continuous step, reduce the robot's gait frequency, speed and stride height; A large language model is used to parse the position of objects in the image, and the trained imperative path planning network is used to generate an optimized motion path based on the object position information.

9. The method for controlling the autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, characterized in that: The robot is subjected to error detection, and if an error is found, the error is corrected. The specific process of the error detection includes: When the robot is moving along the path generated by the command path planning network, if the actual path is found to be deviated, a path deviation error will occur; The large language model re-parses the destination coordinates in the rescue instructions and triggers the trained path planning network to re-plan the path; If the robot encounters an irregular structure terrain, the irregular structure terrain refers to a step whose height is higher than a set threshold, then the control instruction of the large language model is to increase the step height so that the robot can cross the step; If the increased stride height is not enough to cross the high step, it means that a motion command error has occurred. The large model will re-parse the information of the RGB image and the depth image to determine whether the step height is greater than the physical limit of the robot joint. If the stride height increases to the physical limit of the robot joint, the robot is not enough to cross the high step, and the large model will output an instruction that the current step cannot be crossed.

10. The autonomous behavior control system of legged robot based on multi-modal large model is characterized by: include: An acquisition module is configured to: acquire a rescue instruction, perform instruction analysis on the rescue instruction, and obtain a rescue mission and a rescue destination; The perception module is configured to: perceive the current environment of the robot: first, perform terrain recognition; then, use a mapping algorithm to locate the current position of the robot; finally, perform navigation path planning based on the current position of the robot and the rescue destination; The optimization module is configured to: the robot moves according to the planned navigation path, and in the process of the robot moving, according to the environment adaptation strategy, the robot gait adjustment, speed adjustment and path optimization; A detection module is configured to: perform error detection on the robot and correct the error if an error is found; The output module is configured to: after the robot arrives at the rescue destination, perform the rescue mission.

Images