Autonomous behavior control method and system for leg-foot robot based on multi-modal large model

Abstract

The application discloses a multi-modal large model-based autonomous behavior control method and system for a leg-foot robot, the method is applied to a controller of the robot, and the method comprises the following steps: obtaining a rescue instruction, performing instruction analysis on the rescue instruction to obtain a rescue task and a rescue destination; perceiving an environment in which the robot currently locates; firstly, terrain recognition is performed; then, a mapping algorithm is used to locate a current position of the robot; finally, navigation path planning is performed according to the current position of the robot and the rescue destination; the robot travels according to the planned navigation path, and in the process of traveling of the robot, the robot is adjusted in gait, speed and path according to an environment self-adapting strategy; error detection is performed on the robot, and if an error is found, the error is corrected; after the robot reaches the rescue destination, the rescue task is performed.

Description

Technical Field

[0001] The present invention relates to the technical field of rescue robots, and in particular to a method and system for controlling autonomous behavior of a leg-foot robot based on a multimodal large model. Background Art

[0002] In the field of intelligent robotics, legged robots (bipeds, quadrupeds, hexapods, humanoids, etc.) have attracted considerable attention due to their exceptional mobility and adaptability. As a key branch of intelligent robotics, legged robots, with their unique mobility and adaptability, have demonstrated tremendous potential for application in a variety of scenarios, including search and rescue, home services, and industrial inspection. The design and control technology of legged robots has long been a hot topic in robotics research.

[0003] Although legged robots have broad application prospects in many fields, there are some problems in the existing technology that need to be solved urgently.

[0004] Especially in terms of environmental perception, legged robots have limited target recognition capabilities under conditions of drastic changes in light, making it difficult to accurately capture and analyze environmental information.

[0005] In addition, the current leg-foot robot technology is still rudimentary in terms of natural language interaction, mainly manifested in the lack of understanding and reasoning ability of complex human grammatical structures, context dependencies, new vocabulary expressions, and rhetorical techniques such as implicit semantics. This limits the robot's accurate grasp of user intentions during the interaction process, greatly reducing the efficiency of human-computer interaction and user experience.

[0006] In terms of autonomous behavior control, existing legged robot technology performs poorly in dealing with complex dynamic environments that require real-time decision-making and strategy adjustments. When the robot encounters unknown obstacles or changes in the environmental layout, it often finds it difficult to make optimal decisions, resulting in inefficient navigation and task execution. Summary of the Invention

[0007] In order to address the shortcomings of the existing technology, the present invention provides a method and system for autonomous behavior control of a legged robot based on a multimodal large model; utilizing the powerful semantic understanding ability of the large language model to more deeply parse the user's complex semantic instructions, thereby improving the accuracy and efficiency of human-computer interaction; through the deep fusion and intelligent analysis of multimodal information such as visual images and voice by the large language model, the robot is given a comprehensive understanding and precise perception of complex environments; based on the continuous learning and self-optimization capabilities of the large language model, the robot can quickly generate and execute the optimal behavior strategy based on the current environmental status and expected goals in a complex dynamic environment.

[0008] In one aspect, a multi-modal large model-based autonomous behavior control method for a leg-foot robot is provided, which is applied to a controller of the robot and includes the following steps:

[0009] A rescue instruction is obtained, and the rescue instruction is analyzed to obtain a rescue task and a rescue destination;

[0010] The environment in which the robot is currently located is perceived: first, terrain recognition is performed; then, a mapping algorithm is used to locate the current position of the robot; finally, a navigation path is planned according to the current position of the robot and the rescue destination;

[0011] According to the planned navigation path, the robot travels, and during the travel of the robot, the robot is adjusted in gait, speed, and path according to an environment adaptive strategy;

[0012] The robot is detected for errors, and if errors are found, the errors are corrected;

[0013] After the robot reaches the rescue destination, the rescue task is performed.

[0014] In another aspect, a multi-modal large model-based autonomous behavior control system for a leg-foot robot is provided, which includes the following components:

[0015] An acquisition module configured to: obtain a rescue instruction, analyze the rescue instruction, and obtain a rescue task and a rescue destination;

[0016] A perception module configured to: perceive the environment in which the robot is currently located: first, terrain recognition is performed; then, a mapping algorithm is used to locate the current position of the robot; finally, a navigation path is planned according to the current position of the robot and the rescue destination;

[0017] An optimization module configured to: according to the planned navigation path, the robot travels, and during the travel of the robot, the robot is adjusted in gait, speed, and path according to an environment adaptive strategy;

[0018] A detection module configured to: detect errors of the robot, and if errors are found, correct the errors;

[0019] An output module configured to: after the robot reaches the rescue destination, perform the rescue task.

[0020] The above technical solution has the following advantages or beneficial effects:

[0021] In terms of environmental perception, a multi-modal perception and instruction-based navigation method for leg-foot robots is proposed, which integrates large language models. By integrating radar, camera, and voice sensors, etc., a comprehensive collection of environmental information is achieved. The large language model is used to efficiently fuse and analyze multi-modal perception data, improving the robot's ability to perceive, identify, and understand key features and dynamic changes in the environment. Instruction-based navigation technology combines deep learning and large language models to achieve deep perception and understanding of the environment, as well as analysis and path planning of target instructions. Using an optimal trajectory optimizer and backpropagation mechanism, the cost function is minimized to optimize the motion trajectory, enabling the robot to make real-time decisions in unknown environments for safe and efficient navigation.

[0022] In terms of natural language interaction, a leg-foot robot motion, learning, and emotional interaction method under natural language interaction is invented. By integrating large language model technology, the leg-foot robot is provided with logical reasoning capabilities, enabling it to deeply analyze the semantic information of human natural language instructions and image information, and further accurately understand user intentions and execute corresponding actions or tasks. The large language model also gives the robot the ability to learn and interact emotionally, providing a more personalized interactive experience and greatly improving user experience.

[0023] In terms of autonomous behavior control, a leg-foot robot intelligent behavior scheduling method based on large language models is designed. Input information is converted into semantic information executable by the robot, and task sub-functions are generated to achieve high-level behavior planning and dynamic adjustment. The integrated high-level feedback mechanism ensures the accuracy and execution efficiency of task planning, while the integration of motion state machines and motion planning technology concretizes abstract planning into precise robot motion. Real-time evaluation and adaptive control further improve the smoothness and safety of motion, and the bottom-level feedback mechanism provides strong support for responding to environmental changes and task requirements. In addition, continuous monitoring and optimization mechanisms ensure continuous improvement of robot behavior, and cross-module coordination capabilities ensure overall consistency and synergy of the system. The intelligent behavior scheduling method improves the autonomy and adaptability of leg-foot robots, supporting their application in diverse task scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which form a part of this description, are included to provide further understanding of the application, and are incorporated in and constitute a part of this application. The illustrative embodiments of the application and their description serve to explain the application. They do not, however, limit the present application's scope, nor do they represent that the present application will necessarily achieve undesired results in all circumstances.

[0025] Figure 1 The method flowchart of Example One;

[0026] Figure 2 The semantic analysis method flowchart of natural human-computer interaction of Example One;

[0027] Figure 3 A flow chart of an environment semantic construction based on a large language model for Embodiment I;

[0028] Figure 4 A flow chart of intelligent behavior scheduling based on a large language model for Embodiment I.

[0029] Figure 5 A network structure diagram of an instructive path planning for Embodiment I.

[0030] Figure 6 A hardware electrical connection diagram for Embodiment I. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0032] Embodiment I

[0033] The embodiment provides a multi-modal large model-based autonomous behavior control method of a leg-foot robot;

[0034] As shown in the figure, the multi-modal large model-based autonomous behavior control method of the leg-foot robot is applied to a controller of the robot, and comprises the following steps: Figure 1

[0035] S101: obtaining a rescue instruction, performing instruction analysis on the rescue instruction to obtain a rescue task and a rescue destination;

[0036] S102: perceiving an environment in which the robot is currently located: first, performing terrain recognition; then, positioning a current location of the robot by using a mapping algorithm; and finally, performing navigation path planning according to the current location of the robot and the rescue destination;

[0037] S103: the robot performs marching according to the planned navigation path, and in the process of marching of the robot, the robot is adjusted in gait, speed and path according to an environment self-adaption strategy;

[0038] S104: performing error detection on the robot, and if an error is found, correcting the error and returning to S101;

[0039] S105: after the robot reaches the rescue destination, performing the rescue task.

[0040] Further, the S101: obtaining a rescue instruction, wherein the rescue instruction comprises searching a set region, transporting survival supplies, avoiding potential dangers and detecting life signs.​

[0041] Further, as Figure 2 indicated, the S101: analyzing the rescue instruction to obtain the rescue task and the rescue destination, the specific specification analysis process includes:

[0042] S101-1: converting the rescue instruction from voice form to text form;

[0043] S101-2: pre-processing the text form rescue instruction;

[0044] S101-3: analyzing the pre-processed text rescue instruction in a linear temporal logic manner to obtain a plurality of sub-tasks;

[0045] S101-4: converting each sub-task into a sub-task text through a large language model, and converting all sub-task texts into voice output.

[0046] The large language model mentioned in the application refers to the Baidu intelligent cloud Qianfan large model platform, and the system integration of the robot is realized by online calling of the Baidu Qianfan large language model.

[0047] Further, the S101-1: converting the rescue instruction from voice form to text form, adopting the Baidu Qianfan voice recognition express version (ASR, Automatic Speech Recognition) to process the rescue instruction, authenticating and creating a voice recognition request by obtaining the AppID, API Key, Secret Key of ASR and exchanging token, ASR recognizing the text result, and then using the text result recognized by ASR for natural language processing.

[0048] Further, the S101-2: pre-processing the text form rescue instruction, including text cleaning to remove noise such as extra spaces, special characters and the like in the text; dividing long text into individual words or phrases and extracting keywords such as place, "transportation materials" and the like rescue instruction intent class to improve the accuracy and efficiency of subsequent natural language processing tasks.

[0049] Further, the S101-3: analyzing the pre-processed text rescue instruction in a linear temporal logic manner to obtain a plurality of sub-tasks, specifically including:

[0050]

[0051] wherein, represents an atomic proposition, p i represents the i-th sub-task, N AP represents the number of sub-tasks, and :: = represents a production rule, denotes logical negation, and ∧ and ∨ denote logical and and logical or, respectively, e.g. denotes the atomic proposition itself, denotes the logical and of two Linear Temporal Logic (LTL) formulas. The rescue instruction is denoted by . denotes that a certain condition holds true at all times, denotes that a certain condition will eventually hold true, denotes the relationship between two conditions, where one condition will persist to be true until the other condition holds true. For example, if the rescue instruction contains multiple steps or conditions, one can consider “arriving at location A” as the proposition p A , and “maintaining communication” as the proposition p comm , then “maintaining communication before arriving at location A” can be denoted as This way provides a linear temporal logic-compliant paradigm basis for large model generation subtask text.

[0052] Further, the S101-4: converting each subtask into a subtask text through a large language model is to convert each subtask into a subtask text compliant with a linear temporal logic formula through a large language model.

[0053] Further, the S101-4: converting all subtask texts into voice output adopts the short text speech synthesis (Text To Speech, TTS) technology of Baidu Qianfan to convert text information into voice output.

[0054] It should be understood that in the process of enhancing the human-computer interaction capability of the leg-foot robot, the present application adopts LTL formula to accurately define and regulate the task behavior of the robot. LTL is a formal method for describing the properties of a system over time, which enhances our ability to describe the behavior of the robot through syntax structure. By using LTL formula, complex tasks can be decomposed into a series of logically verifiable subtasks, ensuring that the behavior of the robot meets the user's expectations. This logical expression capability is crucial for improving user experience, as it allows us to define tasks in a precise and consistent manner.

[0055] The complexity of natural language processing (NLP) poses high standards for large language models, requiring them not only to parse the ambiguity of language, complex syntax, and contextual dependencies, but also to adapt to the continuous evolution of language, including understanding and adapting to emerging vocabulary and expression methods. In order to meet these requirements, large language models must have the ability to continuously learn and update, as well as handle the complexity of implied meanings, irony, humor, and other rhetorical devices in language.

[0056] In this context, introducing prompt language models becomes a key strategy to improve the quality and efficiency of natural human-machine interaction. Prompt language models can be visualized as a large model prompt template. First, the robot receives rescue instructions, and then the large model analyzes the instructions, analyzes the instruction type, including single dialogue, motion control, autonomous navigation, etc. According to the parsed rescue instructions, enter different control modes and return to the controller for execution. For example, when the robot receives a rescue instruction, if the large model does not parse the obvious rescue instruction when parsing the instruction, it enters the dialogue mode; if the large model parses the keyword "go to destination", the system will directly enter the autonomous navigation mode, and plan the path according to the instruction path planning network; if the large model parses "cross high steps", it enters the motion control mode, increases the robot step height and reduces the step frequency.

[0057] In this way, the prompt language model not only improves the response speed of AI and optimizes the semantic analysis ability of AI, but also provides strong support for building a more intelligent and strongly adaptive interactive system. This system can adapt to the individual needs of users and the constant changes of language, making human-computer interaction more smooth and natural, and providing users with a more rich and satisfactory interactive experience.

[0058] The prompt language model adopted by the present application is composed of five modules: input, prompt generation, large language model, output and feedback. The input module first receives and preprocesses the original text, performs word segmentation, removes stop words and part-of-speech tagging, and enhances text understanding. Next, the prompt generation module creates task prompts according to user needs, guiding the model to understand the text and generate expected output. The large language model, i.e. the large language model, uses its rich language knowledge base to analyze the text in depth and generate the internal representation required for the task. The output module converts these internal representations into text that conforms to grammatical rules and accurately conveys core information. Finally, the feedback module allows users to provide feedback to ensure the accuracy and relevance of the output results.

[0059] When integrating large models into robot systems, ensuring accurate understanding of non-standardized user voice instructions is a major challenge. The solution proposed by the present application is a prompt engineering method based on large language models, which converts users' natural language requests into robot control instructions. The interaction process is divided into two steps: first, the speech recognition module ASR converts the speech signal into text and identifies key operation words such as "jogging". Second, the keywords are converted into standardized control instructions by the prompt engineering module of the large language model, sent to the robot controller for execution, and the user feedback is provided by the speech synthesis module TTS to optimize the accuracy of the command and achieve more accurate robot control.

[0060] Further, the S102: perceive the environment in which the robot is currently located: first, perform terrain recognition; wherein the specific process of terrain recognition comprises:

[0061] The RGB image and the depth image collected by the depth camera are transmitted to the large language model for analysis, and the large language model extracts feature information in the image to identify the terrain and infer the environment in front of the robot.

[0062] For example, when the robot camera captures the front as a grassland terrain in an outdoor environment, and there are many trees around, the large language model accurately identifies that the front terrain is grassland and the current environment is likely a park by analyzing the color and texture information of the RGB image and the spatial information of the depth image. This way can effectively avoid the problem of inaccurate terrain recognition of the robot in strong light, and quickly and accurately give the terrain recognition result.

[0063] It should be understood that the S102: uses a mapping algorithm to locate the current position of the robot, and the laser radar odometry and mapping algorithm is based on the LOAM (Lida Odometry and Mapping) framework, which uses a point cloud model to predict the dynamic characteristics and environmental changes of the robot in a future period of time within each processing period, and then seeks the optimal feature point detection and matching strategy in the current processing period in real time.

[0064] Further, the S102: uses a mapping algorithm to locate the current position of the robot, specifically comprising:

[0065] S102-1: receives and synchronizes point cloud data, performs data preprocessing, and removes invalid nan value points;

[0066] S102-2: timestamp calibration, calculating the smoothness of points, identifying feature points, the feature points including plane points and edge points;

[0067] S102-3: matching the feature points of the current frame with the feature points of the adjacent frame, performing motion estimation of the robot, and obtaining the position of the robot at the current time.

[0068] Further, the S102-1: receives and synchronizes point cloud data, performs data preprocessing, and removes invalid nan value points, comprising:

[0069] The point cloud data collected by the laser radar is received and synchronized through network communication, and data preprocessing is performed, that is, invalid nan value points (not a number, data points that are not numbers) are removed.

[0070] Further, the S102-2: includes timestamp calibration, comprising:

[0071] During the point cloud data acquisition phase, each point cloud data is assigned a timestamp to record the exact time it was acquired; the timestamps of all point cloud data frames are analyzed to ensure that they are synchronized with a unified time base (such as the time the system was started); and then the data frames are sorted using the timestamps to ensure that they are arranged in chronological order.

[0072] Furthermore, the calculating of the smoothness of the points and identifying the feature points include:

[0073]

[0074] Among them, S represents the i-th point in the k-th frame point cloud data A set of continuous points near, |S| represents the number of midpoints, L represents the laser radar coordinate system, represents the coordinates of the i-th point in the k-th frame of point cloud data in the LiDAR coordinate system, and ||·|| represents the Euclidean norm. If the smoothness is less than the set threshold, the current point is a plane feature; if the smoothness is greater than the set threshold, the current point is an edge feature.

[0075] Furthermore, the step S102-3: matching the feature points of the current frame with the feature points of the adjacent frames to estimate the motion of the robot and obtain the position of the robot at the current moment, includes:

[0076] Select the feature points of the same type with the closest distance between the two frames as the feature points to be matched, and calculate the relative distance d between the two ε , the calculation formula is defined as:

[0077]

[0078] in, represents the predicted coordinates of the i-th point in the k+1-th frame in the lidar coordinate system L, and They represent the reference coordinates of the j-th point and the l-th point in the k-th frame, respectively. × represents the vector cross product, and |·| represents the modulus of the vector.

[0079] Assuming the robot moves at a constant speed, a nonlinear optimization method is used to make the distance d ε The robot's motion between time k and time k+1 can be obtained by minimizing the motion of the robot at time k and by associating it with the motion at the previous k moments to obtain the robot's positioning at the current moment.

[0080] Further, the S102: according to the current position of the robot and the rescue destination, the instruction navigation path planning is carried out, in order to ensure that the robot system can have higher real-time and accuracy when working in unknown environment, an end-to-end path planning method of imperative learning (IL) is proposed, which specifically includes:

[0081] S102-4: The coordinates of the rescue destination are given in the world coordinate system, and the destination coordinates are converted to the robot coordinate system through the conversion relationship between the world coordinate system and the robot coordinate system;

[0082] S102-5: The depth image captured by the camera is preprocessed, and the processed image is input into the trained imperative path planning network to obtain the planned path;

[0083] The trained imperative path planning network comprises a linear layer, a perception layer, a series concatenation layer and a fully connected layer; the perception layer is implemented by Resnet-18; the input end of the linear layer is used to input the coordinates of the starting point and the target position point; the input end of the perception layer is used to input the depth image; the output end of the linear layer and the output end of the perception layer are connected with the input end of the series concatenation layer; the output end of the series concatenation layer is connected with the input end of the fully connected layer; the output end of the fully connected layer outputs the planned path.

[0084] The training process of the trained imperative path planning network includes: constructing a training set, the training set being the starting point and the target point of the known planning path; inputting the training set into the imperative path planning network to train the imperative path planning network; when the loss function value of the imperative path planning network no longer decreases, the training is stopped, and the trained imperative path planning network is obtained.

[0085] The preprocessing of S102-5 includes: reducing image noise using median filtering, adjusting pixel values to ensure that their conversion relationship with actual depth values is correct and remains within a reasonable range, and truncating the depth range as needed to retain appropriate depth information. The range is determined according to the needs, and a too small range will result in insufficient information and a too large range will result in too complex spatial information, both of which will reduce the effect of path planning.

[0086] Further, the design of the imperative path planning network covers multiple key links such as linear layer, perception layer, fully connected layer, loss generation and parameter update, forming a complete system, and the network structure is as shown in Figure 5 .

[0087] The perception layer is based on convolutional neural network (CNN) technology, which uses its powerful feature extraction capability to process the input depth image. The core of the perception layer is the Resnet (Residual Network) network structure, which effectively solves the gradient vanishing and gradient explosion problems in deep network training. The Resnet-18 variant is used to reduce the computational burden while maintaining performance. The features extracted by the perception layer are combined with the target point features extracted by the linear layer to provide input to the planning layer.

[0088] The fully connected layer predicts a path composed of multiple key points, which collectively point to the target position, forming an optimized path K t The generation of the path not only depends on the features of the depth image, but also involves accurate understanding and prediction of the target point. The generated path is then mapped to a pre-constructed Euclidean signed distance field (ESDF) cost map, which converts the path into a quantifiable loss C. The loss C is combined with other task losses L in the network, and is used to update the network parameters θ through the backpropagation mechanism of the network.

[0089] The linear layer is constructed by combining the spatial information of the target point with the output features of the perception network using a fully connected layer. The three-dimensional channel information of the target point is expanded to match the number of channels output by the perception network, ensuring consistency in feature dimensions. Then, a composite structure of convolutional neural network (CNN) and multi-layer perceptron (MLP) is used to process high-dimensional feature information. The CNN part effectively captures local spatial features using its local connection and parameter sharing characteristics, while the MLP part further abstracts and integrates the features through its fully connected layers, ultimately predicting a path K t composed of n key points.

[0090] The cost C obtained by mapping the key point path to the cost map and other task-level losses L are used to construct the network loss function. The cost map uses a Euclidean distance-based signed distance field (ESDF), which is constructed by extending the traditional binary occupancy grid map. Each grid point in the ESDF identifies the presence of an obstacle and records the exact Euclidean distance to the nearest obstacle. Gaussian filtering is applied to improve the smoothness and differentiability of the ESDF, maintaining the distance information while locally smoothing the cost map to ensure its continuity and differentiability in the local area. The path point p i is mapped to the cost map to obtain the collision loss C o :

[0091]

[0092] where H(p i ) is the cost of the point p iThe value of the corresponding position, m is the number of key points in the path. The target loss C G The path K t The distance from the end point to the target point is calculated as:

[0093]

[0094] The function E calculates the Euclidean distance between two given points, and the path smoothness loss C M The calculation formula is:

[0095]

[0096] is the coordinate of the current position, is the coordinate of the target point, and the function represents the path interval length between two given points, and n is the number of path points. The total loss C of the path can be represented as C o , C G and C M :

[0097]

[0098] where α, β and γ are hyperparameters used to balance the cost between different terms.

[0099] The planning network outputs a collision probability to evaluate the risk of collision, and the collision probability can also constitute a fear loss. The loss L(K t ) is obtained by binary cross entropy (BCE):

[0100]

[0101] where the loss function BCELoss(p, y) is:

[0102] BCELoss(p, y) = -y·log(p) + (1-y)·log(1-p) (9) where p is the probability predicted by the model, and y is the true label, usually 0 or 1, indicating whether the sample belongs to the positive class, then equation (8) shows two cases: when y = 1.0, the loss function measures the loss when the actual label is positive (i.e. no collision) when the model predicts the probability u t ; when y = 0.0, the loss function measures the loss when the actual label is negative (i.e. collision) when the model predicts the probability u t . The final training loss F is:

[0103] F = C(K t ) + L(K t ) (10)

[0104] where C(K t ) is the cost of the planned path, and L(K t ) is the fear loss of collision. The training loss F is back-propagated to optimize the network parameters θ by gradient descent, and the optimal path is iteratively solved:

[0105]

[0106] where θ is the current network parameter, θ i+1 is the updated network parameter, ω is a hyperparameter related to the learning rate, controlling the step size of gradient descent, K t is the path, is the loss function F with respect to the network parameter θ i , and the gradient of the training loss F with respect to the network parameter θ i is obtained:

[0107]

[0108] Further, the S103: according to the planned navigation path, the robot travels, and in the process of the robot traveling, the robot is adjusted in gait, speed and path according to the environment adaptive strategy, specifically including:

[0109] After registering the RGB image and the depth image, the position information of the objects in the image is obtained simultaneously;

[0110] The RGB image and the position information are input into the large language model to obtain the information of the objects in the environment of the robot, and the objects or terrain in the environment are identified, and the robot is adjusted in gait and speed according to the environment adaptive strategy: if it is a snow environment, the gait frequency of the robot is increased; if it is a continuous step, the gait frequency, speed and step height of the robot are reduced;

[0111] The large language model is used to analyze the position of the objects in the image, and the optimized motion path is generated according to the object position information using the trained instructive path planning network.

[0112] Exemplarily, Figure 3 a flowchart of environment semantic construction based on a large language model is shown. The environment information is collected using a depth camera Intel Realsense D435i to obtain an RGB image and a depth image. In order to obtain the semantic information and the position information of the objects simultaneously, a corresponding color needs to be added to each pixel in the depth image. The RGB image is input into the GML Calibaration Toolbox (camera calibration software) to calculate the intrinsic parameters and obtain the intrinsic parameter matrix H rgb of the RGB image and the intrinsic parameter matrix H ir of the depth image. The depth image point coordinates Pir = (x, y, z), p ir is the projection coordinate of the point in the image plane. Where (x, y) is the coordinate of the pixel in the depth image, z is the depth value of the pixel. According to the pinhole imaging model, there is a relationship as follows:

[0113] P ir = H ir p ir (13)

[0114] Let P rgb be the spatial coordinates of the same point in the RGB camera coordinate system, P rgb be the projection coordinate of the point in the image plane, since the coordinates of the depth image and the coordinates of the RGB image may be different, it is necessary to use the rotation and translation transformation to connect them, that is

[0115] P rgb = RP ir + T (14)

[0116] Where R is the rotation matrix, T is the translation vector, and H rgb is used to project P rgb , calculate the corresponding coordinate of the point:

[0117] p rgb = H rgb P rgb (15)

[0118] In the construction of p ir , the original pixel coordinates (x, y) need to be multiplied by the depth value z, and after calculating p rgb , P rgb needs to be divided by its z component to get the actual pixel coordinates.

[0119] The extrinsic matrix is composed of the rotation matrix R ir (or R rgb ) and the translation vector T ir (or T rgb ), which represents the transformation of the point in the global coordinate system to the depth image or RGB image coordinate system:

[0120] P ir = R ir P + T ir (16)

[0121] P rgb = R rgb P + T rgb (17)In equation (16), P is expressed as P ir , R ir and T ir , and is brought into equation (17) to get:

[0122]

[0123] Contrast P rgb = RP ir + T, the formula can be obtained:

[0124]

[0125] wherein is the inverse of the rotation matrix R ir , after R, T is obtained, the coordinates P ir under the depth image can be converted to the coordinates P rgb under the RGB image through coordinate transformation, and after multiplying the intrinsic matrix H rgb with it, p rgb can be obtained. And p rgb is also a three-dimensional vector, whose x, y coordinates are pixel coordinates in the RGB image, so the color of the RGB pixel coordinates can be used as the color of the corresponding coordinates in the depth image. After registering the RGB image and the depth image, the semantic information and the position information of the image can be obtained at the same time.

[0126] On this basis, by calling the token of the large language model, the RGB image and the position information are analyzed online, and different objects and terrains such as grassland, sandy land, continuous steps, etc. are accurately identified, and by analyzing the information of the objects around the robot, it is inferred that the environment may be an office, a campus, a park or a road, etc. According to the identified terrain environment, the corresponding control strategy is set, and the robot will automatically switch to the motion state suitable for different terrains, for example, in the snow, the robot will adopt a more frequent gait to avoid slipping; on the grassland, the robot will reduce the step height to adapt to the uneven ground; and on the continuous steps, the robot will reduce the step frequency and increase the step height to realize smooth up and down step movement, etc. The text after the large model analysis is returned to the controller to execute the control instruction.

[0127] The position of the surrounding object is analyzed by using the large language model, and the motion path is generated by using the instructive path planning network according to the object position information, that is, the future motion of the robot is planned, the autonomous navigation strategy is adjusted in real time, and the perception ability of the robot to unknown environment is enhanced.

[0128] Finally, the synthesized text module (TTS, Text-To-Speech) of Baidu Qianan short text is called to synthesize the parsed text into speech, and after obtaining the wav speech file, it is output through the output device Newmann DP01 Bluetooth speaker, further improving the ability of human-computer interaction.

[0129] Further, the S104: error detection is performed on the robot, and if an error is found, the error is corrected, and the specific process of the error detection comprises:

[0130] During the travel of the robot according to the path generated by the instruction path planning network, if the actual travel path is found to deviate, a path yaw error is generated;

[0131] The large language model reanalyzes the destination coordinates in the rescue instruction, and triggers the trained path planning network to re-plan the path;

[0132] If the robot encounters an irregular structure terrain, that is, a step with a height higher than a set threshold, the control instruction of the large language model is to increase the step height to enable the robot to cross the step;

[0133] If the increased step height is not enough to cross the high step, it indicates that a motion instruction error is currently generated, then the large model reanalyzes the information of the RGB image and the depth image, judges whether the step height is greater than the physical limit of the robot joint, if the step height is increased to the physical limit of the robot joint, the robot is not enough to cross the high step, and the large model outputs an instruction that the current step cannot be crossed.

[0134] Further, the S105: after the robot reaches the rescue destination, the rescue task is performed, specifically including: placing rescue materials such as water, medicines, etc. on the robot for material transportation; communicating with the injured personnel to provide psychological construction and emotional support; collecting on-site data such as images, videos, etc. information conducive to rescue, to provide information support for rescue operations.

[0135] Further, as shown in Figure 6 The controller of the robot is connected with the switch through a first network port, the switch is connected with the leg-foot driver of the robot through a second network port, the controller is further connected with the laser radar through a third network port, the controller is further connected with the depth camera through a USB interface, and the controller is further connected with the sound box through Bluetooth.

[0136] Figure 4 The intelligent behavior scheduling flowchart based on the large language model is shown, which shows the relationship between the above robot environment perception and path planning and error detection. This method of using a large model for intelligent behavior scheduling enables the leg-foot robot to achieve efficient task planning under physical and environmental constraints, and still has adaptability and robustness when processing multi-task and multi-target optimization problems.

[0137] After an earthquake or other natural disaster, search and rescue missions are crucial but often challenging. A quadruped robot, by integrating the above key technologies, can be used to perform search and rescue tasks in such extreme environments. First, through the powerful natural human-machine interaction technology of the large language model, the rescue team can issue rescue instructions to the robot, such as "search a specific area or transport supplies while avoiding potential dangers", etc. The large language model's high-efficiency and accurate semantic parsing capabilities can convert complex search and rescue instructions into action sequences that the robot can execute, and complete rescue operations according to linear temporal logic. This natural human-machine interaction capability greatly improves the flexibility and response speed of rescue missions. Second, the robot is equipped with sensors such as laser radar and cameras, and uses laser radar odometry and mapping algorithms positioning strategy to create a three-dimensional map of the surrounding environment in real time and accurately locate its position. Combined with the end-to-end path planning method of instruction-based learning, the robot can automatically plan an optimal path to avoid obstacles and quickly reach areas where survivors may be present. At the same time, through the environment adaptation strategy based on the large language model, the quadruped robot can accurately identify and understand the terrain environment around the disaster site, and adjust its motion behavior (such as switching gaits, adjusting speed, etc.) according to the terrain environment information to adapt to different search and rescue scenarios. In addition, during the search process, the intelligent behavior scheduling method based on the large language model can analyze the radar, camera, and voice sensor data and ontology information collected by the robot in real time. These information is used by the large language model to detect planning errors or execution errors made by the robot during the search and rescue task in real time, and when errors occur, the feedback mechanism of the large language model is triggered to re-plan, allowing the robot to maintain efficient and stable motion in complex terrain environments. Through this integrated technical solution, the quadruped robot exhibits excellent adaptability and robustness in search and rescue tasks, providing a powerful tool for rescue teams and significantly improving the success rate of rescue in complex disaster environments.

[0138] The intelligent behavior scheduling strategy based on the large language model realizes semantic analysis and standardization of voice or text input by online calling the large language model, converts natural language instructions into semantic instructions understandable by the robot, and realizes natural human-robot interaction. Based on the semantic instructions generated by the large language model, the task sub-functions containing the action constraints of the robot are generated through sub-task planning, realizing high-level planning and scheduling of the robot behavior. The errors generated in the task planning process are corrected in time by using the high-level feedback mechanism, ensuring the accuracy and efficiency of task execution. At the same time, by importing the task sub-functions and the action constraints of the robot, the dynamic adjustment and optimization of the robot behavior are realized. In the task execution stage, the high-level planning is converted into specific motion sequences and trajectories through the motion state machine and motion planning. The motion controller evaluates and adjusts the motion execution in real time according to the robot attributes and environmental feedback, ensuring the smoothness and safety of the motion. When encountering motion errors, the system can trigger re-planning through the bottom-level feedback mechanism to adapt to environmental changes and task requirements. In addition, through text input and natural language processing technology, continuous monitoring and optimization of the robot behavior are realized. This method realizes the intelligent behavior scheduling and autonomous decision-making ability of the leg-foot robot in complex environments, providing strong technical support for the intelligence and automation of the robot. This not only improves the autonomy and adaptability of the robot, but also provides more possibilities for the application of the robot in diversified task scenarios.

[0139] The present application gives the robot natural language processing capability by fusing the large language model in the leg-foot robot system, which can accurately analyze the user's instructions and intentions, realize smooth dialogue with the user, quickly respond and execute the user's corresponding action instructions, and enhance the user's interactive experience.

[0140] The present application gives the robot comprehensive understanding and accurate perception ability of complex environment by deep information fusion and analysis of visual and language sensor data through the large language model, which can accurately identify various different terrains and objects such as sand, snow, grass and continuous steps, and infer that the environment may be an office, a campus, a park or a road according to the identification of objects and terrain, so as to switch different motion states to adapt to motion on different terrains.

[0141] The natural language instructions of the user such as "avoid obstacles and go to the designated position" are converted into semantic instructions executable by the robot through the large language model, corresponding action sequences are generated to guide the action execution of the robot, and the motion state and environmental changes of the robot are monitored in real time by the large language model. When encountering dynamic obstacles or environmental changes, the behavior strategy is adjusted quickly, the gait and speed are automatically adjusted, the robot maintains stable and efficient motion in complex terrain, the task execution efficiency of the robot is effectively improved, and the application scenarios of the leg-foot robot are expanded.

[0142] Embodiment two

[0143] The embodiment provides a multi-modal large model-based autonomous behavior control system of a leg-foot robot, and the system comprises the following modules.

[0144] An acquisition module configured to acquire a rescue instruction, perform instruction analysis on the rescue instruction, and obtain a rescue task and a rescue destination.

[0145] A perception module configured to perceive an environment in which the robot is currently located, first, perform terrain identification, then, adopt a mapping algorithm to locate a current position of the robot, and finally, perform navigation path planning according to the current position of the robot and the rescue destination.

[0146] An optimization module configured to make the robot travel according to the planned navigation path, and in the process of traveling of the robot, perform gait adjustment, speed adjustment and path optimization on the robot according to an environment adaptive strategy.

[0147] A detection module configured to detect errors of the robot, and correct the errors if the errors are found.

[0148] An output module configured to make the robot execute the rescue task after reaching the rescue destination.

[0149] The above merely describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. The autonomous behavior control method of legged robots based on a multimodal large model is characterized by: Controllers for robots, including: Obtaining a rescue instruction, performing instruction analysis on the rescue instruction, and obtaining a rescue mission and a rescue destination; Perceive the robot's current environment: First, identify the terrain; then, use a mapping algorithm to locate the robot's current position; finally, plan a navigation path based on the robot's current location and the rescue destination; The robot moves along the planned navigation path. During the movement, the robot adjusts its gait, speed, and path according to the environment adaptation strategy. Perform error detection on the robot and correct the error if any; After the robot arrives at the rescue destination, it performs the rescue mission; Perform error detection on the robot and correct the error if any is found. The specific process of error detection includes: When the robot is moving along the path generated by the command path planning network, if the actual path is found to be deviated, a path deviation error will occur; The large language model re-parses the destination coordinates in the rescue instructions, triggering the trained path planning network to replan the path; If the robot encounters an irregular structured terrain, where the irregular structured terrain refers to a step whose height is higher than a set threshold, the control instruction of the large language model is to increase the stride height so that the robot can cross the step; If the increased stride height is not enough to cross the high step, it means that a motion command error has occurred. The large model will re-parse the RGB image and depth image information to determine whether the step height is greater than the physical limit of the robot joint. If the stride height increases to the physical limit of the robot joint, the robot is not enough to cross the high step, and the large model will output an instruction that the current step cannot be crossed.

2. The method for controlling autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, wherein: The rescue command is analyzed to obtain the rescue mission and rescue destination. The specific analysis process includes: Convert rescue instructions from voice to text; Perform pre-processing operations on rescue instructions in text form; The text rescue instructions after preprocessing are parsed using linear temporal logic to obtain several subtasks; Each subtask is converted into subtask text through a large language model, and all subtask texts are converted into speech output.

3. The method for controlling the autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 2, wherein: The text rescue instructions after preprocessing are parsed using linear temporal logic to obtain several subtasks, including: （1） in, represents an atomic proposition, Indicates the subtasks, Indicates the number of subtasks, represents a production rule, Indicates logical NOT, and are logical AND and logical OR respectively, represents the atomic proposition itself, Represents the logical AND of two linear sequential logic formulas; rescue instructions are used express; Indicates that a condition is always true throughout the entire time range. Indicates that a condition will eventually be met. Expresses a relationship between two conditions where one condition remains true until the other condition is true.

4. The method for controlling autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, wherein: Perceive the robot's current environment: First, perform terrain recognition. The specific process of terrain recognition includes: transmitting the RGB image and depth image captured by the depth camera to the large language model for analysis. The large language model extracts feature information in the image to perform terrain recognition and infer the environment directly in front of the robot.

5. The method for controlling autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, wherein: Use mapping algorithms to locate the robot's current position, including: Receive and synchronize point cloud data, perform data preprocessing, and remove invalid Nan value points; Timestamp calibration, calculation of point smoothness, and identification of feature points, including plane points and edge points; The feature points of the current frame are matched with the feature points of the adjacent frames to estimate the robot's motion and obtain the robot's position at the current moment.

6. The method for controlling autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 5, wherein: The calculating the smoothness of the points and identifying the feature points include: （2） in, Indicates the Frame point cloud data points A set of consecutive points nearby, represents the number of midpoints, represents the laser radar coordinate system, Indicates the first Frame point cloud data The coordinates of the points, Represents the Euclidean norm; if the smoothness is less than the set threshold, it indicates that the current point is a plane feature; if the smoothness is greater than the set threshold, it indicates that the current point is an edge feature; Match the feature points of the current frame with the feature points of the adjacent frames to estimate the robot's motion and obtain the robot's current position, including: Select the feature points of the same type with the closest distance between the two frames as the feature points to be matched, and calculate the relative distance between the two , the calculation formula is defined as: （3） in, Expressed in the lidar coordinate system In Frame No. The predicted coordinates of the points, and Respectively represent Frame No. Point and The reference coordinates of the points, represents vector cross product, Represents the modulus of the vector; using nonlinear optimization methods to make the distance Minimum, you can get the robot at time and time The movement between By correlating the motion conditions at different moments, the robot's positioning at the current moment can be obtained.

7. The method for controlling autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, wherein: Navigation path planning is performed based on the robot's current location and rescue destination, specifically including: The coordinates of the rescue destination are given in the world coordinate system, and the destination coordinates are converted to the robot coordinate system through the conversion relationship between the world coordinate system and the robot coordinate system; The depth image captured by the camera is preprocessed and then fed into the trained instructional path planning network to obtain the planned path. The trained directive path planning network includes: a linear layer, a perception layer, a serial splicing layer, and a fully connected layer; the perception layer is implemented by Resnet-18; the input end of the linear layer is used to input the coordinates of the starting point and the coordinates of the target position point; the input end of the perception layer is used to input the depth image; the output end of the linear layer and the output end of the perception layer are both connected to the input end of the serial splicing layer; the output end of the serial splicing layer is connected to the input end of the fully connected layer; the output end of the fully connected layer outputs the planned path; The training process of the trained imperative path planning network includes: constructing a training set, wherein the training set is the starting point and the target point of the known planned path; inputting the training set into the imperative path planning network, training the imperative path planning network, and stopping the training when the loss function value of the imperative path planning network no longer decreases, thereby obtaining the trained imperative path planning network.

8. The method for controlling autonomous behavior of a legged robot based on a multimodal large model as claimed in claim 1, wherein: The robot moves along the planned navigation path. During the movement, the robot adjusts its gait, speed, and path according to the environment adaptation strategy. Specifically, the following steps are performed: After registering the RGB image with the depth image, the position information of the object in the image is obtained at the same time; The RGB image and position information are input into the large language model to obtain information about objects in the robot's surrounding environment. The robot then identifies objects or terrain in the environment and adjusts its gait and speed according to the environmental adaptation strategy. For example, if the environment is snowy, the robot's gait frequency is increased; if there are continuous steps, the robot's gait frequency, speed, and stride height are reduced. A large language model is used to analyze the position of objects in the image, and the trained instruction-based path planning network is used to generate an optimized motion path based on the object position information.

9. The autonomous behavior control system of a legged robot based on a multimodal large model is characterized by: include: An acquisition module is configured to: acquire a rescue instruction, perform instruction analysis on the rescue instruction, and obtain a rescue mission and a rescue destination; The perception module is configured to: perceive the robot's current environment by first identifying the terrain; then using a mapping algorithm to locate the robot's current position; and finally, planning a navigation path based on the robot's current position and the rescue destination; an optimization module configured to: cause the robot to move along the planned navigation path, and during the movement of the robot, adjust the robot's gait, adjust its speed, and optimize its path according to an environmental adaptation strategy; a detection module configured to: perform error detection on the robot and correct the error if an error is found; The specific process of the error detection includes: When the robot is moving along the path generated by the command path planning network, if the actual path is found to be deviated, a path deviation error will occur; The large language model re-parses the destination coordinates in the rescue instructions, triggering the trained path planning network to replan the path; If the robot encounters an irregular structured terrain, where the irregular structured terrain refers to a step whose height is higher than a set threshold, the control instruction of the large language model is to increase the stride height so that the robot can cross the step; If the increased stride height is not enough to cross the high step, it means that a motion command error has occurred. The large model will re-parse the RGB image and depth image information to determine whether the step height is greater than the physical limit of the robot joint. If the stride height increases to the physical limit of the robot joint, the robot is not able to cross the high step, and the large model will output a command that the current step cannot be crossed. The output module is configured to: after the robot arrives at the rescue destination, perform the rescue mission.

Images