Method and system for controlling intelligent robot based on large cloud model, robot, readable medium and program product

Abstract

The invention discloses a control method and system for controlling an intelligent robot based on a cloud large model, the intelligent robot is controlled mainly through the cloud large model and dynamic cue word generation, and the task planning efficiency is greatly improved through the intellectualization and the dynamics of the cue word generation; the cloud large model can quickly convert natural languages into intelligent robot actions, the execution precision and the execution efficiency are remarkably improved, the task adaptability in complex scenes is improved, the cloud large model is combined with a multi-mode sensor of the intelligent robot to observe and collect environment information, and the intelligent robot is more intelligent. Multi-modal data support and real-time closed-loop optimization enable the intelligent robot to adapt to dynamic scenes and complex environment tasks; the cloud large model has a more friendly tendency in use cost of enterprises or users and is not affected by hard requirements of hardware such as a traditional server or a third-party processor, and the intelligent robot is remotely and wirelessly connected with the cloud large model and is wider in use scene coverage.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robot control technology, and in particular to a method for controlling an intelligent robot based on a cloud-based large model. Background Art

[0002] With the rapid development of science and technology, the technologies related to artificial intelligence are flourishing. Among them, intelligent robots, as an important fulcrum of artificial intelligence output, have shown a diversified trend in many fields and have huge potential and value. On the one hand, intelligent robots can help humans get rid of some heavy, dangerous or repetitive work, such as construction agriculture, fire protection chemical industry, manufacturing assembly lines, etc. On the other hand, they can replace humans to perform some high-precision and high-sustainability work, such as precision manufacturing, medical surgery, etc. It can be said that the value of intelligent robots is huge. Nowadays, intelligent robots are almost all based on large models. Large models are machine learning models with ultra-large parameter scales, which can help intelligent robots learn quickly, but the development of intelligent robots has also been affected by some factors, resulting in certain limitations.

[0003] The large models used by intelligent robots in the prior art include two types. One is that the large model is directly deployed on the intelligent robot, is independent of the network, and has good real-time performance, but has high hardware cost and is limited by the hardware resources of the intelligent robot, making it difficult to support ultra-large-scale models or tasks with high computing requirements. The second is that the large model is set on a third-party processor, such as a computer host. It mainly has the characteristics of being economical and affordable, and is suitable for intelligent robots with low power consumption and computing power. Its cost is also more economical than that directly deployed on the intelligent robot. However, both types of large models need to be used locally and within a certain range, and are limited by different hardware, resulting in inconsistent experience effects and relatively expensive costs. It has certain limitations, which is a major drawback for some companies with limited economies and demand for computing power. In this regard, the present invention provides a method for controlling an intelligent robot by combining a cloud server with a large model to solve the pain points and difficulties in the prior art. Summary of the invention

[0004] In view of the shortcomings of the above-mentioned technologies that traditional intelligent robots are limited by the inability to be implemented efficiently, the present invention provides a method for controlling intelligent robots based on a cloud-based large model, aiming to solve the problem of efficient conversion of natural language instructions into physical actions. Through the powerful semantic parsing capability of the cloud-based large model and the local real-time feedback mechanism of the intelligent robot, the intelligent robot can perform efficiently in complex and diverse task scenarios.

[0005] To achieve the above object, the present invention provides a method for controlling an intelligent robot based on a cloud-based large model, comprising the following steps: Get task instructions; The task instructions and the environmental information collected by the intelligent robot are uploaded to the cloud-based big model. The cloud-based big model combines the task instructions and environmental information to analyze and generate feedback. The intelligent robot uploads the received feedback to the cloud-based big model. The cloud-based big model dynamically adjusts the task plan and converts it into control instructions that can be executed by the intelligent robot. The main control module of the intelligent robot receives the control instruction, and controls the corresponding actuator to perform the corresponding action based on the control instruction.

[0006] As an improved solution of the present invention, the steps of obtaining the task instruction are as follows: Users wake up the intelligent robot through voice; Input voice task instructions to the intelligent robot, and record and save the voice task instructions; The cloud-based large model converts voice task instructions into text task instructions As an improved solution of the present invention, the steps of generating dynamic prompt words are as follows: When environmental information changes, the intelligent robot collects environmental information through sensors and uploads it to the cloud-based large model; The cloud-based large model generates new dynamic prompts based on environmental information and task instructions so that task planning can implement task instructions.

[0007] As an improved solution of the present invention, the steps of converting the task planning into the control instructions are as follows: Feedback the dynamic prompt words to the cloud-based big model, and the cloud-based big model optimizes and adjusts the task planning; Generate control instructions that meet the task plan through the cloud-based large model.

[0008] As an improved solution of the present invention, the steps of the actuator executing the control instruction are as follows: When the main control module receives the control instruction, the control instruction is mapped to the corresponding hardware control instruction in the task mapping library; The intelligent robot controls the corresponding actuator according to the hardware control instructions of the task mapping library to output task instructions.

[0009] As an improved solution of the present invention, the sensor for collecting environmental information includes a physical sensor, a chemical sensor, a biological sensor, and an electromagnetic sensor to perceive the task scene and generate multimodal prompt words.

[0010] The present invention also provides a system for controlling an intelligent robot based on a cloud-based large model, which is used to execute the method for controlling an intelligent robot based on a cloud-based large model, comprising: The natural language input and conversion module is used to obtain and save the user's voice information to wake up the intelligent robot or input task instructions to the intelligent robot, and convert the voice information into text information; The main control module is used to receive control instructions and convert the control instructions into executable programs for the intelligent robot. The control instructions are converted into hardware control instructions through the task mapping library and sent to the execution module; The acquisition module is used to collect information about the task environment and send it to the prompt word generation module; The prompt word generation module generates dynamic prompt words based on task instructions and environmental information, combined with the cloud-based large model; The cloud feedback module is used to identify and analyze the semantics of voice information and task instructions and generate task plans, and convert the task plans into control instructions that can be executed by the robot; The execution module is used to receive and execute hardware control instructions and convert the hardware control instructions into physical actions.

[0011] The present invention also provides an intelligent robot, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more intelligent robots, the one or more intelligent robots implement the method of controlling the intelligent robot based on a cloud-based large model.

[0012] The present invention also provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for controlling an intelligent robot based on a cloud-based large model as described above is implemented.

[0013] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method of controlling an intelligent robot based on a cloud-based large model.

[0014] The beneficial effects of the present invention are as follows: compared with the prior art, the method for controlling an intelligent robot based on a cloud-based large model provided by the present invention mainly realizes the control of the intelligent robot through the cloud-based large model and dynamic prompt word generation, and the intelligent and dynamic prompt word generation greatly improves the task planning efficiency; the cloud-based large model can realize the rapid conversion of natural language into intelligent robot actions, significantly improves the execution accuracy and execution efficiency, and improves the adaptability to tasks in complex scenarios. The cloud-based large model combines the intelligent robot's own multimodal sensors to observe and collect environmental information. Multimodal data support and real-time closed-loop optimization enable the intelligent robot to adapt to dynamic scenarios and complex environmental tasks; the cloud-based large model has a more friendly tendency in terms of the use cost of enterprises or users, and is not subject to the rigid requirements of hardware such as traditional servers or third-party processors. The intelligent robot remotely wirelessly connects to the cloud-based large model, covering a wider range of usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the method flow of the present invention; Figure 2 This is a structural block diagram of a system for controlling an intelligent robot based on a cloud-based large model according to the present invention; Figure 3 Schematic diagram of the execution flow of the intelligent robot of the present invention. DETAILED DESCRIPTION

[0016] In order to more clearly illustrate the present invention, the present invention is further described below in conjunction with the accompanying drawings.

[0017] In the following description, the example details are given to provide a deeper understanding of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. It should be understood that the specific embodiments are only used to explain the present invention and are not used to limit the present invention.

[0018] It should be understood that when the terms "include" and / or "comprises" are used in this specification, they indicate the existence of the stated features, integers, steps, operations, elements or components, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components or combinations thereof.

[0019] The large models used by intelligent robots in the prior art include two types. One is that the large model is directly deployed on the intelligent robot, is independent of the network, and has good real-time performance, but has high hardware cost. Due to the limitation of the hardware resources of the intelligent robot, it is difficult to support ultra-large-scale models or tasks with high computing requirements. The second is that the large model is set on a third-party processor, such as a computer host. It mainly has the characteristics of economy and is suitable for intelligent robots with low power consumption and computing power. Its cost is more economical than that directly deployed on the intelligent robot. However, both types of large models need to be used locally and within a certain range, and are limited by different hardware, resulting in inconsistent experience effects. The cost is also relatively expensive and has certain limitations. It is a major drawback for some companies with limited economy and demand for computing power. In this regard, the present invention provides a method for controlling an intelligent robot by combining a cloud server with a large model to solve the pain points and difficulties in the prior art.

[0020] See also Figure 1-Figure 3 , a method for controlling an intelligent robot based on a cloud-based large model of the present invention comprises the following steps: Get task instructions; The task instructions and the environmental information collected by the intelligent robot are uploaded to the cloud-based big model, which analyzes the task instructions and environmental information and generates dynamic prompt words; The intelligent robot uploads the received dynamic prompt words to the cloud-based big model, which plans tasks based on the prompt words and converts them into control instructions that can be executed by the intelligent robot; The main control module of the intelligent robot receives the control instruction, and controls the corresponding actuator to perform the corresponding action based on the control instruction.

[0021] The cloud-based large model referred to in the present invention refers to a large-scale machine learning model running on a cloud computing platform, especially a deep learning model, such as OpenAI's GPT-4, Google's Gemini, and Baidu's Qianfan and other large models; since the cloud-based large model is usually provided with computing power and storage by multiple cloud servers, it contains billions or even trillions of parameters, and the model scale is huge, which can provide ultra-fast computing speed for intelligent robots, making the control of intelligent robots smoother and more efficient, and based on the ultra-large cardinality of parameters, it can also achieve more flexible dynamic environment adaptability; it can be understood that the more sensors or actuators the intelligent robot carries itself, the greater the effect of the combination of the intelligent robot and the cloud-based large model.

[0022] In this embodiment, the steps of obtaining task instructions are as follows: Users wake up the intelligent robot through voice; Input voice task instructions to the intelligent robot, and record and save the voice task instructions; The cloud-based large model converts voice task instructions into text task instructions.

[0023] It is understandable that before inputting task instructions to the intelligent robot, it needs to be woken up. A specific wake-up word can be set for the intelligent robot. The voice receiver of the intelligent robot is in working state and continuously captures the wake-up word. After recognizing and capturing the wake-up word, it sends feedback to the user. The user can issue task instructions to the intelligent robot, and the intelligent robot receives the task instructions, records and saves the voice task instructions. The cloud-based large model converts the voice task instructions into text task instructions that are easy for the intelligent robot to understand.

[0024] The steps to generate dynamic prompt words are as follows: When environmental information changes, the intelligent robot collects environmental information through sensors and uploads it to the cloud-based large model; The cloud-based large model generates new dynamic prompts based on environmental information and task instructions so that task planning can implement task instructions.

[0025] Among them, the dynamic generation of prompt words mainly changes according to the changes of task instructions and environment. For example, "Put the blue sponge on the desktop onto the red sponge". First, the intelligent robot will obtain the current picture through the camera and feed it back to the cloud-based large model for processing and recognition, and then generate prompt words for the relative positions of objects, such as "The blue sponge block is on the left side of the red sponge", etc. The prompt words are optimized in combination with the cloud-based large model, and the final execution results are generated according to prompt words such as "blue sponge", "red sponge", "placement" or orientation, such as "Put the blue sponge on the desktop onto the red sponge". The cloud-based large model converts the process and results into underlying physical actions, generates control instructions that match the hardware capabilities of the intelligent robot, and can convert the control instructions into executable programs for the intelligent robot. The intelligent robot then controls each actuator based on the control instructions to complete the corresponding actions.

[0026] The working principle of the present invention is: The natural language input module is mainly used to record the voice commands issued by the user. After the user wakes up the intelligent robot, the intelligent robot begins to capture and recognize the sounds in the environment. When the voice command issued by the user is recognized, it is recorded and saved, and the voice command is sent to the main control module. The main control module then uploads it to the cloud large model. The cloud large model converts the voice command into a text command that the intelligent robot can understand and returns it to the main control module. Since the cloud large model contains a large number of parameter resources and has the characteristics of high performance and high computing power, it can parse and identify the semantics of the text command and generate dynamic feedback. When the intelligent robot receives the feedback, the main control module uploads the collected environmental information to the cloud large model for parsing and generating dynamic prompt words. The cloud large model generates task planning and converts it into control instructions based on the semantics of the text command and the dynamic prompt words; a task mapping library is set in the main control module of the intelligent robot. Each hardware control instruction in the task mapping library corresponds to controlling an actuator. When the main control module receives the control instruction, it converts the control instruction into the physical action of the intelligent robot based on the task mapping library, thereby realizing the task instruction.

[0027] Among them, the intelligent robot provided by the present invention has multimodal fusion capabilities and can process data types such as images, videos, and sounds. The collected environmental information includes physical sensors, chemical sensors, biological sensors, electromagnetic sensors, and other perception task scenarios. For example, sensors used to detect environmental information include but are not limited to visual, auditory, photoelectric, temperature and other sensors. Based on these sensors, the intelligent robot can generate multimodal prompt words and process complex task instructions and task environments; and the execution module of the intelligent robot mainly includes but is not limited to servo motors, steering gears, robotic arms, cameras, sensors and other components that can produce execution results.

[0028] In this embodiment, the steps of converting the task plan into control instructions are as follows: Feedback the dynamic prompt words to the cloud-based big model, and the cloud-based big model optimizes and adjusts the task planning; Generate control instructions that meet the task plan through the cloud-based large model.

[0029] Since the intelligent robot cannot read voice task instructions directly, it is necessary to convert the voice task instructions into text task instructions that the intelligent robot can easily understand and read, so that it can more easily and accurately understand the semantics and context in the task instructions, and thus respond to customer needs more accurately; therefore, when the text task instructions are uploaded to the cloud big model, the cloud big model parses the text semantics and generates a task plan. For example: send a command to the intelligent robot "take a few steps forward, and then a few steps to the right." The intelligent robot will understand the meaning of this sentence based on the big data in the cloud big model, and generate a task plan to implement this sentence. The first step is to parse the user instructions according to the preset prompt words; the second step is to summarize the parsed text to generate control instructions that are easy for the robot to execute; the third step is that the main control module of the humanoid robot receives the control instructions, and controls the actuator to complete the task instructions according to the control instructions.

[0030] Among them, when the intelligent robot performs the task planning stage, it will continuously observe environmental information, such as visual information, auditory information, etc., through its own sensors, and transmit it to the cloud-based large model to generate dynamic prompt words. For example, before issuing a transport instruction to the intelligent robot, the spatial orientation of the transport target is changed or moved. The intelligent robot perceives the change in environmental information through the visual sensor, and the spatial coordinates of the transport target change. Correspondingly, the intelligent robot will regenerate the task planning that will implement the transport instruction, and generate new control instructions corresponding to the task planning. The newly added actions and routes are also equivalent to the generation of dynamic prompt words. Combined with the cloud-based large model, the intelligent robot can flexibly and quickly generate dynamic prompt words based on environmental information. It has strong intelligence and dynamism, which greatly improves the efficiency of task planning.

[0031] The steps for the actuator to execute the control instruction are as follows: When the main control module receives a control instruction, it maps the control instruction to the corresponding hardware control instruction in the task mapping library; the task mapping library contains the driving instructions of all the actuators of the intelligent robot, that is, the hardware control instructions. When the control instruction is generated, the main control module converts the control instruction into the corresponding hardware control instruction based on the task mapping library.

[0032] The intelligent robot controls the corresponding actuators according to the hardware control instructions of the task mapping library to output task instructions. The intelligent robot controls the output of actuators such as robotic arms, servos, sensors, etc. according to the converted hardware control instructions.

[0033] See also Figure 2 The present invention also provides a system for controlling an intelligent robot based on a cloud-based large model, which is used to execute the method for controlling an intelligent robot based on a cloud-based large model, including: The natural language input module is used to obtain and save the user's voice information to wake up the intelligent robot or input task instructions to the intelligent robot.

[0034] The main control module is used to receive control instructions and convert them into executable programs for the intelligent robot. The control instructions are converted into hardware control instructions through the task mapping library and sent to the execution module. The main control module will also upload voice information, dynamic prompts, and environmental information to the cloud-based large model. The feedback generated by the cloud-based large model is sent to each module via the main control module, serving as a bridge for interaction between other modules and the cloud-based large model.

[0035] The acquisition module is used to collect information about the task environment and send it to the prompt word generation module.

[0036] The prompt word generation module generates dynamic prompt words based on task instructions and environmental information, combined with the cloud-based large model. The prompt word generation module receives the environmental information collected by the acquisition module and uploads it to the visual large model for analysis. The visual large model generates dynamic prompt words based on the environmental information and returns them to the prompt word generation module. The prompt word generation module then sends the dynamic prompt words to the main control module, and the main control module then uploads the dynamic prompt words to the language large model, which then plans the generation task.

[0037] The cloud feedback module is used to convert voice task instructions into text task instructions, identify and parse the semantics and task instructions of the text task instructions and generate a task plan, and convert the task plan into corresponding control instructions executable by the robot. The cloud big model mainly includes but is not limited to the voice big model, the language big model and the vision big model. The voice big model is used to convert the input voice task instructions into text task instructions. After the conversion is completed, it returns to the main control module. The main control module then uploads the text task instructions and dynamic prompt words to the language big model. The language big model generates a task plan based on the text task instructions and the dynamic prompt words, and converts the task plan into corresponding control instructions. After the visual big model generates dynamic prompt words based on the environmental information collected by the acquisition module, the language big model will also regenerate the corresponding task plan and control instructions based on the dynamic prompt words.

[0038] The execution module is used to receive and execute hardware control instructions, and convert the control instructions into physical actions based on the hardware control instructions in the task mapping library to realize the task instructions.

[0039] Regarding task input and prompt word generation, the following is an example of the control process of the present invention: The user describes the task to the intelligent robot through natural language, such as "move the red sponge on the table to the blue sponge". The prompt word generation module uploads the preset prompt words, such as "obtain the positional relationship between the sponge blocks on the table in three-dimensional space" and environmental data (such as data collected by cameras and voice sensors) to the cloud. After receiving the prompt words, the cloud-based large model uses its powerful semantic understanding and image understanding and capabilities to generate refined dynamic prompt words such as: "The red sponge is to the left of the blue sponge." The cloud-based large model combines the dynamic prompt words generated in the previous step with the existing prompt words to perform natural language parsing and task planning, and returns the following content: {'action':["pick_up(red)","place_on('blue','top')"],'response':'Start executing the task, put the red sponge on the blue sponge'} The intelligent robot performs actions based on the analysis results, moving the red sponge to the blue sponge to complete the set task objectives.

[0040] The present invention also provides an intelligent robot, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more intelligent robots, the one or more intelligent robots implement a method for controlling the intelligent robot based on a cloud-based large model.

[0041] The present invention also provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, a method for controlling an intelligent robot based on a cloud-based large model is implemented.

[0042] The present invention also provides a computer program product, including a computer program, which implements a method for controlling an intelligent robot based on a cloud-based large model when the computer program is executed by a processor.

[0043] The advantages of the present invention are: 1) Intelligent and dynamic prompt word generation greatly improves task planning efficiency.

[0044] 2) Realize the rapid conversion from natural language feedback to intelligent robot actions, significantly improving execution accuracy.

[0045] 3) The control instructions for the robot are generated based on the cloud-based large model, and the adaptability to complex scene tasks is improved. Multimodal data support and real-time closed-loop optimization enable the intelligent robot to adapt to dynamic scenes and complex environment tasks.

Claims

1. A method for controlling an intelligent robot based on a cloud-based large model, characterized in that: The following steps are involved: Get task instructions; The task instructions and the environmental information collected by the intelligent robot are uploaded to the cloud-based big model, which analyzes the task instructions and environmental information and generates dynamic prompt words; The intelligent robot uploads the received dynamic prompt words to the cloud-based big model, which plans tasks based on the prompt words and converts them into control instructions that can be executed by the intelligent robot; The main control module of the intelligent robot receives the control instruction, and controls the corresponding actuator to perform the corresponding action based on the control instruction.

2. The method for controlling an intelligent robot based on a cloud-based large model according to claim 1, characterized in that: in, The steps to obtain task instructions are as follows: Users wake up the intelligent robot through voice or text; Input voice task instructions or text task instructions to the intelligent robot, and record and save the voice task instructions; The cloud-based large model converts voice task instructions into text task instructions.

3. The method for controlling an intelligent robot based on a cloud-based large model according to claim 1, characterized in that: in, The steps to generate dynamic prompt words are as follows: When environmental information changes, the intelligent robot collects environmental information through sensors and uploads it to the cloud-based large model; The cloud-based large model generates new dynamic prompts based on environmental information and task instructions so that task planning can implement task instructions.

4. The method for controlling an intelligent robot based on a cloud-based large model according to claim 3, characterized in that: in, The steps to convert the task plan into control instructions are as follows: Feedback the dynamic prompt words to the cloud-based big model, and the cloud-based big model optimizes and adjusts the task planning; Generate control instructions that meet the task plan through the cloud-based large model.

5. The method for controlling an intelligent robot based on a cloud-based large model according to claim 1, characterized in that: The steps for the actuator to execute the control command are as follows: When the main control module receives the control instruction, the control instruction is mapped to the corresponding hardware control instruction in the task mapping library; The intelligent robot controls the corresponding actuator according to the hardware control instructions of the task mapping library to output task instructions.

6. The method for controlling an intelligent robot based on a cloud-based large model according to claim 3, characterized in that: Sensors used to collect environmental information include physical sensors, chemical sensors, and biological sensors. Physical sensors, chemical sensors, and biological sensors are used to perceive task scenarios and generate multimodal prompt words.

7. A system for controlling an intelligent robot based on a cloud-based large model, used to execute a method for controlling an intelligent robot based on a cloud-based large model according to claims 1-6, characterized in that: include: A natural language input module is used to obtain and save the user's voice information to wake up the intelligent robot or input task instructions to the intelligent robot; The main control module is used to receive control instructions and convert the control instructions into executable programs for the intelligent robot. The control instructions are converted into hardware control instructions through the task mapping library and sent to the execution module; The acquisition module is used to collect information about the task environment and send it to the prompt word generation module; The prompt word generation module generates dynamic prompt words based on task instructions and environmental information, combined with the cloud-based large model; The cloud feedback module is used to convert voice task instructions into text task instructions, identify and parse the semantics of the text task instructions and the task instructions, generate task plans, and convert the task plans into control instructions that can be executed by the robot; The execution module is used to receive and execute hardware control instructions and convert the hardware control instructions into physical actions.

8. An intelligent robot, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more intelligent robots, the one or more intelligent robots implement the method as described in any one of claims 1-6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images