A method for realizing mobile robot human-computer interaction inspection by using a large language model

Abstract

The application discloses a method for realizing man-machine interaction inspection of a mobile robot by using a large language model, which comprises the following steps: establishing a task scheduling manager; obtaining a natural language instruction of a user through a voice module or a visual interactive interface of the mobile robot; using the large language model to infer an inspection task execution process through the task scheduling manager and the natural language instruction; feeding back the inspection task execution process to the user by using the task scheduling manager; if the user has an opinion, returning to the previous step to iteratively optimize the inspection task execution process, and if the user confirms execution, proceeding to the next step; controlling the mobile robot to execute the inspection task by using the task scheduling manager, iteratively inferring by using the large language model during the inspection process; and feeding back the inspection result to the user after the inspection task is completed. The application improves the interaction efficiency and quality between the mobile robot and the user, and improves the adaptability and intelligence of the mobile robot inspection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mobile robots, and particularly relates to a method for realizing man-machine interactive inspection of a mobile robot by using a large language model. BACKGROUND

[0002] A mobile robot is a robot capable of moving autonomously or semi-autonomously in an environment, which can be used to perform various tasks such as navigation, detection, carrying, inspection, etc. The mobile robot has good application prospects in inspection, and can replace or assist artificial completion of the inspection task in a complex environment. The mobile robot has the advantages of not needing to pre-embed navigation markers, being capable of realizing flexible switching of multiple points and multiple lines, and being capable of realizing remote monitoring and control.

[0003] The inspection demand is usually not fixed and variable, and different inspection requirements are usually made according to the situation on site. A staff member needs to interact with the mobile robot on site and issue different instructions to meet various inspection demands. However, the intelligent level of the mobile robot needs to be improved, especially in terms of man-machine interaction. At present, the interaction between the mobile robot and a person mainly depends on a touch screen, a key, a remote controller and the like. These modes not only have complicated operation, but also have low interaction efficiency, and cannot well understand the intention of the staff member and autonomously plan and execute a task. Therefore, how to realize natural language interaction between the mobile robot and a person is an important research topic in the field of mobile robots. SUMMARY

[0004] In order to solve the above problems, the application provides a method for realizing man-machine interactive inspection of a mobile robot by using a large language model, which improves the interaction efficiency and quality between the mobile robot and a user, and improves the adaptability and intelligence of the mobile robot in inspection.

[0005] To achieve the above purpose, the technical scheme adopted by the application is as follows: a method for realizing man-machine interactive inspection of a mobile robot by using a large language model, comprising the following steps:

[0006] Step 1: establishing a task scheduling manager;

[0007] Step 2: obtaining a natural language instruction of a user through a voice module or a visual interactive interface of the mobile robot;

[0008] Step 3: using a large language model to infer an inspection task execution process through the task scheduling manager and the natural language instruction;

[0009] Step 4: feeding back the inspection task execution process to the user by using the task scheduling manager. If the user has opinions, the inspection task execution process is iteratively optimized in step 3, and if the user confirms execution, step 5 is entered;

[0010] Step 5: Control the mobile robot to perform the inspection task using the task scheduling manager, and use the large language model to iteratively infer during the inspection process;

[0011] Step 6: After the completion of the inspection task, feedback the inspection results to the user.

[0012] Further, the task scheduling manager includes scheduling principles, an inspection function module library, a question and answer module, and a large language model input module. In step 1, the task scheduling manager is established, including the steps of:

[0013] Step 1.1: Establish scheduling principles: provide a standardized format and sequential reasoning for the large language model to infer task steps, and preferentially call the inspection function module library;

[0014] Step 1.2: Establish the inspection function module library: encapsulate basic modules for different inspection sub-tasks, each basic module provides clear name prompts, usage prompts, and input and output prompts, facilitating the large language model to correctly understand and call;

[0015] Step 1.3: Establish the question and answer module: the user interacts with the question and answer module through voice instructions or interactive interfaces, and the question and answer module stores the user's current natural language instructions and historical natural language instructions for subsequent large language model reasoning;

[0016] Step 1.4: Establish the large language model input module: aggregate the history of hidden questions and answers and the history of reasoning results of the large language model, and convert the output information of each basic module into a natural language form understood by the large language model for subsequent reasoning.

[0017] Further, in step 1.1, the scheduling principles are established, including:

[0018] (1) The task scheduling manager can access the inspection function module library, allowing the large language model to infer the inspection task flow or execute the instruction code according to the prompts of the basic modules;

[0019] (2) The task scheduling manager generates a unique serial number for each inspection task and adds a suffix prompt after the serial number indicating which basic modules the task is composed of, to avoid the large language model reasoning errors caused by confusing task names;

[0020] (3) The task scheduling manager uses chain thinking to assist the large language model in reasoning or scheduling basic modules, that is, after generating an inspection task, it asks the large language model a question to inquire whether more basic modules are needed to improve or optimize the inspection task.

[0021] Further, in the step 1.2, the inspection function module library assists the task scheduling manager in scheduling the base model through the base model prompt, so that the large language model accurately understands the base model, and the base model prompt specifically includes: (1) a name prompt: provides a summary of the overall function of each base module, helping the large language model understand the purpose of each base module concisely, and serving as an invocation entry of the base module; (2) a use prompt: describes the use scenario of each base module, helping the large language model decide which base module to use for a specific task; (3) an input / output prompt: describes the input and output formats required by each base module, helping the mobile robot correctly execute the base module to provide explicit definitions.

[0022] According to the types and prompts of the base model, the mobile robot perception host module is defined to include: a simultaneous localization and mapping module, a map management module, a path planning and motion control module, a voice interaction module, a personnel position recognition module based on deep learning, a personnel wearing standard recognition module based on deep learning, and a fire and smoke detection module based on deep learning.

[0023] Further, in the step 2, the user's natural language instruction is obtained through the microphone array voice module, including the steps of:

[0024] Step 2.1: The microphone array voice module collects user voice input; the natural language instruction of the user voice input is collected through the microphone array voice module and converted into a digital signal;

[0025] Step 2.2: The digital signal will be sent to the speech processing module for preprocessing, which includes noise removal and speech signal enhancement;

[0026] Step 2.3: Speech recognition; the preprocessed signal is sent to the speech recognition module, which uses acoustic model, language model and pronunciation dictionary technology to convert the natural language instruction voice into text.

[0027] Further, in the step 3: using the large language model to infer the inspection task execution process, including the steps of:

[0028] Step 3.1: Query the user's historical natural language instructions of the question and answer module and the historical hidden question and answer and historical reasoning results of the output module;

[0029] Step 3.2: Use the task scheduling manager to input the contents queried in step 3.1, the user's natural language instruction, the scheduling principle and the base module prompt into the large language model for iterative reasoning to obtain the inspection task execution process;

[0030] Step 3.3: Store the hidden question and answer of the large language model iterative reasoning process and the user's current natural language instruction.

[0031] Further, in the step 4: the task scheduling manager is used to feed back the inspection task execution process to the user, including the steps of:

[0032] Step 4.1: the inspection task execution process is presented to the user through the voice module or visual interactive interface;

[0033] Step 4.2: the user's natural language instructions are obtained again through the voice module or visual interactive interface of the mobile robot; if the user issues a command to execute the task, step 5 is entered; if the user proposes a modification, step 3 is returned to reason the inspection task execution process again; if the user does not respond or issues a command to cancel the task, the task is canceled.

[0034] Further, in the step 5: the task scheduling manager is used to control the mobile robot to execute the inspection task, and the large language model is used to iteratively reason during the inspection process, including the steps of:

[0035] Step 5.1: according to the inspection task execution process and the inspection function module library, the large language model is used to generate task execution code;

[0036] Step 5.2: the task execution code is issued to the robot through the task scheduling manager to control the basic module to execute the inspection task;

[0037] Step 5.3: the execution information of the basic module is received, and the execution information is processed using the large language model input module;

[0038] Step 5.4: the processed execution information of the basic module, the inspection task execution process, the scheduling principle, and the inspection function module library are input into the large language model to reason whether the inspection task is completed using the task scheduling manager, and the reasoning result and the basic module execution information are stored;

[0039] Step 5.5: according to the reasoning result, the corresponding output is executed; if the inspection task is completed, step 6 is entered; if the inspection task is not completed, the execution continues; if an abnormal situation occurs, a new inspection task execution code is reasoned, and step 5.2 is entered.

[0040] Further, in the step 6: after the completion of the inspection task, the inspection result is fed back to the user, including the steps of:

[0041] Step 6.1: the historical execution information of the basic module and the historical reasoning result of the large language model during the inspection process are input into the large language model through the task scheduling manager to reason the inspection result report content;

[0042] Step 6.2: the task scheduling manager presents the report content to the user through the voice module or visual interactive interface.

[0043] The beneficial effects of adopting the technical solution are:

[0044] The present application realizes efficient, intelligent and flexible human-computer interaction between the large language model and the user by constructing the task scheduling manager and integrating multiple inspection function modules, and realizes more diversified and personalized inspection content and methods by using the powerful generation ability and creativity of the large language model. The present application can improve the inspection efficiency and quality of mobile robots in the locomotive repair shop scene, enhance the adaptability and intelligence of mobile robots, and improve user experience and satisfaction.

[0045] The present application realizes efficient, intelligent and flexible human-computer interaction between the large language model and the user by constructing the task scheduling manager, and improves the inspection efficiency and quality of mobile robots in the industrial scene.

[0046] The present application expands the ability of the large language model to handle different types of inspection tasks by integrating multiple inspection function modules, and enhances the adaptability and intelligence of mobile robots in the industrial scene.

[0047] The present application simplifies the interaction between the user and the mobile robot by using the voice interface and the model output module, and improves the user experience and satisfaction. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A method flowchart for realizing mobile robot human-computer interaction inspection using a large language model according to the present application;

[0049] Figure 2 A connection relationship diagram between modules in an embodiment of the present application;

[0050] Figure 3 A connection relationship diagram of a mobile robot in an embodiment of the present application;

[0051] Figure 4 A complete execution flowchart of an embodiment of the present application;

[0052] Figure 5 A flowchart of the large language model reasoning inspection task execution process in an embodiment of the present application;

[0053] Figure 6 A flowchart of the large language model reasoning in the task execution process in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described below with reference to the drawings.

[0055] The hardware used by the present application mainly includes a mobile quadruped robot, an inference server, a microphone array, a laser radar, a depth camera and a wireless AP, wherein the GPU of the large language model deployment server is an NVIDIA V100 32G, and a hardware connection diagram is as shown in Figure 3 .

[0056] In the embodiment, as shown in Figure 1 , the present application provides a method for realizing mobile robot human-computer interaction inspection by using a large language model, which comprises the following steps:

[0057] Step 1: establishing a task scheduling manager;

[0058] Step 2: obtaining a natural language instruction of a user through a voice module or a visual interactive interface of the mobile robot;

[0059] Step 3: using a large language model to infer an inspection task execution process through the task scheduling manager and the natural language instruction;

[0060] Step 4: feeding back the inspection task execution process to the user by using the task scheduling manager. If the user proposes an opinion, then returning to step 3 to iteratively optimize the inspection task execution process, and if the user confirms execution, then entering step 5;

[0061] Step 5: controlling the mobile robot to execute the inspection task by using the task scheduling manager, and iteratively inferring by using the large language model during the inspection process;

[0062] Step 6: feeding back the inspection result to the user after the inspection task is completed.

[0063] The specific implementation process is as shown in Figure 4 , by which the user issues an instruction to mobilize the inference inspection task execution process, the user iteratively solves the direct current and safety of the solution in the loop, and the inference execution code executes the task.

[0064] As an optimization scheme of the above embodiment, the task scheduling manager comprises a scheduling principle, an inspection function module library, a question and answer module and a large language model input module, the task scheduling manager is established in step 1, and comprises the following steps:

[0065] Step 1.1: establishing a scheduling principle: providing a standard format and a sequential inference of a task step for the large language model, and preferentially calling the inspection function module library;

[0066] Step 1.2: establishing an inspection function module library: encapsulating a basic module for different inspection sub-tasks, each basic module providing an explicit name prompt, a use purpose prompt and an input and output prompt, so as to facilitate the large language model to correctly understand and call;

[0067] Step 1.3: Establishing a question and answer module: the user interacts with the question and answer module through voice instructions or an interactive interface, and the question and answer module stores the user's current natural language instructions and historical natural language instructions for subsequent large language model reasoning;

[0068] Step 1.4: Establishing a large language model input module: summarizing the historical hidden question and answer and historical reasoning results of the large language model reasoning, and converting the output information of each basic module into a natural language form understood by the large language model for subsequent reasoning.

[0069] Among them, the establishment of the scheduling principle includes:

[0070] (1) The task scheduling manager can access the inspection function module library, so that the large language model reasons the inspection task flow or executes the instruction code according to the prompt of the basic module;

[0071] (2) The task scheduling manager generates a unique serial number for each inspection task, and adds a suffix prompt behind the serial number to indicate which basic modules the task is composed of, so as to avoid the large language model reasoning error caused by confusing task names;

[0072] (3) The task scheduling manager uses chain thinking to assist the large language model in reasoning or scheduling basic modules, that is, after generating an inspection task each time, it asks the large language model a question to ask whether more basic modules are needed to improve or optimize the inspection task.

[0073] Among them, in the step 1.2, the inspection function module library assists the task scheduling manager in scheduling the basic model through the prompt of the basic model, so that the large language model accurately understands the basic module, and the basic model prompt specifically includes:

[0074] (1) Name prompt: provides a summary of the overall function of each basic module, helping the large language model to understand the purpose of each basic module concisely, and serving as the calling entry of the basic module;

[0075] (2) Use prompt: describes the use scenario of each basic module, helping the large language model to decide which basic module to use for a specific task;

[0076] (3) Input and output prompt: describes the input and output format required by each basic module, helping the mobile robot to correctly execute the basic module to provide clear definition.

[0077] As Figure 2As shown, according to the category and prompt of the base model, defining the mobile robot perception host module includes: a simultaneous localization and mapping module, a map management module, a path planning and motion control module, a voice interaction module, a deep learning-based personnel position recognition module, a deep learning-based personnel wearing standard recognition module, and a deep learning-based fire smoke detection module.

[0078] (1) Simultaneous localization and mapping module m1:

[0079] Name prompt: This module is used to obtain the position of the robot and the map of the surrounding environment in real time.

[0080] Usage usage prompt: This module can be used for real-time positioning of the robot, and possible usage: after obtaining the real-time position, the area information of the location can be obtained through the map management module m2.

[0081] Input and output prompt: This module does not require additional input, and the output is the position and map. The code for obtaining the pose is "getLocation()", and the output type is the defined location class, representing the current position; the code for obtaining the map is "getMap()", and the output type is the map type, representing the surrounding map information.

[0082] (2) Map management module m2:

[0083] Name prompt: This module is used to store and load map information, indicating the area information of different positions, such as positions (1, 2, 3) in the office area, etc.

[0084] Usage usage prompt: This module can be used to understand the area information of different positions.

[0085] Input and output prompt: The input of this module is position information, and the output is area information. The code for obtaining area information is "getAreaName(locaiton)", and the output type is string, representing the area name. The input needs to provide location, which is of the location class, representing the position to be queried.

[0086] (3) Path planning and motion control module m3:

[0087] Name prompt: This module is used to plan a safe, efficient, and smooth path according to the target position of the robot and map information, and control the robot to go to the specified location. In the process, the module will automatically call the simultaneous localization and mapping module to obtain its own position and map information without additional calls. At the same time, the module will automatically loop to achieve dynamic obstacle avoidance.

[0088] Usage usage hint: This module can be used to implement the autonomous movement of the robot.

[0089] Input / Output hint: This module takes in the end location information and outputs whether the end has been reached when the life cycle ends. The code to get the path sequence is "setDestination(goalLocation)", the output type is bool, which represents whether the end has been reached, and the input needs to provide goalLocation, which is of type location, representing the location of the destination.

[0090] (4) Voice interaction module m4:

[0091] Name hint: This module is used to implement voice interaction between the robot and humans, including speech recognition and speech synthesis.

[0092] Usage usage hint: This module can be used to implement natural language interaction between users and robots, by speech recognition to obtain user instruction information, and by speech synthesis to provide voice feedback.

[0093] Input / Output hint: This module includes two sub-modules, including speech recognition and speech synthesis, where speech recognition does not require additional input, and the call code is "getSpeech()", the output is of type string, representing the user's speech. The call code for speech synthesis is "speak(string)", with no output, and the input needs to provide a string type, representing the text to be spoken to the user.

[0094] (5) Personnel location recognition module based on deep learning m5:

[0095] Name hint: This module is used to identify the location information of personnel in a complex environment based on the camera data of the robot.

[0096] Usage usage hint: Use when you need to know the location of personnel, such as: Is there a person in front of me? Where is the location in the map?

[0097] Input / Output hint: This module does not require additional input, and outputs the personnel pose. The call code is "getPersonLocationAndPose()". The output is a defined person class, which contains the personnel number and location information.

[0098] (6) Personnel wearing standard recognition module based on deep learning m6:

[0099] Name hint: This module is used to identify whether the personnel within the camera's field of view meet the wearing standards, such as whether they wear masks, safety helmets, etc.

[0100] Usage usage prompt: this module can be used to enable the robot to supervise the wearing of personnel, and is used when it is necessary to check whether the personnel are wearing standards.

[0101] Input and output prompt: this module does not require additional input, and the output is whether the personnel are not wearing standards. The calling code is "getPersonWearingStandardRecognitionResult()", and the output is a bool type, representing whether the personnel are not wearing standards.

[0102] (7) Fire smoke detection module m7 based on deep learning:

[0103] Name prompt: this module can detect whether there is a sign of fire smoke in the camera range.

[0104] Usage usage prompt: used when it is necessary to check whether there is a fire in the area.

[0105] Input and output prompt: this module does not require additional input, and the output is whether there is a sign of fire smoke. The calling code is "getFireSmokeDetectionResult()", and the output is a bool type, representing whether there is a sign of fire smoke.

[0106] Step 1.3: Establish a question and answer module. The user interacts with the question and answer module through voice instructions or an interactive interface, and the question and answer module stores the current natural language instruction and the historical natural language instruction of the user, so that the subsequent large language model can reason again;

[0107] Step 1.4: Establish a large language model input module. The large language model input module summarizes the historical hidden question and answer and historical reasoning results of the large language model reasoning, and converts the output information of the basic module into a natural language form that the large language model can understand, so as to facilitate subsequent reasoning again.

[0108] As an optimization scheme of the above embodiment, in the step 2: the natural language instruction of the user is obtained through the microphone array voice module, and specifically includes the following steps:

[0109] Step 2.1: The microphone array voice module collects the voice input of the user; the natural language instruction of the user voice input is collected through the microphone array voice module, and is converted into a digital signal;

[0110] Step 2.2: The digital signal will be sent to the speech processing module for preprocessing, and the preprocessing includes noise elimination and speech signal enhancement;

[0111] Step 2.3: Speech recognition; the preprocessed signal is sent to the speech recognition module, and the acoustic model, language model and pronunciation dictionary technology are used to convert the natural language instruction voice into text.

[0112] As an optimization solution of the above embodiment, in step 3: using a large language model to infer the inspection task execution process includes the following steps:

[0113] Step 3.1: Query the user's historical natural language instructions for the question-answering module and output the module's historical hidden questions and answers and historical reasoning results;

[0114] Step 3.2: Use the task scheduling manager to input the query content of step 3.1, the user's natural language instructions, scheduling principles, and basic module prompts into the large language model for iterative reasoning to obtain the inspection task execution process;

[0115] The inference function is defined as:

[0116]

[0117] Where i is the dialogue round, j is the i-th dialogue round m j The output, represents the output of the question-answering module, f is the task scheduling manager function, P is the scheduling principle described in step 1.1, M = {m1, m2, ..., m9} is the inspection function module library described in step 1.2, Q i is the question and answer content of the user in round i, H i A is the user's historical natural language instructions, including all user natural language instructions before round i; j To solve historical hidden questions and answers, the large language model relies on the task scheduling manager to gradually call the functions of various basic modules to complete the generation of inspection tasks. This calling process will generate multiple hidden question and answer interactions. For the reasoning history, for the i-th round of dialogue, the reasoning content of all basic modules before will be called. The schematic diagram of the reasoning process of the inspection task execution process is as follows Figure 5 shown.

[0118] Step 3.3: Store the hidden question and answer A of the large language model iterative reasoning process j and the user's current natural language instruction Q i .

[0119] As an optimization solution of the above embodiment, in step 4: using the task scheduling manager to feed back the inspection task execution process to the user, the following steps are included:

[0120] Step 4.1: Present the inspection task execution process to the user through a voice module or a visual interactive interface;

[0121] Step 4.2: Obtain the natural language instruction of the user again through the voice module or visual interactive interface of the mobile robot; if the user issues a command to execute the task, go to Step 5; if the user proposes a modification, return to Step 3 to reason the task execution flow again; if the user does not respond or issues a command to cancel the task, cancel the task.

[0122] As an optimization of the above embodiment, in Step 5: use the task scheduling manager to control the mobile robot to execute the inspection task, and use the large language model to iteratively reason during the inspection process, as shown in the following steps: Figure 6

[0123] Step 5.1: generate the task execution code using the large language model according to the inspection task execution flow and the inspection function module library;

[0124] Step 5.2: issue the task execution code to the robot through the task scheduling manager to control the basic module to execute the inspection task;

[0125] Step 5.3: receive the execution information of the basic module and process the execution information using the large language model input module;

[0126] Step 5.4: input the processed execution information of the basic module, the inspection task execution flow, the scheduling principle, and the inspection function module library into the large language model to reason whether the inspection task is completed using the task scheduling manager, and store the reasoning result and the execution information of the basic module;

[0127] Step 5.5: execute the corresponding output according to the reasoning result; if the inspection task is completed, go to Step 6; if the inspection task is not completed, continue to execute; if an abnormal situation occurs, reason new inspection task execution code and go to Step 5.2.

[0128] As an optimization of the above embodiment, in Step 6: after the completion of the inspection task, feedback the inspection result to the user, including the following steps:

[0129] Step 6.1: input the historical execution information of the basic module and the historical reasoning result of the large language model during the inspection process into the large language model through the task scheduling manager to reason the report content of the inspection result;

[0130] Step 6.2: present the report content to the user through the voice module or visual interactive interface of the task scheduling manager.

[0131] ​The application utilizes the powerful generation capability and creativity of the large language model to realize more diversified and personalized inspection contents and methods, and meets the needs of different scenes and tasks. For example, new inspection requirements and targets can be referenced or described through natural language queries and feedback, realizing zero-shot or few-shot inspection learning. Through this method, the inspection robot can more intelligently, efficiently and flexibly complete the inspection task in the industrial scene, improving the inspection efficiency and safety.

[0132] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for implementing human-machine interactive inspection of a mobile robot using a large language model, characterized in that: Including steps: Step 1: Create a task scheduling manager; The task scheduling manager includes a scheduling principle, an inspection function module library, a question-answering module, and a large language model input module. The task scheduling manager is established in step 1, including the following steps: Step 1.1: Establish scheduling principles: Provide a standardized format and sequential reasoning for the large language model to determine the task steps, and prioritize the use of the inspection function module library. In step 1.1, the scheduling principles are established, including: (1) The task scheduling manager can access the inspection function module library, allowing the large language model to infer the inspection task process or execute instruction codes based on the prompts of the basic modules; (2) The task scheduling manager generates a unique serial number for each inspection task and adds a suffix to the serial number to indicate which basic modules the task is composed of, so as to avoid large language model inference errors caused by confusing task names; (3) The task scheduling manager uses chain thinking to assist the large language model in reasoning or scheduling basic modules. That is, each time an inspection task is generated, a question is asked to the large language model to ask whether more basic modules are needed to improve or optimize the inspection task. Step 1.2: Establish an inspection function module library: Encapsulate basic modules for different inspection subtasks. Each basic module provides a clear name prompt, usage prompt, and input and output prompt to facilitate the correct understanding and call of the large language model; Step 1.3: Establish a question-answering module: Users interact with the question-answering module through voice commands or an interactive interface. The question-answering module stores the user's current natural language commands and historical natural language commands for subsequent large language model inference. Step 1.4: Establish a large language model input module: Summarize the historical hidden questions and answers and historical reasoning results of the large language model, and convert the output information of each basic module into a natural language form understood by the large language model for subsequent reasoning; Step 2: Obtain the user's natural language instructions through the mobile robot's voice module or visual interactive interface; Step 3: Use the large language model to infer the inspection task execution process through the task scheduling manager and natural language instructions; Step 4: Use the task scheduling manager to provide feedback to the user on the inspection task execution process. If the user raises any concerns, return to step 3 to iteratively optimize the inspection task execution process. If the user confirms the execution, proceed to step 5. Step 5: Use the task scheduling manager to control the mobile robot to perform the inspection task, and use the large language model for iterative reasoning during the inspection process; Step 6: After the inspection task is completed, the inspection results are fed back to the user.

2. The method for realizing human-machine interactive inspection of a mobile robot using a large language model according to claim 1, characterized in that: In the step 1.2, the inspection function module library assists the task scheduling manager in scheduling the basic model through the basic model prompt, so that the large language model can accurately understand the basic module. The basic model prompt specifically includes: (1) Name prompt: provides a summary of the overall function of each basic module, helps the large language model to concisely understand the purpose of each basic module, and serves as the calling entry of the basic module; (2) Usage prompt: describes the usage scenario of each basic module, helps the large language model to decide which basic module to use for a specific task; (3) Input and output prompt: describes the input and output format required by each basic module, and helps the mobile robot provide a clear definition for correctly executing the basic module; According to the types and prompts of the basic model, the mobile robot perception host modules are defined as follows: real-time positioning and map construction module, map management module, path planning and motion control module, voice interaction module, personnel location recognition module based on deep learning, personnel clothing standard recognition module based on deep learning, and fire smoke detection module based on deep learning.

3. The method for realizing human-machine interactive inspection of a mobile robot using a large language model according to claim 1, characterized in that: In step 2, obtaining the user's natural language instructions through the microphone array voice module includes the following steps: Step 2.1 The microphone array voice module collects user voice input; the microphone array voice module collects the natural language instructions of the user voice input and converts them into digital signals; Step 2.2 The digital signal will be sent to the voice processing module for preprocessing, which includes noise elimination and voice signal enhancement; Step 2.3 Speech Recognition: The preprocessed signal is fed into the speech recognition module, which uses acoustic models, language models, and pronunciation dictionary technology to convert natural language instruction speech into text.

4. The method for realizing human-machine interactive inspection of a mobile robot using a large language model according to claim 1, characterized in that: In step 3: using the large language model to infer the inspection task execution process, the steps include: Step 3.1: Query the user's historical natural language instructions for the question-answering module and output the module's historical hidden questions and answers and historical reasoning results; Step 3.2: Use the task scheduling manager to input the query content of step 3.1, the user's natural language instructions, scheduling principles, and basic module prompts into the large language model for iterative reasoning to obtain the inspection task execution process; Step 3.3: Store the hidden questions and answers of the large language model iterative reasoning process and the user's current natural language instructions.

5. The method for realizing human-machine interactive inspection of a mobile robot using a large language model according to claim 4, characterized in that: In step 4: using the task scheduling manager to feed back the inspection task execution process to the user, the steps include: Step 4.1: Present the inspection task execution process to the user through a voice module or a visual interactive interface; Step 4.2: The mobile robot obtains the user's natural language instructions again through the voice module or visual interactive interface. If the user issues a command to execute the task, the process proceeds to step 5. If the user proposes a modification, the process returns to step 3 and the inspection task execution process is inferred again. If the user does not respond or issue a command to cancel the task, the task is canceled.

6. The method for implementing a mobile robot human-machine interactive inspection using a large language model according to claim 5, characterized in that: In step 5: using a task scheduling manager to control the mobile robot to perform the inspection task, and using a large language model for iterative reasoning during the inspection process, including the following steps: Step 5.1: Generate task execution code using the large language model based on the inspection task execution process and inspection function module library; Step 5.2: Send the task execution code to the robot through the task scheduling manager to control the basic module to perform the inspection task; Step 5.3: Receive the execution information of the basic module and process the execution information using the large language model input module; Step 5.4: Use the task scheduling manager to input the processed basic module execution information, inspection task execution process, scheduling principles, and inspection function module library into the large language model to infer whether the inspection task is completed, and store the inference results and basic module execution information; Step 5.5: Execute the corresponding output according to the inference result; if the inspection task has been completed, proceed to step 6; if the inspection task has not been completed, continue execution; If an abnormal situation occurs, the new inspection task execution code is inferred and the process goes to step 5.

2.

7. The method for implementing a mobile robot human-machine interactive inspection using a large language model according to claim 6, characterized in that: In step 6: after the inspection task is completed, the inspection result is fed back to the user, including the steps of: Step 6.1: Input the historical execution information of the basic module and the historical inference results of the large language model during the inspection process into the large language model through the task scheduling manager, and infer the content of the inspection result report; Step 6.2: The task scheduling manager presents the report content to the user through a voice module or a visual interactive interface.

Images