Unmanned aerial vehicle industrial intelligent agent system based on multi-modal large model and construction method

Abstract

The invention discloses an unmanned aerial vehicle industrial intelligent agent system based on a multi-modal large model and a construction method. The unmanned aerial vehicle industrial intelligent agent system comprises an unmanned aerial vehicle executing multiple tasks, a universal system framework, a main program taking the multi-modal large model as an agent and mobile terminal software. Hardware of the unmanned aerial vehicle comprises an airborne computer, a flight control module, unmanned aerial vehicle rotors and motors, a visual camera, a voice output module and a power module. The system framework comprises an unmanned aerial vehicle, a task object, personnel and mobile terminal software of an edge part, and a cloud server and a monitoring center of a cloud part; a main program taking the multi-modal large model as an agent is carried on an airborne computer, and tasks in various scenes are realized through interaction with the multi-modal large model and other hardware of the unmanned aerial vehicle; the functions of the mobile terminal software comprise a communication center, data visualization, preset task scene design, task deployment, cloud synchronization, manual takeover and the like; the control method of the unmanned aerial vehicle executing the multiple tasks adopts a position loop and attitude loop double-closed-loop control scheme.

Description

Technical Field

[0001] The present invention belongs to the field of aircraft and the Internet of Things, and specifically relates to an unmanned aerial vehicle industrial intelligent agent system based on a multimodal large model and a construction method. Background Art

[0002] In the existing industrial field, there are many scenarios where tedious and single tasks are completed manually, such as evaluating workers' operations and checking the storage of equipment and tools. Many tasks have high safety requirements and require people to complete operations accurately. For example, the inspection of electrical facilities and equipment such as power system transmission lines requires inspectors to climb up the towers for inspection and maintenance, or the switching operation of the power system requires ensuring that the operation and maintenance personnel perform the switching operation correctly under safe conditions. All of the above tasks are visual tasks, that is, by observing the environment and task objects, important information such as risks, operation accuracy, equipment and personnel status can be evaluated and judged. These tasks are highly repetitive, tedious, and demanding. Relying solely on manual execution is very inefficient and risky.

[0003] There are many works that integrate cameras and robots into these industrial environments to assist humans in performing corresponding tasks. For example, wheeled robots are used to inspect substation lines, cameras are set up at various locations in the factory to monitor workers' operations, and drones are used to inspect power poles. These methods rely on manual remote monitoring or traditional visual algorithms and models for judgment. Their intelligence is often weak, and the algorithms they carry often cannot cope with a variety of different scenarios, that is, they have almost no generalization ability. At the same time, ground robots are limited by the movement space and cannot obtain more fields of view. It is difficult to lay cameras in scenes such as construction sites. In most industrial scenes, a large number of cameras need to be laid, which is quite costly.

[0004] With the emergence and widespread application of large multimodal models, it has become possible to combine them with highly flexible equipment such as drones to perform tasks in various industrial scenarios. However, there is currently no general system framework that can solve multiple industrial vision tasks at the same time. Summary of the invention

[0005] In order to solve the shortcomings of the existing technology, realize a strong generalization ability in a variety of industrial vision application scenarios, effectively perform tasks, improve benefits at a lower cost, and have the purpose of modularization, easy operation, easy expansion, and high versatility, the present invention adopts the following technical solutions:

[0006] A method for constructing UAV industrial intelligent agents based on multimodal large models, applied to the UAV side;

[0007] Obtain preset task scenarios from the cloud server through the mobile terminal, start and execute tasks based on the preset task scenarios, obtain task information based on task objects, and perform calculations and analysis through a multimodal large model;

[0008] The images generated by the multimodal large model, real-time mission progress, safety analysis, drone and / or personnel status information are sent to the mobile terminal so that the mobile terminal can display and / or send to the cloud server.

[0009] Furthermore, the task scenario includes task content, function library, prior knowledge base, and remark information;

[0010] The task content includes the basic elements of the task;

[0011] The function library includes the adjustment control function and state feedback function required to perform the task, so that the multimodal large model calls the function corresponding to the task content, adjusts and controls the drone and provides state feedback, and obtains the function return value, which includes the image obtained by the visual task, the flag bit of the function call completion, and the state of the drone and / or personnel;

[0012] The prior knowledge base includes the prior knowledge required to perform the task, which is set according to the task scenario so that the knowledge is outside the multimodal large model training data set and is closely related to the task; at least includes the answer format output by the multimodal large model, and the answer format at least includes the function to be called by the next main program and the step description currently being executed by the drone;

[0013] The remark information supplements the execution task with details so that the multimodal large model can communicate as required to output valid data that can be parsed by the program.

[0014] Furthermore, the main program using the multimodal large model as an agent is installed on the drone side, and its execution process is as follows:

[0015] Initialization and preset loading; the main program initializes the drone hardware, including the flight control module, visual camera, and voice output module, so that the program can control the drone's position and posture to control the drone's movement, receive environmental images to perform the visual tasks, and output voice broadcasts to convey the drone's mission status and safety warnings at any time; the main program and the mobile software establish a network communication link, load the preset mission scenarios, and can pass the prior information required for all preset mission executions to the multimodal large model in subsequent information;

[0016] Dialogue construction: Convert the preset task scenario information into prompt sentences and send them to the multimodal large model through the API;

[0017] Task execution based on interaction: After completing the preset scenario, the multimodal large model will return data in a specified format as required, which at least includes the function and its parameters to be called by the main program in the next step, and the description of the steps currently being executed by the drone; the main program receives the data and parses the function and its parameters to be called, and after executing the function, it will call the corresponding hardware to execute the specified steps, and wait for the return value of the function before sending it to the multimodal large model; after receiving the return value, the multimodal large model continues to return new data, including the function and its parameters to be called by the main program in the next step, and the description of the steps currently being executed by the drone; this process repeats itself, and the main program gradually completes the task steps under the agency of the multimodal large model to achieve the preset task scenario requirements;

[0018] Data synchronization: While implementing steps 1) to 3), the main program can synchronize data to the mobile software operated by personnel through a network link as required.

[0019] The UAV industrial intelligent agent system based on the multimodal large model includes a multi-task UAV, a system framework, a main program with the multimodal large model as an agent, and a mobile terminal;

[0020] The system framework includes drones, mobile terminals and their related edge parts, as well as cloud servers and their related cloud parts.

[0021] The mobile terminal obtains a preset task scenario from the cloud server, obtains the information generated by the main program through the multimodal large model as an agent, and displays and / or sends it to the cloud server;

[0022] The multi-task drone is equipped with a multimodal large model as the main program of the agent, starts and executes tasks based on the preset task scenarios of the mobile terminal, obtains task information based on the task objects and performs calculation and analysis through the multimodal large model to generate images, real-time task progress, safety analysis, drone and / or personnel status information.

[0023] Furthermore, the task scenario includes task content, function library, prior knowledge base, and remark information;

[0024] The task content includes the basic elements of the task;

[0025] The function library includes the adjustment control function and state feedback function required to perform the task, so that the multimodal large model calls the function corresponding to the task content, adjusts and controls the drone and provides state feedback, and obtains the function return value, which includes the image obtained by the visual task, the flag bit of the function call completion, and the state of the drone and / or personnel;

[0026] The prior knowledge base includes the prior knowledge required to perform the task, which is set according to the task scenario so that the knowledge is outside the multimodal large model training data set and is closely related to the task; at least includes the answer format output by the multimodal large model, and the answer format at least includes the function to be called by the next main program and the step description currently being executed by the drone;

[0027] The remark information supplements the execution task with details so that the multimodal large model can communicate as required to output valid data that can be parsed by the program.

[0028] Furthermore, the hardware of the multi-task drone includes an onboard computer, a flight control module, a drone rotor and motor, a visual camera, a voice output module, and a power module;

[0029] The onboard computer is used to process the state data of the drone and the image data of the camera, realize the proxy task of the multimodal large model, communicate with the mobile terminal software through the network to realize data transmission and task deployment, and control the movement trajectory and posture of the drone by controlling the flight control module;

[0030] The flight control module is used to control the flight of the drone at the bottom level, and directly controls the motor and rotor of the drone by giving torque, current or speed; the flight control module includes a controller and a sensor element for detecting the state of the drone;

[0031] The sensor element includes a GPS positioning module, an altimeter, and a gyroscope, which can obtain the three-dimensional coordinates (x, y, z) and Euler angle attitude (roll, pitch, yaw) of the drone in real time and transmit the data to the onboard computer;

[0032] The visual camera is used to capture environmental images and transmit the images to the onboard computer for input into the multi-modal large model to perform multi-scenario tasks;

[0033] The speech output module is used for tasks such as status broadcast and warning broadcast, and can convert text into speech sequence output;

[0034] The power module can power all the hardware of the multi-task UAV, including the onboard computer, flight control module, UAV rotor and motor, visual camera, voice output module, etc. Among them, the onboard computer, flight control module, UAV rotor and motor should be powered by the main power supply, and the visual camera and voice output module can be powered by the onboard computer interface.

[0035] Furthermore, the function library includes a UAV flight control function, which is used to control the UAV to fly to the destination along a specific trajectory; a UAV attitude adjustment function, which is used to control the attitude of the UAV to adjust the camera's viewing angle to obtain a suitable field of view; a function for calling the camera to obtain an image, which is used to obtain the field of view of the environment and the task object to perform a visual task; a function for calling the voice output module, which enables the UAV to broadcast the current task status and issue a safety warning for an emergency; the set of all callable functions is the total function library, and each set of preset function libraries should be a subset of the total function library to ensure that the function is callable;

[0036] The task includes a visual task of locking, identifying and evaluating the task object. The visual camera, in cooperation with the flight control module, captures the complete field of view of the task object. The flight control module adjusts the posture so that the visual camera receives the image of the task object at the correct angle and distance, thereby transmitting the appropriate image information to the onboard computer.

[0037] The information returned by the multimodal large model server is visualized and fed back on the mobile device. The status information of the drone includes information obtained by the GPS module, altimeter, and gyroscope carried by the drone, and the status information of the personnel includes information obtained by the positioning module and barometer carried on the mobile device of the personnel.

[0038] Furthermore, the UAV performs multi-task control, adopts a dual closed-loop control scheme of position loop and attitude loop, and the attitude control of the UAV can be divided into pitch, yaw, and roll;

[0039] The pitch motion of the drone is controlled by the controller in the flight control module, and the motor flight control module transmits the command to change the deflection angle θ of the drone around the y-axis to control the pitch pitch; the yaw motion of the drone is controlled by the controller in the flight control module, and the motor receives the command transmitted by the flight control module to change the deflection angle of the drone around the z-axis To control the yaw angle yaw; the rolling motion of the drone is controlled by the controller in the flight control module, and the motor receives the command transmitted by the flight control module to change the deflection angle φ of the drone around the x-axis to control the rolling angle roll.

[0040] Furthermore, the execution process of the dual closed-loop control is as follows:

[0041] The position loop:

[0042] u pos =a t +K d (v t -v)+K p (pt -p)

[0043] Among them, v t represents the trajectory speed, a t represents the trajectory acceleration, p t represents the trajectory position, u pos It represents the signal output to the flight control module, which converts it into actual thrust (throttle) and outputs it to the motor. v and p represent the current speed and position respectively. The position is obtained by the GPS positioning module, and then the speed is calculated. The speed is calculated by the inertial measurement unit IMU and / or the camera-based visual odometer VO; K d and K p Respectively represent the differential coefficient and proportional coefficient in proportional-integral-derivative PID (Proportional-Integral-Derivative) control; a t As feedforward, K d (v t -v)+K p (p t -p) serves as feedback and together form a position loop, so that the UAV can complete the flight of the specified trajectory and stabilize at the final position;

[0044] The attitude loop is realized by using the angle and angular velocity errors between the target attitude and the actual attitude:

[0045] u a =K p,a ·e a +K i,a ·e a ·t+K d,a ·(e a -e′ a )

[0046] u w =K p,w ·e w +K i,w ·e w ·t+K d,w ·(e w -e′ w )

[0047] Among them, the subscript a represents the parameter of the angle loop, w represents the parameter of the angular velocity loop, and u a Indicates the control output of the angle loop, u w represents the control output of the angular velocity loop. The error between the angle loop and the actual angular velocity value will be used as the input of the angular velocity loop. The angular velocity and angle are obtained by obtaining data from the gyroscope and are calculated. p,a and K p,wThey represent the proportional coefficients of the angle loop and angular velocity loop in the proportional integral differential control PID control, respectively, and K i,a and K i,w They represent the integral coefficients of the angle loop and angular velocity loop in the proportional integral differential control PID control, respectively. d,a and K d,w They represent the differential coefficients of the angle loop and angular velocity loop in the proportional integral differential control PID control, respectively. a and e w Represent the angle error and angular velocity error of the target attitude and actual attitude respectively, e′ a and e′ w Respectively represent the angle error and angular velocity error of the target posture and actual posture in the previous time step, and t represents the time step. ;

[0048] These two rings form an inner and outer ring structure, allowing the drone to stabilize in an accurate posture.

[0049] Furthermore, the functions of the mobile terminal software include at least communication hub, data visualization, preset task scenario design, task deployment, cloud synchronization, and manual takeover;

[0050] The communication hub, the mobile software communicates with the drone and the cloud server respectively through the network link, and assumes the function of the communication hub and bridge between the edge and the cloud. The software receives the data transmitted by the drone and can selectively synchronize it to the cloud server;

[0051] The data visualization, after the mobile terminal software receives the data from the drone and analyzes it, displays it through various UI components. The UI used for data visualization includes an image display component, a status monitoring component, and a preset scene component, which can display the image captured by the camera carried by the drone, the status of the drone and personnel, and the preset mission scene currently set for the drone and its details;

[0052] The preset task scenario design allows personnel to quickly and easily add, delete, modify and view the preset task scenarios through the UI of the mobile software. The specific objects of execution include task content, function library, prior knowledge base and remarks information.

[0053] For the task deployment, the personnel selects the existing preset task scenario through the mobile software and deploys it to the connected drone, so that the drone starts to automatically execute the target task;

[0054] The cloud synchronization means that the data received by the mobile software from the drone can be synchronized to the cloud server, and all the preset mission scenarios and total function libraries on the cloud or mobile software can be synchronized in both directions;

[0055] The manual takeover allows personnel to manually take over the control of the drone through the mobile software to ensure that when the main program fails, the safety of the drone, personnel, on-site environment and equipment can be guaranteed through direct control.

[0056] The advantages and beneficial effects of the present invention are:

[0057] (1) The system framework of the present invention has a modular feature. The edge part can run independently from the cloud part, and the cloud part does not need to be built to save costs. On the contrary, the existence of the cloud part can make the architecture suitable for most actual projects, and facilitate the industrial background to perform complex subsequent processing and archiving synchronization of data. At the same time, the drone can also run independently after the task is deployed through the mobile software, and no manual access is required if there is no failure.

[0058] (2) The main program of the present invention uses a large multimodal model deployed on a cloud server as an agent. Therefore, the onboard computer of the drone of the present invention does not need to store a large number of model parameters, nor does it need to perform complex reasoning tasks, and the required configuration requirements are low.

[0059] (3) The mobile software of the present invention can easily implement tasks such as task deployment, monitoring, and design scene presets. The operation is simple for personnel, and they only need to carry mobile devices such as mobile phones and tablets.

[0060] (4) Thanks to the multimodal large model, the present invention can adapt to various types of visual task scenarios in the industry and has strong generalization performance. It can also quickly understand tasks in unfamiliar fields by providing prior knowledge. The large model replaces the auxiliary role of the traditional algorithm and becomes an intelligent agent of the program. Its ability to understand tasks is stronger than that of traditional visual algorithms.

[0061] (5) The hardware equipment carried by the drone of the present invention is mainly composed of basic sensors and flight control components. The equipment is quite basic and low in cost.

[0062] (6) The system framework of the present invention is highly scalable. On its basis, more hardware can be added to the drone, more professional industrial field prior knowledge can be provided for the multimodal large model, and more functions can be provided for the cloud and mobile software, so that it can not only cope with visual tasks, but also cope with more complex industrial task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a schematic diagram of the architecture of a drone that performs multiple tasks.

[0064] Figure 2 It is a general system framework diagram.

[0065] Figure 3It is a schematic diagram of a general system framework with power system switching operation monitoring as a specific scenario.

[0066] Figure 4 It is a general system framework with power system tools collection and evaluation as specific scenario diagram.

[0067] Figure 5 It is a schematic diagram of the operation process and principle of the main program with the multimodal large model as the agent in the switching operation supervision scenario.

[0068] Figure 6 It is a schematic diagram of the operation process and principle of the main program with a multimodal large model as the agent in the tool collection and evaluation scenario.

[0069] Figure 7 It is a schematic diagram of the mobile software UI and its functions. DETAILED DESCRIPTION

[0070] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0071] Embodiment 1:

[0072] Taking the task of monitoring the switching operation of the power system as an example, a general industrial intelligent agent system of a drone based on a multimodal large model is specifically described below, including a drone that performs multiple tasks, a general system framework, a main program with a multimodal large model as an agent, and mobile terminal software. The construction method thereof includes the following steps:

[0073] Step 1: Build Figure 1 The multi-task drone architecture shown in the figure, the hardware of the multi-task drone includes an onboard computer, a flight control module, a drone rotor and motor, a visual camera, a voice output module, and a power module;

[0074] The onboard computer is used to process the status data of the drone and the image data of the camera, realize the proxy task of the multi-modal large model, communicate with the mobile software through the network to realize data transmission and task deployment, and control the movement trajectory and posture of the drone by controlling the flight control module;

[0075] The flight control module is used to control the flight of the drone at the bottom level. It directly controls the motor and rotor of the drone by giving torque, current or speed. The flight control module includes a controller and sensor components for detecting the status of the drone.

[0076] The sensor components include a GPS positioning module, an altimeter, and a gyroscope, which can obtain the drone’s three-dimensional coordinates (x, y, z) and Euler angle attitude (roll, pitch, yaw) in real time and transmit the data to the onboard computer;

[0077] The visual camera is used to capture environmental images and transmit them to the onboard computer for input into the multi-modal large model to perform multi-scenario tasks;

[0078] The voice output module is used for tasks such as status broadcast and warning broadcast, and can convert text into voice sequence output;

[0079] The power module can power all the hardware of the multi-task UAV, including the onboard computer, flight control module, UAV rotor and motor, visual camera, voice output module, etc. Among them, the onboard computer, flight control module, UAV rotor and motor should be powered by the main power supply, and the visual camera and voice output module can be powered by the onboard computer interface.

[0080] Step 2: Figure 2 The general system framework design shown in the figure is slightly modified to determine and build Figure 3 The system framework shown, the left and right dotted boxes represent the edge part and the cloud part of the present invention respectively. The system framework includes the edge part composed of drones, task objects, personnel, and mobile terminal software, and the cloud part composed of cloud servers and monitoring centers. The task objects are embodied as electrical equipment and inspection personnel;

[0081] The overall working principle of the framework is as follows:

[0082] 1) Communication construction and task deployment: The personnel operate the mobile software to connect to the drone's onboard computer through the network to establish a link with the drone. Through this link, the drone presets the task scenario, and the drone program starts to execute the task, performing visual tasks such as locking, identifying, and evaluating the task object;

[0083] 2) Data feedback and visualization: Principle 1) During the mission, the drone will continuously return data through the network link. The data should include at least images, status, messages, safety warnings and other forms. The data will be visualized on the UI of the mobile software to facilitate personnel monitoring and taking actions;

[0084] 3) Data cloud synchronization: The mobile software can connect to the cloud server in the cloud through the network and transmit the data generated during task execution to the cloud server. The mobile software and the cloud server can also realize two-way synchronization of preset information by mutually transmitting existing preset task scenarios and total function libraries.

[0085] 4) Cloud monitoring: The monitoring center can access the cloud server through the network to obtain the data described in principle 3), realize remote monitoring of tasks, be able to know and handle special situations in a timely manner, and further analyze, process and retain the data;

[0086] 5) Multimodal large model access: The drone mainly accesses the multimodal large model through a network interface. The model can be deployed on the cloud server of the cloud part or other network servers.

[0087] Step 3: The main program with the multimodal large model as the agent is installed on the onboard computer. The main program intelligently implements tasks in various scenarios by interacting with the multimodal large model and other hardware of the drone. The specific implementation steps are as follows:

[0088] 1) Initialization and preset loading: The main program first initializes the drone hardware, including the flight control module, visual camera, and voice output module, so that the program can control the drone position and posture to control the drone movement, receive environmental images to perform the switching monitoring task, and output voice broadcasts to convey the drone mission status and safety warnings at any time;

[0089] At the same time, the main program and the mobile software establish a network communication link and load the preset task scenarios, which can subsequently pass the prior information required for the execution of all preset tasks to the multimodal large model.

[0090] 2) Dialogue construction: After the main program completes step 1), it connects to the multimodal big model through the network API, converts the preset scene information into prompt sentences and sends them to the multimodal big model through the API. A feasible format of the prompt sentence is as follows:

[0091] Your task is {task content}, the executable functions are {function library}, prior knowledge: {prior knowledge base}, what needs attention is {remark information}, let's get started now!

[0092] The specific switching operation monitoring task content, function library, prior knowledge base, and remarks are in Figure 5 There are hints in the text;

[0093] 3) Task execution based on interaction: e.g. Figure 5In the left dialogue part, after the preset scenario is transmitted to the multimodal large model in step 2), the large model will return data in a specified format as required, which at least includes the function and its parameters to be called by the main program in the next step, and the description of the steps currently being executed by the drone; the main program receives the data and parses the function and its parameters to be called, and after executing the function, calls the corresponding hardware to execute the specified steps, and waits for the return value of the function before sending it to the multimodal large model; after receiving the return value, the multimodal large model continues to output new data, including the function and its parameters to be called by the main program in the next step, and the description of the steps currently being executed by the drone; in this way, the main program gradually completes the task steps under the agency of the multimodal large model to achieve the preset task scenario requirements; Figure 5 The intelligent agent first controls the drone to fly to the specified position and then determines the angle deviation through the image. It then continues to control the drone's yaw angle to make corrections. After obtaining the correct field of view, it begins to monitor the switching operation to determine whether there are any safety issues.

[0094] 4) Data synchronization: While implementing steps 1) to 3), the main program can synchronize the data to the mobile software operated by personnel through a network link as required.

[0095] Step 4: The responsible personnel use the mobile software to connect to the drone and deploy the switching operation monitoring task to start the main program in step 3 so that the intelligent agent can take over the task execution;

[0096] The functions of the mobile software include at least communication hub, data visualization, preset task scenario design, task deployment, cloud synchronization, manual takeover, etc. Its UI is shown below: Figure 7 As shown, it is implemented in the form of WeChat applet, but the form of mobile software is not limited to WeChat applet;

[0097] 1) Communication hub: The mobile software communicates with the drone and cloud server through network links. It needs to serve as the communication hub and bridge between the edge and the cloud. The software receives data transmitted by the drone and can selectively synchronize it to the cloud server.

[0098] 2) Data visualization: After the mobile software receives the data from the drone and parses it, it displays it through various UI components. The UI used for data visualization includes image display components, status monitoring components, preset scene components, etc. It can display the images captured by the camera on the drone, the status of the drone and personnel, and the preset mission scenes currently set for the drone and their details;

[0099] 3) Preset task scenario design: Personnel can add, delete, modify and view preset task scenarios conveniently and quickly through the UI of the mobile software. The executable specific objects include task content, function library, prior knowledge base, note information, etc.

[0100] 4) Task deployment: Personnel can select existing preset task scenarios through mobile software and deploy them to connected drones, so that the drones can automatically perform the target tasks;

[0101] 5) Cloud synchronization: The data received by the mobile software from the drone can be synchronized to the cloud server. All preset mission scenarios and total function libraries on the cloud or mobile software can be synchronized in both directions;

[0102] 6) Manual takeover: Personnel can manually take over the control of the drone through the mobile software to ensure the safety of the drone, personnel, on-site environment and equipment through direct control when the main program fails.

[0103] Step five: After the mission is completed, the responsible personnel can manually take over the drone and return to the origin, or directly deploy a new mission for the intelligent agent to continue executing. The data generated by the monitoring can also be synchronized to the cloud for recording and archiving, etc., thus completing the functional implementation of the invention in the power system switching operation monitoring scenario.

[0104] Embodiment 2:

[0105] Taking the tool collection evaluation task as an example, a general industrial intelligent agent system for drones based on a multimodal large model is specifically described below, including a drone that performs multiple tasks, a general system framework, a main program with a multimodal large model as an agent, and mobile terminal software. The construction method thereof includes the following steps:

[0106] Step 1: Build Figure 1 The multi-task drone architecture shown in the figure, the hardware of the multi-task drone includes an onboard computer, a flight control module, a drone rotor and motor, a visual camera, a voice output module, and a power module;

[0107] The onboard computer is used to process the state data of the drone and the image data of the camera, realize the proxy task of the multimodal large model, communicate with the mobile terminal software through the network to realize data transmission and task deployment, and control the movement trajectory and posture of the drone by controlling the flight control module;

[0108] The flight control module is used to control the flight of the drone at the bottom level, and directly controls the motor and rotor of the drone by giving torque, current or speed; the flight control module includes a controller and a sensor element for detecting the state of the drone;

[0109] The sensor element includes a GPS positioning module, an altimeter, and a gyroscope, which can obtain the three-dimensional coordinates (x, y, z) and Euler angle attitude (roll, pitch, yaw) of the drone in real time and transmit the data to the onboard computer;

[0110] The visual camera is used to capture environmental images and transmit the images to the onboard computer for input into a multi-modal large model to perform multi-scenario tasks;

[0111] The voice output module is used for tasks such as status broadcast and warning broadcast, and can convert text into voice sequence output;

[0112] The power module can power all the hardware of the multi-task UAV, including the onboard computer, flight control module, UAV rotor and motor, visual camera, voice output module, etc. Among them, the onboard computer, flight control module, UAV rotor and motor should be powered by the main power supply, and the visual camera and voice output module can be powered by the onboard computer interface.

[0113] Step 2: Figure 2 The general system framework design shown in the figure is slightly modified to determine and build Figure 4 The system framework shown, the left and right dotted boxes represent the edge part and the cloud part of the present invention respectively. The system framework includes the edge part composed of drones, task objects, personnel, and mobile terminal software, and the cloud part composed of cloud servers and monitoring centers. The task object is specifically represented by the storage status of tools;

[0114] The overall working principle of the framework is as follows:

[0115] 1) Communication construction and task deployment: The personnel operate the mobile software to connect to the drone's onboard computer through the network to establish a link with the drone. Through this link, the drone presets the task scenario, and the drone program starts to execute the task, performing visual tasks such as locking, identifying, and evaluating the task object;

[0116] 2) Data feedback and visualization: During the mission described in principle 1), the drone will continuously return data via the network link. The data should include at least images, status, messages, safety warnings, etc. The data will be visualized on the UI of the mobile software to facilitate personnel monitoring and taking actions;

[0117] 3) Data cloud synchronization: The mobile software can connect to the cloud server in the cloud through the network and transmit the data generated during task execution to the cloud server. The mobile software and the cloud server can also realize two-way synchronization of preset information by mutually transmitting existing preset task scenarios and total function libraries.

[0118] 4) Cloud monitoring: The monitoring center can access the cloud server through the network to obtain the data described in principle 3), realize remote monitoring of tasks, be able to know and handle special situations in a timely manner, and further analyze, process and retain the data;

[0119] 5) Multimodal large model access: The drone mainly accesses the multimodal large model through a network interface. The model can be deployed on the cloud server of the cloud part or other network servers.

[0120] Step 3: The main program with the multimodal large model as the agent is installed on the onboard computer. The main program intelligently implements tasks in various scenarios by interacting with the multimodal large model and other hardware of the drone. The specific implementation steps are as follows:

[0121] 1) Initialization and preset loading: The main program first initializes the drone hardware, including the flight control module, visual camera, and voice output module, so that the program can control the drone position and posture to control the drone movement, receive environmental images to perform the switching monitoring task, and output voice broadcasts to convey the drone mission status and safety warnings at any time;

[0122] At the same time, the main program and the mobile software establish a network communication link and load the preset task scenarios, which can subsequently pass the prior information required for the execution of all preset tasks to the multimodal large model.

[0123] 2) Dialogue construction: After the main program completes step 1), it connects to the multimodal big model through the network API, converts the preset scene information into prompt sentences and sends them to the multimodal big model through the API. A feasible format of the prompt sentence is as follows:

[0124] Your task is {task content}, the executable functions are {function library}, prior knowledge: {prior knowledge base}, what needs attention is {remark information}, let's get started now!

[0125] The specific tool contains the task content, function library, prior knowledge base, and notes of the evaluation. Figure 6 There are hints in the text;

[0126] 3) Task execution based on interaction: e.g. Figure 6In the left dialogue part, after the preset scenario is transmitted to the multimodal large model in step 2), the large model will return data in a specified format as required, which at least includes the function and its parameters to be called by the main program in the next step, and the description of the steps currently being executed by the drone; the main program receives the data and parses the function and its parameters to be called, and after executing the function, calls the corresponding hardware to execute the specified steps, and waits for the return value of the function before sending it to the multimodal large model; after receiving the return value, the multimodal large model continues to output new data, including the function and its parameters to be called by the main program in the next step, and the description of the steps currently being executed by the drone; in this way, the main program gradually completes the task steps under the agency of the multimodal large model to achieve the preset task scenario requirements; Figure 6 The intelligent agent first controls the drone to fly to the specified position and then uses the image to determine if there is an angle deviation. It then continues to control the drone's yaw angle to make corrections. After obtaining the correct field of view, it begins to evaluate the storage of tools, determine whether the tools are placed correctly, whether there are any missing tools, and give a score.

[0127] 4) Data synchronization: While implementing steps 1) to 3), the main program can synchronize the data to the mobile software operated by personnel through a network link as required.

[0128] Step 4: The responsible person uses the mobile software to connect to the drone and deploy the tool collection and evaluation task to start the main program in step 3 so that the intelligent agent takes over the task execution;

[0129] The functions of the mobile software include at least communication hub, data visualization, preset task scenario design, task deployment, cloud synchronization, manual takeover, etc. Its UI is shown below: Figure 7 As shown, it is implemented in the form of WeChat applet, but the form of mobile software is not limited to WeChat applet;

[0130] 1) Communication hub: The mobile software communicates with the drone and cloud server through network links. It needs to serve as the communication hub and bridge between the edge and the cloud. The software receives data transmitted by the drone and can selectively synchronize it to the cloud server.

[0131] 2) Data visualization: After the mobile software receives the data from the drone and parses it, it displays it through various UI components. The UI used for data visualization includes image display components, status monitoring components, preset scene components, etc. It can display the images captured by the camera on the drone, the status of the drone and personnel, and the preset mission scenes currently set for the drone and their details;

[0132] 3) Preset task scenario design: Personnel can add, delete, modify and view preset task scenarios conveniently and quickly through the UI of the mobile software. The executable specific objects include task content, function library, prior knowledge base, note information, etc.

[0133] 4) Task deployment: Personnel can select existing preset task scenarios through mobile software and deploy them to connected drones, so that the drones can automatically perform the target tasks;

[0134] 5) Cloud synchronization: The data received by the mobile software from the drone can be synchronized to the cloud server. All preset mission scenarios and total function libraries on the cloud or mobile software can be synchronized in both directions;

[0135] 6) Manual takeover: Personnel can manually take over the control of the drone through the mobile software to ensure the safety of the drone, personnel, on-site environment and equipment through direct control when the main program fails.

[0136] Step five: After the mission is completed, the responsible personnel can manually take over the drone and return to the origin, or directly deploy a new mission for the intelligent agent to continue executing. The data generated by the evaluation can also be synchronized to the cloud for recording and archiving, etc., thus completing the functional implementation of the invention in the tool storage evaluation scenario.

[0137] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an industrial intelligent agent for drones based on a multimodal large model, applied to the drone side, is characterized by: Obtain preset task scenarios from the cloud server through the mobile terminal, start and execute tasks based on the preset task scenarios, obtain task information based on task objects, and perform calculations and analysis through a multimodal large model; The images generated by the multimodal large model, real-time mission progress, safety analysis, drone and / or personnel status information are sent to the mobile terminal so that the mobile terminal can display and / or send to the cloud server.

2. The method for constructing an unmanned aerial vehicle industrial intelligent agent based on a multimodal large model according to claim 1, characterized in that: The task scenario includes task content, function library, prior knowledge base, and remark information; The task content includes the basic elements of the task; The function library includes the adjustment control function and state feedback function required to perform the task, so that the multimodal large model calls the function corresponding to the task content, adjusts and controls the drone and provides state feedback, and obtains the function return value, which includes the image obtained by the visual task, the flag bit of the function call completion, and the state of the drone and / or personnel; The prior knowledge base includes the prior knowledge required to perform the task, which is set according to the task scenario so that the knowledge is outside the multimodal large model training data set and is closely related to the task; The remark information supplements the execution task with details so that the multimodal large model can communicate as required to output valid data that can be parsed by the program.

3. The method for constructing an unmanned aerial vehicle industrial intelligent agent based on a multimodal large model according to claim 2 is characterized in that: The main program using the multimodal large model as an agent is installed on the drone side, and its execution process is as follows: Initialization and preset loading; the main program initializes the drone to control the drone movement, receives environmental images to perform the visual task, and outputs the drone task status; the main program establishes a network communication link with the mobile terminal software and loads the preset task scene; Dialogue construction: convert the preset task scenario information into prompt sentences and send them to the multimodal large model; Task execution based on interaction: After completing the preset scenario, the multimodal large model will return data as required, including at least the function to be called by the main program in the next step and its parameters, and the description of the steps currently being executed by the drone; the main program receives the data and parses the function to be called and its parameters, and after executing the function, it will call the corresponding hardware to execute the specified steps, and wait for the return value of the function and send it to the multimodal large model; the multimodal large model continues to return new data after receiving the return value; this goes back and forth, and the main program gradually completes the task steps under the agency of the multimodal large model to achieve the preset task scenario requirements; Data synchronization: The main program synchronizes data to the mobile terminal through the network link as required.

4. The UAV industrial intelligent agent system based on the multimodal large model includes a multi-task UAV, a system framework, a main program with the multimodal large model as an agent, and a mobile terminal, and is characterized by: The system framework includes drones, mobile terminals and their related edge parts, as well as cloud servers and their related cloud parts. The mobile terminal obtains a preset task scenario from the cloud server, obtains the information generated by the main program through the multimodal large model as an agent, and displays and / or sends it to the cloud server; The multi-task drone is equipped with a multimodal large model as the main program of the agent, starts and executes tasks based on the preset task scenarios of the mobile terminal, obtains task information based on the task objects and performs calculation and analysis through the multimodal large model to generate images, real-time task progress, safety analysis, drone and / or personnel status information.

5. The UAV industrial intelligent agent system based on multimodal large model according to claim 4 is characterized by: The task scenario includes task content, function library, prior knowledge base, and remark information; The task content includes the basic elements of the task; The function library includes the adjustment control function and state feedback function required to perform the task, so that the multimodal large model calls the function corresponding to the task content, adjusts and controls the drone and provides state feedback, and obtains the function return value, which includes the image obtained by the visual task, the flag bit of the function call completion, and the state of the drone and / or personnel; The prior knowledge base includes the prior knowledge required to perform the task, which is set according to the task scenario so that the knowledge is outside the multimodal large model training data set and is closely related to the task; The remark information supplements the execution task with details so that the multimodal large model can communicate as required to output valid data that can be parsed by the program.

6. The UAV industrial intelligent agent system based on multimodal large model according to claim 5 is characterized by: The hardware of the multi-task drone includes an onboard computer, a flight control module, a drone rotor and motor, a visual camera, a voice output module, and a power module; The onboard computer is used to process the state data of the drone and the image data of the camera, realize the proxy task of the multimodal large model, communicate with the mobile terminal software through the network to realize data transmission and task deployment, and control the movement trajectory and posture of the drone by controlling the flight control module; The flight control module is used to control the flight of the drone at the bottom level, and directly controls the motor and rotor of the drone by giving torque, current or speed; the flight control module includes a controller and a sensor element for detecting the state of the drone; The sensor element includes a GPS positioning module, an altimeter, and a gyroscope, which can obtain the three-dimensional coordinates and Euler angle attitude of the drone in real time and transmit the data to the onboard computer; The visual camera is used to capture environmental images and transmit the images to the onboard computer for input into the multi-modal large model to perform multi-scenario tasks; The speech output module is used for task status broadcast and warning broadcast, and can convert text into speech sequence output; The power module can provide power for the hardware of a multi-task drone.

7. The UAV industrial intelligent agent system based on multimodal large model according to claim 6 is characterized by: The function library includes a UAV flight control function, which is used to control the UAV to fly to a destination along a specific trajectory; a UAV attitude adjustment function, which is used to control the attitude of the UAV to adjust the viewing angle of the camera to obtain a suitable field of view; Call the function of acquiring images from the camera to obtain the field of view of the environment and task objects to perform visual tasks; Call the function of the voice output module to enable the drone to broadcast the current mission status and issue safety warnings for emergency situations; The task includes a visual task of locking, identifying and evaluating the task object. The visual camera, in cooperation with the flight control module, captures the complete field of view of the task object. The flight control module adjusts the posture so that the visual camera receives the image of the task object at the correct angle and distance, thereby transmitting the appropriate image information to the onboard computer. The information returned by the multimodal large model server is visualized and fed back on the mobile device. The status information of the drone includes information obtained by the GPS module, altimeter, and gyroscope carried by the drone, and the status information of the personnel includes information obtained by the positioning module and barometer carried on the mobile device of the personnel.

8. The UAV industrial intelligent agent system based on multimodal large model according to claim 6 is characterized by: The multi-task control of the UAV adopts dual closed-loop control of position loop and attitude loop. The attitude control of the UAV can be divided into pitch, yaw angle and roll angle; The pitch motion of the drone is controlled by the controller in the flight control module, and the motor flight control module transmits the command to change the deflection angle θ of the drone around the y-axis to control the pitch; the yaw motion of the drone is controlled by the controller in the flight control module, and the motor receives the command transmitted by the flight control module to change the deflection angle of the drone around the z-axis To control the yaw angle; the rolling motion of the drone is controlled by the controller in the flight control module, and the motor receives the command transmitted by the flight control module to change the deflection angle φ of the drone around the x-axis to control the rolling angle.

9. The UAV industrial intelligent agent system based on multimodal large model according to claim 8 is characterized by: The execution process of the dual closed-loop control is as follows: The position loop: u pos =a t +K d (v t -v)+K p (p t -p) Among them, v t represents the trajectory speed, a t represents the trajectory acceleration, p t represents the trajectory position, u pos It represents the signal output to the flight control module, which converts it into actual thrust and outputs it to the motor. v and p represent the current speed and position respectively. The position is obtained by the GPS positioning module, and then the speed is calculated. The speed is calculated by the inertial measurement unit IMU and / or the camera-based visual odometer VO; K d and K p Respectively represent the differential coefficient and proportional coefficient in proportional-integral-differential control; a t As feedforward, K d (v t -v)+K p (p t -p) serves as feedback and together form a position loop, so that the UAV can complete the flight of the specified trajectory and stabilize at the final position; The attitude loop is realized by using the angle and angular velocity errors between the target attitude and the actual attitude: u a =K p,a ·e a +K i,a ·e a ·t+K d,a ·(e a -e ′ a ) u w =K p,w ·e w +K i,w ·e w ·t+K d,w ·(e w -e ′ w ) Among them, the subscript a represents the parameter of the angle loop, w represents the parameter of the angular velocity loop, and u a Indicates the control output of the angle loop, u w represents the control output of the angular velocity loop. The error between the angle loop and the actual angular velocity value will be used as the input of the angular velocity loop. The angular velocity and angle are obtained by obtaining data from the gyroscope and are calculated. p,a and K p,w They represent the proportional coefficients of the angle loop and angular velocity loop in the proportional-integral-derivative control, respectively, and K i,a and K i,w They represent the integral coefficients of the angle loop and angular velocity loop in the proportional-integral-derivative control, respectively, and K d,a and K d,w They represent the differential coefficients of the angle loop and angular velocity loop in the proportional integral differential control PID control, respectively. a and e w They represent the angle error and angular velocity error of the target attitude and the actual attitude respectively, e ′ a and e ′ w They represent the angle error and angular velocity error of the target posture and actual posture in the previous time step respectively, and t represents the time step.

10. The UAV industrial intelligent agent system based on multimodal large model according to claim 6 is characterized by: The functions of the mobile terminal software include at least communication hub, data visualization, preset task scenario design, task deployment, cloud synchronization, and manual takeover; The communication hub, the mobile software communicates with the drone and the cloud server respectively through the network link, and assumes the function of the communication hub and bridge between the edge and the cloud. The software receives the data transmitted by the drone and can selectively synchronize it to the cloud server; The data visualization, after the mobile terminal software receives the data from the drone and analyzes it, displays it through various UI components. The UI used for data visualization includes an image display component, a status monitoring component, and a preset scene component, which can display the image captured by the camera carried by the drone, the status of the drone and personnel, and the preset mission scene currently set for the drone and its details; The preset task scenario design allows personnel to quickly and easily add, delete, modify and view the preset task scenarios through the UI of the mobile software. The specific objects of execution include task content, function library, prior knowledge base and remarks information. For the task deployment, the personnel selects the existing preset task scenario through the mobile software and deploys it to the connected drone, so that the drone starts to automatically execute the target task; The cloud synchronization means that the data received by the mobile software from the drone can be synchronized to the cloud server, and all the preset mission scenarios and total function libraries on the cloud or mobile software can be synchronized in both directions; The manual takeover allows personnel to manually take over control of the drone through mobile software.

Images