ROS-based four-rotor tour guide unmanned aerial vehicle system and tour guide auxiliary method thereof

Abstract

The invention discloses a four-rotor tour guide unmanned aerial vehicle system based on an ROS and a tour guide auxiliary method thereof. According to the method, whether an input external instruction is an interaction instruction or not is judged based on an ROS module; if yes, generating a control parameter signal in combination with unmanned aerial vehicle state information collected by a sensing module to drive the unmanned aerial vehicle to adjust the state; and if the instruction is a non-control type interaction instruction, uploading the instruction and state information to the large model module. The large model module generates personalized tour guide script information, the ROS module converts the personalized tour guide script information into tour guide auxiliary voice signals, and the tour guide auxiliary voice signals are presented to tourists through the output module. According to the method, the external instruction and the unmanned aerial vehicle state can be understood and responded, personalized and situational tour guide content is dynamically generated, the man-machine interaction process is more natural, and multiple intelligent interaction and personalized tour guide requirements can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle services, and in particular to a ROS-based four-rotor tour guide unmanned aerial vehicle system and a tour guide assisting method thereof. BACKGROUND

[0002] With the rapid development of smart tourism and unmanned system technology, more and more scenic spots have begun to introduce intelligent equipment to improve the tour guide service experience. The Robot Operating System (ROS) provides a solid technical foundation for realizing tour guide unmanned aerial vehicles with high collaboration, autonomous decision-making capabilities, and multi-language interaction functions due to its modularity, scalability, and powerful communication mechanisms. However, despite the great potential of ROS in the technical aspect, the current ROS-based tour guide unmanned aerial vehicle systems applied in tourism guide scenes still have significant deficiencies in terms of intelligent degree and service mode innovation. Specifically, the intelligent degree is insufficient, making it difficult to deeply understand and dynamically generate content based on tourist characteristics, real-time environmental changes, or points of interest, and lacking natural, smooth, and context-aware conversation capabilities, resulting in poor tourist experience; the service mode is single, mostly limited to fixed route voice explanations or simple demonstrations, and unable to provide personalized, immersive, or collaborative guide services. These technical problems have affected the tour guide unmanned aerial vehicle in the smart tourism scene to play its due value, making it difficult to meet the growing demand for deep, personalized, and interactive experiences for tourists. SUMMARY

[0003] The present application provides a ROS-based four-rotor tour guide unmanned aerial vehicle system and a tour guide assisting method thereof, which solves the technical problem of the current tour guide unmanned aerial vehicle in the smart tourism scene, which lacks intelligent interactive capabilities and has a single service mode, making it difficult to meet the needs of tourists' play experience.

[0004] The present application provides a ROS-based four-rotor tour guide unmanned aerial vehicle system, comprising: an input module, a sensing module, a ROS module, a large model module, a driving module, an output module, and a four-rotor structure.

[0005] The input module, the sensing module, the ROS module, the driving module, and the output module are all installed on the four-rotor structure.

[0006] The input module is configured to collect external instructions.

[0007] The sensing module is configured to collect state information of the four-rotor structure.

[0008] The ROS module is connected to the input module, the sensing module, the driving module, the large model module, and the output module.

[0009] The ROS module is configured to determine whether the external instruction is an interactive instruction, determine whether the external instruction is a control instruction if the external instruction is the interactive instruction, generate a control parameter signal according to the external instruction and the state information and send the control parameter signal to the driving module if the external instruction is the control instruction, and upload the external instruction and the state information to the large model module, generate a tour guide auxiliary output voice signal according to personalized tour guide script information returned by the large model module, and send the tour guide auxiliary output voice signal to the output module if the external instruction is not the control instruction.

[0010] The large model module is configured to generate personalized tour guide script information according to the external instruction and the state information.

[0011] The driving module is configured to drive the quadrotor structure according to the control parameter signal to adjust the state of the quadrotor structure.

[0012] Further, the quadrotor structure comprises a fuselage, a front arm, a rear arm and a landing gear assembly.

[0013] The fuselage is provided with the front arm and the rear arm on both sides.

[0014] One side of the front arm and the rear arm is hinged to the landing gear assembly for folding the landing gear assembly.

[0015] The landing gear assembly is provided with a rotor at a connecting end of the front arm or the rear arm.

[0016] Further, a power supply module is further included.

[0017] The power supply module is arranged in the fuselage, a wireless charging receiving coil is mounted at the bottom of the rear arm, and the wireless charging receiving coil is electrically connected to the power supply module.

[0018] Further, the sensing module comprises a camera, a positioning unit, a pressure sensor, a gravity sensor, a pressure sensor, an inertial measurement unit and a ranging sensor.

[0019] The camera is mounted at the front end of the fuselage for collecting environmental images around the quadrotor structure.

[0020] The positioning unit is arranged in the front arm for collecting positioning information of the quadrotor structure.

[0021] The pressure sensor is arranged at the bottom of the landing gear assembly for collecting pressure state of the landing gear assembly, so as to monitor load distribution when the quadrotor structure lands.

[0022] The gravity sensor is arranged on the top of the landing gear assembly, and is configured to collect attitude data of the quadrotor structure relative to a gravity field.

[0023] The inertial measurement unit is arranged on the top of the fuselage, and is configured to collect flight angular velocity and flight acceleration of the quadrotor structure.

[0024] The ranging sensor is arranged on the front end of the fuselage, and is configured to measure a distance between the quadrotor structure and an obstacle.

[0025] Further, the input module includes a text recognition unit and a voice recognition unit.

[0026] The text recognition unit is configured to collect a text input external instruction.

[0027] The voice recognition unit is configured to collect a voice input external instruction, perform noise reduction processing on the voice input external instruction, generate a noise-reduced external instruction, and convert the noise-reduced external instruction into a text corresponding external instruction according to a preset conversion format.

[0028] Further, the ROS module includes an API interface searching unit, a dynamic modification unit, a format conversion unit, and a program execution unit.

[0029] The API interface searching unit is configured to search for an API interface according to the external instruction and the state information.

[0030] The dynamic modification unit is configured to adjust a control parameter according to the external instruction and the state information, and generate a control parameter signal.

[0031] The format conversion unit is configured to convert the individualized tour guide script information into tour guide auxiliary output information, and generate a tour guide auxiliary output voice signal.

[0032] The program execution unit is configured to call the API interface searched by the API interface searching unit, and load the external instruction and the state information, or load the control parameter signal and / or the tour guide auxiliary output voice signal, so as to realize the ROS program service.

[0033] Further, the external instruction is specifically a basic flight instruction or a shooting instruction.

[0034] The ROS module is specifically configured to generate a control parameter signal according to the external instruction and the state information, and send the control parameter signal to a corresponding driving node in the driving module.

[0035] Further, the external instruction is specifically a charging instruction.

[0036] The ROS module is specifically configured to generate a charging path plan of an optimal charging pile according to the charging instruction and the state information according to a preset path algorithm, generate a control parameter signal based on the charging path plan and send the control parameter signal to the driving module, so that the driving module adjusts the flight state of the quadrotor structure according to the control parameter signal, and when the state information meets state preset data corresponding to the charging path plan, a charging preparation instruction is generated to the driving module, so that the wireless charging receiving coil is docked with a resident area of the optimal charging pile for charging.

[0037] Further, the external instruction is specifically a service request voice instruction; the service request voice instruction is divided into a tour instruction, a question and answer instruction and a translation instruction;

[0038] The large model module comprises a response optimization unit, a tour unit, a question and answer unit and a translation unit;

[0039] The response optimization unit is configured to extract position information, user interaction data, emotional state and expression state according to the external instruction and the state information, and generate a differentiated interpretation script;

[0040] The tour unit is configured to extract an interest label and a tour target according to the tour instruction, collect passenger flow data of the tour target, perform heat map analysis according to the tour target and the passenger flow data, generate a recommended residence time for an interest point corresponding to the interest label, combine the corresponding differentiated interpretation script, and generate personalized tour guide script information of a tour path plan;

[0041] The question and answer unit is configured to parse question and answer context information according to the question and answer instruction and the state information, match a reply text in a preset natural language question and answer knowledge base through the question and answer context information, combine the differentiated interpretation script, and generate personalized tour guide script information of a question and answer;

[0042] The translation unit is configured to parse a text to be translated and a target language instruction according to the translation instruction, translate the text to be translated according to the target language instruction and fuse cultural background annotations, combine the corresponding differentiated interpretation script, and generate personalized tour guide script information of a translation.

[0043] The application also provides a tour guide auxiliary method of a ROS-based quadrotor tour guide unmanned aerial vehicle system, the tour guide auxiliary method is implemented based on any of the quadrotor tour guide unmanned aerial vehicle systems, and the method comprises the following steps:

[0044] Collecting an external instruction and state information of a quadrotor structure;

[0045] Determining whether the external instruction is an interaction instruction; if yes, determining whether the external instruction is a control instruction;

[0046] If yes, a control parameter signal is generated according to the external instruction and the state information; and the quadrotor structure is driven by the driving module according to the control parameter signal to adjust the state of the quadrotor structure.

[0047] If no, personalized tour guide script information is generated by the large model module according to the external instruction and the state information; and the personalized tour guide script information is converted and output as tour guide auxiliary output voice signals.

[0048] From the above technical solutions, the present application has the following advantages:

[0049] The present application provides a ROS-based quadrotor tour guide unmanned aerial vehicle system and a tour guide auxiliary method thereof, wherein the system comprises an input module, a sensing module, a ROS module, a large model module, a driving module, an output module and a quadrotor structure; the input module, the sensing module, the ROS module, the driving module and the output module are all installed on the quadrotor structure; the input module is used to collect external instructions; and the sensing module is used to collect state information of the quadrotor structure.

[0050] The ROS module is connected to the input module, the sensing module, the driving module, the large model module and the output module respectively; the ROS module is used to determine whether the external instruction is an interactive instruction; if yes, it is determined whether the external instruction is a control instruction; if yes, a control parameter signal is generated according to the external instruction and the state information and sent to the driving module; if no, the external instruction and the state information are uploaded to the large model module, a tour guide auxiliary output voice signal is generated according to the personalized tour guide script information returned by the large model module, and the tour guide auxiliary output voice signal is sent to the output module; the large model module is used to generate personalized tour guide script information according to the external instruction and the state information; and the driving module is used to drive the quadrotor structure according to the control parameter signal to adjust the state of the quadrotor structure.

[0051] In the present application, through the cooperative work of software and hardware modules, the ROS module is used as the task scheduling center, and the large model module is introduced at the same time, which can understand and respond to the external instruction and the state of the unmanned aerial vehicle, dynamically generate personalized and situational tour guide content, make the human-machine interaction process more natural, and meet the user's demand for personalized tour guide, such as multi-language interpretation, interest navigation and ecological observation, thereby solving the technical problem that the existing tour guide unmanned aerial vehicle has insufficient intelligent interaction capability and a single service mode in the smart tourism scene, and is difficult to meet the tour experience demand of tourists. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative labor based on the embodiments in the present application shall fall within the scope of protection of the present application.

[0053] Figure 1 A structural schematic diagram of an embodiment of a ROS-based four-rotor tour guide unmanned aerial vehicle system provided by the present application;

[0054] Figure 2 A module block diagram of an embodiment of a ROS-based four-rotor tour guide unmanned aerial vehicle system provided by the present application;

[0055] Figure 3 An interaction schematic diagram of a ROS module and a large model module of a ROS-based four-rotor tour guide unmanned aerial vehicle system provided by the present application;

[0056] Figure 4 A step flowchart of a tour guide assisting method of a ROS-based four-rotor tour guide unmanned aerial vehicle system provided by the present application;

[0057] Wherein, the reference signs are: front arm 1, rear arm 2, landing gear assembly 3, rotor 4 and camera 5. DETAILED DESCRIPTION

[0058] The embodiments of the present application provide a ROS-based four-rotor tour guide unmanned aerial vehicle system and a tour guide assisting method thereof, and are used for solving the technical problem that the existing tour guide unmanned aerial vehicle has insufficient intelligent interaction capability and single service mode in the intelligent tourism scene, and is difficult to meet the playing experience demand of tourists.

[0059] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative labor based on the embodiments in the present application shall fall within the scope of protection of the present application.

[0060] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0061] Unless otherwise expressly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0062] Please refer to Figure 1 and Figure 2 , the present application provides an embodiment of a ROS-based four-rotor tour guide unmanned aerial vehicle system, comprising: an input module, a sensing module, a ROS module, a large model module, a driving module, an output module and a four-rotor structure;

[0063] The input module, the sensing module, the ROS module, the driving module and the output module are all mounted on the four-rotor structure;

[0064] The input module is used to collect external instructions;

[0065] The sensing module is used to collect state information of the four-rotor structure;

[0066] The ROS module is connected to the input module, the sensing module, the driving module, the large model module and the output module respectively;

[0067] The ROS module is used to determine whether the external instruction is an interactive instruction; if so, it is determined whether the external instruction is a control instruction; if so, a control parameter signal is generated according to the external instruction and the state information and sent to the driving module; if not, the external instruction and the state information are uploaded to the large model module, a tour guide auxiliary output voice signal is generated according to the personalized tour guide script information returned by the large model module, and the tour guide auxiliary output voice signal is sent to the output module;

[0068] The large model module is used to generate personalized tour guide script information according to the external instruction and the state information;

[0069] A driving module is configured to drive the quadrotor structure according to the control parameter signal to adjust the state of the quadrotor structure.

[0070] It should be noted that the external instruction can be a voice instruction, a text instruction, a noise instruction, etc. If the external instruction is not an interactive instruction, it indicates that the external instruction is an invalid instruction (such as environmental noise), and the ROS module can continue to be in a state of waiting for an instruction; if the external instruction is an interactive instruction, it is further determined whether the external instruction needs to be parsed by the large model module; if the external instruction is a simple interactive instruction (i.e., a control instruction, such as flying up 2 meters), the ROS module only needs to perform simple parsing processing and execute a corresponding control process; if the external instruction is a complex interactive instruction (such as recommending a comfortable parent-child play route), the large model module needs to perform personalized parsing.

[0071] The state information includes an environmental image of the quadrotor structure, positioning information, a pressure state, attitude data, flight angular velocity, flight acceleration, and ranging information, etc., and the state information reflects the flight state of the quadrotor structure and the environmental state in which the quadrotor structure is located.

[0072] In addition, the large model module mainly implements a personalized inference function based on a Qwen series large model, and the large model module provided in this embodiment is deployed on a Jetson embedded platform, can realize edge computing inference, reduce response delay, and improve privacy protection.

[0073] The present application aims to provide more intelligent, flexible, and safe tour guide services for scenes such as scenic spots and museums, improve the experience of tourists, and reduce operating costs. In this embodiment, the quadrotor tour guide unmanned aerial vehicle system has intelligentization, high adaptability, and strong interaction capability, and through the collaborative work of software and hardware modules, the ROS module is used as a task scheduling center to realize functions such as task decomposition and instruction forwarding; at the same time, the large model module is introduced, which can generate personalized tour guide scripts according to real-time states and instruction contexts, respond to semantic intentions of users, and provide differentiated service experiences; the perception, decision-making, execution, and interaction are organically integrated, which can meet the needs of users for personalized tour guide services such as multi-language interpretation, interest navigation, and ecological observation, thereby improving the experience of users during the tour.

[0074] In a preferred embodiment, please refer to Figure 1 The quadrotor structure includes a fuselage, a front arm 1, a rear arm 2, and a landing gear assembly 3.

[0075] The fuselage is provided with the front arm 1 and the rear arm 2 on both sides.

[0076] One side of the front arm 1 and the rear arm 2 is hinged with the landing gear assembly 3, for realizing folding of the landing gear assembly 3.

[0077] The landing gear assembly 3 is provided with a rotor 4 on the connecting end of the front arm 1 or the rear arm 2.

[0078] In the preferred embodiment, the front arm 1 and the rear arm 2 are symmetrically distributed on both sides of the fuselage, and the fuselage is connected by the front arm 1 and the rear arm 2 while carrying the rotor 4. The front arm 1 and the rear arm 2 constitute the flight framework of the unmanned aerial vehicle system, and provide sufficient extension space for the rotor 4 to ensure that the rotors 4 do not interfere with each other during rotation, and at the same time play a role in balancing and stabilizing the unmanned aerial vehicle during flight. Among them, the front arm 1 and the rear arm 2 adopt a telescopic structure, and the landing gear assembly 3 of the rear arm 2 can be made of carbon fiber + shock-absorbing rubber.

[0079] In a preferred embodiment, it also includes a power supply module;

[0080] The power supply module is arranged in the fuselage; the bottom of the rear arm 2 is provided with a wireless charging receiving coil; and the wireless charging receiving coil is electrically connected with the power supply module.

[0081] In the preferred embodiment, the quad-rotor tour guide unmanned aerial vehicle system integrates a wireless charging docking function. When the wireless charging receiving coil of the rear arm 2 is aligned with the ground transmitting end in the residence area of the charging pile on the ground, the automatic charging function is realized, thereby prolonging the endurance of the quad-rotor tour guide unmanned aerial vehicle. It can be understood that when the quad-rotor tour guide unmanned aerial vehicle system is charging, the quad-rotor structure needs to be safely locked to prevent displacement during charging and thereby interrupt the charging.

[0082] In addition, the power supply module includes a battery and a battery fixing assembly, wherein the battery fixing assembly includes a buckle type fixing protrusion, a spring with adjustable pre-tightening force, a cylinder with an insulating coating, and a base provided with anti-slip patterns, and the spring pressure is self-adaptive to the size of the battery during installation. Considering the heat generated during wireless charging and component operation, the rear arm 2 is also provided with a motor cooling hole.

[0083] In a preferred embodiment, please refer to Figure 1 The sensing module includes a camera 5, a positioning unit, a pressure sensor, a gravity sensor, a pressure sensor, an inertial measurement unit, and a ranging sensor.

[0084] The camera 5 is installed at the front end of the fuselage and is used to collect environmental images around the quad-rotor structure; wherein the camera 5 supports video live broadcast, photography, and provides a privacy protection function; the camera 5 can adopt an Intel RealSense depth camera 5 to provide three-dimensional environmental perception through USB connection.

[0085] The positioning unit is arranged in the front arm 1 and is used for collecting positioning information of the quadrotor structure; wherein, the positioning unit can adopt a double-antenna design to realize centimeter-level positioning accuracy, thereby enhancing the signal stability of the GPS Beidou dual-mode navigation and improving the positioning accuracy;

[0086] The pressure sensor is arranged at the bottom of the landing gear assembly 3 and is used for collecting the pressure state of the landing gear assembly 3, thereby monitoring the load distribution of the quadrotor structure during landing;

[0087] The gravity sensor is arranged at the top of the landing gear assembly 3 and is used for collecting attitude data of the quadrotor structure relative to the gravity field, thereby ensuring the take-off and landing safety of the unmanned aerial vehicle;

[0088] The inertial measurement unit (IMU) or barometer is arranged at the top of the fuselage and is used for collecting flight angular velocity and flight acceleration of the quadrotor structure; wherein, the inertial measurement unit can adopt Bosch BNO055 type inertial measurement unit to provide attitude and motion detection through an I2C interface connection.

[0089] The ranging sensor is arranged at the front end of the fuselage and is used for measuring the distance between the quadrotor structure and the obstacle; wherein, the ranging sensor can be a laser radar (LiDAR), which detects the front obstacle in real time based on the active remote sensing technology (RPLIDAR) and realizes ranging. In addition, the environmental map information can also be constructed according to the real-time ranging information and environmental image information.

[0090] Through the collaborative layout of the multiple sensors of the sensing module, comprehensive and high-precision state information input is provided for the decision and control of the ROS module, so that the unmanned aerial vehicle system can realize stable flight, autonomous navigation, safe take-off and landing, and intelligent interaction functions. Specifically, the front camera and the ranging sensor ensure the front environment perception and obstacle avoidance; the pressure sensor and the gravity sensor distributed on the landing gear assembly provide key data for smooth landing and ground attitude monitoring; the inertial measurement unit (IMU) and the positioning unit jointly provide core navigation data of flight attitude and position.

[0091] In a preferred embodiment, the input module includes a text recognition unit and a voice recognition unit;

[0092] The text recognition unit is used for collecting external instructions of text input;

[0093] The voice recognition unit is used for collecting external instructions of voice input and performing noise reduction processing on the external instructions of voice input to generate noise-reduced external instructions; and converting the noise-reduced external instructions into external instructions corresponding to text according to a preset conversion format.

[0094] It can be understood that the quadcopter tour guide unmanned aerial vehicle system further comprises a Bluetooth module and a microphone, the Bluetooth module is arranged in the front arm 1, and the microphone is installed at the front end of the fuselage; the text recognition unit is connected with the Bluetooth module, the voice recognition unit is connected with the Bluetooth module and the microphone respectively, so that the text recognition unit can connect the user's mobile phone APP or the remote control equipment of the control station by means of the Bluetooth / WIFI antenna technology of the Bluetooth module to obtain the external instruction of the text input; the voice recognition unit can connect the user's Bluetooth earphone by means of the Bluetooth module to obtain the external instruction of the user's voice input, and the voice recognition unit can also receive the voice instruction of the user by means of the microphone. In addition, the external instruction of the voice input also needs to be subjected to noise reduction and format conversion processing by the voice recognition unit, so as to improve the efficiency and accuracy of subsequent instruction processing.

[0095] In the preferred embodiment, the output module is connected with the Bluetooth module and the microphone respectively, and the tour guide auxiliary output voice signal can be transmitted to the user through the Bluetooth module or the microphone; meanwhile, the output module is based on a TTS engine, and the supported languages and dialects include but are not limited to English, Japanese, Russian, Korean, French, Portuguese, Spanish, German, Italian and other foreign languages, as well as Sichuan dialect, Cantonese, Shanghai dialect, Jiangxi dialect, Hunan dialect, Hakka dialect, Minnan dialect and other domestic dialects, so as to meet the language needs of different users. In addition, the Bluetooth module supports the BLE 5.2 protocol.

[0096] In a preferred embodiment, please refer to Figure 3 , the ROS module comprises an API interface searching unit, a dynamic modification unit, a format conversion unit and a program execution unit;

[0097] The API interface searching unit is used for searching for an API interface according to the matching of the external instruction and the corresponding API interface information;

[0098] The dynamic modification unit is used for adjusting the control parameter according to the external instruction and the state information, and generating a control parameter signal;

[0099] The format conversion unit is used for converting the individualized tour guide script information into tour guide auxiliary output information, and generating a tour guide auxiliary output voice signal;

[0100] The program execution unit is used for calling the API interface searched by the API interface searching unit, and loading the external instruction and the state information, or loading the control parameter signal and / or the tour guide auxiliary output voice signal, so as to realize the ROS program service.

[0101] In the preferred embodiment, the API interface searching unit finds the control interface node under ROS based on natural language analysis of the external instruction, and realizes automatic conversion of "instruction -> behavior" according to the analysis result; the dynamic modification unit modifies the flight / navigation related parameters (such as flight height, heading adjustment) through dynamic script, and improves the flexibility of task response; the format conversion unit converts the personalized tour guide script information in text form into voice form of tour guide auxiliary output information according to the preset conversion format, and converts the readable text instructions generated by the large model module into voice signals for TTS broadcast, so as to realize voice output of personalized tour guide information; the program execution unit executes the modular task distribution function, and runs the navigation, shooting, voice broadcast and other functional modules in parallel based on the ROS node, so as to realize multi-task scheduling and execution.

[0102] Taking the shooting instruction as an example, the API interface searching unit maps the natural language keywords (such as "shooting panorama") in the shooting instruction to the shooting API interface information, so as to find the shooting API interface ( / camera_control); then, the dynamic modification unit adjusts the control parameters according to the shooting instruction and the state information of the quadcopter structure, such as adjusting the flight height to 10m and the flight speed to 5m / s and the shooting angle in real time through the Python script; finally, the program execution unit calls the ROS service (such as / navigate_to_poi) to output the control parameter signal to the driving module to trigger the shooting planning.

[0103] The application can provide users with functions such as shooting control, question answering, scenic commentary and route planning. The following further describes the functions involved in the basic flight instruction, the shooting instruction, the charging instruction and the service request voice instruction.

[0104] In a preferred embodiment, the external instruction is specifically a basic flight instruction or a shooting instruction.

[0105] The ROS module is specifically configured to generate a control parameter signal according to the external instruction and the state information, and send the control parameter signal to the corresponding driving node in the driving module.

[0106] The basic flight instruction can be a basic instruction for controlling the take-off, landing, hovering, flight direction adjustment and the like of the quadcopter structure; and the ROS module can determine the control parameters based on the PID algorithm, such as the rotation speed of the propeller of each rotor 4, the folding angle of the landing gear assembly 3 and the rotation angle of the gimbal of the camera 5, etc.

[0107] Tourists can watch the live video or take photos of the scenic spot through the mobile phone APP or other terminal devices in real time, which can record the beautiful moment. The application also provides a privacy protection setting of closing the face recognition, and the user can choose not to identify and record the face information when taking photos, so as to protect the personal privacy. In addition, the endangered animals and plants can be observed through the low-interference shooting mode, which can not only meet the observation demand of the user for the rare species, but also will not cause interference to the living environment of the rare species, so as to protect the ecological environment of the park.

[0108] In a preferred embodiment, the external instruction is specifically a charging instruction;

[0109] The ROS module is specifically used for generating a charging path planning of the best charging pile according to the charging instruction and the state information according to a preset path algorithm, generating a control parameter signal based on the charging path planning and sending the control parameter signal to the driving module, so that the driving module adjusts the flight state of the quadrotor structure according to the control parameter signal; when the state information meets the state preset data corresponding to the charging path planning, a charging preparation instruction is generated to make the driving module make the wireless charging receiving coil and the residence area of the best charging pile butt joint to charge.

[0110] The four-rotor guide unmanned aerial vehicle system provided by the application has the functions of low-power automatic return and automatic obstacle avoidance, which are used to ensure the flight safety of the unmanned aerial vehicle; for the low-power alarm and automatic return function, when the battery power is lower than the set threshold (such as 20%), the unmanned aerial vehicle automatically sends an alarm, and plans the shortest path to return to the take-off point or the designated safe landing point, so as to ensure that the unmanned aerial vehicle will not crash due to power consumption; for the obstacle detection and obstacle avoidance function, the front obstacles are detected in real time through devices such as laser radar and ultrasonic sensor, and when the obstacles are detected, the flight path is automatically adjusted or the unmanned aerial vehicle is suspended to avoid collision.

[0111] The state information meeting the state preset data corresponding to the charging path planning specifically includes that the positioning information, pressure state, attitude data, flight angular velocity and flight acceleration of the quadrotor structure all meet the state preset data, and the overlap rate of the environment image of the quadrotor structure and the environment image of the best charging pile in the state preset data meets a preset overlap value, which indicates that the four-rotor guide unmanned aerial vehicle system can perform the charging operation.

[0112] In a preferred embodiment, please refer to Figure 3 The external instruction is specifically a service request voice instruction; the service request voice instruction is divided into a guide instruction, a question and answer instruction and a translation instruction;

[0113] The large model module includes a response optimization unit, a guide unit, a question and answer unit and a translation unit;

[0114] A response optimization unit is configured to extract location information, user interaction data, emotional state and expression state according to external instructions and state information, and generate a differentiated interpretation script;

[0115] A guide unit is configured to extract an interest label and a guide target according to guide instructions, collect passenger flow data of the guide target, perform heat map analysis according to the guide target and the passenger flow data, generate a recommended stay time for an interest point corresponding to the interest label, and generate a personalized tour guide script information of a guide path planning in combination with the corresponding differentiated interpretation script;

[0116] A question and answer unit is configured to parse question and answer context information according to question and answer instructions and state information, match a reply text in a preset natural language question and answer knowledge base through the question and answer context information, generate a personalized tour guide script information of a question and answer in combination with the differentiated interpretation script;

[0117] A translation unit is configured to parse a text to be translated and a target language instruction according to a translation instruction, translate the text to be translated according to the target language instruction and fuse cultural background annotations, and generate a personalized tour guide script information of a translation in combination with the corresponding differentiated interpretation script.

[0118] It should be noted that the large model has powerful semantic understanding and content generation capabilities, can fuse image, voice, text and other information, and realize advanced functions such as natural language driven multi-round interaction, interest guide and dynamic planning.

[0119] In the large model module of the preferred embodiment, the response optimization unit realizes a user attribute alignment function, which extracts a GPS location (such as a tourist being located in a museum), historical interaction data (such as a preference for art), a real-time emotional state (such as a rapid voice tone indicating a shortened answer) and an expression state (such as an anxious state) according to external instructions and state information, thereby generating a corresponding differentiated interpretation script (such as a brief and clear interpretation script) for the user; the question and answer unit can answer a question raised by the user according to a preset natural language question and answer knowledge base, such as "Who is the author of this sculpture?"; the guide unit can recommend a route in combination with a tourist interest (such as a historical and cultural preference) and dynamically avoid a congested area, such as performing heat map analysis to dynamically avoid a congested area and recommending a stay time based on a point of interest (POI) according to a tourist interest label (such as "Tang Dynasty architecture") and real-time passenger flow based on LiDAR detection; and the translation unit provides a real-time translation accuracy of greater than or equal to 98% (Chinese-English mutual translation), supports cultural background annotations of a scenic spot, and can accurately translate scenic spot information into a language specified by a tourist and provide detailed interpretation.

[0120] Considering the differences in functional requirements of the quadcopter tour guide unmanned aerial vehicle system in different intelligent tourism scenarios, the following further describes the functional expansion of the present application:

[0121] 1. Large Model Module

[0122] Other large models that support multilingual and multimodal interaction (such as GPT-4o, Claude-3, Yi-VL, etc.) can be used as alternatives to the Qwen series; at the same time, large models can be lightweighted through model quantization, pruning, knowledge distillation and other techniques to make them more suitable for embedded deployment and reduce the computing power requirements and energy consumption of airborne equipment; a continuous learning mechanism based on user feedback can be introduced to improve the personalization of the guide content and the accuracy of interaction.

[0123] 2. ROS module

[0124] Robot middleware such as ROS 2 or CyberRT, which are more suitable for industrial applications, can be used to improve real-time communication and system stability; a task management framework (such as SMACH or behavior tree) can be introduced to achieve state control and fault-tolerant recovery for complex tour guide tasks.

[0125] 3. Wireless charging mechanism

[0126] Visual-guided docking charging can be used, which combines visual recognition to locate ground charging points; or, long-distance magnetic resonance charging can be used to improve the charging range and docking fault tolerance; or, solar wings can be used to assist charging to improve range in open-air scenarios.

[0127] 4. Enhanced Flight and Positioning Systems

[0128] Multi-source fusion positioning methods such as RTK-GPS, visual inertial SLAM, and lidar SLAM can be introduced to improve navigation accuracy in dense crowds or weak GNSS environments; infrared / ultrasonic modules are integrated into the landing gear to achieve accurate landing and obstacle detection in nighttime / high reflective scenarios.

[0129] 5. Camera

[0130] Add a thermal imaging camera or near-infrared module for nighttime ecological observation or security patrol; optional variable zoom gimbal for both high-altitude inspection and close-up interpretation.

[0131] The present invention provides a ROS-based quadcopter tour guide drone system, which has the following advantages:

[0132] 1. This invention integrates the dynamic parameter adjustment and node execution mechanism under the ROS system, which can match navigation, shooting, narration and other tasks in real time according to natural language instructions and dynamically adjust the execution process to achieve semantic-driven task execution.

[0133] 2. The Jetson platform deploys a multi-modal large model to realize inference and control loop on the end side, avoid data transmission to the cloud, protect user privacy and security, and improve the real-time performance and robustness of the system response; natural language understanding is realized through the large model module, personalized content is dynamically generated, such as multi-lingual recognition of tourist voice instructions, semantic interaction and stylized interpretation, the interaction intelligence is improved, and compared with the existing system with only keyword triggering or fixed voice playback, the naturalness of interaction and the personalization of service are greatly enhanced.

[0134] 3. The application combines tourist interest tags (such as historical and cultural preferences, ecological concerns) with real-time data of scenic spots, generates customized routes and guide content with the help of a large model module, optimizes the guide path, and significantly improves the personalized service level and tourist satisfaction compared with the uniform guide path in the prior art.

[0135] 4. The application realizes intelligent energy compensation between tasks through the automatic wireless charging docking mechanism of the landing gear and the residence area, significantly improves the endurance efficiency and the degree of automation of the unmanned aerial vehicle system.

[0136] 5. Based on privacy protection functions (such as face recognition off) and silent observation modes (suitable for endangered animal shooting), the application solves the problems of public privacy protection and ecological disturbance control in the prior art, and expands the use boundary of unmanned aerial vehicles in natural protection scenarios.

[0137] Please refer to Figure 4 The application also provides a guide assisting method of the ROS-based quad-rotor guide unmanned aerial vehicle system, which is implemented based on the quad-rotor guide unmanned aerial vehicle system.

[0138] Step 101: Collect external instructions and state information of the quad-rotor structure.

[0139] It can be understood that the external instructions can be voice instructions, text instructions, noise instructions, etc.; the state information includes environmental images, positioning information, pressure states, attitude data, flight angular velocity and flight acceleration of the quad-rotor structure, and the state information reflects the flight state of the quad-rotor structure and the environmental state thereof.

[0140] If the external instruction is a voice instruction, the external instruction also needs to be preprocessed, such as denoising and format conversion;

[0141] Taking a scenic spot guide instruction as an example, in order to reduce the interference of background noise and make the voice content easier to understand, voice denoising is performed before the voice instruction is converted into a text instruction, and the voice denoising is a process of improving the voice quality by reducing or eliminating the noise in the voice signal. The time domain denoising method is adopted, which can be realized through a moving average filter, the window size is dynamically adjusted according to the signal characteristics, and the voice denoising process can be described as formula (1):

[0142] (1)

[0143] wherein, is the denoised signal, is the original signal, is the window radius, indicating that points are taken from the current point and points are taken forward and backward.

[0144] Then the voice recognition model is used to convert the user voice instruction input into a text input instruction . Wherein, the generation process can be described as formula (2):

[0145] (2)

[0146] Step 102, judge whether the external instruction is an interactive instruction; if yes, judge whether the external instruction is a control instruction.

[0147] In this embodiment, it is judged whether the external instruction is an interactive instruction; if it is an interactive instruction, it is an effective instruction; then it is judged whether the external instruction is a control instruction, if not, the large model module needs to be used for instruction analysis processing.

[0148] Step 103, if yes, generate a control parameter signal according to the external instruction and the state information; drive the quadrotor structure through the driving module according to the control parameter signal to adjust the state of the quadrotor structure.

[0149] In this embodiment, the quadrotor structure is driven through the driving module according to the control parameter signal to adjust the state of the quadrotor structure.

[0150] Step 104, if not, generate personalized tour guide script information through the large model module according to the external instruction and the state information; convert and output the tour guide auxiliary output voice signal.

[0151] In this embodiment, the personalized tour guide script information is generated through the large model module according to the external instruction and the state information; the personalized tour guide script information is converted and output as a tour guide auxiliary output voice signal.

[0152] ​Continuing with the example of scenic spot guide instructions, the large model module generates text for a specific theme, input sequence , output sequence , the specific process can be described as formula (3):

[0153] (3)

[0154] For route planning, consider soft and hard constraints and typical routes.

[0155] Large model planning process: use the encoder function (such as BERT-base) to convert the combined (concatenated, Concat) text of user information, soft and hard constraints, and typical routes and map information into vectors and vectors , respectively.

[0156] Where, soft and hard constraints and typical routes are stored in the vector database Fasis in the form of text, and then obtained through a similarity query, i.e. Loc(location) represents the location (name string) obtained through GPS information, then the most similar soft and hard constraint name is obtained through Top-k(CosineSimilarity()), and then the typical route pi is obtained through the soft and hard constraints cons and the location; finally, ; this process converts each text instruction sequence into a vector representation for easy processing by the large model module, as shown in formula (4):

[0157]

[0158] (4)

[0159] The vector after the encoder process is input into the encoder part of the large model. This process processes the entire input sequence, generating a hidden state representing the entire input information, as shown in formula (5):

[0160] (5)

[0161] Then use the decoder in the large model to process the hidden state and the current time step , and generate an operation step . This process can be seen as the model reasoning on the input information and generating output based on the reasoning results, as shown in formula (6):

[0162] (6)

[0163] Finally, the generated sequence of operation plans is returned as the final output. This sequence contains the specific steps, taking into account the efficient use of time and resources, in the form of .

[0164] In summary, the entire process can be represented by equation (7):

[0165] (7)

[0166] Finally, Qwen generates a conversation based on the obtained plan and user attributes, i.e. PlanAdvise(Plan, UserProfile), where PlanAdvise() is the response generation function; UserProfile and the text generation process are described in the introduction section of the AttractIntro() function.

[0167] The application provides a tour guide auxiliary method of a ROS-based four-rotor tour guide unmanned aerial vehicle system, which responds to external instructions according to requirements, dynamically provides personalized tour guide services and realizes simple control interaction instructions in combination with state information, and significantly improves the intelligence, naturalness and richness of the tourist tour experience.

[0168] The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A ROS-based quadcopter guide drone system, characterized in that, The application relates to a four-rotor unmanned aerial vehicle (UAV) with a large model module, and belongs to the field of UAVs. The application relates to a four-rotor unmanned aerial vehicle (UAV) with a large model module, and belongs to the field of UAVs. The input module is used for collecting external instructions. The sensing module is used for collecting state information of the four-rotor structure. The ROS module is connected with the input module, the sensing module, the driving module, the large model module and the output module. If the external instruction is not an interactive instruction, the external instruction and the state information are uploaded to the large model module, a personalized tour guide script information is generated according to the large model module, a tour guide auxiliary output voice signal is generated according to the personalized tour guide script information, and the tour guide auxiliary output voice signal is sent to the output module. The large model module is used for generating personalized tour guide script information according to the external instruction and the state information. The driving module is used for driving the four-rotor structure according to the control parameter signal to adjust the state of the four-rotor structure. The four-rotor structure comprises a fuselage, front arms, rear arms and a landing gear assembly. The fuselage is provided with the front arms and the rear arms on both sides.

2. The quadcopter tour guide drone system of claim 1, wherein, The front arms and the rear arms are hinged with the landing gear assembly on one side, so that the landing gear assembly can be folded. The landing gear assembly is provided with a rotor on the connecting end of the front arm or the rear arm. The power supply module is arranged in the fuselage, and a wireless charging receiving coil is arranged at the bottom of the rear arm and electrically connected with the power supply module. The sensing module comprises a camera, a positioning unit, a pressure sensor, a gravity sensor, a pressure sensor, an inertial measurement unit and a distance measuring sensor.

3. The quadcopter tour guide drone system of claim 2, wherein, The camera is arranged at the front end of the fuselage and used for collecting environmental images around the four-rotor structure. The positioning unit is arranged in the front arm and used for collecting positioning information of the four-rotor structure.

4. The rotor guided tour UAV system of claim 2, wherein, The pressure sensor is arranged at the bottom of the landing gear assembly and used for collecting the pressure state of the landing gear assembly, so as to monitor the load distribution of the four-rotor structure during landing. The gravity sensor is arranged at the top of the landing gear assembly and used for collecting attitude data of the four-rotor structure relative to the gravity field. The inertial measurement unit is arranged at the top of the fuselage and used for collecting flight angular velocity and flight acceleration of the four-rotor structure. The distance measuring sensor is arranged at the front end of the fuselage and used for measuring the distance between the four-rotor structure and an obstacle. The input module comprises a text recognition unit and a voice recognition unit. The text recognition unit is used for collecting external instructions of text input. ​ 5. The quadcopter tour guide drone system of claim 1, wherein, ​ ​ The voice recognition unit is configured to collect voice input external instructions, perform noise reduction processing on the voice input external instructions, and generate noise-reduced external instructions. The noise-reduced external instructions are converted into text corresponding external instructions according to a preset conversion format.

6. The quadcopter tour guide drone system of claim 1, wherein, The ROS module includes an API interface searching unit, a dynamic modification unit, a format conversion unit, and a program execution unit. The API interface searching unit is configured to search for an API interface according to the external instructions and matching API interface information. The dynamic modification unit is configured to adjust a control parameter according to the external instructions and the state information, and generate a control parameter signal. The format conversion unit is configured to convert the individualized tour guide script information into tour guide auxiliary output information, and generate a tour guide auxiliary output voice signal. The program execution unit is configured to call the API interface searched by the API interface searching unit, and load the external instructions and the state information, or load the control parameter signal and / or the tour guide auxiliary output voice signal, so as to realize the ROS program service.

7. The quadcopter tour guide drone system of claim 1, wherein, The external instructions are specifically basic flight instructions or shooting instructions. The ROS module is specifically configured to generate a control parameter signal according to the external instructions and the state information, and send the control parameter signal to a corresponding driving node in the driving module.

8. The quadcopter tour guide drone system of claim 3, wherein, The external instructions are specifically charging instructions. The ROS module is specifically configured to generate a charging path plan of an optimal charging pile according to the charging instructions and the state information according to a preset path algorithm, generate a control parameter signal based on the charging path plan and send the control parameter signal to the driving module, so that the driving module adjusts the flight state of the quadcopter structure according to the control parameter signal; when the state information meets state preset data corresponding to the charging path plan, a charging preparation instruction is generated to the driving module, so that the wireless charging receiving coil is docked with a resident area of the optimal charging pile for charging.

9. The quadcopter tour guide drone system of claim 1, wherein, The external instructions are specifically service request voice instructions; the service request voice instructions are divided into tour guide instructions, question and answer instructions, and translation instructions. The large model module includes a response optimization unit, a tour guide unit, a question and answer unit, and a translation unit. The response optimization unit is configured to extract location information, user interaction data, emotional state, and expression state according to the external instructions and the state information, and generate a differential interpretation script. The tour guide unit is configured to extract an interest label and a tour guide target according to the tour guide instructions, collect passenger flow data of the tour guide target, perform heat map analysis according to the tour guide target and the passenger flow data, generate a recommended residence time for an interest point corresponding to the interest label, combine a corresponding differential interpretation script, and generate individualized tour guide script information of a tour guide path plan. The question and answer unit is configured to parse question and answer context information according to the question and answer instructions and the state information, match a reply text in a preset natural language question and answer knowledge base through the question and answer context information, combine the differential interpretation script, and generate individualized tour guide script information of a question and answer. The translation unit is used for parsing the text to be translated and target language instructions according to the translation instructions; and generating a translated personalized tour guide script information by translating the text to be translated according to the target language instructions, fusing cultural background annotations, and combining corresponding differential explanation scripts.

10. A tour guide assisting method based on a ROS-based quad-rotor tour guide UAV system, the tour guide assisting method being implemented based on the quad-rotor tour guide UAV system of any one of claims 1-9, characterized in that, The method comprises: Collecting external instructions and state information of the quadrotor structure; Determining whether the external instructions are interactive instructions; if yes, determining whether the external instructions are control instructions; If yes, generating a control parameter signal according to the external instructions and the state information; and driving the quadrotor structure to adjust the state of the quadrotor structure by a driving module according to the control parameter signal; If no, generating personalized tour guide script information by the large model module according to the external instructions and the state information; and converting and outputting tour guide auxiliary output voice signals from the personalized tour guide script information.

Images