A method and system for collaborative interaction between a manned vehicle and an unmanned vehicle based on voice recognition

Abstract

The application discloses a manned aircraft and unmanned aircraft cooperative interaction method and system based on voice recognition, and the method comprises the following steps: collecting the voice signal of the control instruction of the operator to the controlled unmanned aircraft at the manned aircraft end, and performing voice recognition according to the voice recognition model constructed in advance based on the unmanned aircraft cooperative task instruction set to obtain text data; performing semantic understanding on the obtained text data, and converting the text data into a task command and corresponding control parameters; encoding the obtained task command and control parameters, and sending the task command and control parameters to the controlled unmanned aircraft to control the unmanned aircraft to perform actions according to the task command and control parameters; and collecting the feedback information of the controlled unmanned aircraft at the manned aircraft end, encoding the feedback information to form a feedback instruction, and obtaining the feedback information of the controlled unmanned aircraft after analyzing the feedback instruction. The application has the advantages of simple implementation method, high intelligent degree, high interaction control efficiency, strong flexibility, safety and reliability and the like.

Description

Technical Field

[0001] This invention relates to the field of collaborative control technology for flight systems, and in particular to a method and system for collaborative interaction between manned and unmanned aircraft based on voice recognition. Background Technology

[0002] With the continuous development of sensor technology, wireless communication technology, and intelligent control technology, drones have been widely used in various application fields. Current technologies typically employ a "remote control from a drone ground station" model, where a ground station communicates with the drone to remotely control it to perform required tasks. However, due to current limitations in intelligence, drones cannot yet autonomously perform diverse tasks in complex environments. Furthermore, communication between the ground station and the drone suffers from drawbacks such as command delays, control and data link interruptions.

[0003] To address the aforementioned issues, a manned-unmanned aerial vehicle (UAV) collaborative approach can be adopted. This involves a manned aircraft commanding and controlling the UAV, effectively shortening the UAV data link, reducing the risk to manned pilots, and compensating for the UAV's lack of autonomy, thus significantly improving its adaptability in various complex environments. However, current technologies for manned-unmanned aerial vehicle (UAV) collaboration typically involve operators on the manned aircraft processing information from the ground control station, operating the aircraft itself, and handling environmental information during mission execution. They then manually control the UAV via a console for mission planning and coordination. This approach is not only complex and difficult to implement, but also inefficient in terms of interactive control, and poses certain risks to the safe flight of both manned and UAVs. Summary of the Invention

[0004] The technical problem to be solved by this invention is: in view of the technical problems existing in the prior art, this invention provides a manned and unmanned aerial vehicle collaborative interaction method and system based on voice recognition that is simple to implement, highly intelligent, has high efficiency in interactive control, is flexible, and safe and reliable.

[0005] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows: A method for collaborative interaction between manned and unmanned aerial vehicles based on speech recognition, comprising: On the manned aircraft side, the voice signals of the operator's control commands to the controlled UAV are collected, and the voice is recognized according to a pre-built voice recognition model based on the UAV collaborative task instruction set, which is converted into text data. The UAV collaborative task instruction set is built according to the instructions and parameters required for collaborative tasks between the manned aircraft and the UAV. The converted text data is semantically understood and converted into task commands and corresponding control parameters for controlling the execution of tasks by the controlled UAV. The converted task commands and control parameters are encoded and sent to the controlled UAV to control the controlled UAV to perform actions according to the task commands and control parameters. Additionally, on the manned aircraft side, feedback instructions are generated by encoding feedback information from the controlled drone. After parsing the feedback instructions, the feedback information of the controlled drone is obtained.

[0006] Furthermore, the UAV collaborative task instruction set includes instruction parameters for controlling the parameters required for the required task or action, task instructions for controlling the execution of the required task, and action instructions for controlling the execution of the required action.

[0007] Furthermore, the command parameters include any combination of position parameters, attitude parameters, speed parameters, heading, mission point parameters, UAV number, target parameters, mission parameters, payload status, formation parameters, area parameters, threat parameters, and mission constraint parameters. The formation parameters are used to characterize the formation status, the area parameters are used to characterize the type of area, and the threat parameters are used to characterize the status of threat targets that pose a threat to the UAV.

[0008] Furthermore, the action commands include any combination of categories such as forced execution, position control, attitude control, speed control, and program call. The forced execution action commands include power on / off commands. The position control action commands include commands for ascending and descending, moving forward and backward, turning left and right, and moving in a specified direction. The attitude control action commands include commands for yaw left and right. The speed control action commands include commands for accelerating and decelerating. The program call action commands include commands for takeoff, landing, hovering, and returning to base.

[0009] Furthermore, the mission instructions include any combination of instructions such as cruise, reconnaissance search, payload mode, target tracking and surveillance, reporting current status, attacking target, instruction confirmation, threat warning / incident assessment, increasing constraints, safe return, hovering and waiting, threat avoidance / coordination, mission cancellation, formation flight / formation change, damage assessment, and target guidance and location.

[0010] Furthermore, when performing speech recognition, a convolutional neural network is used to train the recognition model. The input of the convolutional neural network is a 1024-dimensional one-dimensional vector, which is first converted from one-dimensional to two-dimensional to a 32*32 two-dimensional vector before being input into the convolutional network for recognition.

[0011] Furthermore, it also includes generating drones, obstacles, and targets using the XTDrone simulation environment, performing voice recognition on manned / drone voice commands in the XTDrone simulation environment, controlling the actions of the controlled drone based on the recognized task commands and corresponding control parameters, and the drone providing real-time feedback on the current area and status to the manned drone. The XTDrone simulation environment adopts a layered modular architecture, integrating dynamic models, sensor models, control algorithms, state estimation algorithms, and 3D scene models.

[0012] A voice recognition-based manned-machine and unmanned aerial vehicle (UAV) collaborative interaction system includes a manned-machine control device connected to the control system of the controlled UAV. The manned-machine control device includes: The command sending module is used on the manned aircraft to collect the voice signals of the operator's control commands to the controlled UAV, and to perform voice recognition based on a pre-built voice recognition model based on the UAV collaborative task command set to convert the voice signals into text data. The UAV collaborative task command set is constructed based on the command parameters, action commands, and task commands required for collaborative tasks between the manned aircraft and the UAV. The module performs semantic understanding on the converted text data and converts it into task commands and corresponding control parameters for controlling the execution of tasks by the controlled UAV. The module encodes the converted task commands and control parameters and sends them to the controlled UAV. The feedback receiving module is used to collect feedback instructions formed by the encoding of feedback information from the controlled UAV, and to obtain the feedback information of the controlled UAV after parsing the feedback instructions.

[0013] Furthermore, it also includes a UAV control device connected to the manned aircraft control device, used to receive coded data packets of task commands and control parameters sent by the manned aircraft at the controlled UAV end, parse them to obtain task instructions and corresponding control parameters, control the UAV to act according to the task instructions and corresponding control parameters, and encode feedback information and send it to the manned aircraft end.

[0014] A computer system includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method described above.

[0015] Compared with existing technologies, the advantages of this invention are as follows: This invention enables collaborative interaction between manned and unmanned aerial vehicles (UAVs) based on speech recognition. A set of UAV collaborative task instructions oriented towards speech recognition is pre-established and trained to obtain a speech recognition model. During the collaborative interaction between manned and UAVs, the operator's speech signal is collected on the manned aircraft and its features are extracted. The speech recognition model is then used to recognize and convert the data into text data. Semantic understanding is then performed on the text data to extract task commands and control parameters, which are encoded and sent to the controlled UAV to control it to perform corresponding actions. Simultaneously, the manned aircraft receives feedback information from the controlled UAV. This combination of speech recognition and speech recognition enables intelligent collaborative interaction between manned and UAVs, eliminating the need for manual operation by the operator. This significantly improves the convenience, flexibility, and efficiency of collaborative interaction, making it easily applicable to various complex environments for efficient execution of diverse tasks. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the implementation process of the manned-machine and unmanned-machine collaborative interaction method and system based on speech recognition in this embodiment.

[0017] Figure 2 This embodiment presents a schematic diagram illustrating the specific implementation principle of the voice command-controlled drone system.

[0018] Figure 3 This is a schematic diagram illustrating the principle of constructing a drone collaborative instruction set based on speech recognition in this embodiment.

[0019] Figure 4 This is a schematic diagram of the specific structure of the speech recognition network in this embodiment.

[0020] Figure 5 This is a schematic diagram of the specific structure of the XTDrone standalone simulation in this embodiment.

[0021] Figure 6 This is a schematic diagram of the applicable scenarios and route design in specific application embodiments of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0023] like Figure 1 As shown in the figure, the manned-machine and unmanned aerial vehicle collaborative interaction method based on speech recognition in this embodiment includes: On the manned aircraft side, the voice signals of the operator's control commands to the controlled UAV are collected, and the voice is recognized and converted into text data according to the speech recognition model pre-built based on the UAV collaborative task instruction set. The UAV collaborative task instruction set is built according to the instructions and parameters required for collaborative tasks between the manned aircraft and the UAV. The converted text data is semantically understood and converted into task commands and corresponding control parameters for controlling the execution of tasks by the controlled UAV. The converted task commands and control parameters are encoded and sent to the controlled UAV. On the manned aircraft side, the feedback information of the controlled UAV is collected and encoded into feedback commands. The feedback commands are parsed to obtain the feedback information of the controlled UAV. The controlled UAV receives encoded data packets of task commands and control parameters sent by the manned UAV, parses them to obtain the task instructions and corresponding control parameters, controls the UAV to act according to the task instructions and corresponding control parameters, and encodes the feedback information and sends it to the manned UAV.

[0024] Utilizing speech recognition for automatic control offers advantages such as flexibility and natural interaction. This embodiment leverages the characteristics of speech recognition to achieve intelligent collaborative interaction between manned and unmanned aerial vehicles (UAVs). This embodiment pre-establishes a set of UAV collaborative task instructions oriented towards speech recognition and trains it to obtain a speech recognition model. During manned-UAV collaborative interaction, the operator's speech signal is collected on the manned aircraft, and features are extracted. The speech recognition model is then used to recognize and convert the speech signal into text data. Semantic understanding is then performed on the text data to extract task commands and control parameters, which are encoded and sent to the controlled UAV to control it to perform corresponding actions. Simultaneously, the manned aircraft receives feedback information from the controlled UAV. This combination of speech recognition and speech recognition enables intelligent collaborative interaction between manned and unmanned aircraft, eliminating the need for manual operator intervention. This significantly improves the convenience, flexibility, and efficiency of collaborative interaction, making it easily applicable to various complex environments for efficient execution of diverse tasks.

[0025] In a specific application embodiment, the manned aircraft collects the voice signal of the operator's task commands through a human-machine interface. The command understanding module recognizes the voice signal of the task commands and converts it into commands and parameters for task execution. The task instruction encoding module encodes the commands and parameters and sends them to the automatic control system on the UAV platform through a data link to control the UAV to execute the corresponding task. The UAV platform sends platform information and response information to the task instruction encoding module based on the task environment and the inquiry from the manned aircraft. The task instruction encoding module sends the information fed back by the UAV to the task instruction parsing module on the manned aircraft platform through a data link, parses it, and then sends the feedback information to the manned aircraft operator.

[0026] like Figure 2As shown, in this embodiment, when the drone system is controlled based on voice commands, the operator speaks the voice commands to the system according to the commands given in the drone command set. The voice is collected by the microphone and sampled. Then, the sampled digital voice signal is sent to the A / D converter, which performs quantization and encoding. The digitized voice signal is then sent to the PC for data storage and transmitted to the voice command recognition module in file format. The result of voice command recognition is sent to the drone platform through the communication interface. After receiving the result, the controller in the drone platform sends control commands to the drone's actuators, and the drone begins to perform specific actions.

[0027] like Figure 3 As shown, this embodiment, by designing a drone collaborative instruction set, enables simple, flexible, and natural voice command recognition, and achieves high accuracy in voice recognition, minimizing the use of easily confused words; the results of voice recognition make it easier to accurately control the drone.

[0028] The UAV collaborative mission instruction set is specifically constructed based on manned / UAV collaborative missions (collaborative reconnaissance and surveillance, collaborative target strike, collaborative formation). The UAV collaborative mission instruction set includes instruction parameters for controlling the required tasks or actions, task instructions for controlling the execution of the required tasks, and action instructions for controlling the execution of the required actions. In specific application embodiments, the collaboration methods for each collaborative task are as follows: (1) Cooperative reconnaissance and surveillance: When carrying out cooperative reconnaissance and surveillance missions, the manned aircraft, as the decision-making center, is located in the rear safe area, while the UAVs enter the required reconnaissance and surveillance area in formation to carry out preliminary detection tasks. The formation of the UAV formation is adjusted by the manned aircraft, and the flight parameters of the UAVs are adjusted to conduct reconnaissance and surveillance of the target from the overall optimal angle. For multiple different targets in the same mission area, due to the different mission payloads carried by different UAVs and the limitation of the detection range of the sensing equipment, the manned aircraft operator makes reasonable and flexible task allocation for the UAV formation, using different payloads to detect different targets, so as to maximize the completion of the reconnaissance and surveillance mission.

[0029] (2) Coordinated strike on targets: Manned fighter jets and one or more unmanned aerial vehicles (UAVs) carry out coordinated strike on targets. The manned aircraft can be in front of the UAV formation, strike the target and then quickly withdraw from the mission area. Then the UAVs enter the mission area to assess the strike effect on the ground targets. The manned aircraft then issues the order to withdraw from the area or carry out supplementary strikes.

[0030] (3) Coordinated formation: The UAV, as the wingman, takes priority in protecting the manned aircraft as the lead aircraft. When the UAV is damaged, an assessment is conducted to determine whether to continue the mission. The formation needs to be reorganized and the mission needs to be planned. If the manned aircraft is damaged, a decision is made after the damage assessment whether to continue the mission. After it is determined that the mission should continue, the lead aircraft needs to re-establish contact with the ground control station, and the control of the UAV formation is handed over to the ground station.

[0031] In this embodiment, the command parameters specifically include position parameters, attitude parameters, velocity parameters, heading, mission point parameters, UAV number, target parameters, mission parameters, payload status, formation parameters, area parameters, threat parameters, and mission constraint parameters. Among these, formation parameters characterize the formation state, area parameters characterize the type of area, and threat parameters characterize the status of threatening targets that pose a threat to the UAV. The command parameters are shown in Table 1. Table 1 Command Parameters

[0032] In this embodiment, based on commonly used flight maneuvers, the specific configuration of the maneuver commands includes mandatory execution, position control, attitude control, speed control, and program call commands. Maneuver commands in the mandatory execution category include power on / off commands; position control commands include commands for ascent and descent, forward and backward movement, left and right movement, and movement in a specified direction; attitude control commands include commands for yaw; speed control commands include commands for acceleration and deceleration; and program call commands include commands for takeoff, landing, hovering, and return to base. The specific maneuver commands are shown in Table 2.

[0033] Table 2 Action Instructions

[0034] In the table above, the "Power On" and "Power Off" commands are mandatory to ensure flight safety. They have the highest priority when invoked and will interrupt the execution of other flight control programs. "Take Off" and "Land" are commands that directly invoke the takeoff and landing flight control programs without requiring other parameters. "Accelerate" and "Decelerate" are commands to adjust speed, with the parameter being the speed adjustment value in m / s. "Ascend," "Descend," "Forward," "Backward," "Left," "Right," "North," "South," "East," and "West" are commands to control the drone's movement, with parameters being the distance traveled and the speed at which it moves. "Yaw Left" and "Yaw Right" are commands to adjust the drone's attitude, with parameters including the yaw angle and the angular velocity during yaw; the angular velocity can be preset. The "Hover" command will interrupt the drone's ongoing flight maneuver, putting the drone into a hovering state. The "Return to Home" command will return the drone to its starting point and then hover at a predetermined altitude.

[0035] Mission commands are top-level commands designed at the mission level and are relatively complex. In this embodiment, combined with command parameters, the specific configuration of mission commands includes commands such as cruise, reconnaissance and search, payload mode, target tracking and surveillance, reporting current status, attacking target, command confirmation, threat warning / unexpected event assessment, adding constraints, safe return to base, circling and waiting, threat avoidance / coordination, mission cancellation, formation flight / formation change, damage assessment, and target guidance and location. The specific mission commands are shown in Table 3.

[0036] Table 3 Task Instructions

[0037] This embodiment constructs a UAV collaborative task instruction set according to the above-mentioned instruction parameters, action instructions, and task instructions. It can cover various types of tasks and action types. When a manned aircraft needs to control a UAV to perform a specified task, it issues a control command according to the corresponding instructions and control parameters in the UAV collaborative task instruction set. After voice recognition, semantic understanding, and task instruction encoding, the command is sent to the UAV, thereby achieving precise control of various complex tasks.

[0038] Considering the limited number of command sets for manned / unmanned vehicles and the potential for interference in speech signals, this embodiment employs a convolutional neural network to train the recognition model for speech recognition, thereby further improving recognition accuracy. Figure 4As shown, the input to the convolutional neural network is a 1024-dimensional one-dimensional vector. After a one-dimensional to two-dimensional transformation, it is converted into a 32*32 two-dimensional vector, and then fed into the convolutional network for recognition. The aforementioned convolutional neural network can specifically be GG, LeNet, ResNet, etc. Traditional convolutional neural networks take images or two-dimensional matrices as input, while speech signals are one-dimensional signals. This embodiment first transforms the input 1024-dimensional vector into a 32*32 two-dimensional vector before feeding it into the convolutional network for recognition.

[0039] In this embodiment, the speech recognition specifically adopts deep learning-based speech recognition, and the specific steps are as follows: (1) Simplify the instruction set to suit speech recognition: For example, the (longitude, latitude, altitude) in the instruction parameters and location parameters are converted into Region 1 and Region 2, and the threat alarm / accident assessment in the task instruction is converted into two instructions: threat alarm and accident assessment. (2) Speech signal acquisition: Speech is acquired through a speech sensor, and the A / D converter converts the speech analog signal. Each acquired speech segment is marked by a pause in speech. (3) Speech signal preprocessing: each segment of data is preprocessed, including data sampling and Fourier transform, and the output is a 1024-dimensional one-dimensional vector; (4) Speech recognition: By converting a one-dimensional vector into a two-dimensional vector and inputting it into a convolutional neural network for training and testing, the final output of speech recognition results is obtained.

[0040] This embodiment also includes generating drones, obstacles, and targets using the XTDrone simulation environment. Within the XTDrone simulation environment, voice commands from manned / drone operators are recognized. Based on the recognized task commands and corresponding control parameters, the controlled drone's actions are controlled, and the drone provides real-time feedback on the current area and status to the manned drone. For example... Figure 5 As shown, the XTDrone simulation environment in this embodiment adopts a layered modular architecture, integrating models such as dynamics models, sensor models, control algorithms, state estimation algorithms, and 3D scenes. Building a simulated UAV dynamics model through the XTDrone platform can reduce simulation implementation costs, while offering high reliability, efficiency, and ease of maintenance and expansion. Based on ROS, PX4, and Gazebo, XTDrone supports multi-machine and multi-type simulations, including multi-rotor UAVs, fixed-wing UAVs, and compound-wing UAVs, as well as unmanned vehicles, unmanned boats, robotic arms, and various sensors. It allows for flexible simulation verification of various UAV mission algorithms, facilitating the control of the UAV platform using simulated language commands.

[0041] In this embodiment, voice commands are used to control the drone in a simulation environment through a robot operating system. The specific steps are as follows: (1) Generate UAVs, obstacles, and targets using the XTDrone simulation environment; (2) Recognize manned / unmanned voice commands; (3) Control the movement of the drone after voice recognition; (4) The drone provides real-time feedback on the current area and status to prepare for the next step of speech recognition.

[0042] like Figure 6 As shown, in a specific application embodiment, the method described above is used for manned-machine and unmanned aerial vehicle (UAV) collaborative interaction. The route is divided into five segments based on the task (reaching the target location) and obstacles (walls, pillars). The corresponding voice commands for the actions are: 60 degrees to the left, 60 degrees to the right, 0 degrees to the right, 150 degrees to the right, and 0 degrees upward. The UAV provides real-time feedback on the current area and status to prepare for the next step of voice recognition. Table 4 shows the voice commands corresponding to the actions. Testing demonstrates that the method of this invention can accurately and efficiently control manned-machine and UAV collaborative interaction, greatly improving the convenience, flexibility, and efficiency of control.

[0043] Table 4 Voice commands corresponding to the action process

[0044] This embodiment of a voice recognition-based manned-machine and unmanned aerial vehicle (UAV) collaborative interaction system includes a manned-machine control device connected to the control system of the controlled UAV. The manned-machine control device includes: The command sending module is used on the manned aircraft side to collect the voice signal of the operator's control commands to the controlled UAV and perform voice recognition to convert it into text data. The control commands are in a pre-built UAV collaborative task command set, which includes command parameters, action commands, and task commands. The module performs semantic understanding on the converted text data and converts it into task commands and corresponding control parameters for controlling the execution of tasks by the controlled UAV. The module encodes the converted task commands and control parameters and sends them to the controlled UAV. The feedback receiving module is used to collect feedback information encoded from the controlled UAV into feedback commands, and after parsing the feedback commands, obtain the feedback information from the controlled UAV.

[0045] In this embodiment, a UAV control device connected to the manned aircraft control device is also included. The UAV device receives encoded data packets of task commands and control parameters sent by the manned aircraft, parses them to obtain task instructions and corresponding control parameters, controls the UAV to act according to the task instructions and corresponding control parameters, and encodes feedback information and sends it to the manned aircraft.

[0046] This embodiment also provides a computer system, including a processor and a memory, wherein the memory is used to store a computer program and the processor is used to execute the computer program to perform the method described above.

[0047] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for collaborative interaction between manned and unmanned aerial vehicles based on speech recognition, characterized in that, include: On the manned aircraft side, the voice signals of the operator's control commands to the controlled UAV are collected, and the voice is recognized and converted into text data according to the speech recognition model pre-built and trained based on the UAV collaborative task instruction set. The UAV collaborative task instruction set is constructed based on the instructions and parameters required for collaborative tasks between the manned aircraft and the UAV. The UAV collaborative task instruction set includes instruction parameters for controlling the required tasks or actions, task instructions for controlling the execution of the required tasks, and action instructions for controlling the execution of the required actions. The converted text data is semantically understood and converted into task commands and corresponding control parameters for controlling the execution of tasks by the controlled UAV. The converted task commands and control parameters are encoded and sent to the controlled UAV to control the controlled UAV to perform actions according to the task commands and control parameters. On the manned aircraft side, feedback instructions are generated by encoding feedback information from the controlled UAV. After parsing the feedback instructions, feedback information from the controlled UAV is obtained and sent to the manned aircraft operator. On the controlled UAV side, the coded data packets of task commands and control parameters sent by the manned aircraft are received, parsed to obtain task commands and corresponding control parameters, and the UAV is controlled to act according to the task commands and corresponding control parameters. The feedback information is also encoded and sent to the manned aircraft side.

2. The manned-machine and unmanned aerial vehicle collaborative interaction method based on speech recognition according to claim 1, characterized in that, The command parameters include any combination of position parameters, attitude parameters, velocity parameters, heading, mission point parameters, UAV number, target parameters, mission parameters, payload status, formation parameters, area parameters, threat parameters, and mission constraint parameters. The formation parameters are used to characterize the formation status, the area parameters are used to characterize the type of area, and the threat parameters are used to characterize the status of threatening targets that pose a threat to the UAV.

3. The manned-machine and unmanned aerial vehicle collaborative interaction method based on speech recognition according to claim 1, characterized in that, The action commands include any combination of categories such as forced execution, position control, attitude control, speed control, and program call. The forced execution action commands include power on / off commands. The position control action commands include commands for ascending and descending, moving forward and backward, turning left and right, and moving in a specified direction. The attitude control action commands include commands for yaw left and right. The speed control action commands include commands for accelerating and decelerating. The program call action commands include commands for takeoff, landing, hovering, and returning to home.

4. The manned-machine and unmanned aerial vehicle collaborative interaction method based on speech recognition according to claim 1, characterized in that, The mission commands include any combination of commands such as cruise, reconnaissance and search, payload mode, target tracking and surveillance, reporting current status, attacking target, command confirmation, threat warning / incident assessment, increasing constraints, safe return, circling and waiting, threat avoidance / coordination, mission cancellation, formation flight / formation change, damage assessment, and target guidance and location.

5. The manned-machine and unmanned aerial vehicle collaborative interaction method based on speech recognition according to any one of claims 1 to 4, characterized in that, When performing speech recognition, a convolutional neural network is used to train the recognition model. The input of the convolutional neural network is a 1024-dimensional one-dimensional vector, which is first converted from one-dimensional to two-dimensional to a 32*32 two-dimensional vector before being input into the convolutional network for recognition.

6. The manned-machine and unmanned aerial vehicle collaborative interaction method based on speech recognition according to any one of claims 1 to 4, characterized in that, It also includes generating drones, obstacles, and targets using the XTDrone simulation environment, performing voice recognition on manned / drone voice commands in the XTDrone simulation environment, controlling the actions of the controlled drone according to the recognized task commands and corresponding control parameters, and the drone providing real-time feedback on the current area and status to the manned drone. The XTDrone simulation environment adopts a layered modular architecture, integrating dynamic models, sensor models, control algorithms, state estimation algorithms, and 3D scene models.

7. A manned-machine and unmanned aerial vehicle (UAV) collaborative interaction system based on speech recognition, characterized in that, The system includes a manned aircraft control device, which is connected to the control system of the controlled unmanned aerial vehicle (UAV). The manned aircraft control device includes: The command transmission module is used to collect the voice signals of the operator's control commands to the controlled UAV on the manned aircraft side, and perform voice recognition based on a pre-built and trained voice recognition model based on the UAV collaborative task command set to convert the voice signals into text data. The UAV collaborative task command set is constructed with parameters based on the commands required for collaborative tasks between the manned aircraft and the UAV. The UAV collaborative task command set includes command parameters for controlling the required tasks or actions, task commands for controlling the execution of the required tasks, and action commands for controlling the execution of the required actions. The module performs semantic understanding on the converted text data and converts it into task commands and corresponding control parameters for controlling the execution of tasks by the controlled UAV. The converted task commands and control parameters are encoded and sent to the controlled UAV. The feedback receiving module is used to collect feedback instructions formed by the encoding of feedback information from the controlled UAV, parse the feedback instructions to obtain the feedback information of the controlled UAV, and send the feedback information to the manned UAV operator. It also includes a UAV control device connected to the manned aircraft control device, which is used to receive coded data packets of task commands and control parameters sent by the manned aircraft at the controlled UAV end, parse them to obtain task instructions and corresponding control parameters, control the UAV to act according to the task instructions and corresponding control parameters, and encode feedback information and send it to the manned aircraft end.

Images