Unmanned aerial vehicle intelligent interaction method and system based on multi-modal data

Abstract

The invention particularly relates to an unmanned aerial vehicle intelligent interaction method and system based on multi-modal data. The method comprises the following steps: acquiring voice control data of a target unmanned aerial vehicle; performing voice recognition processing on the voice control data, and converting the voice control data into corresponding control text data; performing semantic analysis processing on the control text data to obtain instruction data of the unmanned aerial vehicle; the instruction data of the unmanned aerial vehicle comprises a task instruction executed by the unmanned aerial vehicle and a control parameter corresponding to the task instruction; the control parameters comprise action instruction data and / or flight instruction data; and encoding the instruction data of the unmanned aerial vehicle to obtain remote control encoded data, and sending the remote control encoded data to the target unmanned aerial vehicle to enable the target unmanned aerial vehicle to execute the instruction data of the unmanned aerial vehicle. According to the method, efficient, safe and flexible unmanned aerial vehicle control can be realized, and diversified tasks can be efficiently completed in a complex and changeable environment.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle flight control, and in particular to a method and system for intelligent interaction of unmanned aerial vehicles based on multimodal data. Background Art

[0002] In related technologies, with the rapid development of drone technology, drones have been used in a variety of different application scenarios. Traditional drone control systems mostly rely on human-computer interaction based on preset instructions or graphic click operations. This interaction mode based on preset input instructions limits the freedom of the operator and the drone system, resulting in low interaction efficiency and difficulty in coping with the changing needs in the battlefield environment.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present invention provides a method and system for intelligent interaction of unmanned aerial vehicles based on multimodal data, a storage medium, and a computer program product, which can realize simple and fast interactive control and faster system response, thereby overcoming the defects existing in the prior art to a certain extent.

[0005] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0006] According to a first aspect of the present invention, a method for intelligent interaction of unmanned aerial vehicles based on multimodal data is provided, the method comprising:

[0007] Collect voice control data of the target drone;

[0008] Performing speech recognition processing on the speech control data to convert it into corresponding control text data;

[0009] Perform semantic parsing on the control text data to obtain command data of the drone; the command data of the drone includes: a task instruction for the drone to execute, and control parameters corresponding to the task instruction; the control parameters include: action instruction data and / or flight instruction data;

[0010] The command data of the drone is encoded to obtain remote control encoded data, and the remote control encoded data is sent to the drone so that the drone executes the command data of the drone.

[0011] In some exemplary embodiments, performing speech recognition processing on the speech control data includes:

[0012] Performing speech recognition processing on the speech control data using the trained Whisper speech recognition model;

[0013] The Whisper speech recognition model is trained based on command sample data; the command sample data includes a list of drone remote control commands based on different dialects.

[0014] In some exemplary embodiments, the remote control instruction list includes mission instructions, action instruction data, and flight instruction data.

[0015] In some exemplary embodiments, the control text data is semantically parsed using a trained general information extraction model to extract the task instructions corresponding to the control text data and the control parameters corresponding to the task instructions.

[0016] In some exemplary embodiments, the action instruction data includes any one or a combination of multiple of the following action instructions: communication configuration type, flight control type, recovery control type, link control type, navigation control type, and mission planning type;

[0017] The communication configuration action instructions include instructions for drone initialization and drone command cycle;

[0018] Flight control commands include take-off, level and direct flight, and orientation;

[0019] Recovery control commands include parking and opening the parachute cabin;

[0020] Link control action instructions include adjusting the servo azimuth angle and link power;

[0021] Navigation control action instructions include emergency time setting and emergency return mode setting;

[0022] Mission planning action instructions include binding cruise routes, binding recovery routes, and binding electronic fences.

[0023] In some exemplary embodiments, the mission instructions include any one or a combination of any two of the following tasks: UAV cruise, target reconnaissance, payload mode, target tracking, target attack, command confirmation, return recovery, and damage assessment.

[0024] In some exemplary embodiments, the flight instruction data includes any one or a combination of any two or more of the following:

[0025] Position parameters, attitude parameters, speed parameters, drone number, target parameters, payload type, area parameters, route parameters, mission point parameters, and mission constraint parameters;

[0026] Among them, the regional parameters are used to describe the type of area where the drone is located and its related characteristics.

[0027] In some exemplary embodiments, the method further comprises:

[0028] Acquire a current image fed back by the UAV; the current image includes any one or a combination of multiple items of flight parameters, location information, sensor data, mission status, and battlefield situation information;

[0029] Performing OCR recognition processing on the current image data to obtain text feature data corresponding to the current image;

[0030] Performing semantic analysis on the text feature data to obtain a semantic analysis result;

[0031] An auxiliary control instruction is generated based on the semantic parsing result, the auxiliary control instruction is encoded to obtain auxiliary coding data, and the auxiliary coding data is sent to the drone.

[0032] In some exemplary embodiments, the method further comprises:

[0033] Get the task plan data corresponding to the current task;

[0034] Performing semantic parsing on the task plan data to obtain corresponding task requirement data and operation instruction data;

[0035] A task instruction set is constructed based on the task requirement data and the operation instruction data.

[0036] According to a second aspect of the present invention, there is provided a drone intelligent interaction system based on multimodal data, comprising:

[0037] Voice data acquisition module, used to collect voice control data of the target UAV;

[0038] A speech recognition module, used to perform speech recognition processing on the speech control data and convert it into corresponding control text data;

[0039] A semantic parsing module, used to perform semantic parsing on the control text data to obtain command data of the UAV; the command data of the UAV includes: a task instruction for the UAV to execute, and control parameters corresponding to the task instruction; the control parameters include: action instruction data and / or flight instruction data;

[0040] The encoding module is used to encode the command data of the drone to obtain remote control coded data, and send the remote control coded data to the drone so that the drone executes the command data of the drone.

[0041] According to a third aspect of the present invention, there is provided a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the above-mentioned intelligent interaction method of drones based on multimodal data is implemented.

[0042] According to a fourth aspect of the present invention, there is provided a computer program product having a computer program stored thereon, which implements the above-mentioned intelligent interaction method of unmanned aerial vehicles based on multimodal data when the computer program is executed by a processor.

[0043] The intelligent interaction method for drones based on multimodal data provided by the embodiment of the present invention collects the user's voice control data of the target drone in real time, performs voice recognition processing on it and converts it into control text data; then performs semantic parsing processing on the control text data to obtain the command data of the drone; and encodes the command data and sends the corresponding remote control coded data to the corresponding target drone, so that the target drone can execute the command data in time. This method can realize the semantic understanding of voice control data to generate dynamic commands, assist users in adjusting and optimizing control commands, and improve the flexibility of command input, so that the drone reduces the reliance on manual operation of the operator during task execution, and realizes efficient, safe and flexible drone control, so that it can efficiently complete diversified tasks in complex and changeable environments.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0046] Figure 1 A schematic diagram schematically illustrates an exemplary embodiment of the present invention, a method for intelligent interaction of unmanned aerial vehicles based on multimodal data;

[0047] Figure 2 A schematic diagram schematically illustrates a drone interaction process according to an exemplary embodiment of the present invention;

[0048] Figure 3 A schematic diagram of an intelligent interaction system for unmanned aerial vehicles based on multimodal data according to an exemplary embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0049] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present invention will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0050] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0051] Among the related technologies, speech recognition and natural language processing technologies have been gradually applied to smart devices. However, there are still many challenges in directly applying them to drone control systems. Especially when facing the scenario of drones performing specific tasks, voice input needs to be quickly and accurately converted into automated operation instructions. However, the existing system has limited command generation and understanding capabilities and cannot meet complex voice interaction requirements. At the same time, there are also problems in actual scenarios such as recognizing regional dialects and Mandarin with local accents, which puts higher requirements on the speech recognition system.

[0052] In view of the shortcomings and deficiencies of the prior art, this example implementation provides a drone intelligent interaction method based on multimodal data, which realizes drone intelligent interaction that is easy to use, convenient to interact, has a fast system response speed, and is adaptable to diversified voice input. Figure 1 As shown, the UAV intelligent interaction method based on multimodal data may specifically include the following steps:

[0053] Step S11, collecting voice control data of the target drone;

[0054] Step S12, performing speech recognition processing on the speech control data and converting it into corresponding control text data;

[0055] Step S13, performing semantic parsing processing on the control text data to obtain command data of the UAV; the command data of the UAV includes: task instructions for the UAV to execute, and control parameters corresponding to the task instructions; the control parameters include: action instruction data and / or flight instruction data;

[0056] Step S14, encoding the command data of the drone to obtain remote control coded data, and sending the remote control coded data to the target drone so that the target drone executes the command data of the drone.

[0057] Below, each step of the drone intelligent interaction method based on multimodal data in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0058] In step S11, voice control data of the target UAV is collected.

[0059] Exemplarily, the above intelligent interaction method can be applied to an intelligent terminal device or a background server to collect the operator's voice commands to the drone, and send the data to the corresponding target drone after data processing. The intelligent terminal device or the background server can communicate data with the drone device.

[0060] For example, the target drone may be a drone device; or multiple drone devices, such as multiple drone devices in a drone formation. Taking the smart terminal device as a ground station as an example, the operator's voice control data of the target drone can be collected in real time. The voice control data may include specific command information.

[0061] In step S12, the voice control data is processed by voice recognition and converted into corresponding control text data.

[0062] Exemplarily, performing speech recognition processing on the voice control data includes: performing speech recognition processing on the voice control data using a trained Whisper speech recognition model; wherein the Whisper speech recognition model is trained based on instruction sample data; and the instruction sample data includes a list of drone remote control instructions based on different dialects.

[0063] Specifically, the Whisper speech recognition model based on the Transformer architecture can be pre-trained for speech recognition of the voice control data collected in real time. Among them, multiple drone remote control command lists containing different dialects can be pre-configured, and the list can include voice data; and the drone remote control command list can be constructed based on the drone mission execution requirements, including specific instructions and required parameters; and it is used as training sample data, and the Whisper speech recognition model is trained using the training sample data based on multiple different dialects. By considering the language differences between different users and the noise in the ground station scene, the Whisper speech recognition model is used to recognize the real-time collected voice data, and a customized training strategy for Chinese content and different dialects is adopted to enhance the model performance of the speech recognition model. It supports multi-language speech recognition, has strong robustness, and can efficiently process speech signals in noisy environments.

[0064] Exemplarily, the remote control instruction list includes mission instructions, action instruction data, and flight instruction data.

[0065] In step S13, the control text data is semantically parsed to obtain the command data of the UAV; the command data of the UAV includes: mission instructions for the UAV to execute, and control parameters corresponding to the mission instructions; the control parameters include: action instruction data and / or flight instruction data.

[0066] Exemplarily, the semantic recognition processing of the control text data includes: using a trained general information extraction model to perform semantic analysis on the control text data to extract task instructions corresponding to the control text data, and control parameters corresponding to the task instructions.

[0067] Specifically, a Universal Information Extraction (UIE) model for semantic analysis of control text data can be pre-trained to extract key task elements contained in the control text data, such as task type and specific execution parameters of the current task.

[0068] Exemplarily, the mission instructions include: any one or a combination of any two of the following missions: UAV cruise, target reconnaissance, payload mode, target tracking, target attack, command confirmation, return recovery, and damage assessment.

[0069] Specifically, the mission command can be a command designed for the drone to complete based on the mission to be performed, which is used to describe the specific mission type of the flight mission currently performed by the target drone. Combined with the drone flight command parameters, the specific configuration of the mission command includes drone cruise, target reconnaissance, payload mode, target tracking, target attack, command confirmation, return recovery, damage assessment and other commands. The mission command and the corresponding specific command parameters are shown in Table 1.

[0070] Table 1

[0071]

[0072]

[0073] Exemplarily, the action instruction data includes any one or a combination of multiple of the following action instructions: communication configuration class, flight control class, recovery control class, link control class, navigation control class, and mission planning class.

[0074] Specifically, the action instruction data can be used to indicate the specific action instruction for currently controlling the UAV. Referring to Table 2, in combination with commonly used flight actions, the categories of the action instructions may include: any one or a combination of multiple action instructions of communication configuration, flight control, recovery control, link control, navigation control, and mission planning. Each type of action instruction may include a variety of different action instruction names and corresponding parameter compositions.

[0075] Table 2

[0076]

[0077]

[0078] Among them, the action instructions of communication configuration include instructions for UAV initialization and UAV command cycle; the action instructions of flight control include instructions for take-off, level flight, direct flight, orientation, manual control, program control and specifying flight direction; the action instructions of recovery control include parking, opening the parachute cabin, throwing the parachute, etc.; the action instructions of link control include instructions for adjusting the servo azimuth angle, adjusting the link power, etc.; the action instructions of navigation control include instructions for emergency time setting, emergency return mode setting, etc.; the action instructions of mission planning include instructions for binding cruise routes, binding recovery routes, binding electronic fences, etc.

[0079] Among them, in the above instructions, different priorities are pre-configured between each instruction. If there are multiple instructions with different priorities at the same time, each instruction will be executed in sequence according to the priority of the instruction. For example, the "bind front end" and "bind ground station position" instructions have the highest priority when called to ensure normal communication between the aircraft and the ground station. "Parking", "opening the parachute cabin", and "shooting the parachute" are recovery control instructions, and the positions of each point in the recovery process must be planned in advance in the route. "Near-end antenna servo azimuth angle adjustment", "far-end antenna servo azimuth angle adjustment", "link uplink high power", "near-end omnidirectional", and "near-end directional" are link control instructions, which ensure the stable locking of the communication channel by adjusting the servo state. "Bind cruise route", "Bind recovery route", and "Bind electronic fence" are mission planning instructions, and the parameters include preset routes and airspaces.

[0080] Exemplarily, the flight instruction data includes any one or a combination of any two of the following: position parameters, attitude parameters, speed parameters, UAV number, target parameters, load type, area parameters, route parameters, mission point parameters, and mission constraint parameters; wherein the area parameters are used to describe the type of area in which the UAV is located and its related characteristics.

[0081] Specifically, the above-mentioned flight instruction data parameters may be one or more specific flight instructions for remotely controlling a UAV. Each type of flight instruction parameter and corresponding parameter content are shown in Table 3.

[0082] Table 3

[0083] Serial number Parameter Type Parameter content 1 Positional parameters Longitude, Latitude, Altitude 2 Attitude parameters Pitch angle, roll angle, yaw angle, heading angle 3 Speed ​​Parameters Take-off speed, level flight speed, descent speed, target speed 4 Drone Number Used to describe and distinguish different drones 5 Target parameters Target number, target type, target location, target damage level 6 Load Type Optoelectronics, radar, infrared 7 Regional parameters Airspace scope, electronic fence 8 Route parameters Take-off route, recovery route 9 Mission point parameters Mission point coordinates, entry and exit methods 10 Task Constraint Parameters Time constraints, fuel constraints, distance constraints

[0084] For example, a drone mission instruction set is constructed through the above-mentioned mission instructions, action instructions and flight instructions, which can cover various types of tasks and action types. When it is necessary to control the drone to perform a task, commands are issued according to the corresponding instructions and parameters in the drone mission instruction set. Through voice recognition, semantic understanding and image processing technology, intelligent interactive control and multimodal information processing of the drone can be realized.

[0085] In step S14, the command data of the drone is encoded to obtain remote control coded data, and the remote control coded data is sent to the target drone so that the target drone executes the command data of the drone.

[0086] For example, after obtaining accurate instruction data through semantic analysis, it can be encoded to generate remote control coded data; then the remote control coded data is sent to one or more target drones. Thus, the target drone can parse the remote control coded data after receiving it, and perform corresponding actions according to the set remote control instructions and parameters, thereby achieving flexible control of the target drone.

[0087] Exemplarily, the method further includes:

[0088] Step S21, obtaining a current image fed back by the UAV; the current image includes any one or a combination of multiple items of flight parameters, location information, sensor data, mission status, and battlefield situation information;

[0089] Step S22, performing OCR recognition processing on the current image data to obtain text feature data corresponding to the current image;

[0090] Step S23, performing semantic analysis on the text feature data to obtain a semantic analysis result;

[0091] Step S24: generating an auxiliary control instruction based on the semantic parsing result, encoding the auxiliary control instruction to obtain auxiliary coding data, and sending the auxiliary coding data to the drone.

[0092] Specifically, when the UAV performs a task, the current image data can be collected in real time through the camera, and the current flight parameters, position information, sensor data, task status, and battlefield situation information can be embedded in the current image data. For example, the collected current image and the corresponding flight parameters, position information, sensor data, task status, and battlefield situation information can be displayed in the display interface or display control interface of the UAV. The UAV can feedback images containing various types of information to the ground station or the intelligent control terminal. After receiving the current image fed back in real time, the ground station can use the OCR (Optical Character Recognition) module to perform character feature recognition on the current image, extract key visual information and convert it into a corresponding text description. As shown in Table 4, the flight parameters may include the current flight altitude, speed, heading angle, pitch angle and other real-time flight data of the UAV; the position information may include longitude and latitude, GPS coordinates, etc.; the sensor data may be the temperature, humidity, air pressure, altitude and other data collected by the temperature and humidity sensors installed on the UAV; the mission status may be the remaining ammunition and fuel remaining of the UAV; the battlefield situation data may be the identified local location, friendly position, terrain data, etc.

[0093] Table 4

[0094] Serial number Image information type Specific content 1 Flight parameters Flight altitude, speed, heading angle, pitch angle, etc. 2 Location Latitude and longitude, GPS coordinates, etc. 3 Sensor data Temperature, humidity, air pressure, etc. 4 Task Status Ammunition remaining, fuel status, etc. 5 Battlefield situation Enemy position, friendly position, terrain

[0095] Specifically, the current image fed back by the UAV is first preprocessed by denoising and contrast enhancement to improve the accuracy of OCR recognition; secondly, the text area in the image is located using image processing algorithms such as edge detection and contour analysis; then the located text area is identified using OCR technology to extract flight parameters, geographic location information, etc. as shown in Table 4; finally, the extracted text data is parsed into structured information. The extracted structured information is input into the UIE semantic understanding model to parse the image information content and generate auxiliary control instructions. For example, the flight path and target indication can be adjusted according to the battlefield situation information, the reconnaissance interval can be adjusted according to the sensor data, and the warning prompt of insufficient ammunition remaining can be issued according to the mission status information. Combined with real-time visual feedback and parsing results, users can obtain dynamic auxiliary information to adjust the operation instructions according to the surrounding environment of the UAV and enhance the real-time response capability of the task.

[0096] By inputting the text data extracted by OCR into the UIE semantic understanding model to parse the image information content and generate auxiliary control instructions. Combining real-time visual feedback and parsing results, users can obtain dynamic auxiliary information to adjust the operation instructions according to the drone's surrounding environment and enhance the real-time response capability of the task.

[0097] By transmitting the generated mission instructions and control parameters to the UAV control system through the encoding module, the UAV performs corresponding operations according to the received instructions. At the same time, by receiving and processing the return data and feedback information from the UAV, the ground station operator can timely understand the progress of the mission and update or adjust the instructions.

[0098] Exemplarily, the method further comprises:

[0099] Step S31, obtaining task plan data corresponding to the current task;

[0100] Step S32, performing semantic parsing on the task plan data to obtain corresponding task requirement data and operation instruction data;

[0101] Step S33, constructing a task instruction set based on the task requirement data and operation instruction data.

[0102] Specifically, before the UAV performs a flight mission, it can first obtain the corresponding mission plan data and extract text information from it. By semantically understanding and processing the extracted text information, it can obtain the corresponding mission requirement data and operation instruction data; wherein, the mission requirement data can be used to determine the corresponding mission type, corresponding to the mission instruction; the operation instruction data can be the specific action instructions, flight instructions, and execution order required to execute the flight mission and complete each mission instruction. According to the determined mission instructions, action instructions, and flight instructions, a specific mission instruction set adapted to the UAV can be generated. Among them, the mission instruction set can include instructions, as well as specific instruction parameters corresponding to each directive.

[0103] In addition, this instruction set can be used for subsequent voice control of the drone.

[0104] The intelligent interaction method of unmanned aerial vehicle based on multimodal data provided by the present invention is referred to Figure 2 As shown, the voice signal of the user's control command to the drone can be collected in real time, and the voice recognition model trained and optimized by the drone task command list can be used to recognize it and convert it into corresponding text data. The drone task command list is constructed based on the drone task execution requirements, including specific instructions and required parameters; the converted text data is semantically understood, the user's intention is extracted, and it is converted into instructions and corresponding parameters for drone task execution to ensure that the drone accurately performs the target operation; the generated task instructions and control parameters are encoded, and the encoded commands and parameters are sent to the drone, so that the drone performs the corresponding actions according to the set task instructions and parameters.

[0105] This method enhances the adaptive ability of UAVs in the process of executing tasks by combining image recognition and semantic understanding technology. By pre-establishing a set of UAV collaborative task instruction sets for image feature extraction and semantic analysis and performing model training, efficient analysis of image feature data is achieved. During the execution of UAV tasks, the real-time collected image data is extracted and converted into text data through feature extraction of the image recognition model, and dynamic instructions are generated through semantic understanding to assist users in adjusting and optimizing control instructions. At the same time, combined with the semantic analysis of the preset combat plan, an executable instruction set is generated, providing highly flexible instruction input for task execution. This method enables the UAV to make dynamic adjustments according to environmental feedback during the execution of the task, improving the accuracy and intelligence level of the task. The whole process reduces the reliance on manual operation of the operator, realizes efficient, safe and flexible UAV control, and enables it to efficiently complete diversified tasks in complex and changing environments.

[0106] It should be noted that the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present invention, rather than for limiting purposes. It is easy to understand that the processes shown in the above-mentioned drawings do not indicate or limit the chronological order of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0107] Furthermore, referring to Figure 3 as shown, in the embodiment of this example, there is also provided an intelligent interaction system 30 for an unmanned aerial vehicle based on multi-modal data, including:

[0108] A voice data acquisition module 301, configured to acquire voice control data for a target unmanned aerial vehicle;

[0109] A voice recognition module 302, configured to perform voice recognition processing on the voice control data and convert it into corresponding control text data;

[0110] A semantic parsing module 303, configured to perform semantic parsing processing on the control text data to obtain instruction data for the unmanned aerial vehicle; the instruction data for the unmanned aerial vehicle includes: task instructions for the unmanned aerial vehicle to execute, and control parameters corresponding to the task instructions; the control parameters include: action instruction data and / or flight instruction data;

[0111] An encoding module 304, configured to perform encoding processing on the instruction data for the unmanned aerial vehicle to obtain remote control encoding data, and send the remote control encoding data to the unmanned aerial vehicle so that the unmanned aerial vehicle executes the instruction data for the unmanned aerial vehicle.

[0112] The functional implementations of the various modules in the intelligent interaction system 30 for the unmanned aerial vehicle have been elaborated in detail in the corresponding method embodiments, and will not be repeated here.

[0113] It should be noted that although several modules or units of devices for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present invention, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0114] Specifically, according to the embodiments of the present invention, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments of the present invention include a computer program product, which includes a computer program carried on a storage medium, and the computer program includes program codes for executing the methods shown in the flowcharts.

[0115] It should be noted that the storage medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any storage medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0116] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0118] It should be noted that, as another aspect, the present application also provides a storage medium, which can be included in an electronic device; or it can exist independently without being installed in the electronic device. The above storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement the following Figure 1 The individual steps of the method are shown.

[0119] In one embodiment, the present application provides a computer program product, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0120] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0121] Other embodiments of the invention will readily occur to those skilled in the art after considering the specification and practicing the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0122] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A drone intelligent interaction method based on multimodal data, characterized in that: The method comprises: Collect voice control data of the target drone; Performing speech recognition processing on the speech control data to convert it into corresponding control text data; Perform semantic parsing on the control text data to obtain command data of the drone; the command data of the drone includes: a task instruction for the drone to execute, and control parameters corresponding to the task instruction; the control parameters include: action instruction data and / or flight instruction data; The command data of the drone is encoded to obtain remote control encoded data, and the remote control encoded data is sent to the target drone so that the target drone executes the command data of the drone.

2. The method according to claim 1, characterized in that: The performing speech recognition processing on the speech control data comprises: Performing speech recognition processing on the speech control data using the trained Whisper speech recognition model; The Whisper speech recognition model is trained based on command sample data; the command sample data includes a list of drone remote control commands based on different dialects.

3. The method according to claim 2, characterized in that The remote control instruction list includes mission instructions, action instruction data, and flight instruction data.

4. The method according to claim 1, characterized in that: The performing semantic recognition processing on the control text data includes: The control text data is semantically parsed using a trained general information extraction model to extract the task instructions corresponding to the control text data and the control parameters corresponding to the task instructions.

5. The method according to claim 3 or 4, characterized in that: The action instruction data includes any one or a combination of multiple ones of the following action instructions: communication configuration, flight control, recovery control, link control, navigation control, and mission planning; Communication configuration action instructions include drone initialization and drone command cycle instructions; Flight control commands include take-off, level and direct flight, orientation, and control mode; Recovery control commands include parking, opening the parachute cabin, and dropping the parachute; Link control action instructions include adjusting the servo azimuth angle and link power; Navigation control action instructions include emergency time setting and emergency return mode setting; Mission planning action instructions include binding cruise routes, binding recovery routes, and binding electronic fences.

6. The method according to claim 3 or 4, characterized in that: The mission instructions include: any one or a combination of any two of the following tasks: UAV cruise, target reconnaissance, payload mode, target tracking, target attack, command confirmation, return recovery, and damage assessment.

7. The method according to claim 3 or 4, characterized in that: The flight instruction data includes any one or a combination of any two or more of the following: Position parameters, attitude parameters, speed parameters, drone number, target parameters, payload type, area parameters, route parameters, mission point parameters, and mission constraint parameters; Among them, the regional parameters are used to describe the type of area where the drone is located and its related characteristics.

8. The method according to claim 1, characterized in that The method further comprises: Acquire a current image fed back by the UAV; the current image includes any one or a combination of multiple items of flight parameters, location information, sensor data, mission status, and battlefield situation information; Performing OCR recognition processing on the current image data to obtain text feature data corresponding to the current image; Performing semantic analysis on the text feature data to obtain a semantic analysis result; An auxiliary control instruction is generated based on the semantic parsing result, the auxiliary control instruction is encoded to obtain auxiliary coding data, and the auxiliary coding data is sent to the drone.

9. The method according to claim 1, characterized in that: The method further comprises: Get the task plan data corresponding to the current task; Performing semantic parsing on the task plan data to obtain corresponding task requirement data and operation instruction data; A task instruction set is constructed based on the task requirement data and the operation instruction data.

10. An intelligent interactive system for unmanned aerial vehicles based on multimodal data, characterized in that: The system comprises: Voice data acquisition module, used to collect voice control data of the target UAV; A speech recognition module, used to perform speech recognition processing on the speech control data and convert it into corresponding control text data; A semantic parsing module, used to perform semantic parsing on the control text data to obtain command data of the UAV; the command data of the UAV includes: a task instruction for the UAV to execute, and control parameters corresponding to the task instruction; the control parameters include: action instruction data and / or flight instruction data; The encoding module is used to encode the command data of the drone to obtain remote control coded data, and send the remote control coded data to the drone so that the drone executes the command data of the drone.

Images