Unmanned system control method, multi-modal virtual rocker assembly, and related devices

Abstract

This invention belongs to the field of human-computer interaction technology for mobile terminals, and discloses a control method for unmanned systems, a multimodal virtual joystick component, and related devices. The method includes receiving control inputs from various modalities and parsing them into instructions with a unified data format. These instructions include an intent field, a control parameter field, a source field, a confidence field, and a timestamp field. Based on a preset control strategy, the current instruction is arbitrated and fused with the next instruction to obtain the control instructions for the unmanned system, which are then sent to the unmanned system. The control strategy includes a priority arbitration strategy and a complementary fusion strategy for combining different instructions. By representing instructions with a unified data format and combining the dual strategies of priority arbitration and complementary fusion, the limitations of single-modal control and the conflict problem of multimodal inputs are solved, achieving organic fusion and collaborative decision-making of multimodal inputs, effectively improving the control efficiency, accuracy, and robustness of unmanned systems in complex interactive scenarios.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology for mobile terminals, and relates to a control method for unmanned systems, a multimodal virtual joystick component, and related devices. Background Technology

[0002] With the rapid development of human-computer interaction (HCI) technology, users' demand for efficient, convenient, and natural input methods is increasing daily. Multimodal interaction, by integrating multiple input channels such as touch, voice, and gestures, can significantly improve the naturalness and efficiency of HCI and has become an important development direction in the field of HCI. In the control scenarios of unmanned systems (such as drones and autonomous vehicles), virtual joysticks are widely used as an intuitive control component. However, there are still many shortcomings in the development of multimodal virtual joystick components in existing technologies.

[0003] Specifically, the single touch operation of existing virtual joystick components often fails to meet users' demands for efficiency and accuracy in complex human-computer interaction scenarios, especially when multiple input sources need to be processed simultaneously. The lack of effective spatiotemporal synchronization and conflict arbitration mechanisms between different input sources greatly limits the performance and application scope of multimodal interaction. For example, when processing multimodal input, a simple sequential or fixed priority strategy is often adopted. This simple conflict handling method cannot adapt to complex and ever-changing interaction scenarios and cannot truly achieve the fusion and collaborative work of multimodal inputs. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an unmanned system control method, a multimodal virtual joystick assembly, and related devices.

[0005] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, the present invention provides a control method for an unmanned system, comprising: receiving control inputs from various modalities; parsing the control inputs from various modalities into instructions with a unified data format; wherein the instructions include an intent field, a control parameter field, a source field, a confidence field, and a timestamp field; arbitrating and fusing the current instruction with the next instruction according to a preset control strategy to obtain control instructions for the unmanned system and sending them to the unmanned system; wherein the control strategy includes a priority arbitration strategy and a complementary fusion strategy for combining different instructions.

[0006] Optionally, the modal includes touch commands, voice commands, and touch and voice fusion commands; the control input for receiving each modal includes: receiving touch commands using a virtual joystick interface, receiving voice commands using a voice recognition module, and receiving touch and voice fusion commands using both a virtual joystick interface and a voice recognition module.

[0007] Optionally, the virtual joystick interface is a virtual joystick interface rendered based on the Canvas component of HarmonyOS ArkUI; the voice recognition module is the HarmonyOS SpeechKit voice recognition service module.

[0008] Optionally, when the modality is a touch command, parsing the control input of each modality into a command with a unified data format includes: obtaining the original coordinates of the touch point of the touch command relative to the screen; converting the original coordinates into relative coordinates relative to the center point of the virtual joystick, and dividing the relative coordinates by the maximum active radius of the virtual joystick to obtain a normalized vector; when the normalized vector is less than a preset dead zone threshold of the center point of the virtual joystick, determining that the control input is invalid; otherwise, applying a preset response curve to the normalized vector and mapping it to the control parameters of the unmanned system, and filling the control parameters of the unmanned system into the control parameter field of the command.

[0009] Optionally, when the modality is a voice command, the step of parsing the control input of each modality into a command with a unified data format includes: acquiring the voice command and converting it into text, and performing natural language understanding processing on the text to obtain the intent of the voice command, the control parameters of the unmanned system, and the confidence level of the voice command, and filling them into the intent field, control parameter field, and confidence level field of the command, respectively.

[0010] Optionally, the priority arbitration strategy includes: obtaining the priority and confidence level of the current instruction and the next instruction; wherein, the priority of an instruction whose modality is touch control is lower than the priority of an instruction whose modality is voice control; when the priority of the next instruction is the highest priority, the next instruction is used as the control instruction of the unmanned system and sent to the unmanned system; otherwise, it is checked whether the current instruction is empty; when the current instruction is empty, the next instruction is used as the control instruction of the unmanned system and sent to the unmanned system; otherwise, it is checked whether the difference between the confidence level of the next instruction and the confidence level of the current instruction is greater than a preset threshold; when the difference between the confidence level of the next instruction and the confidence level of the current instruction is greater than a preset confidence gap threshold, the next instruction is used as the control instruction of the unmanned system and sent to the unmanned system; otherwise, a complementary fusion strategy is executed.

[0011] Optionally, the complementary fusion strategy includes: acquiring the intent of the current instruction and the next instruction; when the intents of the current instruction and the next instruction are complementary, averaging or weighting the control parameters of the current instruction and the next instruction based on priority to generate a fused instruction and send it to the unmanned system as the control instruction of the unmanned system; when the intents of the current instruction and the next instruction conflict, using the instruction with higher priority between the current instruction and the next instruction as the control instruction of the unmanned system and sending it to the unmanned system.

[0012] In a second aspect, the present invention provides a multimodal virtual joystick component, comprising: an interaction module for receiving control inputs from various modalities; a modal parsing module for parsing the control inputs from various modalities into instructions with a unified data format; wherein the instructions include an intent field, a control parameter field, a source field, a confidence field, and a timestamp field; and a fusion decision engine module for arbitrating and fusing the current instruction with the next instruction according to a preset control strategy to obtain control instructions for the unmanned system and sending them to the unmanned system; wherein the control strategy includes a priority arbitration strategy and a complementary fusion strategy for combining different instructions.

[0013] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the unmanned system control method.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the unmanned system control method.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention provides an unmanned system control method that parses control inputs from various modalities into instructions in a unified data format, including intent, control parameters, source, confidence level, and timestamp. This allows heterogeneous inputs from different modalities to be understood and compared at the same semantic level, laying a data foundation for subsequent arbitration and fusion. By employing a control strategy that includes a priority arbitration strategy and a complementary fusion strategy to arbitrate and fuse adjacent instructions, the method addresses two key issues. First, the priority arbitration strategy makes reasonable decisions based on preset rules when multimodal instructions conflict, avoiding the control chaos caused by first-come-first-served or fixed priorities in existing technologies, significantly improving control determinism and security in multimodal scenarios. Second, the complementary fusion strategy combines different instructions to generate a single composite control instruction, achieving synergistic effects of multimodal inputs rather than simple overlap. This effectively solves the limitations of single-modal control and the conflict problem of multimodal inputs in existing technologies, allowing operators to use the most natural modality for control in parallel, achieving organic fusion and intelligent decision-making of multimodal inputs, greatly improving interaction efficiency and task performance, and enabling efficient control of unmanned systems in complex interaction scenarios. Attached Figure Description

[0016] Figure 1 This is a flowchart of an unmanned system control method according to an embodiment of the present invention.

[0017] Figure 2This is a schematic diagram of the instruction data structure for a unified data format according to an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the analysis process in an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram illustrating the design principle of the control strategy in an embodiment of the present invention.

[0020] Figure 5 This is a schematic diagram of the control strategy decision-making process according to an embodiment of the present invention.

[0021] Figure 6 This is a schematic diagram illustrating the implementation process of an emergency safety interruption scenario according to an embodiment of the present invention.

[0022] Figure 7 This is a structural block diagram of a multimodal virtual joystick component according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 In one embodiment of the present invention, an unmanned system control method is provided, which can efficiently integrate control inputs from various modes to generate unified and intelligent control commands for the unmanned system, thereby significantly improving the intuitiveness, efficiency and robustness of unmanned system control.

[0026] Specifically, the unmanned system control method of the present invention includes the following steps: S1: Receives control inputs for each mode.

[0027] S2: Parse the control inputs of each modality into instructions with a unified data format; wherein the instructions include an intent field, a control parameter field, a source field, a confidence field, and a timestamp field.

[0028] S3: According to the preset control strategy, the current instruction and the next instruction are arbitrated and fused to obtain the control instruction of the unmanned system and sent to the unmanned system; wherein, the control strategy includes a priority arbitration strategy and a complementary fusion strategy for combining different instructions.

[0029] Explanatory, see Figure 2 The unified data format defines a standardized UnifiedCommand data structure to map control commands from different modalities into semantic objects containing intent, control parameters, data source, confidence, and timestamps. Intent specifies the purpose of the command. Control parameters represent raw or fused parameters from different modalities, including specific parameters such as motion, target point, and action. Data source indicates the source of the command, identifying whether it is touch, voice, or fusion. Confidence (typically a value between 0 and 1) represents the confidence level in the command's intent recognition, used for subsequent command fusion decisions. Timestamps are used for time-series analysis and command sequencing.

[0030] For example, for a drone, the intent includes movement, hovering, takeoff, landing, and descent. Control parameters include motion parameters, target point parameters, and action parameters. Motion parameters include linear velocity, angular velocity, and reference coordinate system (body coordinate system and world coordinate system, etc.). Target point parameters include latitude, longitude, and altitude information. Action parameters are used for device operation, such as taking photos and recording videos.

[0031] Furthermore, it should be noted that "next instruction" here does not specifically refer to the next instruction, but can be several instructions in the sequence. For ease of description and understanding, the term "next instruction" is used here. Also, "current instruction" does not refer to an instruction currently being executed; it can also be an instruction that has just been received but has not yet begun execution.

[0032] This invention provides an unmanned system control method that parses control inputs from various modalities into instructions in a unified data format, including intent, control parameters, source, confidence level, and timestamp. This allows heterogeneous inputs from different modalities to be understood and compared at the same semantic level, laying a data foundation for subsequent arbitration and fusion. By employing a control strategy that includes a priority arbitration strategy and a complementary fusion strategy to arbitrate and fuse adjacent instructions, the method addresses two key issues. First, the priority arbitration strategy makes reasonable decisions based on preset rules when multimodal instructions conflict, avoiding the control chaos caused by first-come-first-served or fixed priorities in existing technologies, significantly improving control determinism and security in multimodal scenarios. Second, the complementary fusion strategy combines different instructions to generate a single composite control instruction, achieving synergistic effects of multimodal inputs rather than simple overlap. This effectively solves the limitations of single-modal control and the conflict problem of multimodal inputs in existing technologies, allowing operators to use the most natural modality for control in parallel, achieving organic fusion and intelligent decision-making of multimodal inputs, greatly improving interaction efficiency and task performance, and enabling efficient control of unmanned systems in complex interaction scenarios.

[0033] In one possible implementation, the modality includes touch commands, voice commands, and touch and voice fusion commands; the control input for receiving each modality includes: receiving touch commands via a virtual joystick interface, receiving voice commands via a voice recognition module, and receiving touch and voice fusion commands via a virtual joystick interface and a voice recognition module.

[0034] In one possible implementation, the virtual joystick interface is a virtual joystick interface rendered based on the Canvas component of HarmonyOS ArkUI; the voice recognition module is the HarmonyOS SpeechKit voice recognition service module.

[0035] Explaining the virtual joystick interface, this implementation uses the Canvas component of HarmonyOS ArkUI for high-performance rendering to draw the virtual joystick interface. It leverages HarmonyOS's declarative UI and responsive layout to achieve cross-device adaptive display of the joystick interface. It can capture user touch events in real time and convert screen coordinates into normalized control variables, such as yaw angle, pitch angle, and throttle input for drones.

[0036] Explaining the speech recognition module, this implementation integrates the HarmonyOS SpeechKit speech recognition service. This module receives user voice input, converts it into text, and then performs semantic analysis using natural language understanding technology to identify control intentions and control parameters. For example, control intentions for a drone include ascending, turning left, and accelerating, while control parameters include 5 meters and 30 degrees.

[0037] In one possible implementation, when the modality is a touch command, parsing the control input of each modality into a command with a unified data format includes: obtaining the original coordinates of the touch point of the touch command relative to the screen; converting the original coordinates into relative coordinates relative to the center point of the virtual joystick, and dividing the relative coordinates by the maximum active radius of the virtual joystick to obtain a normalized vector; when the normalized vector is less than a preset dead zone threshold of the center point of the virtual joystick, determining that the control input is invalid; otherwise, applying a preset response curve to the normalized vector and mapping it to the control parameters of the unmanned system, and filling the control parameters of the unmanned system into the control parameter field of the command.

[0038] Explanatory, see Figure 3 The system listens for the `onTouch` event of the HarmonyOS system to obtain the original coordinates and timestamp of the touch point. The parsing process includes the following steps: Coordinate transformation: Converting the absolute coordinates relative to the screen to relative coordinates relative to the center point of the virtual joystick. Normalization: Dividing the relative coordinates by the maximum active radius of the virtual joystick to obtain a normalized vector with a value range of [-1, 1], ensuring consistent control sensitivity of the virtual joystick on different screen sizes and devices. Dead zone handling: To avoid misoperations caused by slight jitter, a dead zone is set at the center of the virtual joystick. If the length of the normalized vector is less than the dead zone threshold (e.g., 0.1), it is considered an invalid input. Sensitivity and curve adjustment: Applying response curves (such as linear, exponential, and logarithmic) to the value of the normalized vector to adjust the control feel. For example, using an exponential curve can make the joystick respond smoothly when pushed slightly and respond quickly when pushed hard. Intent mapping: Mapping the processed normalized vector to the control parameters of the unmanned system, and filling the control parameters of the unmanned system into the control parameter field of the command.

[0039] In one possible implementation, when the modality is a voice command, the step of parsing the control input of each modality into a command with a unified data format includes: acquiring the voice command and converting it into text, and performing natural language understanding processing on the text to obtain the intent of the voice command, the control parameters of the unmanned system, and the confidence level of the voice command, and filling them into the intent field, control parameter field, and confidence level field of the command, respectively.

[0040] Explanatory, see Figure 3Using HarmonyOS's CoreSpeechKit, audio streams are converted into text in real time. The recognized text undergoes NLU (Natural Language Understanding) processing, including: Intent recognition: determining the user's overall purpose, such as moving, taking off, or taking a photo; Entity extraction: extracting key parameters from the text, such as direction (left), speed (faster), distance (10 meters), and target (building A). The recognized intent and entities are mapped to specific parameters of the command. Finally, the confidence levels of the final command are determined by combining the confidence levels of speech recognition and NLU parsing, and mapped to the command's confidence field.

[0041] Explanatoryly, for a complete instruction, although it includes an intent field, control parameter field, source field, confidence field, and timestamp field, the intent field, control parameter field, confidence field, and timestamp field can all be set to empty or predefined values.

[0042] For the interpretation of touch and voice fusion commands, when parsing them into commands with a unified data format, the touch commands and voice commands in the touch and voice fusion commands can be parsed separately using the methods described above. Then, the two parsed commands can be merged. This merging can be done by averaging, by taking the larger value, or by other custom merging methods.

[0043] In one possible implementation, the priority arbitration strategy includes: obtaining the priority and confidence level of the current instruction and the next instruction; wherein, the priority of an instruction whose modality is touch control is lower than the priority of an instruction whose modality is voice control; when the priority of the next instruction is the highest priority, the next instruction is used as the control instruction of the unmanned system and sent to the unmanned system; otherwise, it is checked whether the current instruction is empty; when the current instruction is empty, the next instruction is used as the control instruction of the unmanned system and sent to the unmanned system; otherwise, it is checked whether the difference between the confidence level of the next instruction and the confidence level of the current instruction is greater than a preset threshold; when the difference between the confidence level of the next instruction and the confidence level of the current instruction is greater than a preset confidence gap threshold, the next instruction is used as the control instruction of the unmanned system and sent to the unmanned system; otherwise, a complementary fusion strategy is executed.

[0044] The explanatory control strategy incorporates a priority-based arbitration strategy and a complementary fusion strategy for combining different instructions, responsible for arbitrating and fusion between different instructions.

[0045] The core idea of ​​the interpretable, priority arbitration strategy is security and prioritizing critical instructions. When a high-priority instruction with high confidence arrives, the system immediately interrupts the current non-critical operation.

[0046] For details, see Figure 4The design of a priority arbitration strategy mainly considers the following aspects: First, in emergency situations, when the next instruction has the highest priority, it is directly sent to the unmanned system as the control command. For example, if the intent of the next instruction is EMERGENCY_STOP (emergency stop) or RETURN_TO_HOME (automatic return), and the confidence level is greater than a high threshold (e.g., 0.9), then any currently executing instructions are immediately interrupted, and the next instruction is forcibly executed. Simultaneously, UI alarms (such as screen flickering) can be triggered.

[0047] Second, modal priority. Considering that voice commands are typically discrete and purposeful, while touch controls are continuous and exploratory, this implementation sets the priority of touch commands to be lower than that of voice commands. For commands of different priorities, higher-priority commands are generally used to replace or optimize lower-priority commands.

[0048] Third, confidence level. When the difference between the confidence level of the next instruction and the confidence level of the current instruction is greater than a preset confidence gap threshold (e.g., 0.3), the next instruction will be considered as the replacement for the current instruction. This is based on the principle that the system trusts explicit instructions more.

[0049] In one possible implementation, the complementary fusion strategy includes: acquiring the intent of the current instruction and the next instruction; when the intents of the current instruction and the next instruction are complementary, averaging or weighting the control parameters of the current instruction and the next instruction based on priority to generate a fused instruction and sending it to the unmanned system as the control instruction of the unmanned system; when the intents of the current instruction and the next instruction conflict, using the instruction with higher priority between the current instruction and the next instruction as the control instruction of the unmanned system and sending it to the unmanned system.

[0050] The explanatory, complementary fusion strategy is based on the idea of ​​synergy. When different instructions are logically compatible and can work together to accomplish a more complex task, they can be combined.

[0051] For details, see Figure 4 The design of complementary integration strategies mainly considers the following aspects: First, intent compatibility check. The intents of the current instruction and the next instruction are the same or compatible. This applies to both being movement intents. Movement intents can be decomposed into parameters of different dimensions (such as horizontal movement, vertical movement, and rotation), and are compatible with each other.

[0052] Second, parameter space complementarity check. When two commands control parameters with different dimensions, since these parameters are orthogonal in physical space and do not conflict, they can be merged to achieve oblique movement in three-dimensional space. For example, touch commands control forward and backward and left and right, while voice commands control lifting and lowering.

[0053] Third, the fusion rules. Taking the best and compensating for the worst: For the same parameter dimension, if only one instruction provides a valid value, that value is used; if two instructions provide values, the value is selected or weighted averaged based on priority and confidence. Basal motion + correction enhancement: Basal motion refers to the continuous and smooth basic motion instructions provided by the touch joystick; correction enhancement refers to the discrete and explicit motion adjustments provided by voice instructions. Continuously controlled touch instructions provide basal motion, while discrete voice instructions provide correction or enhancement. For example, the horizontal and forward / backward speeds of the touch are used as a basis, superimposed with the vertical speed of the voice.

[0054] In this embodiment, see Figure 5 For a newly arrived instruction, the system determines whether to execute it immediately, merge it with the current instruction, or ignore it, based on a preset control strategy and the current system state. This process includes the following steps: Step 1: Security check (highest priority).

[0055] Judgment Criteria: Is the next instruction a safety emergency instruction? Judgment Basis: Check if the instruction's intent belongs to a predefined list of safety-critical instructions, and if its confidence level is higher than a high threshold (e.g., 0.9). Decision Action: Immediately execute priority arbitration. Regardless of the currently executing instruction, forcibly interrupt it and directly execute the new safety instruction.

[0056] Step 2: Check the current status.

[0057] Judgment condition: Is there an instruction currently being executed in the system? Decision action: If not: It indicates the system is in an idle or standby state. The next instruction does not require contention and is executed immediately as the current instruction. If it exists: Proceed to step three.

[0058] Step 3: Priority arbitration (competitive judgment).

[0059] Judgment Criterion: Is the next instruction higher in priority than the current instruction? Judgment Basis: Compare the priorities of the two. Confidence Advantage: Is the confidence level of the next instruction significantly higher than that of the current instruction? Decision Action: If the next instruction has a higher priority: Proceed to step four to determine if intelligent fusion is possible. If the next instruction has a lower or equal priority: Retain the current instruction and ignore or reject the next instruction.

[0060] Step 4: Complementarity assessment (cooperation assessment).

[0061] Judgment Criteria: Are the next instruction and the current instruction complementary rather than conflicting in terms of parameters? Judgment Basis: Examine the intent and control parameters of both instructions. Typical complementary scenarios: identical intent and orthogonal parameters (e.g., the current instruction controls X-axis (forward / backward) and Y-axis (left / right) movement, while the next instruction controls Z-axis (vertical / vertical) movement). Decision Action: If complementary: Perform complementary fusion, intelligently merging the parameters of the two instructions to generate a new, more complex fused instruction. If conflicting: Use the instruction with higher priority.

[0062] In one possible implementation, the unmanned system control method of the present invention is illustrated using an emergency safety interruption scenario as an example. The scenario is described as follows: An operator is using a virtual joystick to control a drone to observe a building at close range. Suddenly, a gust of wind causes the drone to drift uncontrollably toward a bridge pier, and the operator immediately shouts "Hover!"

[0063] Traditional solutions have the following problems: Operators need to first locate the "hover" button on the screen or return the joystick to center, resulting in slow response and a high risk of errors. Especially under pressure, touch operation may be inaccurate. See also Figure 6 The specific implementation process of the present invention is illustrated. It is evident that the unmanned system control method of the present invention greatly enhances control safety, with voice commands serving as a safety red line and a response speed far exceeding that of manual operation; simultaneously, it aligns with practical applications, as the instinctive reaction of humans in emergency situations is to shout, and the method of the present invention fully utilizes this intuitive interaction.

[0064] In one possible implementation, the unmanned system control method of the present invention is illustrated using a highly efficient collaborative operation scenario as an example. Scenario description: An operator controls a drone to inspect a field, needing to maintain flight while simultaneously taking focused photos of areas suspected of being infested by pests or diseases.

[0065] Traditional solutions suffer from several problems: the operator must use one hand to control the joystick to maintain flight while simultaneously tapping the "take photo" button on the screen with the other, resulting in high operational difficulty and unstable flight paths. Alternatively, hovering before taking a photo and then flying again is inefficient. The proposed method includes: Touch input (basal motion): The left hand continuously pushes the joystick, causing the drone to fly along the field ridges at a stable speed and altitude. Command: {intent: MOVE, linear: {x: 0.6, y: 0, z: 0}, source:TOUCH}. Voice input (correction / enhancement): When a suspicious area is detected, the operator says "take photo". Command: {intent:TAKE_PHOTO, source: VOICE}. Fusion decision: The engine determines that the two commands have different intentions (MOVE vs TAKE_PHOTO) and do not conflict, representing typical non-conflicting command parallelism. Decision: No parameter fusion is performed, but parallel execution is allowed. The fusion engine sends both commands to the drone simultaneously. Execution result: The drone triggers the photo-taking action while maintaining its original flight path. No additional manual operation is required from the operator. Therefore, the method of this invention achieves a significant increase in efficiency, as flight and operational tasks can be carried out simultaneously, greatly improving inspection efficiency. At the same time, it simplifies operation, reducing the operator's cognitive load and operational complexity, effectively lowering the difficulty for a single person to complete complex tasks.

[0066] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0067] See Figure 7 In another embodiment of the present invention, a multimodal virtual joystick component is provided, which can be used to implement the above-mentioned unmanned system control method. Specifically, the multimodal virtual joystick component includes an interaction module, a modal parsing module, and a fusion decision engine module.

[0068] The interaction module receives control inputs from various modalities; the modal parsing module parses the control inputs from various modalities into instructions with a unified data format; wherein the instructions include an intent field, a control parameter field, a source field, a confidence field, and a timestamp field; the fusion decision engine module arbitrates and fuses the current instruction with the next instruction according to a preset control strategy to obtain the control instructions for the unmanned system and sends them to the unmanned system; wherein the control strategy includes a priority arbitration strategy and a complementary fusion strategy for combining different instructions.

[0069] In one possible implementation, the modality includes touch commands, voice commands, and touch and voice fusion commands; the control input for receiving each modality includes: receiving touch commands via a virtual joystick interface, receiving voice commands via a voice recognition module, and receiving touch and voice fusion commands via a virtual joystick interface and a voice recognition module.

[0070] In one possible implementation, the virtual joystick interface is a virtual joystick interface rendered based on the Canvas component of HarmonyOS ArkUI; the voice recognition module is the HarmonyOS SpeechKit voice recognition service module.

[0071] In one possible implementation, when the modality is a touch command, parsing the control input of each modality into a command with a unified data format includes: obtaining the original coordinates of the touch point of the touch command relative to the screen; converting the original coordinates into relative coordinates relative to the center point of the virtual joystick, and dividing the relative coordinates by the maximum active radius of the virtual joystick to obtain a normalized vector; when the normalized vector is less than a preset dead zone threshold of the center point of the virtual joystick, determining that the control input is invalid; otherwise, applying a preset response curve to the normalized vector and mapping it to the control parameters of the unmanned system, and filling the control parameters of the unmanned system into the control parameter field of the command.

[0072] In one possible implementation, when the modality is a voice command, the step of parsing the control input of each modality into a command with a unified data format includes: acquiring the voice command and converting it into text, and performing natural language understanding processing on the text to obtain the intent of the voice command, the control parameters of the unmanned system, and the confidence level of the voice command, and filling them into the intent field, control parameter field, and confidence level field of the command, respectively.

[0073] In one possible implementation, the priority arbitration strategy includes: obtaining the priority and confidence level of the current instruction and the next instruction; wherein, the priority of an instruction whose modality is touch control is lower than the priority of an instruction whose modality is voice control; when the priority of the next instruction is the highest priority, the next instruction is used as the control instruction of the unmanned system and sent to the unmanned system; otherwise, it is checked whether the current instruction is empty; when the current instruction is empty, the next instruction is used as the control instruction of the unmanned system and sent to the unmanned system; otherwise, it is checked whether the difference between the confidence level of the next instruction and the confidence level of the current instruction is greater than a preset threshold; when the difference between the confidence level of the next instruction and the confidence level of the current instruction is greater than a preset confidence gap threshold, the next instruction is used as the control instruction of the unmanned system and sent to the unmanned system; otherwise, a complementary fusion strategy is executed.

[0074] In one possible implementation, the complementary fusion strategy includes: acquiring the intent of the current instruction and the next instruction; when the intents of the current instruction and the next instruction are complementary, averaging or weighting the control parameters of the current instruction and the next instruction based on priority to generate a fused instruction and sending it to the unmanned system as the control instruction of the unmanned system; when the intents of the current instruction and the next instruction conflict, using the instruction with higher priority between the current instruction and the next instruction as the control instruction of the unmanned system and sending it to the unmanned system.

[0075] All relevant content of each step involved in the aforementioned embodiments of the unmanned system control method can be referenced from the functional description of the functional module corresponding to the multimodal virtual joystick component in the embodiments of the present invention, and will not be repeated here.

[0076] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0077] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of an unmanned system control method.

[0078] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the unmanned system control method in the above embodiments.

[0079] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A control method for an unmanned system, characterized in that, include: Receive control inputs from each mode; The control inputs of each modality are parsed into instructions with a unified data format; wherein, the instructions include an intent field, a control parameter field, a source field, a confidence field, and a timestamp field; According to a preset control strategy, the current instruction and the next instruction are arbitrated and fused to obtain the control instructions for the unmanned system and then sent to the unmanned system; wherein, the control strategy includes a priority arbitration strategy and a complementary fusion strategy for combining different instructions.

2. The unmanned system control method according to claim 1, characterized in that, The modalities include touch commands, voice commands, and touch and voice fusion commands; the control input for receiving each modality includes: receiving touch commands using a virtual joystick interface, receiving voice commands using a voice recognition module, and receiving touch and voice fusion commands using a virtual joystick interface and a voice recognition module.

3. The unmanned system control method according to claim 2, characterized in that, The virtual joystick interface is a virtual joystick interface rendered based on the Canvas component of HarmonyOS ArkUI. The speech recognition module is the HarmonyOS SpeechKit speech recognition service module.

4. The unmanned system control method according to claim 1, characterized in that, When the modality is a touch command, the step of parsing the control input of each modality into a command with a unified data format includes: Obtain the original coordinates of the touch point relative to the screen for the touch command; The original coordinates are converted into relative coordinates with respect to the center point of the virtual joystick, and the relative coordinates are divided by the maximum active radius of the virtual joystick to obtain a normalized vector; When the normalized vector is less than the preset dead zone threshold of the virtual joystick center point, the control input is determined to be invalid; otherwise, the normalized vector is applied with the preset response curve and mapped to the control parameters of the unmanned system, and the control parameters of the unmanned system are filled into the control parameter field of the command.

5. The unmanned system control method according to claim 1, characterized in that, When the modality is a voice command, the step of parsing the control input of each modality into commands with a unified data format includes: The system acquires voice commands and converts them into text, and performs natural language understanding processing on the text to obtain the intent of the voice command, the control parameters of the unmanned system, and the confidence level of the voice command, which are then filled into the intent field, control parameter field, and confidence level field of the command, respectively.

6. The unmanned system control method according to claim 1, characterized in that, The priority arbitration strategy includes: Obtain the priority and confidence level of the current command and the next command; wherein, the priority of the command whose modality is touch command is lower than the priority of the command whose modality is voice command; If the next instruction has the highest priority, the next instruction will be used as the control instruction for the unmanned system and sent to the unmanned system; otherwise, check if the current instruction is empty. When the current instruction is empty, the next instruction is used as the control instruction for the unmanned system and sent to the unmanned system; otherwise, it is checked whether the difference between the confidence level of the next instruction and the confidence level of the current instruction is greater than a preset threshold. If the difference between the confidence level of the next instruction and the confidence level of the current instruction is greater than a preset confidence gap threshold, the next instruction will be used as the control instruction for the unmanned system and sent to the unmanned system; otherwise, a complementary fusion strategy will be executed.

7. The unmanned system control method according to claim 6, characterized in that, The complementary fusion strategy includes: Obtain the intent of the current instruction and the next instruction; When the intents of the current instruction and the next instruction are complementary, the control parameters of the current instruction and the next instruction are averaged or weighted based on priority to generate a fused instruction, which is then used as the control instruction of the unmanned system and sent to the unmanned system. When the intents of the current instruction and the next instruction conflict, the instruction with higher priority is used as the control instruction of the unmanned system and sent to the unmanned system.

8. A multimodal virtual joystick component, characterized in that, include: The interaction module is used to receive control inputs from various modes; The modality parsing module is used to parse the control input of each modality into instructions with a unified data format; wherein, the instructions include an intent field, a control parameter field, a source field, a confidence field, and a timestamp field; The fusion decision engine module is used to arbitrate and fuse the current instruction with the next instruction according to a preset control strategy to obtain the control instructions of the unmanned system and send them to the unmanned system; wherein, the control strategy includes a priority arbitration strategy and a complementary fusion strategy for combining different instructions.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the unmanned system control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the unmanned system control method as described in any one of claims 1 to 7.

Images