Method for generating man-machine interaction instruction for remote cooperative control of target drone

Abstract

The invention discloses a man-machine interaction instruction generation method for target aircraft remote cooperative control, and relates to the technical field of man-machine interaction instruction generation. According to the method, a track prediction mechanism based on historical state data and a feedforward control strategy are fused, a response window of a control instruction can be judged in advance by constructing a short-time future state prediction sequence of a target drone, and the time sensitivity and the execution matching degree of the control instruction are greatly improved; structured instruction unit generation and multi-stage sub instruction chain analysis are adopted, so that the composite action is modularly expressed and finely disassembled, and the control precision and logic consistency of complex operation are effectively improved. Besides, by means of space attitude residual error screening, envelope boundary modeling and time tolerance design, a stable control area and a buffer area can be established in multiple directions, and the fault-tolerant capability and controllability in the high-dynamic flight state are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction instruction generation, in particular to a human-computer interaction instruction generation method for remote collaborative control of a target machine. Background Art

[0002] With the continuous development of modern drone control technology, the demand for the application of human-computer interaction systems in high-dynamic, high-response precision scenarios is growing. In particular, in the collaborative control of remote drones, in order to effectively achieve target trajectory simulation and complex action execution, higher requirements are placed on the intelligence, structured, and feedforward nature of the command generation mechanism. Traditional drone control methods mainly rely on operators manually issuing commands or simple rule-based control systems, which have problems such as response lag, large execution deviation, and weak dynamic adaptability. In recent years, with the application of technologies such as data prediction modeling, human-computer collaborative perception, and command structure optimization, remote drone control has gradually evolved towards high real-time, predictive, and data fusion-driven directions. How to build a human-computer interaction command generation system for future situation prediction to improve the response accuracy and trajectory fitting of drone control has become an important research direction in this field.

[0003] CN111857177B discloses a method, device, equipment, and medium for generating remotely controlled target commands. The method involves a pilot generating control commands based on the target's motion characteristics, and converting these commands into equivalent commands through a command modifier to achieve control of the target's flight trajectory. This method utilizes the "equivalence similarity principle" to make the simulated target motion approach the flight characteristics of the target fighter, improving the command matching and motion restoration capabilities to a certain extent. However, this technology relies on the immediacy and accuracy of the pilot's current commands, fails to establish a state prediction model for future time periods, lacks a tolerance identification and correction mechanism for command response lags, and does not perform structured modeling on the command data stream. This makes it prone to command distortion and system delays in complex motion control scenarios.

[0004] CN115127400B discloses a control method for a target reporting and score generation control system, which improves the shooter's training efficiency through the return of shooting data and a real-time feedback mechanism. Its main technical features include the coordinated control of the target drone and single-chip microcomputer, data processing and display, and Bluetooth voice broadcast, embodying the basic form of human-computer interaction. Although this technology has certain practicality in training scenarios, its control mechanism is mainly based on passive response after data return, lacking the ability to predict the future state of the target drone, nor involving the deconstruction of the composite instruction chain and time offset correction, and lacking the refined processing capabilities of a multi-level control unit. Summary of the Invention

[0005] In view of the problems existing in existing target drone control technology in simulating real flight conditions and improving control response accuracy, the present invention is proposed.

[0006] Therefore, the problem to be solved by the present invention is how to solve the problem of delayed instruction response and low action matching in traditional methods.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: In the first aspect, the present invention provides a method for generating human-computer interaction instructions for remote collaborative control of a target machine, which includes: constructing a continuous state space trajectory of the target machine in a short future period based on the historical state data stream returned by the target machine, and outputting a state prediction sequence for pre-judging the instruction response window; receiving the operator's current interaction instruction data stream, and reconstructing the current interaction instruction data stream according to the time frame structure in the state prediction sequence to generate a set of structured instruction units; based on the structured instruction unit set and the state prediction sequence, combined with the instruction response lag tolerance and multi-directional envelope boundary parameters, generating a feedforward envelope instruction stream; after generating the feedforward envelope instruction stream, identifying and template matching the structured instruction units containing composite action labels, and parsing them into a multi-level sub-instruction chain with frame index integrity; combining the state prediction sequence and the historical state data stream returned by the target machine, performing time offset comparison on the multi-level sub-instruction chain, and outputting a dynamically adjusted refined control data packet.

[0008] As a preferred solution of the method for generating human-computer interaction instructions for remote collaborative control of the target aircraft described in the present invention, the generation of the state prediction sequence includes: receiving the continuous state data stream sent back by the target aircraft, aligning the position triples, attitude triples and velocity scalars according to the timestamps, and constructing a state data matrix containing six-dimensional fields of time series, space and attitude; performing piecewise cubic interpolation fitting on the space and attitude sequences in adjacent time periods in the state data matrix, and calculating the first-order derivative curve for each interpolation result to obtain a set of speed-attitude change rate curves; identifying the acceleration or attitude angle change mutation points in the first-order derivative curve as trend mutation nodes, and refitting the continuous trajectory segments between the trend mutation nodes, correcting the interpolation prediction trajectory in combination with the environmental modeling parameters, and generating a state prediction sequence in the future time window.

[0009] As a preferred solution of the human-computer interaction instruction generation method for remote collaborative control of the target machine described in the present invention, the structural reconstruction of the current interaction instruction data stream includes: receiving the operator's current interaction instruction data stream, parsing the timing field, action description field and target state field therein, dividing it into continuous instruction segments according to the time axis, and semantically classifying the action description field to obtain a basic action label set; combining the basic action label set and the target state field, extracting the corresponding standard action unit template from the preset action knowledge template library, and determining the predicted frame index of each target state according to the instruction trigger time, and establishing an action unit-frame index-duration item triple; mapping each action unit-frame index-duration item triple to a structured instruction unit, which includes a standard action type, a target predicted frame index and a standard duration field, to form a structured instruction unit set.

[0010] As a preferred solution of the method for generating human-computer interaction instructions for remote collaborative control of the target aircraft described in the present invention, the generation of the feedforward envelope instruction stream includes: in the state prediction sequence, according to the target prediction frame index and the standard duration field in the structured instruction unit, intercepting the state prediction frame of the corresponding time period as the initial response candidate window; calculating the difference between the position vector and the attitude Euler angle of each state prediction frame in the initial response candidate window and the target state field in the corresponding structured instruction unit, and normalizing them to obtain the spatial dimension residual set and the attitude dimension residual; screening the continuous state prediction frame segments whose residuals are simultaneously lower than the spatial residual threshold and the attitude residual threshold, and defining The state matching window is used; with the command target position vector in the state matching window as the center, the angle range is set along the forward direction and the left and right lateral directions of the target aircraft velocity vector, and a three-directional prediction vector group is generated as a spatial direction reference edge; on each spatial direction reference edge, the maximum controllable displacement distance is expanded according to the inertial dynamic parameters of the target aircraft, and an envelope boundary parameter set with directional attributes is constructed; combined with the time response tolerance range set by the input delay statistical data of the operating end, a time envelope is constructed according to the time response advance and time response lag, and the envelope boundary parameter set is embedded in the time envelope time period to form an envelope command segment, and at the same time, the central control stable area and the edge transition buffer zone are divided.

[0011] As a preferred solution of the method for generating human-computer interaction instructions for remote collaborative control of the target aircraft described in the present invention, the instruction target position vector is the spatial position vector component in the target state field.

[0012] As a preferred solution of the human-computer interaction instruction generation method for remote collaborative control of the target aircraft described in the present invention, the parsing into a multi-level sub-instruction chain with frame index integrity includes: extracting action units containing composite action tags in the structured instruction unit set, and loading a preset high-order tactical action template set with the standard action type field as the index; searching the corresponding template structure in the tactical action template set based on the target prediction frame index and standard duration field associated with the action unit, and outputting a sub-instruction chain containing a sub-action sequence, a relative time offset and a target state transition sequence; for each sub-action in the sub-instruction chain, parameter inheritance is performed according to the target state field in the structured instruction unit, and the absolute frame index and target state field of each sub-instruction are completed in combination with the matching state matching window in the state prediction sequence; all sub-instructions are arranged in frame index order, and a multi-level sub-instruction chain set with time controllability and structural consistency is output.

[0013] As a preferred solution of the human-computer interaction instruction generation method for remote collaborative control of the target aircraft described in the present invention, the time offset comparison includes: extracting the state prediction frame corresponding to the target prediction frame index in each structured instruction unit, and extracting a set of actual state frames consisting of position triples, attitude triples and velocity scalars in adjacent time windows from the historical state data stream returned by the target aircraft; for all actual state frames of each target prediction frame in the time window, calculating the difference vector of the spatial position vector and the attitude Euler angle field, and identifying the frame with the smallest difference vector as the response alignment frame, and recording the timestamp; calculating the timestamp difference between each group of target prediction frames and the corresponding response alignment frame, recording it as the time offset, and constructing a time offset mapping table between the target prediction frame index and the returned state frame index.

[0014] As a preferred solution of the human-computer interaction instruction generation method for remote collaborative control of the target machine described in the present invention, wherein: the output of the dynamically adjusted refined control data packet includes: using the time offset in the time offset mapping table as a reference basis, performing synchronization adjustment on the target prediction frame index bound to each sub-action, so as to maintain time consistency with the actual state response frame; after completing the frame index correction, the sub-actions in the sub-instruction chain are re-sorted in chronological order, and after the sorting is completed, the trigger intervals between the temporally adjacent sub-actions are integrity checked, and for structures below the preset trigger interval threshold, a frame distance extension operation is performed; according to the standard action type field and target state field of each corrected sub-action, the action parameters are confirmed for consistency, and each sub-action structure is converted into a structured instruction unit set with a standard field format.

[0015] In a second aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the human-computer interaction instruction generation method for remote collaborative control of a target machine as described in the first aspect of the present invention are implemented.

[0016] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the human-computer interaction instruction generation method for remote collaborative control of a target machine as described in the first aspect of the present invention are implemented.

[0017] The beneficial effects of the present invention are as follows: the present invention integrates a trajectory prediction mechanism based on historical state data and a feedforward control strategy. By constructing a state prediction sequence for the target aircraft in the short term, it can determine the response window of the control command in advance, greatly improving the time sensitivity and execution matching degree of the control command; the use of structured instruction unit generation and multi-level sub-instruction chain analysis enables modular expression and fine decomposition of compound actions, effectively improving the control accuracy and logical consistency of complex operations. In addition, by utilizing spatial attitude residual screening, envelope boundary modeling, and time tolerance design, stable control areas and buffer zones can be established in multiple directions, improving the fault tolerance and controllability in high-dynamic flight states; and time offset comparison enhances the adaptive adjustment capability for non-ideal factors such as execution errors and delay changes, ensuring that the human-machine instruction chain has high robustness and real-time performance in remote high-frequency collaborative scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 Flowchart of the method for generating human-computer interaction instructions for remote collaborative control of a target drone. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0023] As mentioned in the background technology above, traditional drone control methods rely primarily on manual operator commands or simple rule-based control systems, which can lead to problems such as delayed response, large execution deviations, and weak dynamic adaptability. In recent years, with the application of technologies such as data prediction modeling, human-machine collaborative perception, and command structure optimization, remote drone control has gradually evolved towards high real-time, predictive, and data-fusion-driven capabilities. Developing a human-machine interactive command generation system for future situation prediction to improve the response accuracy and trajectory fit of drone control has become a key research direction in this field.

[0024] Figure 1 Flowchart of the method for generating human-computer interaction instructions for remote collaborative control of a target machine according to an embodiment of the present invention. Figure 1 As shown, in the method for generating human-computer interaction instructions for remote collaborative control of a target drone, the method includes: S1: Based on the historical state data stream sent back by the target machine, the continuous state space trajectory of the target machine in the short future period is constructed, and the state prediction sequence for pre-judgment of the command response window is output.

[0025] S1.1: Receive the continuous state data stream sent back by the target aircraft, align the position triplet, attitude triplet, and velocity scalar fields according to the timestamp, and construct a state data matrix containing six-dimensional fields of time series, space, and attitude.

[0026] Specifically, the field alignment operation requires a high-precision clock as the time base reference. For each data record, the timestamp field is extracted and preliminarily sorted in chronological order. The time difference is then uniformly calibrated with millisecond-level accuracy, and linear time backtracking is performed on the time jitter caused by data link delays.

[0027] The position triplet is converted to an absolute position vector (X, Y, Z) in the geodetic coordinate system using a geographic coordinate conversion function. The attitude triplet (pitch, yaw, and roll) is converted to an angle triplet in a standard rotation order using the Euler angle conversion rule. The velocity scalar is converted to the velocity modulus after calibration with the unit vector direction. After the above fields are formatted and standardized, state data vectors are constructed at each moment and concatenated to form a time series state matrix.

[0028] It should be noted that through field alignment and matrix construction, high-density structured packaging of the target aircraft's historical state data can be effectively achieved. Furthermore, the six-dimensional state data matrix has the advantages of a unified time base and standardized coordinate system, which provides good scalability when integrating multiple platforms or migrating predictive models.

[0029] S1.2: Perform piecewise cubic interpolation fitting on the spatial and attitude sequences in adjacent time periods in the state data matrix, and calculate the first-order derivative curve for each interpolation result to obtain a set of velocity-attitude change rate curves.

[0030] To address the discrete transitions and data gaps in the target drone's state sequence, the present invention implements interpolation modeling using piecewise cubic spline interpolation. Specifically, the data is first divided into multiple continuous intervals based on the time series fields in the state data matrix. The interval length is dynamically adjusted based on the platform's feedback frequency and the command control response time window, with each interpolation segment preferably consisting of 10 frames. Within each interpolation segment, a set of cubic spline functions is constructed for the spatial position vector (3D) and the attitude Euler angles (3D). Each dimension is modeled separately, and the interpolation function ensures that the position and first-order derivatives are continuous at the segment points, ensuring the physical plausibility and dynamic coherence of the trajectory.

[0031] After interpolation is complete, the first-order derivative function is calculated for each dimension's interpolation function. For spatial position, the first-order derivative corresponds to the velocity vector (i.e., the rate of change of position); for attitude angle, the first-order derivative corresponds to the rate of change of attitude angle, reflecting the speed of the platform's attitude adjustment or its rotational trend. All first-order derivative functions are concatenated in time series to form a set of velocity-attitude rate-of-change curves.

[0032] Compared with traditional linear interpolation, cubic spline has high-order continuity characteristics and can more accurately fit the actual flight trend of the target aircraft in nonlinear stages such as rapid attitude switching and trajectory edge adjustment.

[0033] S1.3: Identify the acceleration or attitude angle change mutation points in the first-order derivative curve as trend mutation nodes, and refit the continuous trajectory segments between the trend mutation nodes. Combined with the environmental modeling parameters, the interpolated predicted trajectory is corrected to generate a state prediction sequence within the future time window.

[0034] For the first-order derivative curves that have been constructed, first search for the inflection point of each curve. Perform the following steps: For each data point in the velocity derivative curve, calculate the rate of change between it and the two adjacent points before and after it; If the sign of the rate of change at the current point is opposite to that of the previous and next points (i.e., the direction of the derivative slope is reversed), and the absolute value of the rate of change exceeds twice the average value, it is marked as a velocity mutation point; Perform the above operations on the derivative curves of pitch angle, yaw angle and roll angle respectively, and mark the attitude mutation points.

[0035] All identified mutation points are arranged along the time axis, and the trajectory segment between every two adjacent mutation points is defined as a refitted trajectory segment unit.

[0036] For each trajectory segment, the mean value of velocity derivative (extract all velocity derivative values ​​and take their arithmetic average) and the total value of attitude change (subtract the starting and ending values ​​of pitch angle, yaw angle, and roll angle, take the absolute value and sum them) are calculated respectively.

[0037] If the mean value of the velocity derivative is greater than the empirical threshold value of the velocity derivative, the velocity weight coefficient is set to the mean value of the velocity derivative divided by the empirical threshold value of the velocity derivative; if the total value of the attitude change is greater than the change threshold value, the attitude weight coefficient is set to the total value of the attitude change divided by the change threshold value, and the final weighting factor takes the larger value of the two.

[0038] This weighting factor controls the density of control points during the spline refitting process: the original trajectory segment, which was interpolated every 200ms, is adjusted to a sampling interval of 200 divided by the final weighting factor. For example, if the final weighting factor is 2.0, then interpolation is performed every 100ms, increasing the sampling density.

[0039] Use B-splines (or Catmull-Rom splines) to fit the above trajectory segments. The spline control points are selected from the original trajectory segments at a weighted sampling rate. After the interpolated trajectory is generated, the timestamps, position triplets, and attitude triplets of the interpolated points are combined into a new state sequence, overwriting the original segment content.

[0040] At the same time, external disturbance parameters are introduced into the fitting process. These disturbances include crosswind angle, lift-drag difference, aerodynamic torque, and other parameters. The specific parameters are derived from meteorological data, wind field models, or historical empirical statistical models. These disturbance parameters are mapped into velocity correction terms and attitude adjustment terms through the disturbance response function and superimposed on the refitted trajectory to form an environmentally responsive enhanced state sequence.

[0041] Specifically, the obtained disturbance factors are normalized and combined to form a disturbance vector. The crosswind angle vector and aerodynamic torque coefficient vector in the disturbance vector are multiplied by empirical parameters and added together to obtain an additional pitch angle correction. The lift-to-drag ratio deviation coefficient in the disturbance vector is similarly multiplied by the empirical parameters to obtain an additional speed correction term.

[0042] Add the additional pitch angle correction to the pitch angle value of each fitting point; add the additional speed correction term to the speed value of the fitting point; recalculate the advancement value of the spatial position vector according to the speed change (advance the additional speed correction term multiplied by the time interval in the current speed direction).

[0043] Finally, the processed trajectory segments are concatenated in chronological order to generate a predicted trajectory covering the future time window. This trajectory consists of a position triplet, an attitude triplet, and a velocity scalar. The data structure is consistent with the initially constructed state data matrix. The state prediction sequence not only maintains the spatial continuity and physical consistency of the trajectory but also reflects the flight platform's true responsiveness to disturbances.

[0044] S2: Receive the operator's current interactive instruction data stream, and restructure the current interactive instruction data stream according to the time frame structure in the state prediction sequence to generate a set of structured instruction units; based on the set of structured instruction units and the state prediction sequence, combined with the instruction response lag tolerance and multi-directional envelope boundary parameters, generate a feedforward envelope instruction stream.

[0045] S2.1: Restructure the current interactive instruction data stream.

[0046] Receive the operator's current interaction instruction data stream, parse the timing field, action description field and target state field, divide it into continuous instruction segments according to the time axis, and semantically classify the action description field to obtain the basic action label set.

[0047] Specifically, in actual operation, the operator's interactive instruction data stream is first received through the real-time data bus. The data stream contains continuous instruction segments, and each segment of data consists of an independent instruction packet.

[0048] Each instruction packet consists of three main fields: a timestamp field for timing positioning, a behavioral semantic field for action description, and a target state field for target identification. The timestamp field, based on UTC or system synchronization time, is used to calibrate the time point when the instruction was triggered. The behavioral semantic field is an encoded value in natural language, icon encoding, or multi-level menu selection results, which is used to express the user's action intention. The target state field includes the spatial position vector (3D coordinates) and posture indication parameters (Euler angles, direction, or motion trend) specified or selected by the operator.

[0049] The received interactive data stream is arranged in order according to the timestamp field, and a threshold (such as 200ms) is set according to the time interval of consecutive instructions to divide the entire stream into several consecutive instruction segments.

[0050] Semantic classification is performed on the behavioral semantic fields in each instruction segment. Based on the action semantic graph, complex semantics are parsed into a basic action label set through semantic label mapping (e.g., a multi-level matcher based on a rule set or embedding model). The basic action label set consists of a finite set of standard action labels, such as "attack targeting," "lock-on flight," "track change pursuit," and "attitude switching," which serve as the classification primitives for action units.

[0051] Combining the basic action label set and the target state field, the corresponding standard action unit template is extracted from the preset action knowledge template library, and the predicted frame index of each target state is determined according to the instruction trigger time, and the action unit-frame index-duration term triple is established.

[0052] Furthermore, each action unit-frame index-duration item triple is mapped into a structured instruction unit, which contains the standard action type, target prediction frame index and standard duration fields, forming a structured instruction unit set.

[0053] S2.2: Generation of feedforward envelope instruction stream.

[0054] In the state prediction sequence, according to the target prediction frame index and standard duration field in the structured instruction unit, the state prediction frame of the corresponding time period is intercepted as the initial response candidate window. For example, if the prediction frame index is , the duration field is , the frame rate is (frames / second), the candidate window range is ,in, and The time offset margin indicating the operation response is preferably set to 1 to 3 frames.

[0055] The difference between the position vector and the attitude Euler angle of each state prediction frame in the initial response candidate window and the target state field in the corresponding structured instruction unit is calculated, and normalized to obtain the spatial dimension residual set and the attitude dimension residual.

[0056] Specifically, for each state frame in the candidate window, the difference between it and the target state field in the current structured instruction unit is calculated, and the Euclidean distance between the two in the spatial dimension is calculated respectively. The absolute angle difference from the Euler angle dimension of the posture All differences are normalized to a standard scale, such as , .

[0057] Filter the continuous state prediction frame segments whose residuals are lower than both the spatial residual threshold and the pose residual threshold, define them as state matching windows, and mark the first and last frame indexes.

[0058] With the command target position vector in the state matching window as the center, the angle range is set along the forward direction and the left and right lateral directions of the target aircraft velocity vector to generate a three-directional prediction vector group as the spatial direction reference edge; the command target position vector is the spatial position vector component in the target state field.

[0059] Specifically, the current speed direction of the target aircraft is taken as the forward axis, and with this axis as the center, the lateral angles (such as 30 degrees) are set on the left and right respectively to form a three-directional prediction vector group, which respectively represent: forward control vector, left deviation vector and right deviation vector. The three-directional vectors define the spatial reference edge of the control action.

[0060] The maximum controllable displacement distance is expanded on each spatial direction reference edge according to the target aircraft's inertial dynamic parameters (such as maximum acceleration, maximum roll angular rate, turning radius, etc.). The three form a fan-shaped or trapezoidal spatial envelope structure, and construct an envelope boundary parameter set with directional attributes.

[0061] Combined with the time response tolerance range set by the input delay statistical data of the operating end, the time envelope is constructed according to the time response advance and time response lag. The envelope boundary parameter set is embedded in the time envelope time period to form an envelope instruction segment, and the central control stable area and the edge transition buffer zone are divided at the same time.

[0062] A "control stability zone" is constructed with ±1 frame around the target prediction frame as the center, which allows the action instructions to be executed stably; an "edge transition buffer" is constructed with the edge of the stability zone ± the lag frame range to absorb disturbances such as delays and prediction errors.

[0063] The generated feedforward envelope instruction stream consists of multiple envelope instruction fragments in chronological order, and is nested with structured instruction unit labels and spatial posture window labels, realizing the full-link mapping from operator intention to controllable trajectory response, and ultimately providing a dynamic control pilot mechanism with high robustness and multiple redundancy guarantees.

[0064] S3: After generating the feedforward envelope instruction stream, the structured instruction units containing compound action labels are identified and template matched, and parsed into multi-level sub-instruction chains with frame index integrity.

[0065] S3.1: Extract action units containing compound action tags from the structured instruction unit set, and use the standard action type field as an index to load the preset high-level tactical action template set.

[0066] Among them, compound action labels include but are not limited to action types such as "compound tracking and avoidance", "concurrent attitude switching flight", and "combined reaction trajectory change". The corresponding operations are composed of multiple action primitives in the time and space dimensions and cannot be described by a single action model.

[0067] All compound action units are extracted to form a compound unit list, and a high-level tactical action template set is loaded based on the standard action type field value. The high-level tactical action template set is a preconfigured data structure on the platform, containing several compound action templates. Each template includes: a set of sub-action sequences (e.g., basic actions executed in the order A→B→C); the relative frame offset of each sub-action (e.g., A starts at frame 0, B starts at frame +20); and the spatial state transition rules for each sub-action (including spatial displacement vector, posture change target, duration, etc.). The template set has a two-layer index structure: the first layer searches based on compound action type code, and the second layer performs variant matching based on action context parameters.

[0068] S3.2: Based on the target prediction frame index and standard duration field associated with the action unit, the corresponding template structure is retrieved from the tactical action template set, and a sub-instruction chain containing a sub-action sequence, a relative time offset, and a target state transition sequence is output.

[0069] When executing, first parse the target prediction frame index and standard duration field contained in the current composite action unit. Indicates the anchor point and duration of the entire compound action Indicates the total duration of the entire action. Combined with the relative offset of the sub-action sequence in the template and standard duration allocation ratio , translate the target prediction frame index and assign it to each sub-action node to generate an absolute frame index sequence , and calculate the duration of each sub-action , each sub-action will be executed in this frame segment.

[0070] In terms of the state field, each sub-action in the template includes target state transition logic, such as posture maintenance + speed increase, spatial translation + direction switch, and other structures. The spatial position vector and posture Euler angle triplet in the original target state field of the composite action unit are used as the template's initial state parameters. Combined with the target state transformation rules for each sub-action, the state field is transferred and the specific target state vector corresponding to the sub-action is output.

[0071] S3.3: For each sub-action in the sub-instruction chain, parameter inheritance is performed according to the target state field in the structured instruction unit, and the absolute frame index and target state field of each sub-instruction are completed in combination with the matching state matching window in the state prediction sequence.

[0072] The process begins by searching the predicted state sequence for the closest matching state window for the target state field in each sub-action instruction. Specifically, candidate frames are located by frame index in the predicted sequence. The distance difference between each frame's spatial position vector and the sub-instruction's target state vector, as well as the difference in attitude Euler angles, is calculated. Frames that continuously meet the criteria are then filtered using predefined spatial and attitude residual thresholds. The first and last frame indices are recorded and used as the execution window frames for the sub-action instruction.

[0073] If the target state of sub-action B matches the frame segment arrive , then this segment is defined as the effective control interval of B. Within the effective interval, the instruction activation time point (such as the center frame) and the control buffer boundary (such as 1 frame before and after) are set, ultimately forming a complete time and space dual boundary of the sub-action instruction.

[0074] S3.4: Arrange all sub-instructions in frame index order and output a multi-level sub-instruction chain set with time controllability and structural consistency.

[0075] After determining the valid execution ranges for all sub-actions, these sub-commands are rearranged according to frame index order to form a multi-level set of sub-command chains with consistent temporal logic. Each chain structure contains the following fields: sub-action type identifier, absolute frame index start and end values, duration field, spatial target vector field, posture Euler angle field, matching status window frame segment identifier, and command priority field. The sub-command chain collection serves as a refined command input source for the control scheduler, supporting the invocation of control strategies such as gait adjustment, movement trajectory change, and precision correction.

[0076] Ultimately, the output multi-level sub-instruction chain set has time continuity, frame index integrity and state matching consistency, which can directly drive the control algorithm in future execution cycles, realize the automatic disassembly of the operator's tactical intentions and platform-level action choreography.

[0077] S4: Combining the state prediction sequence and the historical state data stream sent back by the target machine, perform time offset comparison on the multi-level sub-instruction chain and output a dynamically adjusted refined control data packet.

[0078] S4.1: Time offset comparison.

[0079] The state prediction frame corresponding to the target prediction frame index in each structured instruction unit is extracted, and the actual state frame set consisting of position triples, attitude triples and velocity scalars is extracted from the historical state data stream sent back by the target machine within the adjacent time window.

[0080] The historical status data stream sent back by the target machine contains the actual execution status data of the time interval corresponding to the deduced state prediction frame.

[0081] For all actual state frames within the time window of each target prediction frame, the difference vector of the spatial position vector and the attitude Euler angle field is calculated, and the frame with the smallest difference vector is identified as the response alignment frame and the timestamp is recorded.

[0082] Specifically, the predicted state frame contains the spatial position vector triple , attitude Euler angle triplet And the velocity scalar generated by deduction ; Each frame of the time series state frame set records the spatial position vector triplet at the actual execution time , attitude Euler angle triplet and velocity scalar A time window is constructed with the timestamp corresponding to the predicted frame as the center, and all state frames whose timestamps fall into this window are filtered from the return stream to construct the candidate set of actual state frames.

[0083] For each actual state frame in the historical state data stream sent back by the target machine, calculate the spatial difference vector between it and the target prediction frame: and attitude difference vector , and perform normalized weighting respectively to obtain the difference sum.

[0084] The returned frame with the smallest difference sum is selected as the response alignment frame of the current target prediction frame. This frame is considered to be the closest state to the actual execution response of the instruction in the current control cycle.

[0085] Furthermore, the timestamp difference between each set of target prediction frames and the corresponding response alignment frames is calculated and recorded as the time offset, and a time offset mapping table between the target prediction frame index and the return status frame index is constructed.

[0086] Finally, the correspondence between each target prediction frame index and response alignment frame index is recorded to construct a time offset mapping table. This table is a hash-type index table structure, with each entry containing fields such as the predicted frame number, the corresponding return frame number, the timestamp difference between the two, the spatial position residual, and the attitude angle residual. This mapping table will be used in subsequent steps to adjust time offsets, duration corrections, and instruction reordering within the sub-command chain.

[0087] Take a schematic scenario as an example: From T0 to T1, the target aircraft's real-time state is acquired, and the state sequence between T1 and T2 is predicted, forming a state prediction frame. Based on the predicted frame, the operator issues a structured instruction unit (SCU) and generates a feedforward envelope instruction stream. This instruction is executed, generating a sub-command chain (S1 to S3 complete). From T2 to T3, during actual flight, the target aircraft transmits back state data from T1 to T3. At this point, S4 is entered to compare the previous predicted frame (T1T2) with the actual transmitted state (T1T3). This achieves alignment and comparison of previously predicted frames using historically transmitted state data.

[0088] It should be noted that the state prediction frame is an expression of the theoretical control intention; the returned state frame reflects the actual flight control result; after comparing the difference between the two, a time offset mapping table is constructed to identify prediction lag or command response delay; and then the trigger frame index and trigger time of the command are corrected based on the offset, thereby realizing dynamic fine-tuning and synchronous optimization of future sub-command chains.

[0089] S4.2: Output the dynamically adjusted refined control data packet.

[0090] Based on the time offset in the time offset mapping table, synchronization adjustment is performed on the target prediction frame index bound to each sub-action to maintain time consistency with the actual state response frame.

[0091] After completing the frame index correction, the sub-actions in the sub-instruction chain are reordered in chronological order. After the sorting is completed, the integrity of the trigger intervals between temporally adjacent sub-actions is checked. For structures below the preset trigger interval threshold, the frame extension operation is performed to restore the controllability of the action interval.

[0092] After completing the sub-action time correction and sequence arrangement, the consistency of the action parameters is confirmed according to the standard action type field and target state field of each corrected sub-action, and each sub-action structure is converted into a structured instruction unit set with a standard field format.

[0093] This embodiment also provides a computer device, which is suitable for the human-computer interaction instruction generation method for remote collaborative control of a target machine, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the human-computer interaction instruction generation method for remote collaborative control of a target machine as proposed in the above embodiment.

[0094] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0095] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for generating human-computer interaction instructions for realizing remote collaborative control of a target machine as proposed in the above embodiment is implemented.

[0096] In summary, the present invention integrates a trajectory prediction mechanism based on historical state data and a feedforward control strategy. By constructing a state prediction sequence for the target aircraft in the short term, it can determine the response window of the control command in advance, significantly improving the time sensitivity and execution matching degree of the control command. The use of structured instruction unit generation and multi-level sub-instruction chain analysis enables modular expression and fine decomposition of compound actions, effectively improving the control accuracy and logical consistency of complex operations. In addition, by utilizing spatial attitude residual screening, envelope boundary modeling, and time tolerance design, stable control areas and buffer zones can be established in multiple directions, improving the fault tolerance and controllability in high-dynamic flight conditions. Time offset comparison enhances the adaptive adjustment capability for non-ideal factors such as execution errors and delay changes, ensuring that the human-machine instruction chain has high robustness and real-time performance in remote high-frequency collaborative scenarios.

[0097] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for generating human-computer interaction instructions for remote collaborative control of a target drone, characterized by: include: Based on the historical state data stream sent back by the target aircraft, the continuous state space trajectory of the target aircraft in the short future period is constructed, and the state prediction sequence for pre-judging the command response window is output; Receive the operator's current interactive instruction data stream, and restructure the current interactive instruction data stream according to the time frame structure in the state prediction sequence to generate a structured instruction unit set; Based on the structured instruction unit set and the state prediction sequence, combined with the instruction response lag tolerance and the multi-directional envelope boundary parameters, a feedforward envelope instruction stream is generated; After generating the feedforward envelope instruction stream, the structured instruction units containing compound action labels are identified and template matched, and parsed into multi-level sub-instruction chains with frame index integrity; Combining the state prediction sequence and the historical state data stream sent back by the target machine, time offset comparison is performed on the multi-level sub-instruction chain, and a dynamically adjusted refined control data packet is output.

2. The method for generating human-computer interaction instructions for remote collaborative control of a target drone according to claim 1, wherein: The generation of the state prediction sequence includes: Receive the continuous state data stream sent back by the target drone, align the position triplet, attitude triplet, and velocity scalar fields according to the timestamp, and construct a state data matrix containing six-dimensional fields of time series, space, and attitude; Performing piecewise cubic interpolation fitting on the spatial and attitude sequences in adjacent time periods in the state data matrix, and calculating the first-order derivative curve for each interpolation result to obtain a set of velocity-attitude change rate curves; In the first-order derivative curve, mutation points of acceleration or attitude angle are identified as trend mutation nodes, and the continuous trajectory segments between the trend mutation nodes are refitted. The interpolated predicted trajectory is corrected in combination with the environmental modeling parameters to generate a state prediction sequence within the future time window.

3. The method for generating human-computer interaction instructions for remote collaborative control of a target drone according to claim 1, wherein: The structural reconstruction of the current interactive instruction data stream includes: Receive the operator's current interactive instruction data stream, parse the timing field, action description field, and target state field, divide it into continuous instruction segments according to the time axis, and semantically classify the action description field to obtain the basic action label set; Combining the basic action label set and the target state field, the corresponding standard action unit template is extracted from the preset action knowledge template library. The predicted frame index of each target state is determined according to the instruction trigger time, and a triplet of action unit, frame index and duration is established. Each action unit-frame index-duration triplet is mapped into a structured instruction unit, which contains the standard action type, target prediction frame index and standard duration fields, forming a set of structured instruction units.

4. The method for generating human-computer interaction instructions for remote collaborative control of a target drone according to claim 3, wherein: The generation of the feedforward envelope instruction stream includes: In the state prediction sequence, according to the target prediction frame index and the standard duration field in the structured instruction unit, the state prediction frame of the corresponding time period is intercepted as the initial response candidate window; Calculate the difference between the position vector and the attitude Euler angle of each state prediction frame and the target state field in the corresponding structured instruction unit in the initial response candidate window, and normalize them to obtain the spatial dimension residual set and the attitude dimension residual; Filter the continuous state prediction frame segments whose residuals are lower than both the spatial residual threshold and the posture residual threshold, which are defined as the state matching window; With the command target position vector in the state matching window as the center, the angle ranges along the forward direction and the left and right lateral directions of the target aircraft velocity vector are set to generate a three-directional prediction vector group as the spatial direction reference edge; Expand the maximum controllable displacement distance on each spatial direction reference edge according to the target drone's inertial dynamic parameters, and construct an envelope boundary parameter set with directional attributes; Combined with the time response tolerance range set by the input delay statistical data of the operating end, the time envelope is constructed according to the time response advance and time response lag. The envelope boundary parameter set is embedded in the time envelope time period to form an envelope instruction segment, and the central control stable area and the edge transition buffer zone are divided at the same time.

5. The method for generating human-computer interaction instructions for remote collaborative control of a target drone according to claim 4, characterized in that: The instruction target position vector is the spatial position vector component in the target state field.

6. The method for generating human-computer interaction instructions for remote collaborative control of a target drone according to claim 1, wherein: The multi-level sub-instruction chain parsed as having frame index integrity includes: Extract action units containing compound action tags from the structured instruction unit set, and use the standard action type field as an index to load the preset high-level tactical action template set; According to the target prediction frame index and standard duration field associated with the action unit, the corresponding template structure is retrieved from the tactical action template set, and a sub-instruction chain containing sub-action sequences, relative time offsets, and target state transition sequences is output; For each sub-action in the sub-instruction chain, parameter inheritance is performed based on the target state field in the structured instruction unit, and the absolute frame index and target state field of each sub-instruction are completed in combination with the state matching window matched in the state prediction sequence; Arrange all sub-instructions in frame index order and output a multi-level sub-instruction chain set with time controllability and structural consistency.

7. The method for generating human-computer interaction instructions for remote collaborative control of a target drone according to claim 1, wherein: The time offset comparison includes: Extract the state prediction frame corresponding to the target prediction frame index in each structured instruction unit, and extract the actual state frame set consisting of position triples, attitude triples and velocity scalars within adjacent time windows from the historical state data stream sent back by the target machine; For all actual state frames within the time window of each target prediction frame, calculate the difference vector between the spatial position vector and the attitude Euler angle field, identify the frame with the smallest difference vector as the response alignment frame, and record the timestamp; Calculate the timestamp difference between each set of target prediction frames and the corresponding response alignment frames, record it as the time offset, and build a time offset mapping table between the target prediction frame index and the return status frame index.

8. The method for generating human-computer interaction instructions for remote collaborative control of a target drone according to claim 7, wherein: The output dynamically adjusted refined control data packet includes: Based on the time offset in the time offset mapping table, the target prediction frame index bound to each sub-action is synchronized to ensure time consistency with the actual state response frame. After completing the frame index correction, the sub-actions in the sub-command chain are reordered in chronological order. After the sorting is completed, the integrity of the trigger intervals between sub-actions that are close in time is checked. For structures below the preset trigger interval threshold, the frame interval extension operation is performed. According to the standard action type field and target state field of each revised sub-action, the consistency of the action parameters is confirmed, and each sub-action structure is converted into a set of structured instruction units with a standard field format.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for generating human-computer interaction instructions for remote collaborative control of a target machine as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating human-computer interaction instructions for remote collaborative control of a target machine as described in any one of claims 1 to 8 are implemented.

Images