A man-machine interactive instruction generation method for remote cooperative control of a target machine

Abstract

The application discloses a kind of man-machine interaction instruction generation methods of target machine remote cooperative control, it is related to man-machine interaction instruction generation technical field.The application is fused trajectory prediction mechanism and feedforward control strategy based on historical state data, by constructing the state prediction sequence of target machine short future, the response window of control instruction can be judged in advance, the time sensitivity of control instruction and execution matching degree are greatly improved;Using structured instruction unit generation and multi-level sub-instruction chain analysis, so that composite action can be modularized expression and fine disassembly, effectively improve the control precision and logic consistency of complex operation.In addition, using space posture residual error screening, envelope boundary modeling and time tolerance design, stable control area and buffer area can be established in multiple directions, to improve fault tolerance and controllability under high dynamic flight state.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction instruction generation, and particularly relates to a human-computer interaction instruction generation method for remote cooperative control of a target machine. BACKGROUND

[0002] With the continuous development of modern target machine control technology, the application demand of human-computer interaction systems in high dynamic and high response precision scenes is increasing, especially in remote target machine cooperative control. In order to effectively realize target trajectory simulation and complex action execution, higher requirements are put forward for the intelligentization, structurization and feedforward of the instruction generation mechanism. The traditional target machine control method mainly relies on manual instruction issued by operators or simple rule control systems, and has problems such as response lag, large execution deviation and weak dynamic adaptability. In recent years, with the application of data prediction modeling, human-computer collaborative perception and instruction structure optimization, remote target machine control gradually evolves towards high real-time, predictability and data fusion driven direction. How to build a human-computer interaction instruction generation system facing future situation prediction to improve the response precision and trajectory fitting degree of target machine control has become an important direction of research in this field.

[0003] CN111857177B discloses a remote control target instruction generation method, device, equipment and medium. The driver generates a control instruction based on the target motion characteristics, and an instruction corrector is used to convert the control instruction into an equivalent instruction to realize the control of the target flight trajectory. The method uses the “equivalence similarity principle” to make the simulated target motion approach the flight characteristics of the target fighter, thereby improving the matching degree and action restoration ability of the instruction to a certain extent. However, this technology relies on the immediacy and accuracy of the current instruction of the driver, and fails to establish a state prediction model facing the future time period, lacks a tolerance identification and correction mechanism for instruction response lag, and does not structurally model the instruction data stream, which is prone to problems such as instruction distortion and system delay in complex action control scenarios.

[0004] CN115127400B discloses a control method of a target score generation control system, which improves the training efficiency of the shooter through shooting data back transmission and real-time feedback mechanism. The main technical features include target machine and single-chip microcomputer linkage control, data processing display and Bluetooth voice broadcast, which embodies the basic form of human-computer interaction. Although this technology has a certain practicality in the training scene, its control mechanism mainly relies on passive response after data back transmission, lacks the ability to predict the future state of the target machine, and does not involve the deconstruction and time offset correction of the composite instruction chain, and does not have the fine processing ability of multi-level control units. SUMMARY

[0005] In view of the problems of existing target machine control technology in simulating real flight state and improving control response precision, the present application is proposed.

[0006] Therefore, the problem to be solved by the present application is how to solve the problems of instruction response lag and low action matching degree in the conventional method.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] In a first aspect, the present application provides a target machine remote cooperative control human-computer interaction instruction generation method, which comprises: based on the historical state data stream returned by the target machine, constructing a continuous state space trajectory of the target machine in a short future time period, and outputting a state prediction sequence for pre-judging an instruction response window; receiving an operator's current interaction instruction data stream, and performing structural reconstruction on the current interaction instruction data stream according to the time frame structure in the state prediction sequence, to generate a structured instruction unit set; based on the structured instruction unit set and the state prediction sequence, combining instruction response lag tolerance and multi-direction envelope boundary parameters to generate a feedforward envelope instruction stream; after generating the feedforward envelope instruction stream, identifying and template matching the structured instruction unit containing a composite action label, and parsing into a multi-level sub-instruction chain with frame index integrity; combining the state prediction sequence and the historical state data stream returned by the target machine, performing time offset comparison on the multi-level sub-instruction chain, and outputting a fine control data packet after dynamic adjustment.

[0009] As a preferred scheme of the target machine remote cooperative control human-computer interaction instruction generation method of the present application, the generation of the state prediction sequence comprises: receiving the continuous state data stream returned by the target machine, field aligning the position triple, attitude triple and velocity scalar according to the time stamp, and constructing a state data matrix containing six-dimensional fields of time sequence, space and attitude; performing piecewise cubic interpolation fitting on the space and attitude sequence in the adjacent time period in the state data matrix, and calculating the first derivative curve of each piece of interpolation result to obtain a set of velocity-attitude change rate curves; identifying the acceleration or attitude angle change mutation point in the first derivative curve as a trend mutation node, and re-fitting the continuous trajectory segment between the trend mutation nodes, combining with the environment modeling parameter to correct the interpolation prediction trajectory, and generating the state prediction sequence in the future time window.

[0010] As a preferred scheme of the human-machine interaction instruction generation method for remote cooperative control of the target machine according to the application, wherein: the structural reconstruction of the current interaction instruction data stream comprises: receiving the current interaction instruction data stream of the operator, parsing the time sequence field, the action description field and the target state field, cutting into continuous instruction segments according to the time axis, and performing semantic classification on the action description field to obtain a basic action label set; combining the basic action label set and the target state field, extracting the corresponding standard action unit template from the preset action knowledge template library, determining the prediction frame index where each target state is located according to the instruction trigger time, and establishing the action unit-frame index-time length item triple; mapping each action unit-frame index-time length item triple into a structured instruction unit, including the standard action type, the target prediction frame index and the standard duration field, and constituting a structured instruction unit set.

[0011] As a preferred scheme of the human-machine interaction instruction generation method for remote cooperative control of the target machine according to the application, wherein: the generation of the feedforward envelope instruction stream comprises: in the state prediction sequence, according to the target prediction frame index and the standard duration field in the structured instruction unit, intercepting the state prediction frame in the corresponding time period as an initial response candidate window; calculating the difference between each state prediction frame in the initial response candidate window and the target state field in the corresponding structured instruction unit in the position vector and the attitude Euler angle, and normalizing to obtain a spatial dimension residual set and an attitude dimension residual; screening the continuous state prediction frame section whose residual is simultaneously lower than the spatial residual threshold and the attitude residual threshold, and defining it as a state matching window; setting an angle range along the forward direction and the left and right transverse direction of the target machine velocity vector direction with the instruction target position vector in the state matching window as the center, generating a three-direction prediction vector group as a spatial direction reference edge; expanding the maximum controllable displacement distance on each spatial direction reference edge according to the inertial dynamic parameters of the target machine, and constructing an envelope boundary parameter set with direction attribute; combining the time response tolerance range set according to the input delay statistical data of the operation end, constructing the time envelope according to the time response advance and the time response lag, embedding the envelope boundary parameter set in the time envelope time period to form an envelope instruction segment, and simultaneously dividing the center control stable area and the edge transition buffer area.

[0012] As a preferred scheme of the human-machine interaction instruction generation method for remote cooperative control of the target machine according to the application, wherein: the instruction target position vector is the spatial position vector component in the target state field.

[0013] As a preferred scheme of the human-computer interaction instruction generation method for remote cooperative control of the target machine according to the application, wherein: the parsing is a multi-level sub-instruction chain with frame index integrity, including: in the structured instruction unit set, extracting the action unit containing the composite action label, and loading the preset high-order tactical action template set with the standard action type field as the index; according to the target prediction frame index and the standard duration field associated with the action unit, searching the corresponding template structure in the tactical action template set, and outputting the sub-instruction chain containing the sub-action sequence, the relative time offset and the target state transition sequence; for each sub-action in the sub-instruction chain, inheriting the parameters according to the target state field in the structured instruction unit, and combining the matched state matching window in the state prediction sequence to complete the absolute frame index and the target state field of each sub-instruction; arranging all the sub-instructions in the order of frame index, and outputting the multi-level sub-instruction chain set with time controllability and structural consistency.

[0014] As a preferred scheme of the human-computer interaction instruction generation method for remote cooperative control of the target machine according to the application, wherein: the time offset comparison includes: extracting the state prediction frame corresponding to the target prediction frame index in each structured instruction unit, and extracting the actual state frame set composed of the position triple, the attitude triple and the velocity scalar in the adjacent time window from the historical state data stream returned by the target machine; for each target prediction frame in the time window, calculating the difference vector of the spatial position vector and the attitude Euler angle field, and identifying the frame with the minimum difference vector as the response alignment frame, and recording the timestamp; calculating the timestamp difference between each group of target prediction frames and the corresponding response alignment frame, and recording it as the time offset, and constructing the time offset mapping table between the target prediction frame index and the returned state frame index.

[0015] As a preferred scheme of the human-computer interaction instruction generation method for remote cooperative control of the target machine according to the application, wherein: the output of the dynamically adjusted fine control data packet includes: taking the time offset in the time offset mapping table as a reference, synchronously adjusting the target prediction frame index bound to each sub-action to maintain time consistency with the actual state response frame; after completing the frame index correction, reordering the sub-actions in the sub-instruction chain according to the time sequence, and after the sorting is completed, performing integrity verification on the trigger interval between the time adjacent sub-actions, and performing frame distance extension operation for the structure lower than the preset trigger interval threshold; according to the standard action type field and the target state field of each corrected sub-action, the action parameters are consistent, and each sub-action structure is converted into a structured instruction unit set with standard field format.

[0016] In a second aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program instructs the processor to implement the steps of the method for generating human-machine interaction instructions for remote cooperative control of a target machine according to the first aspect of the present application.

[0017] In a third aspect, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program instructs the processor to implement the steps of the method for generating human-machine interaction instructions for remote cooperative control of a target machine according to the first aspect of the present application.

[0018] The present application has the following advantages: the present application combines a trajectory prediction mechanism based on historical state data and a feedforward control strategy, can judge the response window of the control instruction in advance by constructing a state prediction sequence of the target machine in the short future, and greatly improves the time sensitivity and execution matching degree of the operation instruction; the use of structured instruction unit generation and multi-level sub-instruction chain analysis enables modular expression and fine disassembly of complex actions, effectively improving the control accuracy and logical consistency of complex operations. In addition, the use of spatial pose residual screening, envelope boundary modeling and time tolerance design can establish stable control zones and buffer zones in multiple directions, improve the fault tolerance and controllability in high dynamic flight states; and the time offset comparison enhances the adaptive adjustment capability to non-ideal factors such as execution error and delay change, ensuring that the human-machine instruction chain has high robustness and real-time performance in the remote high-frequency cooperative scene. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 The flowchart of the method for generating human-machine interaction instructions for remote cooperative control of a target machine. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0022] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from the description, and those skilled in the art can make similar generalizations without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0023] Second, the "one embodiment" or "an embodiment" referred to herein can include a particular feature, structure, or characteristic. The various embodiments are not mutually exclusive, but a single embodiment can be selected from a plurality of mutually exclusive or alternative embodiments.

[0024] As described in the above background, the traditional target control method mainly relies on the operator's manual instruction or simple rule control system, and there are problems such as response lag, large execution deviation, and weak dynamic adaptation. In recent years, with the application of data prediction modeling, human-computer collaborative perception, and instruction structure optimization technologies, remote target control gradually evolves towards high real-time, predictability, and data fusion driven direction. How to build a human-computer interaction instruction generation system for future situation prediction to improve the response accuracy and trajectory fitting degree of target control has become an important direction of research in this field.

[0025] Figure 1 The flow chart of the target remote collaborative control human-computer interaction instruction generation method according to the embodiment of the application. As shown in Figure 1 The target remote collaborative control human-computer interaction instruction generation method includes the following steps:

[0026] S1: Based on the historical state data stream returned by the target, the continuous state space trajectory of the target in the short future period is constructed, and the state prediction sequence for pre-judging the instruction response window is output.

[0027] S1.1: Receive the continuous state data stream returned by the target, align the position triple, attitude triple and velocity scalar fields according to the time stamp, and construct a state data matrix containing six-dimensional fields of time sequence, space and attitude.

[0028] Specifically, the field alignment operation needs to use a high-precision clock as the time base reference. For each data record, the time stamp field is extracted, and a preliminary sorting is completed according to the time sequence. Then, the time difference is uniformly calibrated with millisecond-level precision, and the linear time backtracking processing is performed for the time jitter caused by data link delay.

[0029] The position triple is unified as an absolute position vector (X, Y, Z) in the geodetic coordinate system through a geographic coordinate conversion function, the attitude triple (pitch angle, yaw angle, roll angle) is unified as an angle triple in the standard rotation order through the Euler angle conversion rule, and the velocity scalar is converted into the velocity module value after direction calibration of the unit vector. After the above fields are standardized, the state data vector is constructed at each time, and the time sequence state matrix is spliced.

[0030] It should be noted that through field alignment and matrix construction, high-density structured packaging of target machine historical state data can be effectively realized. In addition, the six-dimensional state data matrix has the advantages of unified time base and standardized coordinate system, and has good scalability when multi-platform fusion or prediction model migration is performed.

[0031] S1.2: Perform piecewise cubic interpolation fitting on the space and attitude sequence in the adjacent time period in the state data matrix, and calculate the first derivative curve of each piecewise interpolation result to obtain a set of velocity-attitude change rate curves.

[0032] To solve the discrete jump and data window problem of the target machine in the state sequence, the interpolation modeling operation of the present application is implemented, and the method adopted is piecewise cubic spline interpolation technology. In specific operation, first, according to the time sequence field in the state data matrix, all data are divided into multiple continuous intervals, the interval length is dynamically adjusted according to the platform return frequency and the instruction control response time window, and preferably each 10 frames form an interpolation segment. Within each interpolation segment, cubic spline function groups are constructed for the spatial position vector (three-dimensional) and the attitude Euler angle (three-dimensional), respectively, each dimension is modeled separately, and the interpolation function satisfies the continuity of position and first derivative at the segment point, ensuring the physical reasonableness and dynamic coherence of the trajectory.

[0033] After interpolation, the first derivative function is calculated for each dimension of the interpolation function. For the spatial position part, the first derivative corresponds to the velocity vector (i.e. the rate of change of position); for the attitude angle part, the first derivative corresponds to the attitude angle change rate, reflecting the platform attitude adjustment speed or rotation trend. All first derivative functions are spliced according to the time sequence to form a set of velocity-attitude change rate curves.

[0034] Compared with traditional linear interpolation, cubic spline has high-order continuity characteristics, which can more accurately fit the real flight trend of the target machine in the nonlinear stage of rapid attitude switching and trajectory edge adjustment.

[0035] S1.3: Identify acceleration or attitude angle change mutation points in the first derivative curve as trend mutation nodes, and re-fit the continuous trajectory segment between the trend mutation nodes, combine with the environment modeling parameter to correct the interpolation prediction trajectory, and generate a state prediction sequence in the future time window.

[0036] For the constructed set of first derivative curves, first, perform inflection point search for each curve. Perform the following steps:

[0037] For each data point in the velocity derivative curve, calculate the change rate between the adjacent two points before and after it;

[0038] If the current point and the rate of change of the sign of the opposite (i.e. derivative slope direction reversal), and the absolute value of the rate of change more than twice the average, marked as a speed mutation point;

[0039] The derivative curve of the pitch angle, yaw angle, roll angle is operated respectively, and the attitude mutation point is marked.

[0040] All identified mutation points are arranged in time axis, and the trajectory segment between each two adjacent mutation points is defined as a re-fitting trajectory segment unit.

[0041] For each trajectory segment, the average value of the velocity derivative (extract all the velocity derivative values, take the arithmetic mean) and the total value of the attitude change (the difference between the initial value and the final value of the pitch angle, yaw angle, roll angle, take the absolute value and sum) are calculated respectively.

[0042] If the average value of the velocity derivative is greater than the empirical threshold value of the velocity derivative, set the velocity weight coefficient as the average value of the velocity derivative divided by the empirical threshold value of the velocity derivative; if the total value of the attitude change is greater than the threshold value of the change, set the attitude weight coefficient as the total value of the attitude change divided by the threshold value of the change, and the final weighting factor takes the larger value of the two.

[0043] This weighting factor controls the control point density in the spline re-fitting process: the original trajectory segment is adjusted to 200 divided by the final weighting factor for every 200ms interpolation sampling interval. For example, if the final weighting factor is 2.0, interpolate once every 100ms to increase the sampling density.

[0044] Use B-spline (or Catmull-Rom spline) to fit the above trajectory segment, and the spline control points are the trajectory points selected from the original trajectory segment at the weighted sampling rate. After generating the interpolated trajectory, the time stamp, position triple, and attitude triple of the interpolated points are combined to form a new state sequence, which covers the original segment content.

[0045] At the same time, external disturbance parameters are introduced in the fitting process, and the disturbance content includes wind angle, lift-drag difference, aerodynamic moment, etc. The specific parameters come from meteorological data, wind field model or historical experience statistical model. These disturbance parameters are mapped into velocity correction term and attitude adjustment term through disturbance response function, superimposed in the re-fitting trajectory, forming an environment response enhanced state sequence.

[0046] Specifically, the obtained disturbance factor is normalized, and the normalized factor is combined into a disturbance vector. The wind angle vector and the aerodynamic moment coefficient vector in the disturbance vector are multiplied by the empirical parameters and added to obtain the additional pitch angle correction; the lift-drag ratio deviation coefficient in the disturbance vector is also multiplied by the empirical parameter to obtain the additional velocity correction term.

[0047] An additional pitch angle correction is added to the pitch angle value of each fitting point; an additional speed correction term is added to the fitting point speed value; the propulsion value of the spatial position vector is recalculated according to the speed change (propel the additional speed correction term multiplied by the time interval in the current speed direction).

[0048] Finally, the processed trajectory segments are spliced in chronological order to generate a prediction trajectory covering the future time window, composed of position triplets, attitude triplets and velocity scalars, consistent with the data structure and the initially constructed state data matrix. The state prediction sequence not only maintains the spatial continuity and physical consistency of the trajectory, but also reflects the true response capability of the flight platform in a disturbed environment.

[0049] S2: Receive the operator's current interaction instruction data stream, and reconstruct the structure of the current interaction instruction data stream according to the time frame structure in the state prediction sequence to generate a set of structured instruction units; based on the set of structured instruction units and the state prediction sequence, generate a feedforward envelope instruction stream in combination with the instruction response lag tolerance and the multi-directional envelope boundary parameters.

[0050] S2.1: Reconstruct the structure of the current interaction instruction data stream.

[0051] Receive the operator's current interaction instruction data stream, parse the time sequence field, action description field and target state field, and divide it into continuous instruction segments according to the time axis, and perform semantic classification on the action description field to obtain a set of basic action labels.

[0052] Specifically, in actual operation, first, the operator's interaction instruction data stream is received through the real-time data bus, and the data stream contains continuous instruction segments, each segment consisting of an independent instruction packet.

[0053] Each instruction packet includes three main fields: a timestamp field for time sequence positioning, a behavior semantic field for action description, and a target state field for target identification. The timestamp field is based on UTC time or system synchronization time, used to mark the time point of the instruction trigger; the behavior semantic field is a natural language, icon code or multi-level menu selection result code value, used to represent the user's action intention; the target state field includes the spatial position vector (three-dimensional coordinates) and attitude indication parameters (Euler angles, direction pointing or motion trend) specified or selected by the operator.

[0054] The received interaction data stream is arranged in order according to the timestamp field, and a threshold value (such as 200ms) is set according to the time interval of continuous instructions to divide the entire stream into several continuous instruction segments.

[0055] The semantic categorization operation is performed on the behavior semantic field in each instruction segment, and the complex semantics is parsed into a set of basic action labels based on the action semantic atlas through semantic label mapping (such as a multi-level matcher based on a rule set or an embedded model). The set of basic action labels is composed of a limited set, such as standard action labels such as "attack aiming", "lock flight", "orbit switching pursuit", "pose switching", etc., as the classification primitives of action units.

[0056] In combination with the set of basic action labels and the target state field, a corresponding standard action unit template is extracted from a preset action knowledge template library, and the prediction frame index where each target state is located is determined according to the instruction trigger time, and a triple of action unit-frame index-duration item is established.

[0057] Further, each action unit-frame index-duration item triple is mapped to a structured instruction unit, including a standard action type, a target prediction frame index, and a standard duration field, to form a set of structured instruction units.

[0058] S2.2: Generation of feedforward envelope instruction stream.

[0059] In the state prediction sequence, according to the target prediction frame index and the standard duration field in the structured instruction unit, the state prediction frame in the corresponding time period is intercepted as an initial response candidate window. For example, if the prediction frame index is , the duration field is , and the frame rate is (frames / second), the candidate window range is , wherein and represent the time offset margin of the operation response, which is preferably set to 1-3 frames.

[0060] The difference between each state prediction frame in the initial response candidate window and the target state field in the corresponding structured instruction unit in the position vector and the attitude Euler angle is calculated, and the spatial dimension residual set and the attitude dimension residual are obtained by normalization.

[0061] Specifically, for each state frame in the candidate window, the difference between the target state field in the current structured instruction unit is calculated, and the Euclidean distance in the spatial dimension and the absolute angle difference in the attitude Euler angle dimension are calculated. All differences are normalized to a standard scale, such as , .

[0062] The continuous state prediction frame segment whose residual is simultaneously lower than the spatial residual threshold and the attitude residual threshold is screened, defined as a state matching window, and the first and last frame indexes are marked.

[0063] The instruction target position vector in the state matching window is taken as the center, and angle ranges are set in the forward direction and the left and right transverse directions along the target speed vector direction to generate a three-direction prediction vector group as a spatial direction reference edge. The instruction target position vector is a spatial position vector component in the target state field.

[0064] Specifically, the current speed direction of the target is taken as the forward axis, and transverse angles (such as 30 degrees) are set to the left and right of the axis to form a three-direction prediction vector group, which respectively represents a forward control vector, a left bias vector and a right bias vector. The three-direction vectors define the spatial reference edge of the control action.

[0065] The maximum controllable displacement distance is expanded on each spatial direction reference edge according to the inertial dynamic parameters (such as maximum acceleration, maximum roll angular rate, turning radius, etc.) of the target, and the three form a fan-shaped or trapezoidal spatial envelope structure to construct an envelope boundary parameter set with direction attributes.

[0066] The time envelope is constructed according to the time response advance and the time response lag in combination with the time response tolerance range set by the input delay statistical data of the operating end, and the envelope boundary parameter set is embedded in the time envelope to form an envelope instruction segment, and the center control stable area and the edge transition buffer area are divided.

[0067] The "control stable area" is constructed around the target prediction frame ±1 frame, and the interval allows stable execution of the action instruction; the "edge transition buffer area" is constructed in the ± lag frame range at the edge of the stable area, which is used to absorb disturbances such as delay and prediction error.

[0068] The generated feedforward envelope instruction stream is composed of multiple envelope instruction segments in time sequence, and is nested with structured instruction unit tags and spatial attitude window tags to realize full-link mapping from operator intent to controllable trajectory response, and finally provide a dynamic control leading mechanism with high robustness and multiple redundant guarantees.

[0069] S3: After generating the feedforward envelope instruction stream, the structured instruction unit containing the composite action tag is identified and template matched to be parsed into a multi-level sub-instruction chain with frame index integrity.

[0070] S3.1: In the structured instruction unit set, extract the action unit containing the composite action tag, and load the preset high-order tactical action template set with the standard action type field as the index.

[0071] The composite action tag includes but is not limited to "composite tracking avoidance", "concurrent attitude switching flight", "combined reaction orbit change", etc. The corresponding operation is composed of multiple action primitives in time and space dimensions, and cannot be described by a single action model.

[0072] All composite action units are extracted to form a composite unit list, and a high-level tactical action template set is loaded according to the standard action type field value. The high-level tactical action template set is a platform preconfigured data structure, containing a plurality of composite action templates, each template including: a sub-action sequence set (such as basic actions executed in the order of A→B→C); the relative frame offset of each sub-action (such as A starting at frame 0 and B starting at frame +20); and the spatial state transition rule of each sub-action (including spatial displacement vector, attitude change target, duration, etc.). The template set has a double-layer index structure, the first layer is retrieved according to the composite action type code, and the second layer is matched according to the action context parameters.

[0073] S3.2: According to the target predicted frame index and the standard duration field associated with the action unit, retrieve the corresponding template structure in the tactical action template set, and output a sub-instruction chain containing a sub-action sequence, a relative time offset, and a target state transition sequence.

[0074] In specific implementation, first, the target predicted frame index and the standard duration field contained in the current composite action unit are parsed. The predicted frame index represents the anchor point of the entire composite action, and the duration represents the total duration covered by the entire action. Combined with the relative offset of the sub-action sequence in the template and the standard duration allocation ratio , the target predicted frame index is shifted and allocated to each sub-action node to generate an absolute frame index sequence , and the duration of each sub-action is calculated , each sub-action will be executed in this frame segment.

[0075] In terms of the state field, each sub-action in the template contains target state transition logic, such as attitude holding + speed improvement, spatial translation + direction switching, etc. The spatial position vector and the Euler angle triplet in the original target state field in the composite action unit are input as the template initial state parameters, combined with the target state transformation rule of each sub-action, the state field is transferred, and the specific target state vector corresponding to the sub-action is output.

[0076] S3.3: For each sub-action in the sub-instruction chain, the parameters are inherited according to the target state field in the structured instruction unit, and combined with the matched state matching window in the state prediction sequence, the absolute frame index and the target state field of each sub-instruction are completed.

[0077] In the operation process, firstly, the target state field in each sub-action instruction is searched for the closest state matching window in the state prediction sequence. The specific method is as follows: locating the candidate frame segment in the prediction sequence according to the frame index, calculating the distance difference value of the spatial position vector of each frame and the target state vector of the sub-instruction, and the attitude Euler angle difference value. The frame segment continuously satisfying the condition is filtered through the set spatial residual threshold and attitude residual threshold, and the first and last frame indexes are recorded as the execution window frame segment of the sub-action instruction.

[0078] If the frame segment matched with the target state of the sub-action B is to , the segment is defined as the effective control interval of B. The instruction activation point (for example, the center frame) and the control buffer boundary (for example, 1 frame before and after) are set in the effective interval, and finally the complete time and space double boundary of the sub-action instruction is formed.

[0079] S3.4: Arrange all sub-instructions in the order of frame index, and output the multi-level sub-instruction chain set with time controllability and consistent structure.

[0080] After determining the effective execution segments of all sub-actions, the sub-instructions are rearranged in the order of frame index to form a multi-level sub-instruction chain set with consistent time logic. Each chain structure includes the following fields: sub-action type identifier, absolute frame index start and end value, duration field, spatial target vector field, attitude Euler angle field, matching state window frame segment identifier, and instruction priority field. The sub-instruction chain set is used as the refined instruction input source of the control scheduler, supporting future control strategies such as gait adjustment, action trajectory change, or precision correction.

[0081] Finally, the output multi-level sub-instruction chain set has time continuity, frame index integrity, and state matching consistency, and can directly drive the control algorithm in future execution cycles to realize the automatic decomposition of the operator's tactical intention and platform-level action arrangement.

[0082] S4: Combine the state prediction sequence and the historical state data stream returned by the target machine to perform time offset comparison on the multi-level sub-instruction chain, and output the dynamically adjusted refined control data package.

[0083] S4.1: Time offset comparison.

[0084] Extract the state prediction frame corresponding to the target prediction frame index in each structured instruction unit, and extract the actual state frame set composed of position triplets, attitude triplets and velocity scalars in the adjacent time window from the historical state data stream returned by the target machine.

[0085] The historical state data stream returned by the target machine contains the actual execution state data of the time interval corresponding to the deduced state prediction frame.

[0086] For each target prediction frame, the difference vectors of spatial position vector and attitude Euler angle field are calculated for all actual state frames within the time window, and the frame with the minimum difference vector is identified as the response alignment frame, and the timestamp is recorded.

[0087] Specifically, the prediction state frame contains a spatial position vector triplet , an attitude Euler angle triplet and a deduced generated velocity scalar ; the time series state frame set records the spatial position vector triplet , the attitude Euler angle triplet and the velocity scalar at the actual execution time of each frame. A time window is constructed around the timestamp of the prediction frame, and all state frames with timestamps falling within this window are selected from the backstream to construct the actual state frame candidate set.

[0088] For each actual state frame in the historical state data stream returned by the target, the spatial difference vector between the target prediction frame and the actual state frame is calculated:

[0089] and the attitude difference vector , and the difference vector is obtained by normalizing and weighting respectively.

[0090] The backstream frame with the minimum difference vector is selected as the response alignment frame of the current target prediction frame, and this frame is considered to be the closest state of the actual execution response of the instruction in the current control period.

[0091] Further, the timestamp difference between each group of target prediction frames and the corresponding response alignment frame is calculated, denoted as time offset, and a time offset mapping table between the target prediction frame index and the backstream state frame index is constructed.

[0092] Finally, the corresponding relationship between each group of target prediction frame index-response alignment frame index is recorded, and a time offset mapping table is constructed. The table is a hash index table structure, each item contains the prediction frame number, the corresponding backstream frame number, the timestamp difference between the two, the spatial position residual, the attitude angle residual and other fields. This mapping table will be used in subsequent steps to adjust the time offset, duration correction and instruction reordering operation in the sub-instruction chain.

[0093] Take an example of a simulated scenario: T0~T1 moment: Obtain the real-time return state of the target machine, predict the state sequence between T1~T2, and form a state prediction frame; the operator issues a structured instruction unit according to the prediction frame, and generates a feedforward envelope instruction stream; execute the instructions to generate a sub-instruction chain (S1~S3 are completed). T2~T3 moment: During the actual flight process, the target machine returns the state data during T1~T3; at this time, enter S4, compare the previous prediction frame (T1T2) with the actual return state (T1T3). At this time, the alignment comparison of the historical return state data to the once predicted frame is realized.

[0094] It should be noted that the state prediction frame is an expression of the theoretical control intention; the return state frame is a reflection of the actual flight control result; the time offset mapping table is constructed after comparing the difference between the two, in order to identify the prediction lag or instruction response delay; and then the trigger frame index and trigger time of the instruction are corrected based on the offset, so as to realize the dynamic fine-tuning and synchronous optimization of the future sub-instruction chain.

[0095] S4.2: Output the fine-tuned control data package after dynamic adjustment.

[0096] Take the time offset in the time offset mapping table as a reference to perform synchronization adjustment on the target prediction frame index bound to each sub-action, so that the actual state response frame is kept in time consistency.

[0097] After completing the frame index correction, reorder the sub-actions in the sub-instruction chain according to the time sequence, and after the sorting is completed, perform integrity check on the trigger interval between time-adjacent sub-actions, and perform frame distance extension operation on the structure below the preset trigger interval threshold to restore the controllability of the action interval.

[0098] After completing the time correction and sequence arrangement of the sub-actions, according to the standard action type field and target state field of each corrected sub-action, the action parameters are consistent, and each sub-action structure is converted into a structured instruction unit set with a standard field format.

[0099] The embodiment also provides a computer device suitable for the human-computer interaction instruction generation method for remote cooperative control of a target machine, which comprises a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the human-computer interaction instruction generation method for remote cooperative control of a target machine proposed in the above embodiment.

[0100] The computer device can be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved by WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0101] The embodiment also provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the man-machine interaction instruction generation method for remote cooperative control of a target machine.

[0102] To sum up, the application combines the trajectory prediction mechanism based on historical state data and the feedforward control strategy, can judge the response window of the control instruction in advance by constructing the state prediction sequence of the short future of the target machine, greatly improves the time sensitivity and execution matching degree of the operation instruction; Using structured instruction unit generation and multi-level sub-instruction chain analysis, the compound action can be modularized and finely disassembled, effectively improving the control accuracy and logical consistency of complex operations. In addition, by using spatial attitude residual error screening, envelope boundary modeling and time tolerance design, stable control area and buffer area can be established in multiple directions to improve the fault tolerance and controllability in high dynamic flight state; and the time offset comparison enhances the adaptive adjustment capability to non-ideal factors such as execution error and delay change, ensuring that the man-machine instruction chain has high robustness and real-time performance in the remote high-frequency cooperative scene.

[0103] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A method for generating human-computer interaction commands for remote collaborative control of a target drone, characterized in that: include: Based on the historical state data stream transmitted back by the target drone, a continuous state space trajectory of the target drone in a short future period is constructed, and a state prediction sequence is output for pre-judging the command response window. Receive the current interactive instruction data stream from the operator, and reconstruct the structure of the current interactive instruction data stream according to the time frame structure in the state prediction sequence to generate a set of structured instruction units; Based on the structured instruction unit set and state prediction sequence, combined with instruction response hysteresis tolerance and multi-directional envelope boundary parameters, a feedforward envelope instruction stream is generated. After generating the feedforward envelope instruction stream, the structured instruction units containing composite action tags are identified and template matched, and parsed into a multi-level sub-instruction chain with frame index integrity; By combining the state prediction sequence and the historical state data stream transmitted back by the target machine, the execution time offset of the multi-level sub-instruction chain is compared, and a dynamically adjusted fine-grained control data packet is output. The time offset comparison includes: extracting the state prediction frame corresponding to the target prediction frame index in each structured instruction unit; extracting the actual state frame set composed of position triples, attitude triples, and velocity scalars from the historical state data stream transmitted back from the target machine within adjacent time windows; calculating the difference vector between the spatial position vector and the attitude Euler angle field for all actual state frames of each target prediction frame within the time window, and identifying the frame with the smallest difference vector as the response alignment frame, and recording the timestamp; calculating the timestamp difference between each group of target prediction frames and the corresponding response alignment frame, recording it as the time offset, and constructing a time offset mapping table between the target prediction frame index and the transmitted state frame index; The dynamically adjusted refined control data packet includes: using the time offset in the time offset mapping table as a reference, performing synchronization adjustment on the target prediction frame index bound to each sub-action to maintain time consistency with the actual state response frame; after completing the frame index correction, reordering each sub-action in the sub-instruction chain according to the chronological order, and after the sorting is completed, performing integrity verification on the trigger interval between sub-actions that are close in time, and performing frame distance extension operation on structures that are lower than the preset trigger interval threshold; confirming the consistency of action parameters according to the standard action type field and target state field of each corrected sub-action, and converting each sub-action structure into a set of structured instruction units with standard field format.

2. The method for generating human-machine interaction commands for remote collaborative control of a target drone as described in claim 1, characterized in that: The generation of the state prediction sequence includes: Receive the continuous state data stream transmitted back by the target drone, align the position triplet, attitude triplet and velocity scalar according to the timestamp, and construct a state data matrix containing six-dimensional fields of time, space and attitude. Piecewise cubic interpolation fitting is performed on the spatial and attitude sequences within adjacent time periods in the state data matrix, and the first derivative curve is calculated for each interpolation result to obtain a set of velocity-attitude change rate curves. The system identifies abrupt changes in acceleration or attitude angle in the first derivative curve as trend change nodes, refits the continuous trajectory segments between trend change nodes, and corrects the interpolated predicted trajectory by combining environmental modeling parameters to generate a state prediction sequence within the future time window.

3. The method for generating human-machine interaction commands for remote collaborative control of a target drone as described in claim 1, characterized in that: The structural reconstruction of the current interactive command data stream includes: Receive the current interactive instruction data stream from the operator, parse the timing field, action description field, and target state field, divide it into continuous instruction segments according to the time axis, and perform semantic classification on the action description field to obtain a basic action tag set; Combining the basic action tag set and the target state field, the corresponding standard action unit template is extracted from the preset action knowledge template library, and the prediction frame index of each target state is determined according to the instruction trigger time, and an action unit-frame index-duration item triplet is established. Each action unit – frame index – duration triplet is mapped to a structured instruction unit, which includes a standard action type, a target prediction frame index, and a standard duration field, forming a set of structured instruction units.

4. The method for generating human-machine interaction commands for remote collaborative control of a target drone as described in claim 3, characterized in that: The generation of the feedforward envelope command stream includes: In the state prediction sequence, state prediction frames for the corresponding time period are extracted as initial response candidate windows according to the target prediction frame index and standard duration field in the structured instruction unit. Calculate the difference between the position vector and the attitude Euler angles of the target state field in each state prediction frame and the corresponding structured instruction unit in the initial response candidate window, and normalize them to obtain the spatial dimension residual set and the attitude dimension residual. A continuous state prediction frame segment whose residuals are simultaneously below both the spatial residual threshold and the attitude residual threshold is defined as the state matching window; Using the command target position vector in the state matching window as the center, set the angle ranges in the forward direction and the left and right lateral directions along the target drone velocity vector direction to generate a three-direction prediction vector group, which serves as a spatial direction reference edge. On each spatial reference edge, the maximum controllable displacement distance is extended according to the target's inertial dynamic parameters to construct an envelope boundary parameter set with directional attributes; Based on the time response tolerance range set by the input delay statistics of the operator terminal, a time envelope is constructed according to the time response advance and time response lag. The envelope boundary parameter set is embedded within the time envelope period to form an envelope instruction fragment, while the central control stable area and the edge transition buffer are divided.

5. The method for generating human-machine interaction commands for remote collaborative control of a target drone as described in claim 4, characterized in that: The command target position vector is the spatial position vector component in the target state field.

6. The method for generating human-machine interaction commands for remote collaborative control of a target drone as described in claim 1, characterized in that: The parsing of a multi-level sub-instruction chain with frame index integrity includes: In the set of structured instruction units, extract the action units containing composite action tags, and load the preset high-level tactical action template set using the standard action type field as the index. Based on the target prediction frame index and standard duration field associated with the action unit, the corresponding template structure is retrieved from the tactical action template set, and a sub-instruction chain containing sub-action sequences, relative time offsets, and target state transition sequences is output. For each sub-action in the sub-instruction chain, parameters are inherited based on the target state field in the structured instruction unit, and the absolute frame index and target state field of each sub-instruction are completed by combining the state matching window matched in the state prediction sequence. Arrange all sub-instructions in frame index order and output a multi-level sub-instruction chain set that has time controllability and structural consistency.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the human-computer interaction instruction generation method for remote collaborative control of target machines as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the human-computer interaction instruction generation method for remote collaborative control of target machines as described in any one of claims 1 to 6.

Images