A sensing and control method for human-machine hybrid system based on large model

Abstract

The present invention belongs to the technical field of unmanned cluster systems, and specifically relates to a large-scale model-based sensing and control method for human-machine hybrid systems. The method inputs cluster status information, images with target borders, target categories, target positioning information, and situation assessment complexity indicators into a multimodal large-scale model. The multimodal large-scale model extracts key information from a large amount of data collected by the unmanned cluster, generates a streamlined natural language situation assessment for a human operator, and the human operator issues cluster control voice instructions that conform to human language habits. The human operator's voice instructions are converted into natural language text through a speech recognition algorithm and input into the multimodal large-scale model. The multimodal large-scale model outputs high-level cluster control instructions that meet the format of the unmanned platform's autonomous control algorithm, enabling the unmanned cluster to take action directly according to the human operator's intentions, thereby improving the perception and control efficiency of the human-machine hybrid system.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned cluster systems, and in particular relates to a sensing and control method for a human-machine hybrid system based on a large model. Background Art

[0002] Unmanned swarm systems have significant application value in defense, geology, and rescue operations. The increasing number of unmanned platforms within swarm systems places higher demands on their situational awareness and control. Pre-trained large models, with their powerful natural language understanding capabilities, are crucial for applying these models to situational awareness and control in human-machine hybrid systems.

[0003] Currently, unmanned swarm systems controlled by a single human operator often consist of dozens to hundreds of heterogeneous unmanned platforms. This creates a large amount of situational and state information, making swarm control complex, placing a heavy cognitive load on the human operator, and making it difficult to ensure the safety of the swarm. Therefore, a large-scale model-based sensing and control method for human-machine hybrid systems is urgently needed to improve the perception and control efficiency of these systems. Summary of the Invention

[0004] (1) Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is that in an unmanned swarm system controlled by a single human operator, comprising dozens to hundreds of heterogeneous unmanned platforms, the amount of situation and status information is large, the swarm control is complex, the cognitive load on the human operator is high, and the safety of the swarm is difficult to ensure. Therefore, in order to improve the perception and control efficiency of human-machine hybrid systems, it is necessary to propose a large-scale model-based sensing and control method for human-machine hybrid systems. Cluster status information, images with target bounding boxes, target categories, target positioning information, and situation assessment complexity indicators are input into a multimodal large-scale model. The multimodal large-scale model extracts key information from the large amount of data collected by the unmanned swarm and provides it to the human operator. The human operator issues swarm control voice commands that conform to human language habits. The human operator's voice commands are converted into natural language text through a speech recognition algorithm and input into the multimodal large-scale model. The multimodal large-scale model outputs high-level swarm control commands that meet the format of the unmanned platform's autonomous control algorithm. This allows the unmanned swarm to take actions directly based on the human operator's intentions, effectively improving the efficiency of human-machine collaboration.

[0006] (2) Technical solution

[0007] To solve the above technical problems, the present invention provides a large-scale model-based human-machine hybrid system sensing and control method, which is characterized in that the large-scale model-based human-machine hybrid system sensing and control method is used to improve the perception and control efficiency of the human-machine hybrid system;

[0008] The human-machine hybrid system includes an unmanned swarm system and a human operator; the unmanned swarm system collects environmental information and sends it to the human operator, and receives control instructions from the human operator; the human operator makes control decisions based on the environmental information collected by the unmanned swarm system and sends control instructions to the unmanned swarm system;

[0009] The unmanned swarm system consists of multiple drones and unmanned vehicles, each of which is an unmanned platform. The unmanned platform is equipped with an embedded computer, a posture measurement module, a motion control module, a battery, a battery monitoring module, a communication radio, a binocular camera, and a self-destruct module.

[0010] The embedded computer is the control core of a single unmanned platform, exchanging data with the posture measurement module, motion control module, battery, battery monitoring module, communication radio, binocular camera, and self-destruct module. The embedded computer receives position and posture information from the posture measurement module, battery voltage information from the battery monitoring module, and visual images from the binocular camera. It receives high-level cluster control instructions from the human operator via the communication radio, sends cluster status information and image information to the human operator via the communication radio, sends motion control instructions to the motion control module, and sends self-destruct instructions to the self-destruct module. An autonomous control algorithm with path planning, obstacle avoidance, task allocation, and self-destruct functions is deployed in the embedded computer to receive high-level cluster control instructions and data from other modules, and converts high-level cluster control instructions into motion control instructions and self-destruct instructions for the unmanned platform through task allocation.

[0011] The posture measurement module measures the position and posture information of the unmanned platform, including positioning information, speed, acceleration, angular velocity and posture information; the posture measurement module sends the measured position and posture information to the embedded computer;

[0012] The motion control module receives motion control instructions from the embedded computer and controls the output power of the unmanned platform motor;

[0013] Batteries provide power to unmanned platforms;

[0014] The battery monitoring module measures the battery voltage and converts it into remaining power information, which is then sent to the embedded computer;

[0015] The communication radio receives high-level control instructions from the human operator and sends them to the embedded computer, and receives the unmanned platform status information and image information summarized by the embedded computer and sends them to the human operator;

[0016] The binocular camera captures the image in front of the unmanned platform and sends the image to the embedded computer;

[0017] The self-destruct module receives the self-destruct command from the embedded computer and triggers the self-destruct operation according to the self-destruct command, thus enabling the unmanned platform to break through obstacles.

[0018] The human operator carries a backpack server, heart rate sensor, microphone, operation terminal, and communication radio. The human operator reads the situation assessment results through the operation terminal and issues cluster control voice commands through the microphone.

[0019] The backpack server exchanges data with the heart rate sensor, microphone, communication radio, and operation terminal; the backpack server receives heart rate data measured by the heart rate sensor, receives the voice signal of the human operator through the microphone, receives the cluster status information and image information of the unmanned cluster through the communication radio, sends advanced cluster control instructions to the unmanned cluster through the communication radio, and displays the natural language situation assessment to the operation terminal; a multimodal large model, speech recognition algorithm, target detection and positioning algorithm, and security assessment algorithm are deployed in the backpack server; the multimodal large model adjusts the language complexity of the output situation assessment according to the heart rate data of the human operator, outputs the natural language situation assessment according to the cluster status information and image information of the unmanned cluster, and generates advanced cluster control instructions according to the natural language cluster control instructions of the human operator; the speech recognition algorithm converts the voice signal of the human operator into natural language text; the target detection and positioning algorithm detects humans and objects in the image based on the image collected by the cluster and measures the relative position of humans and objects; the security assessment algorithm evaluates the advanced cluster control instructions according to the security policy;

[0020] The heart rate sensor measures the heart rate of the human operator and is installed at the heart of the human operator;

[0021] The microphone receives a voice signal from a human operator;

[0022] The communication radio receives status information and image information from the unmanned cluster and sends it to the backpack server, and receives high-level control instructions from human operators and sends them to the unmanned cluster;

[0023] The operation terminal includes a display screen and operation buttons. The display screen displays the human operator's high-level control instructions and natural language situation assessment, and the operation buttons control the start and stop of the unmanned swarm.

[0024] The human-machine hybrid system sensing control method includes the following steps:

[0025] Step 1: A human operator wears a heart rate sensor and microphone, carries a communication radio and backpack server, and holds an operation terminal. The human operator activates the unmanned swarm system through the operation terminal.

[0026] Step 2: The heart rate sensor collects the operator's heart rate data, calculates the situation assessment complexity index, and inputs it into the multimodal large model. The multimodal large model adjusts the language complexity of the situation assessment based on the complexity index. The situation assessment complexity index is calculated based on the heart rate to reflect the human cognitive load. The multimodal large model adaptively adjusts the language complexity of the situation assessment based on the size of the human cognitive load, improving the human operator's situational awareness experience and decision-making efficiency, thereby improving the perception and control efficiency of the human-machine hybrid system. The situation assessment complexity index is calculated by the following formula:

[0027]

[0028] Where x is the instantaneous heart rate, k1 is the resting heart rate of the human operator, k2 is the age of the human operator, ε is a small positive number to prevent the denominator from being zero, and φ is the complexity index of the situation assessment;

[0029] Step 3: All unmanned platforms in the unmanned swarm system collect platform status information and environmental images, and send them to the backpack server via the communication radio;

[0030] Step 4: The piggyback server aggregates the platform status information into cluster status information and adjusts it into the text input format of the multimodal large model;

[0031] Step 5: The piggyback server runs the object detection and localization algorithm on the image, outputting an image with object bounding boxes, object categories, and object location information.

[0032] Step 6: Input the cluster state information, images with target bounding boxes, target categories, target positioning information, and situation assessment complexity indicators into the multimodal large model, which generates a natural language situation assessment. The language complexity of the situation assessment is related to the situation assessment complexity indicator, and in turn to the human cognitive load. Therefore, the language complexity of the situation assessment can be adjusted according to the human cognitive load, improving the human situation perception experience and decision-making efficiency. The multimodal large model extracts key information from the large amount of data collected by the unmanned swarm system, discards invalid or redundant information, and outputs a streamlined natural language situation assessment to the human operator, thereby improving the utilization efficiency of the unmanned swarm system information and enhancing the perception efficiency of the human-machine hybrid system.

[0033] Step 7: The human operator reads the natural language situation assessment through the operation terminal and issues cluster control voice commands through the microphone;

[0034] Step 8: The backpack server uses a speech recognition algorithm to convert the operator's voice signal into natural language text and inputs it into the multimodal large model, which then outputs high-level cluster control instructions. The operator issues voice instructions that conform to human language habits. The high-level cluster control instructions meet the format of direct input to the autonomous control algorithm of the unmanned platform, allowing the unmanned cluster system to take actions directly based on the operator's intentions, thereby improving the control efficiency of the human-machine hybrid system.

[0035] Step 9: Input the advanced cluster control instructions generated by the multimodal large model into the safety assessment algorithm, which evaluates the trajectory safety and self-destruction safety of the advanced cluster control instructions. If the advanced cluster control instructions do not pose a safety risk, the advanced cluster control instructions are sent to the unmanned cluster system for execution. Otherwise, the multimodal large model regenerates the advanced cluster control instructions. A safety assessment is performed on the advanced cluster control instructions generated by the multimodal large model to reduce the risk of operational errors, thereby reducing the possibility of accidents in the unmanned cluster system and improving the safety of the unmanned cluster system control.

[0036] Step 9-1: Evaluate the trajectory safety of high-level cluster control instructions, determine whether collisions between unmanned platforms occur, and calculate the trajectory safety index based on cluster status information; calculate all predicted trajectories within the prediction time range:

[0037] X pre,t =X+t·V,t=dt,2dt,...,T

[0038] Among them, X pre,t is the predicted trajectory under t prediction steps, X is the cluster position matrix, V is the cluster velocity matrix, T is the prediction time range, t is the prediction time, and dt is the prediction time resolution;

[0039] The minimum spacing of the unmanned platform's predicted trajectory at each prediction step within the prediction time range is:

[0040] d min =min(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T

[0041] Among them, d min The minimum distance between the unmanned platform prediction trajectories for each prediction step, M+N is the total number of unmanned platforms, M is the number of unmanned vehicles, N is the number of drones, i and j are the numbers of the unmanned platforms, and min() means finding the minimum value;

[0042] The maximum distance between the unmanned platform's predicted trajectories at each prediction step within the prediction time range is:

[0043] d max =max(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T

[0044] Among them, d max The maximum distance between the unmanned platform's predicted trajectories for each prediction step;

[0045] The trajectory safety index is calculated as follows:

[0046]

[0047] Among them, safe traj is the trajectory safety index, d1 is the minimum safe distance, d2 is the absolute safe distance, 'notsafe' means the trajectory is unsafe, 'complete safe' means the trajectory is absolutely safe, and std() means calculating the variance of each column of the matrix;

[0048] The conditions for unmanned swarm trajectory safety are:

[0049] safe traj ='complete safe' or safe traj >safe traj_threshold

[0050] Among them, safe traj_threshold is the trajectory safety threshold;

[0051] Step 9-2: If the high-level cluster control command includes a self-destruct command, evaluate the self-destruct safety of the high-level cluster control command and determine whether the unmanned platform self-destruction threatens the human operator and other unmanned platforms. Otherwise, skip this step; calculate the self-destruct safety index based on the cluster status information; the self-destruct damage radius is calculated by the following formula:

[0052]

[0053] Among them, r is the self-destruction damage radius, K is the damage coefficient, and Q is the TNT equivalent loaded in the self-destruction module of the unmanned platform;

[0054] The position of the unmanned platform that is about to self-destruct is calculated by the following formula:

[0055] X blast =X·C

[0056] Among them, X blast is the position matrix of the self-destructing unmanned platform, X is the cluster position matrix, and C is the self-destruction control matrix;

[0057] Calculate the minimum distance between the unmanned platform that is about to self-destruct and other unmanned platforms:

[0058] d blast_platform =min(||X(i)-X blast (k)||2),i=1,2,...,M+N,k=1,2,...,L and i≠k

[0059] Where k is the number of the self-destructing unmanned platform, and L is the number of self-destructing unmanned platforms;

[0060] Calculate the minimum distance between the unmanned platform that is about to self-destruct and the human operator:

[0061] d blast_human =min(||X human -X blast (k)||2),k=1,2,...,L

[0062] Among them, X human For human operator position;

[0063] The self-destruction safety index is calculated by the following formula:

[0064]

[0065] Among them, safe blast is the self-destruction safety indicator, 'not safe' means self-destruction is unsafe, and 'safe' means self-destruction is safe;

[0066] The conditions for the safety of unmanned swarm self-destruction are:

[0067] safe blast ='safe'.

[0068] (3) Beneficial effects

[0069] Compared with the existing technology, the present invention proposes a large-scale model-based human-machine hybrid system sensing and control method, which inputs cluster status information, images with target borders, target categories, target positioning information and situation assessment complexity indicators into a multimodal large model. The multimodal large model extracts key information from the large amount of data collected by the unmanned cluster, generates a streamlined natural language situation assessment for the human operator, and the human operator issues cluster control voice instructions that conform to human language habits. The human operator's voice instructions are converted into natural language text through a speech recognition algorithm and input into the multimodal large model. The multimodal large model outputs high-level cluster control instructions that meet the format of the unmanned platform's autonomous control algorithm, enabling the unmanned cluster to take action directly according to the human operator's intentions, thereby improving the perception and control efficiency of the human-machine hybrid system.

[0070] The present invention proposes a large-model-based sensing and control method for a human-machine hybrid system. The method inputs cluster status information, images with target bounding boxes, target categories, target positioning information, and situation assessment complexity indicators into a multimodal large model. The multimodal large model extracts key information from the large amount of data collected by the unmanned cluster, discards invalid or redundant information, and generates a streamlined natural language situation assessment for the human operator, thereby improving the utilization efficiency of the unmanned cluster information and the perception efficiency of the human-machine hybrid system.

[0071] The present invention proposes a large-scale model-based sensing and control method for a human-machine hybrid system. A human operator issues cluster control voice instructions that conform to human language habits. The human operator's voice instructions are converted into natural language text through a speech recognition algorithm and input into a multimodal large model. The multimodal large model outputs high-level cluster control instructions that meet the format of the unmanned platform's autonomous control algorithm, enabling the unmanned cluster to take action directly according to the human operator's intentions, thereby improving the control efficiency of the human-machine hybrid system.

[0072] This paper proposes a large-scale model-based sensing and control method for a human-machine hybrid system. A heart rate sensor collects heart rate data from the human operator, calculates a situation assessment complexity index, and feeds this data into a multimodal large-scale model. The multimodal large-scale model then adjusts the language complexity of the situation assessment based on the complexity index. The situation assessment complexity index calculated from the heart rate reflects the human cognitive load, enabling the multimodal large-scale model to adaptively adjust the language complexity of the situation assessment based on the human cognitive load. This improves the human operator's situational awareness and decision-making efficiency, thereby enhancing the perception and control efficiency of the human-machine hybrid system.

[0073] The present invention proposes a large-model-based human-machine hybrid system sensing and control method, which inputs high-level cluster control instructions generated by a multimodal large model into a safety assessment algorithm. The safety assessment algorithm evaluates the trajectory safety and self-destruction safety of the high-level cluster control instructions, reducing the risk of operational errors, thereby reducing the possibility of accidents in unmanned clusters and improving the safety of unmanned cluster control.

[0074] The patent of this invention processes the voice instructions of human operators through a multimodal large model and a speech recognition algorithm, and outputs advanced cluster control instructions. Replacing the multimodal large model and the speech recognition algorithm with a multimodal large model that can directly input audio signals can be considered as the same solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a schematic diagram of the structure of a human-machine hybrid system;

[0076] Figure 2 This is a flow chart of a large-scale model-based human-machine hybrid system sensing and control method of the present invention;

[0077] Figure 3This is a schematic diagram of a large-scale model-based human-machine hybrid system situation assessment process of the present invention;

[0078] Figure 4 This is a schematic diagram of an unmanned cluster control process of a large-scale human-machine hybrid system according to the present invention;

[0079] Figure 5 This is a schematic diagram of a language complexity flow chart for adaptively adjusting situation assessment of a large-scale model-based human-machine hybrid system according to the present invention;

[0080] Figure 6 This is a schematic diagram of the operation terminal display interface. DETAILED DESCRIPTION

[0081] In order to make the purpose, content, and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0082] To solve the problems of the prior art, the present invention provides a large-scale model-based human-machine hybrid system sensing and control method, characterized in that the large-scale model-based human-machine hybrid system sensing and control method is used to improve the perception and control efficiency of the human-machine hybrid system;

[0083] The human-machine hybrid system includes an unmanned swarm system and a human operator; the unmanned swarm system collects environmental information and sends it to the human operator, and receives control instructions from the human operator; the human operator makes control decisions based on the environmental information collected by the unmanned swarm system and sends control instructions to the unmanned swarm system;

[0084] The unmanned swarm system consists of multiple drones and unmanned vehicles, each of which is an unmanned platform. The unmanned platform is equipped with an embedded computer, a posture measurement module, a motion control module, a battery, a battery monitoring module, a communication radio, a binocular camera, and a self-destruct module.

[0085] The embedded computer is the control core of a single unmanned platform, exchanging data with the posture measurement module, motion control module, battery, battery monitoring module, communication radio, binocular camera, and self-destruct module. The embedded computer receives position and posture information from the posture measurement module, battery voltage information from the battery monitoring module, and visual images from the binocular camera. It receives high-level cluster control instructions from the human operator via the communication radio, sends cluster status information and image information to the human operator via the communication radio, sends motion control instructions to the motion control module, and sends self-destruct instructions to the self-destruct module. An autonomous control algorithm with path planning, obstacle avoidance, task allocation, and self-destruct functions is deployed in the embedded computer to receive high-level cluster control instructions and data from other modules, and converts high-level cluster control instructions into motion control instructions and self-destruct instructions for the unmanned platform through task allocation.

[0086] The posture measurement module measures the position and posture information of the unmanned platform, including positioning information, speed, acceleration, angular velocity and posture information; the posture measurement module sends the measured position and posture information to the embedded computer;

[0087] The motion control module receives motion control instructions from the embedded computer and controls the output power of the unmanned platform motor;

[0088] Batteries provide power to unmanned platforms;

[0089] The battery monitoring module measures the battery voltage and converts it into remaining power information, which is then sent to the embedded computer;

[0090] The communication radio receives high-level control instructions from the human operator and sends them to the embedded computer, and receives the unmanned platform status information and image information summarized by the embedded computer and sends them to the human operator;

[0091] The binocular camera captures the image in front of the unmanned platform and sends the image to the embedded computer;

[0092] The self-destruct module receives the self-destruct command from the embedded computer and triggers the self-destruct operation according to the self-destruct command, thus enabling the unmanned platform to break through obstacles.

[0093] The human operator carries a backpack server, heart rate sensor, microphone, operation terminal, and communication radio. The human operator reads the situation assessment results through the operation terminal and issues cluster control voice commands through the microphone.

[0094] The backpack server exchanges data with the heart rate sensor, microphone, communication radio, and operation terminal; the backpack server receives heart rate data measured by the heart rate sensor, receives the voice signal of the human operator through the microphone, receives the cluster status information and image information of the unmanned cluster through the communication radio, sends advanced cluster control instructions to the unmanned cluster through the communication radio, and displays the natural language situation assessment to the operation terminal; a multimodal large model, speech recognition algorithm, target detection and positioning algorithm, and security assessment algorithm are deployed in the backpack server; the multimodal large model adjusts the language complexity of the output situation assessment according to the heart rate data of the human operator, outputs the natural language situation assessment according to the cluster status information and image information of the unmanned cluster, and generates advanced cluster control instructions according to the natural language cluster control instructions of the human operator; the speech recognition algorithm converts the voice signal of the human operator into natural language text; the target detection and positioning algorithm detects humans and objects in the image based on the image collected by the cluster and measures the relative position of humans and objects; the security assessment algorithm evaluates the advanced cluster control instructions according to the security policy;

[0095] The heart rate sensor measures the heart rate of the human operator and is installed at the heart of the human operator;

[0096] The microphone receives a voice signal from a human operator;

[0097] The communication radio receives status information and image information from the unmanned cluster and sends it to the backpack server, and receives high-level control instructions from human operators and sends them to the unmanned cluster;

[0098] The operation terminal includes a display screen and operation buttons. The display screen displays the human operator's high-level control instructions and natural language situation assessment, and the operation buttons control the start and stop of the unmanned swarm.

[0099] The human-machine hybrid system sensing control method includes the following steps:

[0100] Step 1: A human operator wears a heart rate sensor and microphone, carries a communication radio and backpack server, and holds an operation terminal. The human operator activates the unmanned swarm system through the operation terminal.

[0101] Step 2: The heart rate sensor collects the operator's heart rate data, calculates the situation assessment complexity index, and inputs it into the multimodal large model. The multimodal large model adjusts the language complexity of the situation assessment based on the complexity index. The situation assessment complexity index is calculated based on the heart rate to reflect the human cognitive load. The multimodal large model adaptively adjusts the language complexity of the situation assessment based on the size of the human cognitive load, improving the human operator's situational awareness experience and decision-making efficiency, thereby improving the perception and control efficiency of the human-machine hybrid system. The situation assessment complexity index is calculated by the following formula:

[0102]

[0103] Where x is the instantaneous heart rate, k1 is the resting heart rate of the human operator, k2 is the age of the human operator, ε is a small positive number to prevent the denominator from being zero, and φ is the complexity index of the situation assessment;

[0104] Step 3: All unmanned platforms in the unmanned swarm system collect platform status information and environmental images, and send them to the backpack server via the communication radio;

[0105] Step 4: The piggyback server aggregates the platform status information into cluster status information and adjusts it into the text input format of the multimodal large model;

[0106] Step 5: The piggyback server runs the object detection and localization algorithm on the image, outputting an image with object bounding boxes, object categories, and object location information.

[0107] Step 6: Input the cluster state information, images with target bounding boxes, target categories, target positioning information, and situation assessment complexity indicators into the multimodal large model, which generates a natural language situation assessment. The language complexity of the situation assessment is related to the situation assessment complexity indicator, and in turn to the human cognitive load. Therefore, the language complexity of the situation assessment can be adjusted according to the human cognitive load, improving the human situation perception experience and decision-making efficiency. The multimodal large model extracts key information from the large amount of data collected by the unmanned swarm system, discards invalid or redundant information, and outputs a streamlined natural language situation assessment to the human operator, thereby improving the utilization efficiency of the unmanned swarm system information and enhancing the perception efficiency of the human-machine hybrid system.

[0108] Step 7: The human operator reads the natural language situation assessment through the operation terminal and issues cluster control voice commands through the microphone;

[0109] Step 8: The backpack server uses a speech recognition algorithm to convert the operator's voice signal into natural language text and inputs it into the multimodal large model, which then outputs high-level cluster control instructions. The operator issues voice instructions that conform to human language habits. The high-level cluster control instructions meet the format of direct input to the autonomous control algorithm of the unmanned platform, allowing the unmanned cluster system to take actions directly based on the operator's intentions, thereby improving the control efficiency of the human-machine hybrid system.

[0110] Step 9: Input the advanced cluster control instructions generated by the multimodal large model into the safety assessment algorithm, which evaluates the trajectory safety and self-destruction safety of the advanced cluster control instructions. If the advanced cluster control instructions do not pose a safety risk, the advanced cluster control instructions are sent to the unmanned cluster system for execution. Otherwise, the multimodal large model regenerates the advanced cluster control instructions. A safety assessment is performed on the advanced cluster control instructions generated by the multimodal large model to reduce the risk of operational errors, thereby reducing the possibility of accidents in the unmanned cluster system and improving the safety of the unmanned cluster system control.

[0111] Step 9-1: Evaluate the trajectory safety of high-level cluster control instructions, determine whether collisions between unmanned platforms occur, and calculate the trajectory safety index based on cluster status information; calculate all predicted trajectories within the prediction time range:

[0112] X pre,t =X+t·V,t=dt,2dt,...,T

[0113] Among them, X pre,t is the predicted trajectory under t prediction steps, X is the cluster position matrix, V is the cluster velocity matrix, T is the prediction time range, t is the prediction time, and dt is the prediction time resolution;

[0114] The minimum spacing of the unmanned platform's predicted trajectory at each prediction step within the prediction time range is:

[0115] d min =min(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T

[0116] Among them, d min The minimum distance between the unmanned platform prediction trajectories for each prediction step, M+N is the total number of unmanned platforms, M is the number of unmanned vehicles, N is the number of drones, i and j are the numbers of the unmanned platforms, and min() means finding the minimum value;

[0117] The maximum distance between the unmanned platform's predicted trajectories at each prediction step within the prediction time range is:

[0118] d max =max(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T

[0119] Among them, d max The maximum distance between the unmanned platform's predicted trajectories for each prediction step;

[0120] The trajectory safety index is calculated as follows:

[0121]

[0122] Among them, safe traj is the trajectory safety index, d1 is the minimum safe distance, d2 is the absolute safe distance, 'notsafe' means the trajectory is unsafe, 'complete safe' means the trajectory is absolutely safe, and std() means calculating the variance of each column of the matrix;

[0123] The conditions for unmanned swarm trajectory safety are:

[0124] safe traj ='complete safe' or safe traj >safe traj_threshold

[0125] Among them, safe traj_threshold is the trajectory safety threshold;

[0126] Step 9-2: If the high-level cluster control command includes a self-destruct command, evaluate the self-destruct safety of the high-level cluster control command and determine whether the unmanned platform self-destruction threatens the human operator and other unmanned platforms. Otherwise, skip this step; calculate the self-destruct safety index based on the cluster status information; the self-destruct damage radius is calculated by the following formula:

[0127]

[0128] Among them, r is the self-destruction damage radius, K is the damage coefficient, and Q is the TNT equivalent loaded in the self-destruction module of the unmanned platform;

[0129] The position of the unmanned platform that is about to self-destruct is calculated by the following formula:

[0130] X blast =X·C

[0131] Among them, X blast is the position matrix of the self-destructing unmanned platform, X is the cluster position matrix, and C is the self-destruction control matrix;

[0132] Calculate the minimum distance between the unmanned platform that is about to self-destruct and other unmanned platforms:

[0133] d blast_platform =min(||X(i)-X blast (k)||2),i=1,2,...,M+N,k=1,2,...,L and i≠k

[0134] Where k is the number of the self-destructing unmanned platform, and L is the number of self-destructing unmanned platforms;

[0135] Calculate the minimum distance between the unmanned platform that is about to self-destruct and the human operator:

[0136] d blast_human =min(||X human -X blast (k)||2),k=1,2,...,L

[0137] Among them, X human For human operator position;

[0138] The self-destruction safety index is calculated by the following formula:

[0139]

[0140] Among them, safe blast is the self-destruction safety indicator, 'not safe' means self-destruction is unsafe, and 'safe' means self-destruction is safe;

[0141] The conditions for the safety of unmanned swarm self-destruction are:

[0142] safe blast ='safe'.

[0143] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A large-scale model-based human-machine hybrid system sensing control method, characterized in that: The human-machine hybrid system sensing control method includes the following steps: Step 1: A human operator wears a heart rate sensor and microphone, carries a communication radio and backpack server, and holds an operation terminal. The human operator activates the unmanned swarm system through the operation terminal. Step 2: The heart rate sensor collects the operator's heart rate data, calculates the situation assessment complexity index, and inputs it into the multimodal large model. The multimodal large model adjusts the language complexity of the situation assessment based on the complexity index. The situation assessment complexity index is calculated based on the heart rate to reflect the human cognitive load. The multimodal large model adaptively adjusts the language complexity of the situation assessment based on the size of the human cognitive load, improving the human operator's situational awareness experience and decision-making efficiency, thereby improving the perception and control efficiency of the human-machine hybrid system. The situation assessment complexity index is calculated by the following formula: Where x is the instantaneous heart rate, k1 is the resting heart rate of the human operator, k2 is the age of the human operator, ε is a small positive number to prevent the denominator from being zero, and φ is the complexity index of the situation assessment; Step 3: All unmanned platforms in the unmanned swarm system collect platform status information and environmental images, and send them to the backpack server via the communication radio; Step 4: The piggyback server aggregates the platform status information into cluster status information and adjusts it into the text input format of the multimodal large model; Step 5: The piggyback server runs the object detection and localization algorithm on the image, outputting an image with object bounding boxes, object categories, and object location information. Step 6: Input the cluster state information, images with target bounding boxes, target categories, target positioning information, and situation assessment complexity indicators into the multimodal large model, which generates a natural language situation assessment. The language complexity of the situation assessment is related to the situation assessment complexity indicator, and in turn to the human cognitive load. Therefore, the language complexity of the situation assessment can be adjusted according to the human cognitive load, improving the human situation perception experience and decision-making efficiency. The multimodal large model extracts key information from the large amount of data collected by the unmanned swarm system, discards invalid or redundant information, and outputs a streamlined natural language situation assessment to the human operator, thereby improving the utilization efficiency of the unmanned swarm system information and enhancing the perception efficiency of the human-machine hybrid system. Step 7: The human operator reads the natural language situation assessment through the operation terminal and issues cluster control voice commands through the microphone; Step 8: The backpack server uses a speech recognition algorithm to convert the operator's voice signal into natural language text and inputs it into the multimodal large model, which then outputs high-level cluster control instructions. The operator issues voice instructions that conform to human language habits. The high-level cluster control instructions meet the format of direct input to the autonomous control algorithm of the unmanned platform, allowing the unmanned cluster system to take actions directly based on the operator's intentions, thereby improving the control efficiency of the human-machine hybrid system. Step 9: Input the advanced cluster control instructions generated by the multimodal large model into the safety assessment algorithm, which evaluates the trajectory safety and self-destruction safety of the advanced cluster control instructions. If the advanced cluster control instructions do not pose a safety risk, the advanced cluster control instructions are sent to the unmanned cluster system for execution. Otherwise, the multimodal large model regenerates the advanced cluster control instructions. A safety assessment is performed on the advanced cluster control instructions generated by the multimodal large model to reduce the risk of operational errors, thereby reducing the possibility of accidents in the unmanned cluster system and improving the safety of the unmanned cluster system control.

2. The large-scale model-based human-machine hybrid system sensing control method according to claim 1, characterized in that: The step 9 includes: Step 9-1: Evaluate the trajectory safety of high-level cluster control instructions, determine whether collisions between unmanned platforms occur, and calculate the trajectory safety index based on cluster status information; calculate all predicted trajectories within the prediction time range: X pre,t =X+t·V,t=dt,2dt,...,T Among them, X pre,t is the predicted trajectory under t prediction steps, X is the cluster position matrix, V is the cluster velocity matrix, T is the prediction time range, t is the prediction time, and dt is the prediction time resolution; The minimum spacing of the unmanned platform's predicted trajectory at each prediction step within the prediction time range is: d min =min(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T Among them, d min The minimum distance between the unmanned platform prediction trajectories for each prediction step, M+N is the total number of unmanned platforms, M is the number of unmanned vehicles, N is the number of drones, i and j are the numbers of the unmanned platforms, and min() means finding the minimum value; The maximum distance between the unmanned platform's predicted trajectories at each prediction step within the prediction time range is: d max =max(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T Among them, d max The maximum distance between the unmanned platform's predicted trajectories for each prediction step; The trajectory safety index is calculated as follows: Among them, safe traj is the trajectory safety index, d1 is the minimum safe distance, d2 is the absolute safe distance, 'not safe' means the trajectory is unsafe, 'complete safe' means the trajectory is absolutely safe, and std() means calculating the variance of each column of the matrix; The conditions for unmanned swarm trajectory safety are: safe traj = 'complete safe' or safe traj > safe traj_threshold Among them, safe traj_threshold is the trajectory safety threshold; Step 9-2: If the high-level cluster control command includes a self-destruct command, evaluate the self-destruct safety of the high-level cluster control command and determine whether the unmanned platform self-destruction threatens the human operator and other unmanned platforms. Otherwise, skip this step; calculate the self-destruct safety index based on the cluster status information; the self-destruct damage radius is calculated by the following formula: Among them, r is the self-destruction damage radius, K is the damage coefficient, and Q is the TNT equivalent loaded in the self-destruction module of the unmanned platform; The position of the unmanned platform that is about to self-destruct is calculated by the following formula: X blast =X·C Among them, X blast is the position matrix of the self-destructing unmanned platform, X is the cluster position matrix, and C is the self-destruction control matrix; Calculate the minimum distance between the unmanned platform that is about to self-destruct and other unmanned platforms: d blast_platform =min(||X(i)-X blast (k)||2),i=1,2,...,M+N,k=1,2,...,L and i≠k Where k is the number of the self-destructing unmanned platform, and L is the number of self-destructing unmanned platforms; Calculate the minimum distance between the unmanned platform that is about to self-destruct and the human operator: d blast_human =min(||X human -X blast (k)||2),k=1,2,...,L Among them, X human For human operator position; The self-destruction safety index is calculated by the following formula: Among them, safe blast is the self-destruction safety indicator, 'not safe' means self-destruction is unsafe, and 'safe' means self-destruction is safe; The conditions for the safety of unmanned swarm self-destruction are: safe blast ='safe'。 3. The large-scale model-based human-machine hybrid system sensing control method according to claim 1, characterized in that: The large-model-based human-machine hybrid system sensing and control method is used to improve the perception and control efficiency of the human-machine hybrid system.

4. The large-scale model-based human-machine hybrid system sensing control method according to claim 1, characterized in that: The human-machine hybrid system includes an unmanned cluster system and a human operator; the unmanned cluster system collects environmental information and sends it to the human operator, and receives control instructions from the human operator; the human operator makes control decisions based on the environmental information collected by the unmanned cluster system and sends control instructions to the unmanned cluster system.

5. The large-scale model-based human-machine hybrid system sensing control method according to claim 4, characterized in that: The unmanned swarm system consists of multiple drones and multiple unmanned vehicles, each of which is an unmanned platform; the unmanned platform is equipped with an embedded computer, a posture measurement module, a motion control module, a battery, a battery monitoring module, a communication radio, a binocular camera, and a self-destruct module.

6. The large-scale model-based human-machine hybrid system sensing control method according to claim 5, characterized in that: The embedded computer is the control core of a single unmanned platform, exchanging data with the posture measurement module, motion control module, battery, battery monitoring module, communication radio, binocular camera, and self-destruct module; The embedded computer receives position and posture information from the posture measurement module, receives battery voltage information from the battery monitoring module, receives visual images from the binocular camera, receives advanced cluster control instructions from the human operator through the communication radio, sends cluster status information and image information to the human operator through the communication radio, sends motion control instructions to the motion control module, and sends self-destruction instructions to the self-destruction module; an autonomous control algorithm with path planning, obstacle avoidance, task allocation, and self-destruction functions is deployed in the embedded computer to receive advanced cluster control instructions and data from other modules, and converts the advanced cluster control instructions into motion control instructions and self-destruction instructions for the unmanned platform through task allocation.

7. The large-scale model-based human-machine hybrid system sensing control method according to claim 6, characterized in that: The position and attitude measurement module measures the position and attitude information of the unmanned platform, including positioning information, speed, acceleration, angular velocity and attitude information; the position and attitude measurement module sends the measured position and attitude information to the embedded computer.

8. The large-scale model-based human-machine hybrid system sensing control method according to claim 7, characterized in that: The motion control module receives motion control instructions from the embedded computer and controls the output power of the unmanned platform motor; Batteries provide power to unmanned platforms; The battery monitoring module measures the battery voltage and converts it into remaining power information, which is then sent to the embedded computer; The communication radio receives high-level control instructions from the human operator and sends them to the embedded computer, and receives the unmanned platform status information and image information summarized by the embedded computer and sends them to the human operator; The binocular camera captures the image in front of the unmanned platform and sends the image to the embedded computer; The self-destruct module receives the self-destruct command from the embedded computer and triggers the self-destruct operation according to the self-destruct command to achieve obstacle removal for the unmanned platform.

9. The large-scale model-based human-machine hybrid system sensing control method according to claim 8, characterized in that: The human operator carries a backpack server, heart rate sensor, microphone, operation terminal, and communication radio. The human operator reads the situation assessment results through the operation terminal and issues cluster control voice commands through the microphone. The backpack server exchanges data with the heart rate sensor, microphone, communication radio, and operation terminal; The backpack server receives heart rate data measured by a heart rate sensor, receives the voice signal of a human operator through a microphone, receives cluster status information and image information of the unmanned cluster through a communication radio, sends advanced cluster control instructions to the unmanned cluster through the communication radio, and displays a natural language situation assessment to the operation terminal; a multimodal large model, a speech recognition algorithm, a target detection and positioning algorithm, and a safety assessment algorithm are deployed in the backpack server; the multimodal large model adjusts the language complexity of the output situation assessment based on the heart rate data of the human operator, outputs a natural language situation assessment based on the cluster status information and image information of the unmanned cluster, and generates advanced cluster control instructions based on the natural language cluster control instructions of the human operator; the speech recognition algorithm converts the voice signal of the human operator into natural language text; the target detection and positioning algorithm detects humans and objects in the image based on the images collected by the cluster and measures the relative positions of humans and objects; the safety assessment algorithm evaluates the advanced cluster control instructions according to the security policy; The heart rate sensor measures the heart rate of the human operator and is installed at the heart position of the human operator; The microphone receives a voice signal from a human operator; The communication radio receives status information and image information from the unmanned cluster and sends it to the backpack server, and receives high-level control instructions from human operators and sends them to the unmanned cluster.

10. The large-scale model-based human-machine hybrid system sensing control method according to claim 9, characterized in that: The operation terminal includes a display screen and operation buttons. The display screen displays the human operator's high-level control instructions and natural language situation assessment, and the operation buttons control the startup and shutdown of the unmanned cluster.

Images