Human-machine hybrid system sensing control method based on large model

Abstract

The invention belongs to the technical field of unmanned cluster systems, and particularly relates to a man-machine hybrid system sensing control method based on a large model, which comprises the following steps of: inputting cluster state information, an image with a target frame, a target category, target positioning information and a situation assessment complexity index into a multi-modal large model; the multi-modal large model extracts key information from a large amount of data collected by the unmanned cluster, a simplified natural language situation assessment is generated to a human operator, the human operator sends a cluster control voice instruction conforming to a human language habit, the voice instruction of the human operator is converted into a natural language text through a voice recognition algorithm, and the natural language text is stored in the unmanned cluster. The multi-mode large model outputs an advanced cluster control instruction meeting the autonomous control algorithm format of the unmanned platform, so that the unmanned cluster directly takes action according to the intention of a human operator, and the perception and control efficiency of the man-machine hybrid system is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned cluster systems, and in particular relates to a sensing and control method for a human-machine hybrid system based on a large model. Background Art

[0002] Unmanned swarm systems have important application value in defense, geology, rescue and other fields. The increase in the number of unmanned platforms in swarm systems has put forward higher requirements for the situational awareness and control of swarm systems. The pre-trained large model has a strong natural language understanding ability, and it is of great significance to apply the large model to the situational awareness and control of human-machine hybrid systems.

[0003] At present, in the unmanned swarm system controlled by a single human operator, there are dozens to hundreds of heterogeneous unmanned platforms, the amount of situation and state information is large, the swarm control is complex, the cognitive load of the human operator is large, and the safety of the unmanned swarm is difficult to ensure. Therefore, a large-scale model-based human-machine hybrid system sensing and control method is urgently needed to improve the perception and control efficiency of the human-machine hybrid system. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is: in an unmanned cluster system controlled by a single human operator, which includes dozens to hundreds of heterogeneous unmanned platforms, the amount of situation and state information is large, the cluster control is complex, the cognitive load of the human operator is large, and the safety of the unmanned cluster is difficult to guarantee. Therefore, in order to improve the perception and control efficiency of the human-machine hybrid system, it is necessary to propose a human-machine hybrid system perception and control method based on a large model, input the cluster state information, images with target borders, target categories, target positioning information and situation assessment complexity indicators into a multimodal large model, the multimodal large model extracts key information from a large amount of data collected by the unmanned cluster to the human operator, the human operator issues a cluster control voice command that conforms to human language habits, and the human operator's voice command is converted into a natural language text through a speech recognition algorithm and input into the multimodal large model, the multimodal large model outputs a high-level cluster control command that meets the format of the unmanned platform autonomous control algorithm, so that the unmanned cluster directly takes action according to the intention of the human operator, and effectively improves the efficiency of human-machine collaboration.

[0006] (II) Technical solution

[0007] To solve the above technical problems, the present invention provides a method for sensing and controlling a human-machine hybrid system based on a large model, characterized in that the method for sensing and controlling a human-machine hybrid system based on a large model is used to improve the perception and control efficiency of the human-machine hybrid system;

[0008] The human-machine hybrid system includes an unmanned cluster system and a human operator; the unmanned cluster system collects environmental information and sends it to the human operator, and receives control instructions from the human operator; the human operator makes control decisions based on the environmental information collected by the unmanned cluster system and sends control instructions to the unmanned cluster system;

[0009] The unmanned swarm system consists of multiple drones and unmanned vehicles, each of which is an unmanned platform. The unmanned platform is equipped with an embedded computer, a posture measurement module, a motion control module, a battery, a battery monitoring module, a communication radio, a binocular camera, and a self-destruct module.

[0010] The embedded computer is the control core of a single unmanned platform, and exchanges data with the posture measurement module, motion control module, battery, battery monitoring module, communication radio, binocular camera, and self-destruct module; the embedded computer receives the position and posture information of the posture measurement module, receives the battery voltage information of the battery monitoring module, receives the visual image of the binocular camera, receives the advanced cluster control instructions from the human operator through the communication radio, sends the cluster status information and image information to the human operator through the communication radio, sends the motion control instructions to the motion control module, and sends the self-destruction instructions to the self-destruction module; an autonomous control algorithm with path planning, obstacle avoidance, task allocation, and self-destruction functions is deployed in the embedded computer to receive the advanced cluster control instructions and data of other modules, and convert the advanced cluster control instructions into the motion control instructions and self-destruction instructions of the unmanned platform through task allocation;

[0011] The posture measurement module measures the position and posture information of the unmanned platform, including positioning information, speed, acceleration, angular velocity and posture information; the posture measurement module sends the measured position and posture information to the embedded computer;

[0012] The motion control module receives motion control instructions from the embedded computer and controls the output power of the motor of the unmanned platform;

[0013] Batteries provide power to unmanned platforms;

[0014] The battery monitoring module measures the battery voltage and converts it into remaining power information, which is then sent to the embedded computer;

[0015] The communication radio receives high-level control instructions from the human operator and sends them to the embedded computer, receives the unmanned platform status information and image information summarized by the embedded computer and sends them to the human operator;

[0016] The binocular camera captures the image directly in front of the unmanned platform and sends the image to the embedded computer;

[0017] The self-destruction module receives the self-destruction command from the embedded computer and triggers the self-destruction operation according to the self-destruction command to achieve obstacle removal for the unmanned platform;

[0018] The human operator carries a backpack server, a heart rate sensor, a microphone, an operation terminal, and a communication radio. The human operator reads the situation assessment results through the operation terminal and issues cluster control voice commands through the microphone.

[0019] The backpack server exchanges data with the heart rate sensor, microphone, communication radio, and operation terminal; the backpack server receives the heart rate data measured by the heart rate sensor, receives the voice signal of the human operator through the microphone, receives the cluster status information and image information of the unmanned cluster through the communication radio, sends advanced cluster control instructions to the unmanned cluster through the communication radio, and displays the natural language situation assessment to the operation terminal; the multimodal large model, speech recognition algorithm, target detection and positioning algorithm, and security assessment algorithm are deployed in the backpack server; the multimodal large model adjusts the language complexity of the output situation assessment according to the heart rate data of the human operator, outputs the natural language situation assessment according to the cluster status information and image information of the unmanned cluster, and generates advanced cluster control instructions according to the natural language cluster control instructions of the human operator; the speech recognition algorithm converts the voice signal of the human operator into natural language text; the target detection and positioning algorithm detects humans and objects in the image based on the image collected by the cluster and measures the relative position of humans and objects; the security assessment algorithm evaluates the advanced cluster control instructions according to the security policy;

[0020] The heart rate sensor measures the heart rate of the human operator and is installed at the heart position of the human operator;

[0021] The microphone receives a voice signal from a human operator;

[0022] The communication radio receives status information and image information from the unmanned cluster and sends it to the backpack server, and receives high-level control instructions from human operators and sends them to the unmanned cluster;

[0023] The operation terminal includes a display screen and operation buttons. The display screen displays the high-level control instructions and natural language situation assessment of the human operator, and the operation buttons control the start and stop of the unmanned cluster.

[0024] The human-machine hybrid system sensing control method comprises the following steps:

[0025] Step 1: The human operator wears a heart rate sensor and a microphone, carries a communication radio and a backpack server, and holds an operation terminal. The human operator starts the unmanned cluster system through the operation terminal.

[0026] Step 2: The heart rate sensor collects the heart rate data of the human operator, calculates the situation assessment complexity index, and inputs it into the multimodal large model. The multimodal large model adjusts the language complexity of the situation assessment according to the complexity index; the situation assessment complexity index is calculated through the heart rate to reflect the human cognitive load, so that the multimodal large model can adaptively adjust the language complexity of the situation assessment according to the size of the human cognitive load, improve the situation perception experience and decision-making efficiency of the human operator, and then improve the perception and control efficiency of the human-machine hybrid system; the situation assessment complexity index is calculated by the following formula:

[0027]

[0028] Where x is the instantaneous heart rate, k1 is the resting heart rate of the human operator, k2 is the age of the human operator, ε is a small positive number to prevent the denominator from being 0, and φ is the situation assessment complexity index;

[0029] Step 3: All unmanned platforms in the unmanned cluster system collect platform status information and environmental images, and send the platform status information and images to the backpack server through the communication radio;

[0030] Step 4: The piggyback server aggregates the platform status information into cluster status information and adjusts it into the text input format of the multimodal large model;

[0031] Step 5: The backpack server runs the object detection and localization algorithm on the image and outputs the image with the object bounding box, the object category and the object location information;

[0032] Step 6: Input the cluster state information, images with target bounding boxes, target categories, target positioning information and situation assessment complexity index into the multimodal large model, and the multimodal large model generates a natural language situation assessment; the language complexity of the situation assessment is related to the situation assessment complexity index, and further to the human cognitive load, so that the language complexity of the situation assessment can be adjusted according to the human cognitive load to improve the human situation perception experience and decision-making efficiency; the multimodal large model extracts key information from the large amount of data collected by the unmanned cluster system, discards invalid or redundant information, and outputs the streamlined natural language situation assessment to the human operator, thereby improving the utilization efficiency of the unmanned cluster system information and improving the perception efficiency of the human-machine hybrid system;

[0033] Step 7: The human operator reads the natural language situation assessment through the operation terminal and issues cluster control voice commands through the microphone;

[0034] Step 8: The backpack server converts the human operator's voice signal into natural language text through the voice recognition algorithm and inputs it into the multimodal large model, which outputs high-level cluster control instructions. The human operator issues voice instructions that conform to human language habits, and the high-level cluster control instructions meet the format of the autonomous control algorithm directly input to the unmanned platform, so that the unmanned cluster system can take actions directly according to the intentions of the human operator, thereby improving the control efficiency of the human-machine hybrid system.

[0035] Step 9: Input the advanced cluster control instructions generated by the multimodal large model into the safety assessment algorithm, and the safety assessment algorithm evaluates the trajectory safety and self-destruction safety of the advanced cluster control instructions; if the advanced cluster control instructions have no safety risks, the advanced cluster control instructions are sent to the unmanned cluster system for execution, otherwise the multimodal large model regenerates the advanced cluster control instructions; perform a safety assessment on the advanced cluster control instructions generated by the multimodal large model to reduce the risk of operational errors, thereby reducing the possibility of accidents in the unmanned cluster system and improving the safety of the unmanned cluster system control.

[0036] Step 9-1: Evaluate the trajectory safety of the high-level cluster control command, determine whether there is a collision between the unmanned platforms, and calculate the trajectory safety index based on the cluster status information; calculate all predicted trajectories within the predicted time range:

[0037] X pre,t =X+t·V,t=dt,2dt,...,T

[0038] Among them, X pre,t is the predicted trajectory at t prediction steps, X is the cluster position matrix, V is the cluster velocity matrix, T is the prediction time range, t is the prediction time, and dt is the prediction time resolution;

[0039] The minimum spacing of the unmanned platform prediction trajectory at each prediction step within the prediction time range is:

[0040] d min =min(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T

[0041] Among them, d min The minimum spacing of the unmanned platform prediction trajectory for each prediction step, M+N is the total number of unmanned platforms, M is the number of unmanned vehicles, N is the number of drones, i and j are the numbers of unmanned platforms, and min() means finding the minimum value;

[0042] The maximum spacing of the unmanned platform prediction trajectory at each prediction step within the prediction time range is:

[0043] d max =max(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T

[0044] Among them, d max The maximum distance between the unmanned platform's predicted trajectories for each prediction step;

[0045] The trajectory safety index is calculated by the following formula:

[0046]

[0047] Among them, safe traj is the trajectory safety index, d1 is the minimum safe distance, d2 is the absolute safe distance, 'notsafe' means the trajectory is not safe, 'complete safe' means the trajectory is absolutely safe, std() means to calculate the variance of each column of the matrix;

[0048] The conditions for unmanned cluster trajectory safety are:

[0049] safe traj ='completely safe' or safe traj >safe traj_threshold

[0050] Among them, safe traj_threshold is the trajectory safety threshold;

[0051] Step 9-2: If the high-level cluster control command includes a self-destruction command, evaluate the self-destruction safety of the high-level cluster control command and determine whether the self-destruction of the unmanned platform threatens the human operator and other unmanned platforms. Otherwise, skip this step; calculate the self-destruction safety index based on the cluster status information; the self-destruction damage radius is calculated by the following formula:

[0052]

[0053] Among them, r is the self-destruction damage radius, K is the damage coefficient, and Q is the TNT equivalent loaded in the self-destruction module of the unmanned platform;

[0054] The position of the unmanned platform that is about to self-destruct is calculated by the following formula:

[0055] X blast =X·C

[0056] Among them, X blast is the position matrix of the self-destructing unmanned platform, X is the cluster position matrix, and C is the self-destruction control matrix;

[0057] Calculate the minimum distance between the unmanned platform that is about to self-destruct and other unmanned platforms:

[0058] d blast_platform =min(||X(i)-X blast (k)||2),i=1,2,...,M+N,k=1,2,...,L and i≠k

[0059] Among them, k is the number of the self-destructing unmanned platform, and L is the number of self-destructing unmanned platforms;

[0060] Calculate the minimum distance from the unmanned platform to the human operator that is about to self-destruct:

[0061] d blast_human =min(||X human -X blast (k)||2),k=1,2,...,L

[0062] Among them, X human For human operator position;

[0063] The self-destruction safety index is calculated by the following formula:

[0064]

[0065] Among them, safe blast is the self-destruction safety indicator, 'not safe' means self-destruction is not safe, 'safe' means self-destruction is safe;

[0066] The conditions for the safety of unmanned swarm self-destruction are:

[0067] safe blast ='safe'.

[0068] (III) Beneficial effects

[0069] Compared with the prior art, the present invention proposes a large-model-based human-machine hybrid system sensing and control method, which inputs cluster state information, images with target borders, target categories, target positioning information and situation assessment complexity indicators into a multimodal large model. The multimodal large model extracts key information from a large amount of data collected by the unmanned cluster, generates a concise natural language situation assessment for the human operator, and the human operator issues cluster control voice instructions that conform to human language habits. The human operator's voice instructions are converted into natural language text through a speech recognition algorithm and input into the multimodal large model. The multimodal large model outputs high-level cluster control instructions that meet the format of the unmanned platform autonomous control algorithm, so that the unmanned cluster can take action directly according to the intention of the human operator, thereby improving the perception and control efficiency of the human-machine hybrid system.

[0070] The present invention proposes a large-model-based human-machine hybrid system sensing and control method, which inputs cluster state information, images with target borders, target categories, target positioning information and situation assessment complexity indicators into a multimodal large model. The multimodal large model extracts key information from a large amount of data collected by the unmanned cluster, discards invalid or redundant information, and generates a concise natural language situation assessment for the human operator, thereby improving the utilization efficiency of the unmanned cluster information and improving the perception efficiency of the human-machine hybrid system.

[0071] The present invention proposes a sensing and control method for a human-machine hybrid system based on a large model. The human operator issues a cluster control voice command that conforms to human language habits. The voice command of the human operator is converted into natural language text through a speech recognition algorithm and input into a multimodal large model. The multimodal large model outputs high-level cluster control commands that meet the format of the unmanned platform's autonomous control algorithm, so that the unmanned cluster can take action directly according to the intention of the human operator, thereby improving the control efficiency of the human-machine hybrid system.

[0072] The present invention proposes a sensing and control method for a human-machine hybrid system based on a large model. The heart rate sensor collects the heart rate data of the human operator, calculates the situation assessment complexity index, and inputs it into the multimodal large model. The multimodal large model adjusts the language complexity of the situation assessment according to the complexity index. The situation assessment complexity index is calculated by heart rate to reflect the human cognitive load, so that the multimodal large model can adaptively adjust the language complexity of the situation assessment according to the size of the human cognitive load, improve the situation perception experience and decision-making efficiency of the human operator, and then improve the perception and control efficiency of the human-machine hybrid system.

[0073] The present invention proposes a large-model-based human-machine hybrid system sensing and control method, which inputs the advanced cluster control instructions generated by the multimodal large model into a safety assessment algorithm. The safety assessment algorithm evaluates the trajectory safety and self-destruction safety of the advanced cluster control instructions, reduces the risk of operational errors, and further reduces the possibility of unexpected incidents in the unmanned cluster, thereby improving the safety of unmanned cluster control.

[0074] The patent of the present invention processes the voice commands of human operators through a multimodal large model and a speech recognition algorithm, and outputs advanced cluster control commands. Replacing the multimodal large model and the speech recognition algorithm with a multimodal large model that can directly input audio signals can be considered as the same solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a schematic diagram of the structure of a human-machine hybrid system;

[0076] Figure 2 It is a schematic diagram of a large-model-based human-machine hybrid system sensing control method flow chart of the present invention;

[0077] Figure 3It is a schematic diagram of a situation assessment process of a human-machine hybrid system based on a large model of the present invention;

[0078] Figure 4 It is a schematic diagram of an unmanned cluster control process of a human-machine hybrid system based on a large model of the present invention;

[0079] Figure 5 It is a schematic diagram of a language complexity flow of adaptively adjusting situation assessment of a human-machine hybrid system based on a large model of the present invention;

[0080] Figure 6 It is a schematic diagram of the display interface of the operation terminal. DETAILED DESCRIPTION

[0081] In order to make the purpose, content, and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings and examples.

[0082] In order to solve the problems of the prior art, the present invention provides a large-model-based human-machine hybrid system sensing and control method, characterized in that the large-model-based human-machine hybrid system sensing and control method is used to improve the perception and control efficiency of the human-machine hybrid system;

[0083] The human-machine hybrid system includes an unmanned cluster system and a human operator; the unmanned cluster system collects environmental information and sends it to the human operator, and receives control instructions from the human operator; the human operator makes control decisions based on the environmental information collected by the unmanned cluster system and sends control instructions to the unmanned cluster system;

[0084] The unmanned swarm system consists of multiple drones and unmanned vehicles, each of which is an unmanned platform. The unmanned platform is equipped with an embedded computer, a posture measurement module, a motion control module, a battery, a battery monitoring module, a communication radio, a binocular camera, and a self-destruct module.

[0085] The embedded computer is the control core of a single unmanned platform, and exchanges data with the posture measurement module, motion control module, battery, battery monitoring module, communication radio, binocular camera, and self-destruct module; the embedded computer receives the position and posture information of the posture measurement module, receives the battery voltage information of the battery monitoring module, receives the visual image of the binocular camera, receives the advanced cluster control instructions from the human operator through the communication radio, sends the cluster status information and image information to the human operator through the communication radio, sends the motion control instructions to the motion control module, and sends the self-destruction instructions to the self-destruction module; an autonomous control algorithm with path planning, obstacle avoidance, task allocation, and self-destruction functions is deployed in the embedded computer to receive the advanced cluster control instructions and data of other modules, and convert the advanced cluster control instructions into the motion control instructions and self-destruction instructions of the unmanned platform through task allocation;

[0086] The posture measurement module measures the position and posture information of the unmanned platform, including positioning information, speed, acceleration, angular velocity and posture information; the posture measurement module sends the measured position and posture information to the embedded computer;

[0087] The motion control module receives motion control instructions from the embedded computer and controls the output power of the motor of the unmanned platform;

[0088] Batteries provide power to unmanned platforms;

[0089] The battery monitoring module measures the battery voltage and converts it into remaining power information, which is then sent to the embedded computer;

[0090] The communication radio receives high-level control instructions from the human operator and sends them to the embedded computer, receives the unmanned platform status information and image information summarized by the embedded computer and sends them to the human operator;

[0091] The binocular camera captures the image directly in front of the unmanned platform and sends the image to the embedded computer;

[0092] The self-destruction module receives the self-destruction command from the embedded computer and triggers the self-destruction operation according to the self-destruction command to achieve obstacle removal for the unmanned platform;

[0093] The human operator carries a backpack server, a heart rate sensor, a microphone, an operation terminal, and a communication radio. The human operator reads the situation assessment results through the operation terminal and issues cluster control voice commands through the microphone.

[0094] The backpack server exchanges data with the heart rate sensor, microphone, communication radio, and operation terminal; the backpack server receives the heart rate data measured by the heart rate sensor, receives the voice signal of the human operator through the microphone, receives the cluster status information and image information of the unmanned cluster through the communication radio, sends advanced cluster control instructions to the unmanned cluster through the communication radio, and displays the natural language situation assessment to the operation terminal; the multimodal large model, speech recognition algorithm, target detection and positioning algorithm, and security assessment algorithm are deployed in the backpack server; the multimodal large model adjusts the language complexity of the output situation assessment according to the heart rate data of the human operator, outputs the natural language situation assessment according to the cluster status information and image information of the unmanned cluster, and generates advanced cluster control instructions according to the natural language cluster control instructions of the human operator; the speech recognition algorithm converts the voice signal of the human operator into natural language text; the target detection and positioning algorithm detects humans and objects in the image based on the image collected by the cluster and measures the relative position of humans and objects; the security assessment algorithm evaluates the advanced cluster control instructions according to the security policy;

[0095] The heart rate sensor measures the heart rate of the human operator and is installed at the heart position of the human operator;

[0096] The microphone receives a voice signal from a human operator;

[0097] The communication radio receives status information and image information from the unmanned cluster and sends it to the backpack server, and receives high-level control instructions from human operators and sends them to the unmanned cluster;

[0098] The operation terminal includes a display screen and operation buttons. The display screen displays the high-level control instructions and natural language situation assessment of the human operator, and the operation buttons control the start and stop of the unmanned cluster.

[0099] The human-machine hybrid system sensing control method comprises the following steps:

[0100] Step 1: The human operator wears a heart rate sensor and a microphone, carries a communication radio and a backpack server, and holds an operation terminal. The human operator starts the unmanned cluster system through the operation terminal.

[0101] Step 2: The heart rate sensor collects the heart rate data of the human operator, calculates the situation assessment complexity index, and inputs it into the multimodal large model. The multimodal large model adjusts the language complexity of the situation assessment according to the complexity index; the situation assessment complexity index is calculated through the heart rate to reflect the human cognitive load, so that the multimodal large model can adaptively adjust the language complexity of the situation assessment according to the size of the human cognitive load, improve the situation perception experience and decision-making efficiency of the human operator, and then improve the perception and control efficiency of the human-machine hybrid system; the situation assessment complexity index is calculated by the following formula:

[0102]

[0103] Where x is the instantaneous heart rate, k1 is the resting heart rate of the human operator, k2 is the age of the human operator, ε is a small positive number to prevent the denominator from being 0, and φ is the situation assessment complexity index;

[0104] Step 3: All unmanned platforms in the unmanned cluster system collect platform status information and environmental images, and send the platform status information and images to the backpack server through the communication radio;

[0105] Step 4: The piggyback server aggregates the platform status information into cluster status information and adjusts it into the text input format of the multimodal large model;

[0106] Step 5: The backpack server runs the object detection and localization algorithm on the image and outputs the image with the object bounding box, the object category and the object location information;

[0107] Step 6: Input the cluster state information, images with target bounding boxes, target categories, target positioning information and situation assessment complexity index into the multimodal large model, and the multimodal large model generates a natural language situation assessment; the language complexity of the situation assessment is related to the situation assessment complexity index, and further to the human cognitive load, so that the language complexity of the situation assessment can be adjusted according to the human cognitive load to improve the human situation perception experience and decision-making efficiency; the multimodal large model extracts key information from the large amount of data collected by the unmanned cluster system, discards invalid or redundant information, and outputs the streamlined natural language situation assessment to the human operator, thereby improving the utilization efficiency of the unmanned cluster system information and improving the perception efficiency of the human-machine hybrid system;

[0108] Step 7: The human operator reads the natural language situation assessment through the operation terminal and issues cluster control voice commands through the microphone;

[0109] Step 8: The backpack server converts the human operator's voice signal into natural language text through the voice recognition algorithm and inputs it into the multimodal large model, which outputs high-level cluster control instructions. The human operator issues voice instructions that conform to human language habits, and the high-level cluster control instructions meet the format of the autonomous control algorithm directly input to the unmanned platform, so that the unmanned cluster system can take actions directly according to the intentions of the human operator, thereby improving the control efficiency of the human-machine hybrid system.

[0110] Step 9: Input the advanced cluster control instructions generated by the multimodal large model into the safety assessment algorithm, and the safety assessment algorithm evaluates the trajectory safety and self-destruction safety of the advanced cluster control instructions; if the advanced cluster control instructions have no safety risks, the advanced cluster control instructions are sent to the unmanned cluster system for execution, otherwise the multimodal large model regenerates the advanced cluster control instructions; perform a safety assessment on the advanced cluster control instructions generated by the multimodal large model to reduce the risk of operational errors, thereby reducing the possibility of accidents in the unmanned cluster system and improving the safety of the unmanned cluster system control.

[0111] Step 9-1: Evaluate the trajectory safety of the high-level cluster control command, determine whether there is a collision between the unmanned platforms, and calculate the trajectory safety index based on the cluster status information; calculate all predicted trajectories within the predicted time range:

[0112] X pre,t =X+t·V,t=dt,2dt,...,T

[0113] Among them, X pre,t is the predicted trajectory at t prediction steps, X is the cluster position matrix, V is the cluster velocity matrix, T is the prediction time range, t is the prediction time, and dt is the prediction time resolution;

[0114] The minimum spacing of the unmanned platform prediction trajectory at each prediction step within the prediction time range is:

[0115] d min =min(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T

[0116] Among them, d min The minimum spacing of the unmanned platform prediction trajectory for each prediction step, M+N is the total number of unmanned platforms, M is the number of unmanned vehicles, N is the number of drones, i and j are the numbers of unmanned platforms, and min() means finding the minimum value;

[0117] The maximum spacing of the unmanned platform prediction trajectory at each prediction step within the prediction time range is:

[0118] d max =max(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T

[0119] Among them, d max The maximum distance between the unmanned platform's predicted trajectories for each prediction step;

[0120] The trajectory safety index is calculated by the following formula:

[0121]

[0122] Among them, safe traj is the trajectory safety index, d1 is the minimum safe distance, d2 is the absolute safe distance, 'notsafe' means the trajectory is not safe, 'complete safe' means the trajectory is absolutely safe, std() means to calculate the variance of each column of the matrix;

[0123] The conditions for unmanned cluster trajectory safety are:

[0124] safe traj ='completely safe' or safe traj >safe traj_threshold

[0125] Among them, safe traj_threshold is the trajectory safety threshold;

[0126] Step 9-2: If the high-level cluster control command includes a self-destruction command, evaluate the self-destruction safety of the high-level cluster control command and determine whether the self-destruction of the unmanned platform threatens the human operator and other unmanned platforms. Otherwise, skip this step; calculate the self-destruction safety index based on the cluster status information; the self-destruction damage radius is calculated by the following formula:

[0127]

[0128] Among them, r is the self-destruction damage radius, K is the damage coefficient, and Q is the TNT equivalent loaded in the self-destruction module of the unmanned platform;

[0129] The position of the unmanned platform that is about to self-destruct is calculated by the following formula:

[0130] X blast =X·C

[0131] Among them, X blast is the position matrix of the self-destructing unmanned platform, X is the cluster position matrix, and C is the self-destruction control matrix;

[0132] Calculate the minimum distance between the unmanned platform that is about to self-destruct and other unmanned platforms:

[0133] d blast_platform =min(||X(i)-X blast (k)||2),i=1,2,...,M+N,k=1,2,...,L and i≠k

[0134] Among them, k is the number of the self-destructing unmanned platform, and L is the number of self-destructing unmanned platforms;

[0135] Calculate the minimum distance from the unmanned platform to the human operator that is about to self-destruct:

[0136] d blast_human =min(||X human -X blast (k)||2),k=1,2,...,L

[0137] Among them, X human For human operator position;

[0138] The self-destruction safety index is calculated by the following formula:

[0139]

[0140] Among them, safe blast is the self-destruction safety indicator, 'not safe' means self-destruction is not safe, 'safe' means self-destruction is safe;

[0141] The conditions for the safety of unmanned swarm self-destruction are:

[0142] safe blast ='safe'.

[0143] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A human-machine hybrid system sensing control method based on a large model, characterized in that: The human-machine hybrid system sensing control method comprises the following steps: Step 1: The human operator wears a heart rate sensor and a microphone, carries a communication radio and a backpack server, and holds an operation terminal. The human operator starts the unmanned cluster system through the operation terminal. Step 2: The heart rate sensor collects the heart rate data of the human operator, calculates the situation assessment complexity index, and inputs it into the multimodal large model. The multimodal large model adjusts the language complexity of the situation assessment according to the complexity index; the situation assessment complexity index is calculated through the heart rate to reflect the human cognitive load, so that the multimodal large model can adaptively adjust the language complexity of the situation assessment according to the size of the human cognitive load, improve the situation perception experience and decision-making efficiency of the human operator, and then improve the perception and control efficiency of the human-machine hybrid system; the situation assessment complexity index is calculated by the following formula: Where x is the instantaneous heart rate, k1 is the resting heart rate of the human operator, k2 is the age of the human operator, ε is a small positive number to prevent the denominator from being 0, and φ is the situation assessment complexity index; Step 3: All unmanned platforms in the unmanned cluster system collect platform status information and environmental images, and send the platform status information and images to the backpack server through the communication radio; Step 4: The piggyback server aggregates the platform status information into cluster status information and adjusts it into the text input format of the multimodal large model; Step 5: The backpack server runs the object detection and localization algorithm on the image and outputs the image with the object bounding box, the object category and the object location information; Step 6: Input the cluster state information, images with target bounding boxes, target categories, target positioning information and situation assessment complexity index into the multimodal large model, and the multimodal large model generates a natural language situation assessment; the language complexity of the situation assessment is related to the situation assessment complexity index, and further to the human cognitive load, so that the language complexity of the situation assessment can be adjusted according to the human cognitive load to improve the human situation perception experience and decision-making efficiency; the multimodal large model extracts key information from the large amount of data collected by the unmanned cluster system, discards invalid or redundant information, and outputs the streamlined natural language situation assessment to the human operator, thereby improving the utilization efficiency of the unmanned cluster system information and improving the perception efficiency of the human-machine hybrid system; Step 7: The human operator reads the natural language situation assessment through the operation terminal and issues cluster control voice commands through the microphone; Step 8: The backpack server converts the human operator's voice signal into natural language text through the voice recognition algorithm and inputs it into the multimodal large model, which outputs high-level cluster control instructions. The human operator issues voice instructions that conform to human language habits, and the high-level cluster control instructions meet the format of the autonomous control algorithm directly input to the unmanned platform, so that the unmanned cluster system can take actions directly according to the intentions of the human operator, thereby improving the control efficiency of the human-machine hybrid system. Step 9: Input the advanced cluster control instructions generated by the multimodal large model into the safety assessment algorithm, and the safety assessment algorithm evaluates the trajectory safety and self-destruction safety of the advanced cluster control instructions; if the advanced cluster control instructions have no safety risks, the advanced cluster control instructions are sent to the unmanned cluster system for execution, otherwise the multimodal large model regenerates the advanced cluster control instructions; perform a safety assessment on the advanced cluster control instructions generated by the multimodal large model to reduce the risk of operational errors, thereby reducing the possibility of accidents in the unmanned cluster system and improving the safety of the unmanned cluster system control.

2. The large-model-based human-machine hybrid system sensing control method according to claim 1, characterized in that: The step 9 includes: Step 9-1: Evaluate the trajectory safety of the high-level cluster control command, determine whether there is a collision between the unmanned platforms, and calculate the trajectory safety index based on the cluster status information; calculate all predicted trajectories within the predicted time range: X pre,t =X+t·V,t=dt,2dt,...,T Among them, X pre,t is the predicted trajectory under t prediction steps, X is the cluster position matrix, V is the cluster velocity matrix, T is the prediction time range, t is the prediction time, and dt is the prediction time resolution; The minimum spacing of the unmanned platform prediction trajectory at each prediction step within the prediction time range is: d min =min(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T Among them, d min The minimum spacing of the unmanned platform prediction trajectory for each prediction step, M+N is the total number of unmanned platforms, M is the number of unmanned vehicles, N is the number of drones, i and j are the numbers of unmanned platforms, and min() means finding the minimum value; The maximum spacing of the unmanned platform prediction trajectory at each prediction step within the prediction time range is: d max =max(||X pre,t (i)-X pre,t (j)||2),i,j=1,2,...,M+N and i≠j,t=dt,2dt,...,T Among them, d max The maximum distance between the unmanned platform's predicted trajectories for each prediction step; The trajectory safety index is calculated by the following formula: Among them, safe traj is the trajectory safety index, d1 is the minimum safe distance, d2 is the absolute safe distance, 'not safe' means the trajectory is not safe, 'complete safe' means the trajectory is absolutely safe, std() means to calculate the variance of each column of the matrix; The conditions for unmanned cluster trajectory safety are: safe traj ='complete safe' or safe traj > safe traj_threshold Among them, safe traj_threshold is the trajectory safety threshold; Step 9-2: If the high-level cluster control command includes a self-destruction command, evaluate the self-destruction safety of the high-level cluster control command and determine whether the self-destruction of the unmanned platform threatens the human operator and other unmanned platforms. Otherwise, skip this step; calculate the self-destruction safety index based on the cluster status information; the self-destruction damage radius is calculated by the following formula: Among them, r is the self-destruction damage radius, K is the damage coefficient, and Q is the TNT equivalent loaded in the self-destruction module of the unmanned platform; The position of the unmanned platform that is about to self-destruct is calculated by the following formula: X blast =X·C Among them, X blast is the position matrix of the self-destructing unmanned platform, X is the cluster position matrix, and C is the self-destruction control matrix; Calculate the minimum distance between the unmanned platform that is about to self-destruct and other unmanned platforms: d blast_platform =min(||X(i)-X blast (k)||2),i=1,2,...,M+N,k=1,2,...,L and i≠k Among them, k is the number of the self-destructing unmanned platform, and L is the number of self-destructing unmanned platforms; Calculate the minimum distance from the unmanned platform to the human operator that is about to self-destruct: d blast_human =min(||X human -X blast (k)||2),k=1,2,...,L Among them, X human For human operator position; The self-destruction safety index is calculated by the following formula: Among them, safe blast is the self-destruction safety indicator, 'not safe' means self-destruction is not safe, 'safe' means self-destruction is safe; The conditions for the safety of unmanned swarm self-destruction are: safe blast ='safe'。 3. The large-model-based human-machine hybrid system sensing control method according to claim 1, characterized in that: The large-model-based human-machine hybrid system sensing and control method is used to improve the perception and control efficiency of the human-machine hybrid system.

4. The large-model-based human-machine hybrid system sensing control method according to claim 1, characterized in that: The human-machine hybrid system includes an unmanned cluster system and a human operator; the unmanned cluster system collects environmental information and sends it to the human operator, and receives control instructions from the human operator; the human operator makes control decisions based on the environmental information collected by the unmanned cluster system and sends control instructions to the unmanned cluster system.

5. The large-model-based human-machine hybrid system sensing control method according to claim 4, characterized in that: The unmanned cluster system is composed of multiple drones and multiple unmanned vehicles, each of which is an unmanned platform; the unmanned platform is equipped with an embedded computer, a posture measurement module, a motion control module, a battery, a battery monitoring module, a communication radio, a binocular camera, and a self-destruct module.

6. The large-model-based human-machine hybrid system sensing control method according to claim 5, characterized in that: The embedded computer is the control core of a single unmanned platform, exchanging data with the posture measurement module, motion control module, battery, battery monitoring module, communication radio, binocular camera, and self-destruction module; The embedded computer receives the position and posture information of the posture measurement module, receives the battery voltage information of the battery monitoring module, receives the visual image of the binocular camera, receives the advanced cluster control instructions from the human operator through the communication radio, sends the cluster status information and image information to the human operator through the communication radio, sends the motion control instructions to the motion control module, and sends the self-destruction instructions to the self-destruction module; an autonomous control algorithm with path planning, obstacle avoidance, task allocation, and self-destruction functions is deployed in the embedded computer to receive the advanced cluster control instructions and data from other modules, and convert the advanced cluster control instructions into motion control instructions and self-destruction instructions of the unmanned platform through task allocation.

7. The large-model-based human-machine hybrid system sensing control method according to claim 6, characterized in that: The position and attitude measurement module measures the position and attitude information of the unmanned platform, including positioning information, speed, acceleration, angular velocity and attitude information; the position and attitude measurement module sends the measured position and attitude information to the embedded computer.

8. The large-model-based human-machine hybrid system sensing control method according to claim 7, characterized in that: The motion control module receives motion control instructions from the embedded computer and controls the output power of the motor of the unmanned platform; Batteries provide power to unmanned platforms; The battery monitoring module measures the battery voltage and converts it into remaining power information, which is then sent to the embedded computer; The communication radio receives high-level control instructions from the human operator and sends them to the embedded computer, receives the unmanned platform status information and image information summarized by the embedded computer and sends them to the human operator; The binocular camera captures the image directly in front of the unmanned platform and sends the image to the embedded computer; The self-destruct module receives the self-destruct command from the embedded computer and triggers the self-destruct operation according to the self-destruct command to achieve obstacle breaking for the unmanned platform.

9. The large-model-based human-machine hybrid system sensing control method according to claim 8, characterized in that: The human operator carries a backpack server, a heart rate sensor, a microphone, an operation terminal, and a communication radio. The human operator reads the situation assessment results through the operation terminal and issues cluster control voice commands through the microphone. The backpack server exchanges data with the heart rate sensor, microphone, communication radio, and operation terminal; The backpack server receives the heart rate data measured by the heart rate sensor, receives the voice signal of the human operator through the microphone, receives the cluster status information and image information of the unmanned cluster through the communication radio, sends the advanced cluster control instructions to the unmanned cluster through the communication radio, and displays the natural language situation assessment to the operation terminal; deploys a multimodal large model, a speech recognition algorithm, a target detection and positioning algorithm, and a safety assessment algorithm in the backpack server; the multimodal large model adjusts the language complexity of the output situation assessment according to the heart rate data of the human operator, outputs the natural language situation assessment according to the cluster status information and image information of the unmanned cluster, and generates the advanced cluster control instructions according to the natural language cluster control instructions of the human operator; the speech recognition algorithm converts the voice signal of the human operator into natural language text; the target detection and positioning algorithm detects humans and objects in the image based on the image collected by the cluster and measures the relative position of humans and objects; the safety assessment algorithm evaluates the advanced cluster control instructions according to the security policy; The heart rate sensor measures the heart rate of the human operator and is installed at the heart position of the human operator; The microphone receives a voice signal from a human operator; The communication radio receives status information and image information from the unmanned cluster and sends it to the backpack server, and receives high-level control instructions from human operators and sends them to the unmanned cluster.

10. The large-model-based human-machine hybrid system sensing control method according to claim 9, characterized in that: The operation terminal includes a display screen and operation buttons. The display screen displays the human operator's high-level control instructions and natural language situation assessment, and the operation buttons control the startup and shutdown of the unmanned cluster.

Images