Unmanned aerial vehicle cluster control method, device and equipment and storage medium

Abstract

The invention discloses an unmanned aerial vehicle cluster control method, device and equipment and a storage medium, electroencephalogram decoding of multi-mode signals of electroencephalogram signals and eye movement signals in a video RSVP is completed through a double-branch multi-mode Densnet network, and a video collected by an unmanned aerial vehicle is played to induce nerve electrical activities related to a specific target of an observer. A target classification is obtained by collecting, analyzing and processing electroencephalogram signals of a user; after related electrical activities of a specific target appear, the system captures the key information to automatically complete dynamic planning of the unmanned aerial vehicle group and automatically realize a specific strategy scheme for the target, and when a new execution scheme needs to be further completed, a new strategy scheme for the target can be completed only by blinking and entering an active control panel. According to the method, the accuracy of target identification is improved through the double-branch multi-mode Densnet network, so that the unmanned aerial vehicle cluster has higher task implementation capability, and the implementation of the scheme enables the unmanned aerial vehicle cluster scheme to have a more practical application method.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to a method, device, equipment and storage medium for controlling a drone cluster. Background Art

[0002] At present, the control technology of drone swarms has poor adaptability to unknown environments, resulting in relatively low success rate and efficiency in mission planning. Due to the lack of real-time perception and response capabilities to unknown environments, the swarm may face potential risks such as collisions and crashes. In order to improve the adaptability and safety of drone swarms, introducing technologies in the field of bioinformatics into drone systems is a feasible solution. Among them, EEG signals, as a bioinformatics signal that reflects human cognition and attention status, can reflect the electrophysiological activities of the brain in different states, providing new possibilities for abnormal state detection of drone swarms. By monitoring the EEG signals of operators, it is possible to understand their cognitive load, attention allocation and other information in real time, thereby adjusting the task allocation and execution strategy of the drone swarm.

[0003] The brain-computer interface based on the Rapid Serial Visual Presentation (RSVP) paradigm can obtain the human brain's rapid perception of the environment. The RSVP paradigm quickly presents environmental information to the subject through a high picture presentation rate, and uses electroencephalogram (EEG) signal decoding technology to interpret it, thereby achieving rapid perception of the surrounding environment. The decoding principle is mainly that when observing the target picture, the human brain will generate an event-related potential (ERP) signal, and the classification task is completed by extracting the event components related to the target stimulus.

[0004] However, under the RSVP paradigm, the decoding performance in the single EEG mode is still poor. In some research reports, eye movement signals show reactions such as visual search and target detection in visual processing. When the drone performs a mission and transmits back, it will present the video on the display. The eye tracker can collect eye movement information to complete the multimodal information decoding of RSVP. By recording the operator's gaze point and gaze duration, we can have a deeper understanding of their perception and understanding of the mission environment.

[0005] Compared with the classic RSVP paradigm, video RSVP plays high-frame-rate videos, which are closer to reality, but are more difficult to decode EEG. Dynamic targets appear without time anchors, making it difficult to accurately extract and analyze single-trial EEG signals induced by task goals. In actual video scenes, the location and time of target appearance are random, resulting in unpredictable visual search time for users, exacerbating ERP variation. The target appearance process is more natural, which weakens the visual stimulation to users, and the corresponding ERP amplitude may decrease, increasing the difficulty of decoding EEG signals. Summary of the invention

[0006] In order to solve the above-mentioned technical problems existing in the prior art, the present invention provides a drone cluster control method, device, equipment and storage medium.

[0007] In order to achieve the above object, the technical solution of the embodiment of the present invention is:

[0008] A first aspect of the present invention provides a method for controlling a drone cluster, the method comprising:

[0009] The real-time drone video is combined with the rapid sequence visual presentation paradigm, and RSVP stimulation signals are sent to users through EEG synchronization equipment;

[0010] In response to the RSVP stimulation signal, synchronously collecting a user's target EEG signal and target eye movement signal;

[0011] Determining a strong characteristic signal and a weak characteristic signal based on the signal dimensions and information amounts of the target EEG signal and the target eye movement signal;

[0012] Inputting the strong feature signal and the weak feature signal into a dual-branch multimodal Densnet network for feature fusion to obtain a target fusion feature;

[0013] Based on a preset classification algorithm, feature classification is performed on the target fusion features to obtain a classification result; and the confidence level corresponding to the classification result is calculated;

[0014] If the number of the classification results does not exceed a preset classification result number threshold, judging whether a specific target appears in the real-time video of the drone based on the classification results and the confidence level;

[0015] If the specific target appears in the real-time video of the drone, a target control instruction is executed on the specific target based on a preset strategy set.

[0016] An embodiment of the present invention provides a drone cluster control device, the device comprising:

[0017] The sending module is used to combine the obtained real-time video of the drone with the rapid sequence visual presentation paradigm and send RSVP stimulation signals to the user through the EEG synchronization device;

[0018] A collection module, used for synchronously collecting a user's target EEG signal and target eye movement signal in response to the RSVP stimulation signal;

[0019] A determination module, used to determine a strong characteristic signal and a weak characteristic signal based on the signal dimensions and information amounts of the target EEG signal and the target eye movement signal;

[0020] A fusion module, used for inputting the strong feature signal and the weak feature signal into a dual-branch multimodal Densnet network for feature fusion to obtain a target fusion feature;

[0021] A classification module is used to classify the target fusion features based on a preset classification algorithm to obtain a classification result; and calculate the confidence level corresponding to the classification result;

[0022] A judgment module, configured to judge whether a specific target appears in the real-time video of the drone based on the classification result and the confidence level if the number of the classification results does not exceed a preset classification result number threshold;

[0023] The execution module is used to execute a target control instruction on the specific target based on a preset strategy set if the specific target appears in the real-time video of the drone.

[0024] In some embodiments, the acquisition module is also used to acquire the target EEG signal; the target EEG signal carries a timestamp; the timestamp is sent to the EEG synchronization device; the timestamp is marked on the target EEG signal by the EEG synchronization device, and based on the timestamp, a target eye movement signal is obtained.

[0025] In some embodiments, the determination module is also used to compare the signal dimensions and information amounts of the target EEG signal and the target eye movement signal to obtain a comparison result; if the comparison result is that the information dimension and information amount of the target EEG signal are greater than those of the target eye movement signal, the target EEG signal is determined as a strong feature signal and the target eye movement signal is determined as a weak feature signal.

[0026] In some embodiments, the judgment module is also used to calculate the probability of occurrence of the specific target based on the classification results and the confidence level if the number of the classification results does not exceed a preset classification result number threshold; if the occurrence probability is greater than a preset probability threshold, determine that the specific target appears in the real-time video of the drone.

[0027] An embodiment of the present invention provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned drone cluster control method when executing the executable instructions stored in the memory.

[0028] An embodiment of the present invention provides a computer-readable storage medium, in which at least one executable instruction is stored. The at least one executable instruction is loaded and executed by a processor to implement the above-mentioned drone cluster control method.

[0029] The drone cluster control method, device, equipment and storage medium provided by the present invention complete the EEG decoding of multimodal signals of EEG signals and eye movement signals in video RSVP through a dual-branch multimodal Densnet network, induce the observer's neural electrical activity related to a specific target by playing the video collected by the drone, and obtain the target classification after collecting the user's EEG signal and analyzing and processing it; after the relevant electrical activity of the specific target appears, the system captures this key information to automatically complete the dynamic planning of the drone cluster, and automatically implements the specific strategy plan for the target. When it is necessary to further complete a new execution plan, it only needs to blink to enter the active control panel to complete the new strategy plan for the target. In the present invention, the dual-branch multimodal Densnet network improves the accuracy of target recognition, and the drone cluster control scheme under the video RSVP based on this enables the drone cluster to have a stronger task realization capability, and the implementation of this scheme enables the drone cluster scheme to have a more practical application method. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic diagram of the structure of a drone cluster control system provided by an embodiment of the present invention;

[0031] Figure 2 It is a flowchart of a method for controlling a drone cluster provided by an embodiment of the present invention;

[0032] Figure 3 It is a schematic diagram of the implementation process provided by an embodiment of the present invention;

[0033] Figure 4 It is a schematic diagram of the collection and training process of a multimodal fusion algorithm based on EEG signals and eye movement signals provided in an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of a drone swarm flight verification system provided by an embodiment of the present invention;

[0035] Figure 6 is a structural diagram of a multimodal fusion algorithm based on EEG signals and eye movement signals provided by an embodiment of the present invention;

[0036] Figure 7Schematic diagram of the structure of the drone cluster control device provided by an embodiment of the present invention;

[0037] Figure 8 It is a schematic diagram of the composition structure of the drone cluster control device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0039] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meaning as those commonly understood by those skilled in the art to which the embodiments of the present invention pertain. The terms used in the embodiments of the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0040] The following describes an exemplary application of the drone cluster control device of an embodiment of the present invention. The drone cluster control device provided by the embodiment of the present invention can be implemented as a terminal or a server. In one implementation, the drone cluster control device provided by the embodiment of the present invention can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, mobile devices, etc.; in another implementation, the drone cluster control device provided by the embodiment of the present invention can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present invention. Below, an exemplary application of the drone cluster control device when it is implemented as a server will be described.

[0041] See also Figure 1 , Figure 1It is a structural diagram of the drone cluster control system 10 provided in an embodiment of the present invention. In order to realize the control of the drone cluster, the embodiment of the present invention can provide a drone cluster control platform, which can be implemented as a drone cluster control application. The drone cluster control system 10 provided in the embodiment of the present invention includes a terminal 110, a network 120 and a server 130, wherein the server 130 is a server of the drone cluster control application. The server 130 can constitute the drone cluster control device of the embodiment of the present invention. The terminal 110 is connected to the server 130 through the network 120, and the network 120 can be a wide area network or a local area network, or a combination of the two.

[0042] In some embodiments, please refer to Figure 1 When controlling the drone cluster, the terminal 110 combines the obtained drone real-time video with the rapid sequence visual presentation paradigm through the network 120, sends an RSVP stimulation signal to the user through the EEG synchronization device and sends it to the server 130. The server 130 responds to the RSVP stimulation signal initiated by the terminal 110 and synchronously collects the user's target EEG signal and target eye movement signal; based on the signal dimensions and information content of the target EEG signal and the target eye movement signal, the strong feature signal and the weak feature signal are determined; the strong feature signal and the weak feature signal are input into the double-branch multimodal Densnet network for feature fusion to obtain the target fusion feature; based on the preset classification algorithm, the target fusion feature is feature classified to obtain the classification result; and the confidence corresponding to the classification result is calculated; if the number of classification results does not exceed the preset classification result number threshold, based on the classification result and the confidence, it is determined whether a specific target appears in the drone real-time video; if a specific target appears in the drone real-time video, based on the preset strategy set, the target control instruction is executed on the specific target to obtain the drone cluster control result. After obtaining the drone cluster control result, the server 130 sends the drone cluster control result to the terminal 110 through the network 120 .

[0043] The present invention provides a method for controlling a drone cluster. Figure 2 , Figure 2 is a flowchart of a method for controlling a drone cluster provided by an embodiment of the present invention, which is combined with Figure 2 The steps shown are explained.

[0044] Step S210, combining the obtained drone real-time video with the rapid sequence visual presentation paradigm, and sending RSVP stimulation signals to the user through the EEG synchronization device.

[0045] In some embodiments, the drone real-time video refers to a video that needs to be judged whether a specific target appears. The drone real-time video can be a video collected by an image collector set on the drone, or a video collected by a laser radar. In the present invention, the drone real-time video can be a video clip of about 12-15 seconds; or it can be a long video of about 15 minutes. A clip of the video clip contains only one usage scenario, such as no target, non-specified target, and specified target.

[0046] In some embodiments, the user refers to a person who wears an EEG cap and is able to perform corresponding operations.

[0047] In the present invention, 12-15s video clips and 15min long videos containing or not containing specific targets are played, and the video playback rate is greater than or equal to 30 frames / s. When each frame starts to play, the timestamp of the current moment is sent to the EEG synchronization device.

[0048] Step S220: In response to the RSVP stimulation signal, synchronously collect the user's target EEG signal and target eye movement signal.

[0049] In some embodiments, the target EEG signal refers to a total of 16 leads of EEG signals collected at Fz, FC1, FC2, C3, Cz, C4, CP1, CP2, P7, P3, Pz, P4, P8, O1, Oz, and O2 of the scalp by an EEG synchronization device according to the 10-20 standard. The collected EEG signal is subjected to a 1-30 Hz bandpass filter and then a 49.5-50.5 Hz bandstop filter to obtain a preprocessed EEG signal.

[0050] In some embodiments, the target eye movement signal refers to an eye movement signal collected based on a timestamp. The collected eye movement signal is subjected to a 1-30 Hz bandpass filter and then a 49.5-50.5 Hz bandstop filter to obtain a preprocessed eye movement signal.

[0051] In the present invention, the user watches the real-time video of the drone and pays attention to the appearance of a specific target. At the same time, the EEG signals of 16 leads, including Fz, FC1, FC2, C3, Cz, C4, CP1, CP2, P7, P3, Pz, P4, P8, O1, Oz, and O2, are collected at the scalp according to the 10-20 standard. The EEG synchronization device marks the time tag on the EEG signal and collects the eye movement signal at the same time. The target EEG signal and the target eye movement signal are based on the timestamp, and the EEG data and eye movement data within 1s after each timestamp appears in the real-time video of the drone are intercepted.

[0052] Step S230, determining a strong characteristic signal and a weak characteristic signal based on the signal dimensions and information amounts of the target EEG signal and the target eye movement signal.

[0053] In some embodiments, information dimensions refer to different aspects involved in describing and analyzing EEG signals and eye movement signals. Information dimensions may include the following aspects: time dimension, space dimension, frequency dimension, power and phase dimension, etc.

[0054] In some embodiments, the amount of information contained refers to the EEG information provided by multiple electrodes in the EEG. The potential difference between different electrodes can be compared, and unipolar or bipolar lead recordings can be performed to identify activities in specific areas of the brain. In addition, a variety of electrophysiological signals of the brain in different states can be captured, such as α waves, β waves, θ waves, δ waves, etc. These signals are closely related to different functional states of the brain. Electroencephalography (EEG) can also be used to further explore the network characteristics and information processing mechanisms of the brain through spectrum analysis, functional connectivity analysis and other methods. The eye movement signal generally only includes the pupil diameter of the left and right eyes and the gaze point target. In the analysis of RSVP, only the eye movement tracking speed and the change in pupil diameter when the target is found can be provided.

[0055] In some embodiments, a strong characteristic signal refers to a characteristic signal with a larger information dimension and a larger amount of information contained; a weak characteristic signal refers to a characteristic signal with a smaller information dimension and a smaller amount of information contained.

[0056] In the present invention, EEG signals are classified as strong characteristic signals according to the signal dimension and the amount of information contained, and eye movements are classified as weak characteristic signals because they have less information dimension and information amount compared to EEG signals.

[0057] Step S240: input the strong feature signal and the weak feature signal into a dual-branch multimodal Densnet network for feature fusion to obtain target fusion features.

[0058] In some embodiments, the dual-branch multimodal Densnet network can be pre-stored in a cloud storage or in a server, and the dual-branch multimodal Densnet network refers to a model capable of fusing strong characteristic signals and weak characteristic signals. The dual-branch multimodal Densnet network can be a multimodal dual-branch Densnet network model; it can also be other types of network models, which is not specifically limited in the present invention.

[0059] In some embodiments, the target fusion feature refers to a feature obtained by fusing the EEG feature and the eye movement feature.

[0060] In the present invention, EEG features and eye movement features are input into two branches of a multimodal dual-branch Densnet network. Densnet emphasizes feature reuse and can deeply convolve shallow features with deep features to obtain detailed information. In this network model, EEG features are only extracted and reused in the EEG feature convolution layer, while eye movement signals, as weaker features, complete shallow feature extraction in the first layer alone and then receive guidance from EEG signals for deep feature extraction.

[0061] Step S250, based on a preset classification algorithm, feature classification is performed on the target fusion feature to obtain a classification result; and the confidence level corresponding to the classification result is calculated.

[0062] In some embodiments, the preset classification algorithm may be pre-stored in a cloud storage or in a server. The preset classification algorithm refers to an algorithm that can determine whether a specific target appears in each video frame.

[0063] In the present invention, a classification algorithm can be used to classify features into 1 or 0 according to the classification requirements of "my aircraft" or "enemy aircraft", output the classification result R (R=0 or 1), and output the confidence P (0<P<1) of the classification result.

[0064] Step S260: If the number of the classification results does not exceed the preset classification result number threshold, based on the classification results and the confidence level, determine whether a specific target appears in the real-time video of the drone.

[0065] In some embodiments, the preset classification result quantity threshold refers to a threshold set when performing feature classification.

[0066] In the present invention, two registers L1 and L2 of length L can be prepared in advance, and the classification result R and confidence P obtained in the above step S250 can be stored at the ends of the registers L1 and L2 respectively. Afterwards, if the number of classification results does not exceed the preset classification result number threshold, the probability of the occurrence of a specific target is calculated based on the number of classification results and confidence stored in the register. If the number of classification results exceeds the preset classification result number threshold, the classification results previously stored in the register are deleted, and feature classification of the target fusion features is restarted.

[0067] Step S270: If the specific target appears in the real-time video of the drone, a target control instruction is executed on the specific target based on a preset strategy set.

[0068] In some embodiments, the preset strategy set refers to a collection of control instructions issued to the drone swarm under different tasks.

[0069] When actually performing tasks, the actual operation set given by experts for different tasks is used as the maneuver expert library, and the different operation methods given by experts to the drone swarm for different tasks are used as the strategy library. When determining the specific tasks to be performed by the drone swarm, the model scores the strategy set given by the expert library in the current situation, and the one with the highest score enters the preset maneuver instruction set. For example, when performing a capture task, the strategy set given by the expert library should give priority to capture over attack. When the capture target escapes the maximum range of the capture, the strategy set considers attacking. When the enemy attacks, it can switch to active brain control to complete the separation of the fighter. When the video RSVP is transmitted back, the human brain recognizes the designated target through passive brain control, and the drone swarm automatically enters the preset instruction state according to the preset strategy set. When it is necessary to further enter other strategy set contents, the observer only needs to blink to enter the active control panel. Under the active control panel, the observer can use active brain control instruction methods such as steady-state visual induction to complete the new strategy control of the drone swarm.

[0070] The drone cluster control method, device, equipment and storage medium provided by the present invention complete the EEG decoding of multimodal signals of EEG signals and eye movement signals in video RSVP through a dual-branch multimodal Densnet network, induce the observer's neural electrical activity related to a specific target by playing the video collected by the drone, and obtain the target classification after collecting the user's EEG signal and analyzing and processing it; after the relevant electrical activity of the specific target appears, the system captures this key information to automatically complete the dynamic planning of the drone cluster, and automatically implements the specific strategy plan for the target. When it is necessary to further complete a new execution plan, it only needs to blink to enter the active control panel to complete the new strategy plan for the target. In the present invention, the dual-branch multimodal Densnet network improves the accuracy of target recognition, and the drone cluster control scheme under the video RSVP based on this enables the drone cluster to have a stronger task realization capability, and the implementation of this scheme enables the drone cluster scheme to have a more practical application method.

[0071] In some embodiments, the above step S220 may further include the following contents:

[0072] First, the target EEG signal is collected; the target EEG signal carries a timestamp. Second, the timestamp is sent to the EEG synchronization device. Finally, the timestamp is marked on the target EEG signal by the EEG synchronization device, and the target eye movement signal is obtained based on the timestamp.

[0073] In some embodiments, the above step S230 may further include the following contents:

[0074] First, the signal dimensions and information content of the target EEG signal and the target eye movement signal are compared to obtain a comparison result. Second, if the comparison result shows that the information dimension and information content of the target EEG signal are greater than those of the target eye movement signal, the target EEG signal is determined as a strong feature signal and the target eye movement signal is determined as a weak feature signal.

[0075] In some embodiments, the above step S260 includes the following contents:

[0076] First, if the number of the classification results does not exceed the preset classification result number threshold, the occurrence probability of the specific target is calculated based on the classification results and the confidence level. Second, if the occurrence probability is greater than the preset probability threshold, it is determined that the specific target appears in the real-time video of the drone.

[0077] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0078] The present invention provides a more user-friendly brain-computer interface paradigm video RSVP. Compared with the rapid sequence visual presentation, which plays a high frame rate video, the user observes that the neural electrical activity potential generated by the corresponding target is lower, making decoding more difficult.

[0079] In some embodiments, in the offline stage, a video clip of about 12-15 seconds is used to induce user-related EEG features. A short video clip contains only one usage scenario, such as no target, non-specified target, and specified target. After the video is played, the user is required to complete feedback on the video clip. If there is no target, press 0, if there is a non-specified target, press 1, and if a specified target appears, press 2. After the user completes the selection, feedback on whether it is correct or not is given.

[0080] In the online stage, 6 15-minute drone videos were used to complete the online verification. When the designated target appeared in the video, the user pressed key 1, and when the non-designated target appeared, the key 0 was pressed. The EEG and eye movement signals and labels collected in the offline stage were used to complete the initial training of the model, and the online experimental test optimization was carried out based on the initial training model. The trained network uses a dual-branch Densnet network to complete the feature interaction and deep feature extraction between modalities. After the recognition target appears, it automatically enters the preset command. When the command setting needs to be further completed, the active control panel is called up using the blink switch. The passive brain-computer interface is more friendly under the video RSVP. When an abnormal situation occurs, the control of the drone group can be efficiently completed in combination with the active brain-computer interface.

[0081] In some embodiments, according to Figure 3Build a target detection system, where users wear EEG and eye movement acquisition equipment, watch the video played on the monitor, and pay attention to the appearance of specific targets. Use the EEG synchronization device to generate time tags and user-observed time node tags, and output them to the computer to mark them in the collected EEG signals and eye movement signals.

[0082] The present invention also provides a method for controlling a drone cluster by combining EEG and eye movement under video RSVP. Figure 4 As shown, the method comprises the following steps:

[0083] Step 1: Use the display to play 12-15s video clips and 15min long videos with or without specific targets. The video playback rate is greater than or equal to 30 frames / s. When each frame starts to play, the current time tag is sent to the EEG synchronization device.

[0084] Step 2: The user watches the video and pays attention to the appearance of specific targets. At the same time, the EEG signals of 16 leads, including Fz, FC1, FC2, C3, Cz, C4, CP1, CP2, P7, P3, Pz, P4, P8, O1, Oz, and O2, are collected at the scalp according to the 10-20 standard. The EEG synchronization device marks the time label on the EEG signal and collects eye movement signals at the same time.

[0085] Step 3: Align the EEG and eye movement data according to the timestamps, intercept the EEG and eye movement data 1 second after each timestamp, synchronously downsample to 64 Hz, perform a 1 Hz to 30 Hz bandpass filter, and then perform a 49.5 Hz to 50.5 Hz bandstop filter.

[0086] Step 4: According to the signal dimension and the amount of information contained, the EEG signal is classified as a strong feature signal, and the eye movement signal is classified as a weak feature signal because of its smaller dimension and information content compared to the EEG signal. The EEG signal and the eye movement signal are input into the two branches of the multimodal dual-branch Densnet network. Densnet emphasizes feature reuse and can deeply convolve shallow features with deep features to obtain detailed information. In the preset network framework, the EEG feature is only extracted and reused in the EEG feature convolution layer, while the eye movement signal, as a weaker feature, completes the shallow feature extraction in the first layer alone and then accepts the guidance of the EEG signal to extract deep features.

[0087] Step 5: Use the classification algorithm to classify the features into 1 or 0 according to "my aircraft" or "enemy aircraft", and output the classification result R (R = 0 or 1) and the confidence level P (0 < P < 1) of the result.

[0088] Step 6: During classification, if an enemy aircraft appears, the EEG will show the "erp" feature, and the eye movement signal will show the "rapid visual search and then tracking" feature. When our aircraft appears, the EEG will not show the "erp" feature, and the eye movement signal will still show the "rapid visual search and then tracking" feature.

[0089] Step 7: Construct two registers L1 and L2 of length L according to "my aircraft" and "enemy aircraft", and save the latest classification result R and confidence P at the end of the register. If the number of results retained in the register is greater than the preset classification result number threshold, delete the earliest classification result. If the number of classification results retained in the register exceeds the preset classification result number threshold L, according to the classification results stored in the register Calculate the probability of a specific target appearing When the Q value is greater than the preset probability threshold Q0, it is determined that a specific target appears in the video. Here, L is obtained from multiple training experiences.

[0090] Step 7: The actual operation set given by experts for different tasks during actual mission execution is used as the mobile expert library, and the different operation methods given by experts to the drone swarm for different tasks are set as the strategy library.

[0091] Step 8: The model output is converted into a probability distribution. When the probability of a specific target category appearing is the largest, it is judged that a specific target appears in the video. The results of multiple trainings are used as evaluation indicators for the expert library and strategy set.

[0092] Step 9: When determining the specific tasks to be performed by the drone swarm, the model scores the strategy set given by the expert database in the current situation. The one with the highest score enters the preset maneuver instruction set. For example, when performing a capture mission, the strategy set given by the expert database should prioritize capture over attack. When the capture target escapes the maximum range of the capture, the strategy set considers attacking. When the enemy uses targeted attack, it can switch to active brain control to complete the fighter's separation. In general, when the video RSVP is transmitted back, the human brain recognizes the designated target through passive brain control, and the drone swarm automatically enters the preset instruction state according to the preset strategy set. When it is necessary to further enter other strategy set contents, the observer only needs to blink to enter the active control panel. Under the active control panel, the observer can use active brain control instructions such as steady-state visual induction to complete the new strategy control of the drone swarm.

[0093] The method provided by the present invention completes the EEG decoding of multimodal signals of EEG and eye movement in video RSVP through a dual-branch multimodal Densnet network, formulates a relevant strategy set, and improves the adaptability and safety of the drone swarm through human perception of the environment. Specifically, the video collected by the drone is played to induce the observer's neural electrical activity related to a specific target, and the target classification is obtained by collecting the user's EEG signal and analyzing and processing it. After the relevant electrical activity of the specific target appears, the system captures this key information and automatically completes the dynamic planning of the drone swarm, and automatically implements the specific strategy plan for the target. When it is necessary to further complete a new execution plan, you only need to blink to enter the active control panel to complete the new strategy plan for the target. The dual-branch multimodal Densnet network improves the accuracy of target recognition, and the drone cluster control scheme under the video RSVP based on this enables the drone swarm to have a stronger task realization capability. The implementation of this scheme enables the drone cluster scheme to have a more practical application method.

[0094] In the present invention, EEG signals and eye movement signals are intercepted according to timestamps and preprocessed; labels are generated by the system and the accuracy of labels added by users is achieved based on a brief discrimination algorithm; the preprocessed EEG signals and eye movement signals are sent to a multimodal large-scale dual-branch Densnet convolutional network training model, and the model will extract features through EEG signals and eye movement signal features, and the EEG signals will provide guidance when extracting eye movement features; after feature extraction, deep feature interaction of EEG signals and eye movement signals will be completed in the feature interaction layer; after feature interaction, the discriminator is sent to give a target prediction probability, and the optimization of the model is completed from the initial label to the end; after the model is generated, it can be used online, and the user controls the drone group to perform task practice. The user only needs to observe and does not need to mark labels. When the specified target is found, the model will automatically execute the task for the corresponding target. When the recognition is not successful or the recognition is wrong, the user can blink and switch to the active control mode; the active control mode includes a steady-state visual evoked (SSVEP) control mode under active brain control, an imaginary movement (MI) control mode, and manual takeover.

[0095] Figure 7 FIG. 1 is a schematic diagram of the structure of a drone cluster control device provided by an embodiment of the present invention. Figure 7As shown, the drone cluster control device 700 includes: a sending module 701, which is used to combine the obtained drone real-time video with the rapid sequence visual presentation paradigm, and send an RSVP stimulation signal to the user through the EEG synchronization device; an acquisition module 702, which is used to synchronously acquire the user's target EEG signal and target eye movement signal in response to the RSVP stimulation signal; a determination module 703, which is used to determine the strong feature signal and the weak feature signal based on the signal dimension and information content of the target EEG signal and the target eye movement signal; a fusion module 704, which is used to input the strong feature signal and the weak feature signal into the dual-branch multi-mode The method comprises the following steps: performing feature fusion in a dynamic Densnet network to obtain target fusion features; a classification module 705, which is used to classify the target fusion features based on a preset classification algorithm to obtain classification results; and calculating the confidence corresponding to the classification results; a judgment module 706, which is used to judge whether a specific target appears in the real-time video of the drone based on the classification results and the confidence if the number of the classification results does not exceed the preset classification result number threshold; and an execution module 707, which is used to execute a target control instruction on the specific target based on a preset strategy set if the specific target appears in the real-time video of the drone.

[0096] In some embodiments, the acquisition module 702 is also used to acquire the target EEG signal; the target EEG signal carries a timestamp; the timestamp is sent to the EEG synchronization device; the timestamp is marked on the target EEG signal by the EEG synchronization device, and based on the timestamp, the target eye movement signal is obtained.

[0097] In some embodiments, the determination module 703 is also used to compare the signal dimensions and information amounts of the target EEG signal and the target eye movement signal to obtain a comparison result; if the comparison result is that the information dimension and information amount of the target EEG signal are greater than those of the target eye movement signal, the target EEG signal is determined as a strong feature signal and the target eye movement signal is determined as a weak feature signal.

[0098] In some embodiments, the judgment module 706 is also used to calculate the probability of occurrence of the specific target based on the classification results and the confidence level if the number of the classification results does not exceed the preset classification result number threshold; if the occurrence probability is greater than the preset probability threshold, determine that the specific target appears in the real-time video of the drone.

[0099] It should be noted that the description of the device of the embodiment of the present invention is similar to the description of the above method embodiment, and has similar beneficial effects as the same method embodiment, so it will not be repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present invention for understanding.

[0100] It should be noted that in the embodiment of the present invention, if the above-mentioned drone cluster control method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product, which is stored in a storage medium and includes several instructions for a terminal to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read Only Memory), a magnetic disk or an optical disk. In this way, the embodiment of the present invention is not limited to any specific combination of hardware and software.

[0101] Correspondingly, an embodiment of the present invention provides a drone cluster control device, Figure 8 is a schematic diagram of the composition structure of the drone cluster control device provided by an embodiment of the present invention, such as Figure 8 As shown, the drone cluster control device 800 includes at least: a processor 801 and a computer-readable storage medium 802 configured to store executable instructions, wherein the processor 801 generally controls the overall operation of the drone cluster control device. The computer-readable storage medium 802 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or processed by each module in the processor 801 and the drone cluster control device 800, which can be implemented through flash memory (FLASH) or random access memory (RAM, Random Access Memory).

[0102] An embodiment of the present invention provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the method provided by the embodiment of the present invention, for example, Figure 2 The method shown.

[0103] In some embodiments, the storage medium can be a computer-readable storage medium, for example, a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above memories.

[0104] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0105] As an example, executable instructions may, but need not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network.

[0106] The above description is only an embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement and improvement made within the spirit and scope of the present invention are included in the protection scope of the present invention.

[0107] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present invention, the size of the serial number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. The serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0108] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, an element defined by the statement "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed.

[0109] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for controlling a drone cluster, characterized in that: The method comprises: The real-time drone video is combined with the rapid sequence visual presentation paradigm, and RSVP stimulation signals are sent to users through EEG synchronization equipment; In response to the RSVP stimulation signal, synchronously collecting a user's target EEG signal and target eye movement signal; Determining a strong characteristic signal and a weak characteristic signal based on the signal dimensions and information amounts of the target EEG signal and the target eye movement signal; Inputting the strong feature signal and the weak feature signal into a dual-branch multimodal Densnet network for feature fusion to obtain a target fusion feature; Based on a preset classification algorithm, feature classification is performed on the target fusion features to obtain a classification result; and the confidence level corresponding to the classification result is calculated; If the number of the classification results does not exceed a preset classification result number threshold, judging whether a specific target appears in the real-time video of the drone based on the classification results and the confidence level; If the specific target appears in the real-time video of the drone, a target control instruction is executed on the specific target based on a preset strategy set.

2. The method according to claim 1, characterized in that Synchronously collect the user's target EEG signals and target eye movement signals, including: Collecting the target EEG signal; the target EEG signal carries a timestamp; Sending the timestamp to the EEG synchronization device; The timestamp is marked on the target EEG signal by the EEG synchronization device, and a target eye movement signal is obtained based on the timestamp.

3. The method according to claim 1, characterized in that The determining the strength of the characteristic signal based on the signal dimension and information amount of the target EEG signal and the target eye movement signal includes: Comparing the signal dimensions and information amounts of the target EEG signal and the target eye movement signal to obtain a comparison result; If the comparison result is that the information dimension and the amount of information contained in the target EEG signal are greater than those of the target eye movement signal, the target EEG signal is determined as a strong feature signal and the target eye movement signal is determined as a weak feature signal.

4. The method according to claim 1, characterized in that: If the number of the classification results does not exceed a preset classification result number threshold, judging whether a specific target appears in the real-time video of the drone based on the classification results and the confidence level includes: If the number of the classification results does not exceed a preset classification result number threshold, calculating the occurrence probability of the specific target based on the classification results and the confidence level; If the occurrence probability is greater than a preset probability threshold, it is determined that the specific target appears in the real-time video of the drone.

5. A drone cluster control device, characterized in that: The device comprises: The sending module is used to combine the obtained real-time video of the drone with the rapid sequence visual presentation paradigm and send RSVP stimulation signals to the user through the EEG synchronization device; A collection module, used for synchronously collecting a user's target EEG signal and target eye movement signal in response to the RSVP stimulation signal; A determination module, used to determine a strong characteristic signal and a weak characteristic signal based on the signal dimensions and information amounts of the target EEG signal and the target eye movement signal; A fusion module is used to input the strong feature signal and the weak feature signal into a dual-branch multimodal Densnet network for feature fusion to obtain a target fusion feature; A classification module is used to classify the target fusion features based on a preset classification algorithm to obtain a classification result; and calculate the confidence level corresponding to the classification result; A judgment module, configured to judge whether a specific target appears in the real-time video of the drone based on the classification result and the confidence level if the number of the classification results does not exceed a preset classification result number threshold; The execution module is used to execute a target control instruction on the specific target based on a preset strategy set if the specific target appears in the real-time video of the drone.

6. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the drone cluster control method described in any one of claims 1 to 5 when executing the executable instructions stored in the memory.

7. A computer-readable storage medium, wherein at least one executable instruction is stored in the computer-readable storage medium, and the at least one executable instruction is loaded and executed by a processor to implement the above-mentioned drone cluster control method.

Images