Brain-controlled driving method and system based on deep learning environment perception

Abstract

The invention discloses a brain-controlled driving method and system based on deep learning environment perception. The method comprises the following steps: constructing a brain-computer interface control normal form of sparse gradient evoked potential; obtaining target electroencephalogram signal data based on the brain-computer interface control normal form; processing the target electroencephalogram signal data to obtain a user control intention, and mapping the user control intention to a linear velocity-angular velocity space of the driving equipment; acquiring surrounding environment information of the driving equipment based on a deep learning network; and based on a dynamic environment adaptive control algorithm, integrating the user control intention and the surrounding environment information of the driving equipment, and performing man-machine cooperative control decision. According to the method, the problems of low brain control efficiency and difficulty in adaptability in a complex dynamic environment caused by non-coupling control of the linear speed and the angular speed of an existing traditional brain control driving system can be solved.

Description

Technical Field

[0001] The present invention relates to the fields of brain-computer interface technology, deep learning, automatic control theory and artificial intelligence technology, and in particular to a brain-controlled driving method and system based on deep learning environment perception. Background Art

[0002] With the rapid development of brain science and artificial intelligence technology, brain-computer interface (BCI) technology has become a revolutionary means of connecting the human brain with external devices.

[0003] Traditional BCI technology mainly focuses on decoding simple user intentions, such as motor imagery or visual induction. However, current BCI systems are verified in simple static environments, while actual scenarios face more complex dynamic environments. In order to adapt to the fast pace and high uncertainty of modern warfare, a brain control system is needed that can respond to environmental changes in real time and accurately execute user intentions. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a brain-controlled driving method and system based on deep learning environmental perception, which is used to solve the problems of low control efficiency and poor environmental adaptability caused by the non-coupling of linear velocity and angular velocity in existing traditional brain-controlled driving.

[0005] On the one hand, to achieve the above-mentioned purpose, the present invention provides a brain-controlled driving method based on deep learning environment perception, comprising:

[0006] Constructing a brain-computer interface control paradigm for sparse gradient evoked potentials;

[0007] Based on the brain-computer interface control paradigm, acquiring target EEG signal data;

[0008] Processing the target EEG signal data to obtain a user's control intention, and mapping the user's control intention to a linear velocity-angular velocity space of a driving device;

[0009] Acquire the surrounding environment information of the driving equipment based on the deep learning network;

[0010] Based on the dynamic environment adaptive control algorithm, the user's control intention and the surrounding environment information of the driving device are integrated to make human-machine collaborative control decisions.

[0011] Preferably, constructing the brain-computer interface control paradigm of the sparse gradient evoked potential comprises:

[0012] Under the condition of stimulation by the stimulus source, the user's electroencephalogram (EEG) signal is acquired in real time, a sparse gradient evoked potential (SGEP) brain-computer interface control paradigm is constructed based on the EEG signal, and the brain-computer interface control paradigm is presented;

[0013] The stimulus source is composed of a number of control blocks arranged horizontally, including flickering blocks and non-flickering blocks. The visual stimulus pattern displayed by the flickering blocks changes in the form of simple harmonic waves through the brightness of a preset frequency, and there is a phase difference of π / 2 between adjacent flickering blocks.

[0014] According to the number of control blocks, complete the linear velocity-angular velocity space mapping of each control block to the driving device.

[0015] Preferably, obtaining the user control intention includes:

[0016] The EEG data in the brain-computer interface control paradigm is preprocessed, and the preprocessed EEG data is decoded by a multi-frequency template canonical correlation analysis method to obtain the user control intention; wherein the preprocessing of the EEG data includes EEG data stream interception, baseline removal and filtering processing.

[0017] Preferably, the preprocessed EEG data is decoded by a multi-frequency template canonical correlation analysis method, including a pre-processing stage, a multi-frequency template construction stage and a correlation analysis stage;

[0018] Wherein, the pre-processing stage includes: dividing the pre-processed EEG data into several sub-bands through band-pass filtering;

[0019] The multi-frequency template construction stage includes the construction of individualized templates and the construction of universal templates;

[0020] The correlation analysis stage is used to perform canonical correlation analysis on the data of each sub-band with the individualized template and the universal template, and to perform weighted summation on the CCA correlation coefficients obtained for each sub-band to obtain the correlation coefficient ρ i and ρ c , and the correlation coefficient ρ i and ρ c Perform weighted summation to obtain the MT-CCA correlation coefficient of the Nth control block, and select the ID of the control block with the largest MT-CCA correlation coefficient as the EEG decoding result.

[0021] Preferably, the construction of the individualized template includes: collecting EEG response data of each individual when looking at each flickering block or non-flickering block, and using the EEG response data as training data, and superimposing the training data to obtain the individualized template:

[0022]

[0023] Where Y refi is an individualized template, X ri is the i-th training data of the user, N t is the total number of training data;

[0024] The construction of the universal template includes: constructing a template containing only the simple harmonic wave of the flicker block frequency and its harmonics according to the brain characteristic response intensity induced by the user to the flicker block obtained by the normalized canonical correlation analysis method. The specific calculation method is:

[0025] Define a simple harmonic template function:

[0026]

[0027] Assume that the frequency of the M flicker blocks in the sparse gradient evoked potential paradigm is {f m}, then the brain response generated by the user should be the weighted sum of the simple harmonic templates of M frequencies:

[0028]

[0029] Where Y refc is a generalized template. If m is an even number, φ is π / 2, and if m is an odd number, φ is 0;

[0030] Among them, A m is the brain characteristic response intensity induced by the user to the flickering block obtained by the normalized canonical correlation analysis method:

[0031]

[0032] In the correlation analysis stage, the data of each sub-band is subjected to canonical correlation analysis with the two templates respectively, and the CCA correlation coefficients obtained for each sub-band are weighted and summed. The weights are set according to the results obtained by optimizing the canonical correlation analysis algorithm based on the filter bank, and then the correlation coefficient ρ of the two templates is obtained. i and ρ c :

[0033]

[0034] Among them, f(j) is a weighting factor, or Represents the template EEG signal Y refik or Y refck The jth correlation with the original EEG signal X;

[0035] The two correlation coefficients are weighted and summed to obtain the MT-CCA correlation coefficient of the Nth target block:

[0036] ρ N =αρ iN +(1-α)ρ cN

[0037] Among them, α is the coefficient to be optimized;

[0038] Select the ID of the control block with the largest MT-CCA correlation coefficient as the output of the EEG decoding result for the instruction:

[0039] ρ=max{ρ1,ρ2,...,ρ N}.

[0040] Preferably, obtaining the surrounding environment information of the driving device includes:

[0041] Use a depth camera to obtain image information of the surrounding environment, input the image information into the environmental information recognition network, and output the position and size of the recognized object.

[0042] The position and size information of the identified object are integrated, and the object's movement speed is calculated by the difference method as the output result of environmental perception;

[0043] The environmental information recognition network includes a feature extraction layer, a region proposal subnetwork, a fast R-CNN subnetwork and an output layer.

[0044] Preferably, integrating the user's control intention with the surrounding environment information of the driving device includes:

[0045] Based on the position information and speed information of the surrounding obstacles, the dynamic intervention speed is calculated, and the human-machine collaborative control decision is made according to the dynamic intervention speed and the user's control intention;

[0046] Among them, the method for calculating the dynamic intervention speed is:

[0047] With the driver as the center, draw a circle with the maximum distance that can effectively detect surrounding obstacles as the radius, and determine this circular area as the safety circle;

[0048] Determine the speed of the obstacle relative to the driver, take the obstacle as the starting point, draw two tangents to the safety circle, and determine the area enclosed by the tangents and the safety circle as the danger zone;

[0049] When the obstacle is located in the danger zone, determining whether there is a risk of collision between the obstacle and the driver, and if there is a risk of collision, intervening in the movement of the driver;

[0050] The method for determining whether there is a risk of collision between an obstacle and a driver is as follows:

[0051]

[0052] In the formula, angle() is the function for calculating the angle between two vectors, v ow is the speed of the obstacle relative to the driver, x ow is the position of the obstacle relative to the driver, r s is the radius of the safety circle;

[0053] The method for performing motion intervention on the driver is:

[0054] Apply a distance v to the obstacle ow The velocity v of the nearest tangent line aw , first calculate the obstacle avoidance speed through geometric relationship:

[0055]

[0056] Then, the direction of the obstacle avoidance speed is achieved by rotating the vector through a two-dimensional rotation matrix:

[0057]

[0058] Among them, sign() is the sign function, a is the rotation matrix;

[0059]

[0060] In the actual control process, v avoid =-v aw

[0061] v avoid is the dynamic intervention speed.

[0062] Preferably, making the human-machine collaborative control decision includes:

[0063] When the size of the obstacle relative to the driver is smaller than the detection circle or the speed of the obstacle relative to the driver is greater than the preset threshold, it is considered that the danger level is high and the actual output control command only includes the dynamic intervention speed:

[0064] v out =v avoid

[0065] In the formula, v out is the actual output control instruction;

[0066] Otherwise, the output control command is the sum of the dynamic intervention speed and the brain control command:

[0067] v out =v avoid +v brain

[0068] In the formula, v brainIt is a mind control command.

[0069] On the other hand, to achieve the above-mentioned purpose, the present invention also provides a brain-controlled driving system based on deep learning environment perception, which is applied to a brain-controlled driving method based on deep learning environment perception, comprising: a signal acquisition module, a visual evoked module, an EEG decoding module, a dynamic environment collaborative control module and a driver module;

[0070] The signal acquisition module includes an EEG cap and an amplifier, which are used to acquire the user's EEG signals in real time and amplify them through the amplifier;

[0071] The visual evoked module is used to generate a brain-computer interface control paradigm of sparse gradient evoked potentials and present it to the user;

[0072] The EEG decoding module is used to intercept, filter, and reduce noise of the acquired EEG signals, and decode the EEG data into the user's control intention instructions;

[0073] The dynamic environment collaborative control module is used to obtain surrounding environment information, and use the dynamic environment collaborative control method to intervene, output collaborative control instructions, and complete human-machine collaborative control decisions;

[0074] The driver module is used to respond and act on the collaborative control instructions;

[0075] Wherein, the dynamic environment collaborative control module includes:

[0076] Building an environmental information acquisition submodule based on deep learning: used to obtain the position and speed information of all objects within the detection range in real time;

[0077] Dynamic intervention speed calculation submodule: used to calculate the dynamic intervention speed according to the surrounding environment information of the driving device;

[0078] Human-machine collaborative decision-making submodule: used to complete the human-machine collaborative control strategy and send collaborative control instructions to the driver module.

[0079] Compared with the prior art, the present invention has the following advantages and technical effects:

[0080] The present invention efficiently maps the user's control intention to the speed and angular velocity space of the driving device, and combines the EEG decoding algorithm of multi-frequency template typical correlation analysis to improve the decoding accuracy and speed of the user's intention. In addition, a submodule for environmental information acquisition based on deep learning and a dynamic environment adaptive control algorithm are constructed, so that the brain-controlled driving system can maintain a high degree of adaptability and safety in complex and changeable battlefield environments.

[0081] The present invention can solve the problems of low brain control efficiency caused by the uncoupled control of linear velocity and angular velocity in existing traditional brain-controlled driving systems, as well as the difficulty in adapting to complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0083] Figure 1 This is a flow chart of a brain-controlled driving method based on deep learning environment perception according to an embodiment of the present invention;

[0084] Figure 2 A schematic diagram of a sparse gradient paradigm according to an embodiment of the present invention;

[0085] Figure 3 A schematic diagram of brain response characteristics of a user under a sparse gradient paradigm according to an embodiment of the present invention;

[0086] Figure 4 Schematic diagram of experimental tasks of an online brain-controlled driving experiment in a dynamic environment according to an embodiment of the present invention;

[0087] Figure 5 It is a schematic diagram of recording the obstacle avoidance process of an online brain-controlled driving experiment in a dynamic environment according to an embodiment of the present invention;

[0088] Figure 6 This is a schematic diagram of the structure of a brain-controlled driving system based on deep learning environment perception according to an embodiment of the present invention. DETAILED DESCRIPTION

[0089] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0090] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0091] The present invention proposes a brain-controlled driving method based on deep learning environment perception, such as Figure 1 ,include:

[0092] Constructing a brain-computer interface control paradigm for sparse gradient evoked potentials;

[0093] Based on the brain-computer interface control paradigm, acquiring target EEG signal data;

[0094] Processing the target EEG signal data to obtain a user's control intention, and mapping the user's control intention to a linear velocity-angular velocity space of a driving device;

[0095] Acquire the surrounding environment information of the driving equipment based on the deep learning network;

[0096] Based on a dynamic environment adaptive control algorithm, the user's control intention and the surrounding environment information of the driving device are integrated to make human-machine collaborative control decisions.

[0097] This embodiment uses an innovative sparse gradient brain-computer interface paradigm to efficiently map the user's control intention to the speed and angular velocity space of the driving device, and at the same time combines the EEG decoding algorithm of multi-frequency template canonical correlation analysis to improve the decoding accuracy and speed of the user's intention.

[0098] Furthermore, a sparse gradient evoked potential brain-computer interface control paradigm is constructed, including:

[0099] Under the condition of stimulation by the stimulus source, the user's electroencephalogram (EEG) signal is acquired in real time, a sparse gradient evoked potential (SGEP) brain-computer interface control paradigm is constructed based on the EEG signal, and the brain-computer interface control paradigm is presented;

[0100] The stimulus source is composed of a series of control blocks arranged horizontally, including flickering blocks and non-flickering blocks. The visual stimulus pattern displayed by the flickering blocks changes in the form of simple harmonic waves through the brightness of a preset frequency, and there is a phase difference of π / 2 between adjacent flickering blocks.

[0101] According to the number of control blocks, complete the linear velocity-angular velocity space mapping of each control block to the driving device.

[0102] Specifically, let the number of control blocks be N, and from left to right, the linear velocity v of the i-th control block is i for:

[0103] If N is even:

[0104]

[0105] If N is odd:

[0106]

[0107] Angular velocity of the i-th control block (left turn is positive, right turn is negative) ω i for:

[0108] If N is even:

[0109]

[0110] If N is odd:

[0111]

[0112] Among them, v max and ω max are the maximum linear velocity and angular velocity of the driving device respectively.

[0113] In this embodiment, the stimulus source includes: seven control blocks arranged horizontally, numbered 1 to 7 from left to right, wherein the control blocks No. 1, 4, and 7 are flashing blocks, and the rest are non-flashing blocks. Figure 2 As shown, the three flickering blocks are visual stimulation patterns whose brightness changes in the form of simple harmonics at three frequencies of 11 Hz, 13 Hz and 15 Hz, respectively, and there is a phase difference of π / 2 between adjacent flickering blocks. This phase difference setting can effectively reduce the mutual influence between the EEG features induced by each flickering block.

[0114] Furthermore, obtaining user control intention information includes:

[0115] The EEG data in the brain-computer interface control paradigm is preprocessed, and the preprocessed EEG data is decoded by a multi-frequency template canonical correlation analysis method to obtain user control intention information; wherein, the preprocessing of the EEG data includes EEG data stream interception, baseline removal and filtering processing.

[0116] The preprocessed EEG data were decoded by the multi-frequency template canonical correlation analysis (MT-CCA) method, including the pre-processing stage, the multi-frequency template construction stage and the correlation analysis stage;

[0117] Specifically, the EEG data decoding adopts the multi-frequency template canonical correlation analysis (MT-CCA) algorithm. First, the EEG data of a subject is calculated using the normalized canonical correlation analysis (NCC) method, and the following is obtained: Figure 3 Distribution of brain response features shown.

[0118] This embodiment designs a normalized canonical correlation analysis (NCC) method, and its calculation method is:

[0119] This method can normalize the response intensity of the human brain to the three flicking blocks. Specifically:

[0120] First define:

[0121]

[0122] Among them, CCA() performs canonical correlation analysis on two matrices and returns the first correlation coefficient. X is EEG data, Y f1 ,Y f2 ,Y f3They are templates of flashing frequencies of three flashing blocks respectively;

[0123] Next, define the normalized correlation coefficient NCC i for:

[0124]

[0125] Here, i traverses each element in a, min(a) is the minimum value in list a, and max(a) is the maximum value in list a. In this way, each element NCC in the NCC list i are the normalized values ​​corresponding to the i-th element in list a. This normalization process is often used to scale data to the interval [0, 1] for easy comparison and further processing.

[0126] Furthermore, the pre-processing stage includes: dividing the pre-processed EEG data into several sub-bands by band-pass filtering;

[0127] Specifically, in order to improve the decoding accuracy of EEG data, a section of EEG data X is divided into n sub-bands SB1, SB2...SBn by bandpass filtering with reference to a filter bank based canonical correlation analysis (FBCCA) method.

[0128] The multi-frequency template construction stage includes the construction of individualized templates and the construction of universal templates;

[0129] Specifically, the construction of the individualized template includes: for each individual, collecting the EEG response data of the individual when he looks at each flickering block / non-flickering block as a training data set in advance, and superimposing these training data to obtain the individualized template:

[0130]

[0131] Where Y refi is an individualized template, X ri is the i-th training data of the user, N t is the total number of training data;

[0132] The construction of the universal template includes: constructing a template containing only the simple harmonic wave of the flicker block frequency and its harmonics according to the brain characteristic response intensity induced by the user to the flicker block obtained by the normalized canonical correlation analysis (NCC) method. The specific calculation method is:

[0133] Define a simple harmonic template function:

[0134]

[0135] Assume that the frequency of the M flicker blocks in the sparse gradient evoked potential paradigm is {f m};

[0136] Then the brain response generated by the user should be the weighted sum of the simple harmonic templates of these M frequencies:

[0137]

[0138] If m is even, then is π / 2. If m is an odd number, then is 0;

[0139] A m The brain characteristic response intensity induced by the user to the flickering block is obtained according to the normalized canonical correlation analysis (NCC) method:

[0140]

[0141] In the correlation analysis stage, the data of each sub-band is subjected to canonical correlation analysis (CCA) with the two templates respectively, and the CCA correlation coefficients obtained for each sub-band are weighted and summed, and the weights are set according to the results obtained by optimizing the filter bank-based canonical correlation analysis (FB-CCA) algorithm. In this way, the correlation coefficient ρ of the two templates can be obtained. i and ρ c :

[0142]

[0143]

[0144] Among them, f(j) is a weighting factor, or Represents the template EEG signal Y refik or Y refck The jth correlation between the original EEG signal X and the original EEG signal X is expressed as:

[0145] f(j)=j -a +b

[0146] Where a and b are constants, and their values ​​are the same as those set in FB-CCA.

[0147] Then the two correlation coefficients are weighted and summed to obtain the MT-CCA correlation coefficient of the Nth target block:

[0148] ρ N =αρ iN +(1-α)ρ cN

[0149] Among them, α is the coefficient to be optimized, which can be optimized by using grid search and gradient descent methods to achieve the highest classification accuracy.

[0150] Finally, select the ID of the control block with the largest MT-CCA correlation coefficient as the output of the EEG decoding result for the instruction:

[0151] ρ=max{ρ1,ρ2,...,ρ N}.

[0152] Furthermore, the surrounding environment information of the driving device is obtained, including:

[0153] Use a depth camera to obtain image information of the surrounding environment, input the image information into the environmental information recognition network, and output the position and size of the recognized object.

[0154] The model can identify the positions of objects around the driving device in real time; finally, the position and size information of all identified objects are integrated, and the movement speed of each object is calculated through the difference method as the final environmental perception result output.

[0155] The environmental information recognition network includes a feature extraction layer, a region proposal subnetwork, a fast R-CNN subnetwork and an output layer. The model is trained using an SGD optimizer, and the learning rate is adjusted according to the training progress.

[0156] Specifically, the feature extraction layer uses multiple convolutional layers to extract the features of the image. Each convolutional layer is followed by an activation function and a pooling layer to reduce the dimension of the feature map. It includes convolution calculation and batch normalization.

[0157] Convolution is the use of multiple convolutional layers to extract features of an image. Each convolutional layer is usually followed by an activation function (such as ReLU) and a pooling layer (such as max pooling) to reduce the dimension of the feature map;

[0158] Adding batch normalization between convolutional layers and activation layers helps speed up the training process and improve the generalization ability of the model.

[0159] The region proposal subnetwork is used to quickly and efficiently generate candidate regions and provide accurate location information for subsequent object detection. It includes a full convolutional network, anchor box calculation, and region proposal network output.

[0160] The fully convolutional network (FCN) uses a fully convolutional network to predict whether each sliding window contains an object and the category of the object. The anchor box calculation is to define multiple anchor boxes of different shapes and scales for each sliding window to predict the location of the object. The output of the region proposal network is to output two categories (object and background) and four bounding box regression parameters for each anchor box.

[0161] The Fast R-CNN subnetwork is used to locate and identify the categories and positions of multiple objects in an image. It contains a candidate region pooling layer and a fully connected layer: the candidate region pooling layer is used to convert the candidate regions obtained from the region building network into a fixed-size feature map through the candidate region pooling layer; the fully connected layer is used to further extract features from the output of the candidate region pooling.

[0162] Finally, the position, size and other information of all identified objects are integrated, and the movement speed of each object is calculated by the difference method as the final environmental perception result output.

[0163] Further, integrating the user's control intention with the surrounding environment information of the driving device includes:

[0164] Based on the position information and speed information of the surrounding obstacles, the dynamic intervention speed is calculated, and the human-machine collaborative control decision is made according to the dynamic intervention speed and the user's control intention;

[0165] Among them, the method for calculating the dynamic intervention speed is:

[0166] With the driver as the center, draw a circle with the maximum distance that can effectively detect surrounding obstacles as the radius, and determine this circular area as the safety circle;

[0167] Determine the speed of the obstacle relative to the driver, take the obstacle as the starting point, draw two tangents to the safety circle, and determine the area enclosed by the tangents and the safety circle as the danger zone;

[0168] Specifically, the detection circle is a circle with the driver as the center and the maximum distance at which the driver can effectively detect surrounding obstacles as the radius. It describes the area where the driver can detect environmental information. The safety circle is determined by the geometric shape of the driver itself and is used to describe the area that ensures the driver's safety and will not collide. Its theoretical value is the circular area swept by the driver's rotation in place, and a small safety margin is added to the radius of the circle on this basis. Use a two-dimensional vector x ow describes the position of the obstacle relative to the driver. The purpose of the DEC controller is to apply an additional velocity v avoid Guarantee x ow Greater than the radius r of the safety circle s .

[0169] Assume the speed of the driver is v w , the speed of the obstacle is v obs . Then the speed of the obstacle relative to the driver is:

[0170] v ow =v obs -v w

[0171] Taking the obstacle as the starting point, two tangent lines are drawn to the safety circle. The area enclosed by the two tangent lines and the safety circle is called the danger zone (DR). ow When in the DR, if the driver and the obstacle maintain their original motion state, the obstacle will definitely enter the driver's safety circle, resulting in the risk of collision.

[0172] When an obstacle is in the danger zone, determine whether there is a risk of collision between the obstacle and the driver. If there is a risk of collision, intervene in the driver's motion.

[0173] The method for determining whether there is a risk of collision between an obstacle and a driver is as follows:

[0174]

[0175] In the formula, angle() is the function for calculating the angle between two vectors, v ow is the speed of the obstacle relative to the driver, x ow is the position of the obstacle relative to the driver, r s is the radius of the safety circle;

[0176] Furthermore, the method of performing motion intervention on the driver is:

[0177] Apply a distance v to the obstacle ow The velocity v of the nearest tangent line aw , first calculate the obstacle avoidance speed through geometric relationship:

[0178]

[0179] Then, the direction of the obstacle avoidance speed is achieved by rotating the vector through a two-dimensional rotation matrix:

[0180]

[0181] Among them, sign() is the sign function, a is the rotation matrix;

[0182]

[0183] In the actual control process, v avoid =-v aw

[0184] v avoid is the dynamic intervention speed.

[0185] Furthermore, human-machine collaborative control decision-making is carried out, including:

[0186] A decision circle is set with the driver as the center, whose radius is larger than the safety circle and smaller than the detection circle.ow When the size is smaller than the detection circle or the speed of the obstacle relative to the driver is greater than a certain threshold (usually the threshold is the maximum speed of the driver), it is considered that the danger level is high and the actual control command v out Include only dynamic intervention speeds:

[0187] v out =v avoid

[0188] Otherwise, the output control command is the sum of the dynamic intervention speed and the brain control command:

[0189] v out =v avoid +v brain

[0190] This example verifies the proposed method in a simulation environment developed based on Unity using brain-controlled vehicle driving control as the scenario. An online control experiment in a dynamic environment is designed. The experimental tasks are as follows: Figure 4 There are two obstacles moving in a straight line at a uniform speed, moving back and forth left and right. The subject is required to control the vehicle from the starting point to avoid the moving obstacles and reach the target area.

[0191] The experimental results show that when the obstacle speed is low (lower than the maximum speed of the vehicle), the number of mission failures due to collision is 0, which shows the effectiveness of the proposed method.

[0192] Figure 5 The positions at three time points demonstrate the vehicle's automatic obstacle avoidance behavior when using a dynamic environment cooperative control method.

[0193] The present invention also proposes a brain-controlled driving system based on deep learning environment perception, such as Figure 6 ,include:

[0194] Signal acquisition module, visual evoked module, EEG decoding module, dynamic environment collaborative control module and driver module;

[0195] The signal acquisition module includes an EEG cap and an amplifier, which are used to obtain the user's EEG signals in real time and amplify them through the amplifier;

[0196] The visual evoked module is used to generate a BCI control paradigm of sparse gradient evoked potentials and present them to the user;

[0197] The EEG decoding module is used to intercept, filter, and reduce noise of the acquired EEG signals, and decode the EEG data into the user's control intention instructions;

[0198] The dynamic environment collaborative control module is used to obtain the surrounding environment information, intervene using the dynamic environment collaborative control method, output collaborative control instructions, and complete the human-machine collaborative control decision;

[0199] The driver module is used to respond and act on the cooperative control instructions;

[0200] Among them, the dynamic environment collaborative control module includes:

[0201] Building an environmental information acquisition submodule based on deep learning: used to obtain the position and speed information of all objects within the detection range in real time;

[0202] Dynamic intervention speed calculation submodule: used to calculate the dynamic intervention speed according to the surrounding environment information of the driving device;

[0203] Human-machine collaborative decision-making submodule: used to complete the human-machine collaborative control strategy and send collaborative control instructions to the driver module.

[0204] Specifically, in this embodiment, the 1st, 4th, and 7th control blocks in the sparse gradient brain-computer interface paradigm are flickering blocks, and the rest are non-flickering blocks. Figure 2 As shown, the three flickering blocks are visual stimulation patterns whose brightness changes in the form of simple harmonics at three frequencies of 11 Hz, 13 Hz and 15 Hz, respectively, and there is a phase difference of π / 2 between adjacent flickering blocks.

[0205] The EEG decoding module is used to intercept, filter and reduce noise of the acquired EEG signals, and use the MT-CCA EEG decoding algorithm to decode the EEG data into the user's control intention instructions;

[0206] The dynamic environment collaborative control module is used to obtain the surrounding environment information, and use the dynamic environment collaborative control method to intervene in dangerous situations and complete the human-machine collaborative control decision;

[0207] Among them, the dynamic environment collaborative control module includes:

[0208] Building an environmental information acquisition submodule based on deep learning: used to obtain the position and speed information of all objects within the detection range in real time;

[0209] Dynamic intervention speed calculation submodule: used to calculate the dynamic intervention speed according to the surrounding environment information of the driving device;

[0210] Human-machine collaborative decision-making submodule: used to complete the human-machine collaborative control strategy and send collaborative control instructions to the driver module.

[0211] In this embodiment, the driver module uses the vehicle as the main body of the driving device, and can respond and act on the cooperative control instructions output from the dynamic environment cooperative control module.

[0212] This embodiment constructs an environmental information acquisition submodule and a dynamic environment adaptive control algorithm based on deep learning, so that the brain-controlled driving system can maintain a high degree of adaptability and safety in a complex and changeable battlefield environment.

[0213] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A brain-controlled driving method based on deep learning environment perception, characterized in that: include: Constructing a brain-computer interface control paradigm for sparse gradient evoked potentials; Based on the brain-computer interface control paradigm, acquiring target EEG signal data; Processing the target EEG signal data to obtain a user's control intention, and mapping the user's control intention to a linear velocity-angular velocity space of a driving device; Acquire the surrounding environment information of the driving equipment based on the deep learning network; Based on the dynamic environment adaptive control algorithm, the user's control intention and the surrounding environment information of the driving device are integrated to make human-machine collaborative control decisions.

2. The brain-controlled driving method based on deep learning environment perception according to claim 1 is characterized in that: Constructing the brain-computer interface control paradigm of the sparse gradient evoked potential, including: Under the condition of stimulation by the stimulus source, the user's electroencephalogram (EEG) signal is acquired in real time, a sparse gradient evoked potential (SGEP) brain-computer interface control paradigm is constructed based on the EEG signal, and the brain-computer interface control paradigm is presented; The stimulus source is composed of a number of control blocks arranged horizontally, including flickering blocks and non-flickering blocks. The visual stimulus pattern displayed by the flickering blocks changes in the form of simple harmonic waves through the brightness of a preset frequency, and there is a phase difference of π / 2 between adjacent flickering blocks. According to the number of control blocks, complete the linear velocity-angular velocity space mapping of each control block to the driving device.

3. The brain-controlled driving method based on deep learning environment perception according to claim 1 is characterized in that: Obtaining the user control intention includes: The EEG data in the brain-computer interface control paradigm is preprocessed, and the preprocessed EEG data is decoded by a multi-frequency template canonical correlation analysis method to obtain the user control intention; wherein the preprocessing of the EEG data includes EEG data stream interception, baseline removal and filtering processing.

4. The brain-controlled driving method based on deep learning environment perception according to claim 3 is characterized in that: Decoding the preprocessed EEG data by a multi-frequency template canonical correlation analysis method, including a pre-processing stage, a multi-frequency template construction stage and a correlation analysis stage; Wherein, the pre-processing stage includes: dividing the pre-processed EEG data into several sub-bands through band-pass filtering; The multi-frequency template construction stage includes the construction of individualized templates and the construction of universal templates; The correlation analysis stage is used to perform canonical correlation analysis on the data of each sub-band with the individualized template and the universal template, and to perform weighted summation on the CCA correlation coefficients obtained for each sub-band to obtain the correlation coefficient ρ i and ρ c , and the correlation coefficient ρ i and ρ c Perform weighted summation to obtain the MT-CCA correlation coefficient of the Nth control block, and select the ID of the control block with the largest MT-CCA correlation coefficient as the EEG decoding result.

5. The brain-controlled driving method based on deep learning environment perception according to claim 4 is characterized in that: The construction of the individualized template includes: collecting the EEG response data of each individual when looking at each flickering block or non-flickering block, and using the EEG response data as training data, and superimposing the training data to obtain the individualized template: Where Y refi is an individualized template, X ri is the i-th training data of the user, N t is the total number of training data; The construction of the universal template includes: constructing a template containing only the simple harmonic wave of the flicker block frequency and its harmonics according to the brain characteristic response intensity induced by the user to the flicker block obtained by the normalized canonical correlation analysis method. The specific calculation method is: Define a simple harmonic template function: Assume that the frequency of the M flicker blocks in the sparse gradient evoked potential paradigm is {f m }, then the brain response generated by the user should be the weighted sum of the simple harmonic templates of M frequencies: Where Y refc is a generalized template. If m is an even number, then is π / 2, if m is an odd number, then is 0; Among them, A m is the brain characteristic response intensity induced by the user to the flickering block obtained by the normalized canonical correlation analysis method: In the correlation analysis stage, the data of each sub-band is subjected to canonical correlation analysis with the two templates respectively, and the CCA correlation coefficients obtained for each sub-band are weighted and summed. The weights are set according to the results obtained by optimizing the canonical correlation analysis algorithm based on the filter bank, and then the correlation coefficient ρ of the two templates is obtained. i and ρ c : Among them, f(j) is a weighting factor, or Represents the template EEG signal Y refik or Y refck The jth correlation with the original EEG signal X; The two correlation coefficients are weighted and summed to obtain the MT-CCA correlation coefficient of the Nth target block: r N =ar iN +(1-a)p cN Among them, α is the coefficient to be optimized; Select the ID of the control block with the largest MT-CCA correlation coefficient as the output of the EEG decoding result for the instruction: ρ=max{ρ1,ρ2,...,ρ N }。 6. The brain-controlled driving method based on deep learning environment perception according to claim 1 is characterized in that: Acquiring the surrounding environment information of the driving device, including: Use a depth camera to obtain image information of the surrounding environment, input the image information into the environmental information recognition network, and output the position and size of the recognized object. The position and size information of the identified object are integrated, and the object's movement speed is calculated by the difference method as the output result of environmental perception; The environmental information recognition network includes a feature extraction layer, a region proposal subnetwork, a fast R-CNN subnetwork and an output layer.

7. The brain-controlled driving method based on deep learning environment perception according to claim 1 is characterized in that: Integrating the user control intention with the surrounding environment information of the driving device, including: Calculate the dynamic intervention speed based on the position information and speed information of the surrounding obstacles, and make a human-machine collaborative control decision based on the dynamic intervention speed and the user's control intention; Among them, the method for calculating the dynamic intervention speed is: With the driver as the center, draw a circle with the maximum distance that can effectively detect surrounding obstacles as the radius, and determine this circular area as the safety circle; Determine the speed of the obstacle relative to the driver, take the obstacle as the starting point, draw two tangents to the safety circle, and determine the area enclosed by the tangents and the safety circle as the danger zone; When the obstacle is located in the danger zone, determining whether there is a risk of collision between the obstacle and the driver, and if there is a risk of collision, intervening in the movement of the driver; The method for determining whether there is a risk of collision between an obstacle and a driver is as follows: In the formula, angle() is the function for calculating the angle between two vectors, v ow is the speed of the obstacle relative to the driver, x ow is the position of the obstacle relative to the driver, r s is the radius of the safety circle; The method for performing motion intervention on the driver is: Apply a distance v to the obstacle ow The velocity v of the nearest tangent line aw , first calculate the obstacle avoidance speed through geometric relationship: Then, the direction of the obstacle avoidance speed is achieved by rotating the vector through a two-dimensional rotation matrix: Among them, sign() is the sign function, a is the rotation matrix; In the actual control process, v avoid =-v aw v avoid is the dynamic intervention speed.

8. The brain-controlled driving method based on deep learning environment perception according to claim 7 is characterized in that: Making the human-machine collaborative control decision includes: When the size of the obstacle relative to the driver is smaller than the detection circle or the speed of the obstacle relative to the driver is greater than the preset threshold, it is considered that the danger level is high and the actual output control command only includes the dynamic intervention speed: v out =v avoid In the formula, v out is the actual output control instruction; Otherwise, the output control command is the sum of the dynamic intervention speed and the brain control command: v out =v avoid +v brain In the formula, v brain It is a mind control command.

9. A brain-controlled driving system based on deep learning environment perception, applied to the brain-controlled driving method based on deep learning environment perception according to any one of claims 1 to 8, characterized in that: It includes signal acquisition module, visual evoked module, EEG decoding module, dynamic environment cooperative control module and driver module; The signal acquisition module includes an EEG cap and an amplifier, which are used to acquire the user's EEG signals in real time and amplify them through the amplifier; The visual evoked module is used to generate a brain-computer interface control paradigm of sparse gradient evoked potentials and present it to the user; The EEG decoding module is used to intercept, filter, and reduce noise of the acquired EEG signals, and decode the EEG data into the user's control intention instructions; The dynamic environment collaborative control module is used to obtain surrounding environment information, and use the dynamic environment collaborative control method to intervene, output collaborative control instructions, and complete human-machine collaborative control decisions; The driver module is used to respond and act on the collaborative control instructions; Wherein, the dynamic environment collaborative control module includes: Building an environmental information acquisition submodule based on deep learning: used to obtain the position and speed information of all objects within the detection range in real time; Dynamic intervention speed calculation submodule: used to calculate the dynamic intervention speed according to the surrounding environment information of the driving device; Human-machine collaborative decision-making submodule: used to complete the human-machine collaborative control strategy and send collaborative control instructions to the driver module.

Images