Brain-controlled driving method, device and equipment based on digital twinning

Abstract

The invention discloses a brain-controlled driving method, device and equipment based on digital twinning, and relates to the technical field of information processing and control. The brain-controlled driving method based on digital twinning comprises the following steps: acquiring an electroencephalogram signal, and position information and motion state information of controlled equipment; according to the position information and the motion state information, controlling the position and the motion state of the digital twin of the controlled equipment to be consistent with those of the controlled equipment; predicting the motion trail of the digital twin of the controlled device by combining the electroencephalogram signal with historical data of the controlled device to obtain a predicted motion trail; and when it is detected that a potential risk exists in the predicted motion track, the driving state of the controlled equipment is actively intervened. According to the embodiment of the invention, the control capability of the mobile equipment based on the brain-computer interface can be enhanced by executing the steps, so that the safety and reliability of a brain control technology are improved.

Description

Technical Field

[0001] The present application relates to the field of information processing and control technology, and in particular to a brain-controlled driving method, device and equipment based on digital twins. Background Art

[0002] Brain control technology, also known as brain-computer interface (BCI), is a technology that directly connects the human brain and external devices, allowing users to control external devices through thinking without traditional muscle movements.

[0003] Brain control technology has been widely used and developed in many fields, but it still faces limitations in hardware and algorithm performance. This is mainly reflected in the fact that in order to achieve a high success rate of brain control tasks with seamless interaction between users and controlled devices, it is necessary to decode and classify the acquired real-time, complex, and variable EEG signals with high precision. However, existing pattern recognition algorithms cannot ensure the safety and reliability of the widespread application of brain control technology. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a brain-controlled driving method, device and equipment based on digital twins, aiming to enhance the control capability of mobile devices based on brain-computer interfaces, thereby improving the safety and reliability of brain control technology.

[0005] To achieve the above objectives, one aspect of an embodiment of the present application proposes a brain-controlled driving method based on digital twins, comprising the following steps:

[0006] Acquiring EEG signals, location information of a controlled device, and motion state information of the controlled device;

[0007] Control the position and motion state of the digital twin of the controlled device to be consistent with the controlled device according to the position information and the motion state information;

[0008] Predicting the motion trajectory of the digital twin of the controlled device by combining the EEG signal with historical data of the controlled device to obtain a predicted motion trajectory;

[0009] When it is detected that there is a potential risk in the predicted motion trajectory, actively intervene in the driving state of the controlled device.

[0010] In some embodiments, before controlling the position and motion state of the digital twin of the controlled device to be consistent with the controlled device according to the position information and the motion state information, the method further includes:

[0011] Acquire visual data;

[0012] A digital twin of the controlled device and digital twins of obstacles in the environment are generated based on the visual data.

[0013] In some embodiments, predicting the motion trajectory of the digital twin of the controlled device by combining the EEG signal with historical data of the controlled device to obtain the predicted motion trajectory includes:

[0014] determining a control command based on the EEG signal;

[0015] Inputting the control command and the historical data into a trajectory prediction model to obtain the predicted motion trajectory;

[0016] Wherein, the trajectory prediction model is constructed by artificial intelligence methods.

[0017] In some embodiments, when the predicted motion trajectory is detected to have a potential risk, actively intervening in the driving state of the controlled device includes:

[0018] Determining a distance between the digital twin of the controlled device and the digital twin of the obstacle according to the predicted motion trajectory;

[0019] When the distance is less than a preset distance, actively intervene in the driving state of the controlled device.

[0020] In some embodiments, when the distance is less than a preset distance, actively intervening in the driving state of the controlled device includes:

[0021] When the distance is less than the preset distance, sending a request message, wherein the request message is used to request the driver to send the EEG signal again to reduce the potential risk;

[0022] When the potential risk is not reduced, a deceleration command or a braking command is sent to the controlled device.

[0023] In some embodiments, the EEG signals include a first group of EEG signals and a second group of EEG signals, and determining the control command based on the EEG signals includes:

[0024] Inputting the first group of EEG signals into a first model to obtain a control command of moving forward or backward, wherein the first group of EEG signals are EEG signals generated by a motor imagery mode, and the first model is a model for extracting asymmetric features of EEG signals in left and right brain regions;

[0025] The second group of EEG signals are input into a second model to obtain a control command of turning left or right, wherein the second group of EEG signals are EEG signals generated by an eye blinking pattern, and the second model is a model for extracting pulse features of EEG signals.

[0026] In some embodiments, after predicting the motion trajectory of the digital twin of the controlled device by combining the EEG signal with the historical data of the controlled device and obtaining the predicted motion trajectory, the method further includes:

[0027] When no potential risk is detected in the predicted motion trajectory, the controlled device is controlled to travel based on the control command generated by the electroencephalogram signal.

[0028] To achieve the above objectives, another aspect of the embodiments of the present application provides a brain-controlled driving device based on digital twins, and the brain-controlled driving device based on digital twins includes:

[0029] An acquisition module is used to acquire EEG signals and the position information and motion state information of the controlled device;

[0030] A control module, configured to control the position and motion state of the digital twin of the controlled device to be consistent with the controlled device according to the position information and the motion state information;

[0031] A prediction module, used to predict the motion trajectory of the digital twin of the controlled device by combining the EEG signal with the historical data of the controlled device to obtain a predicted motion trajectory;

[0032] The active intervention module is used to actively intervene in the driving state of the controlled device when it is detected that the predicted motion trajectory has potential risks.

[0033] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned brain-controlled driving method based on digital twins when executing the computer program.

[0034] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a mobile device, which includes a power drive device, and the operation of the mobile device is controlled by a brain-controlled driving method based on digital twins in the above-mentioned embodiment, a brain-controlled driving device based on digital twins in the above-mentioned embodiment, or an electronic device in the above-mentioned embodiment.

[0035] The embodiments of the present application include at least the following beneficial effects:

[0036] The present application provides a brain-controlled driving method, device and equipment based on digital twins. In the embodiments of the present application, first, EEG signals, position information and motion state information of the controlled device are obtained; the position and motion state of the digital twin of the controlled device are controlled to be consistent with the controlled device according to the position information and motion state information; the motion trajectory of the digital twin of the controlled device is predicted by combining the EEG signals with the historical data of the controlled device to obtain the predicted motion trajectory; when the predicted motion trajectory is detected to have potential risks, the driving state of the controlled device is actively intervened. The embodiments of the present application simulate the running state of the controlled device in the virtual space by controlling the position and motion state of the digital twin of the controlled device to be consistent with the controlled device, and predict the motion trajectory of the digital twin according to the brain control signals and historical data in the virtual space, which provides an effective reference for the actual motion trajectory that may be generated by the controlled device according to the current motion trend, and actively intervenes in the driving state of the controlled device when there are potential risks in the predicted motion trajectory, so as to enhance the control ability of the mobile device based on the brain-computer interface, thereby improving the safety and reliability of the brain control technology.

[0037] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0039] Figure 1 is a schematic diagram of a brain control system framework based on digital twins provided in some embodiments of the present application;

[0040] Figure 2 is a flowchart of a brain-controlled driving method based on digital twins provided in some embodiments of the present application;

[0041] Figure 3 is a brain control command recognition block diagram provided by some embodiments of the present application;

[0042] Figure 4 It is a block diagram of an implementation of a brain-controlled driving process based on digital twins provided in some embodiments of the present application;

[0043] Figure 5 is a schematic block diagram of a module of a brain-controlled driving device based on digital twins provided in some embodiments of the present application;

[0044] Figure 6 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the reference to "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0046] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0047] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0049] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0050] In order to make the inventive concept of the present application easy to understand, before describing the embodiments of the present application in detail, the English abbreviations (terms) / related concepts involved in the embodiments of the present application are first described. The English abbreviations (terms) / related concepts involved in the embodiments of the present application are subject to the following explanations.

[0051] Emotiv Epoc: Emotiv Epoc is an advanced wireless electroencephalogram (EEG) device that is designed to capture and analyze electrical brain signals.

[0052] BCI: BCI stands for Brain-Computer Interface, which can also be called brain-control technology. Brain-computer interface technology interprets neural signals from the brain and converts them into electronic commands to control computers, robotic arms or other devices.

[0053] Brain control technology has been widely used and developed in many fields, but it still faces limitations in hardware and algorithm performance. These limitations lead to a single type of brain control signal data, low recognition rate of brain control commands, and poor reliability of the operating system, which cannot fully realize its design potential. This is mainly because the real-time interactive technology accessible between the brain and external physical devices is still in its early stages of development, and the main technical means are far from meeting the needs of the brain.

[0054] On the one hand, when the brain sends a large number of command signals, the brain-computer interface hardware equipment needs to accurately and completely obtain and filter the required signals. However, the current equipment has problems such as large size, high cost, poor comfort, and weak adaptability to obtaining high-quality signals for each user.

[0055] On the other hand, in order to achieve a high success rate of brain control tasks with seamless interaction between users and devices, it is necessary to decode and classify the acquired real-time, complex and variable EEG signals with high precision. However, existing pattern recognition algorithms alone are far from being able to achieve the safety and reliability of widespread application of brain control.

[0056] The analysis found that the key to improving the safety and reliability of brain-computer interfaces lies in developing a technical means to accurately monitor the operating status of the controlled equipment, analyze highly complex and information-rich brain command signals, and achieve "optimal control". Digital twin technology has broken through technical barriers in multiple fields with its advantages such as ultra-high-fidelity simulation prediction, virtual-real interactive mapping, and autonomous feedback regulation.

[0057] In view of this, the present application proposes a brain-controlled driving method, device and equipment based on digital twins. The scheme obtains EEG signals, position information and motion state information of the controlled device; controls the position and motion state of the digital twin of the controlled device to be consistent with the controlled device according to the position information and motion state information; predicts the motion trajectory of the digital twin of the controlled device by combining the EEG signals with the historical data of the controlled device to obtain a predicted motion trajectory; and actively intervenes in the driving state of the controlled device when potential risks are detected in the predicted motion trajectory.

[0058] The embodiment of the present application realizes the simulation of the operating state of the controlled device in the virtual space by controlling the position and motion state of the digital twin of the controlled device to be consistent with the controlled device, and predicts the motion trajectory of the digital twin in the virtual space according to brain control signals and historical data, thereby providing an effective reference for the actual motion trajectory that may be generated by the controlled device according to the current motion trend, and actively intervening in the driving state of the controlled device when there are potential risks in the predicted motion trajectory, thereby enhancing the control capability of mobile devices based on brain-computer interfaces, and thereby improving the safety and reliability of brain control technology.

[0059] The method provided in the embodiment of the present application can be applied to the electronic device provided in the embodiment of the present application, wherein the electronic device can be a terminal or a server.

[0060] The terminal may be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto.

[0061] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, as well as big data and artificial intelligence platforms.

[0062] Before introducing the implementation steps of the digital twin-based brain-controlled driving method provided in this application, it is necessary to first introduce the digital twin-based brain-controlled system provided in this application.

[0063] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the framework of a brain control system based on digital twins provided in some embodiments of the present application. The brain control system based on digital twins consists of three parts, namely, a BCI system, a digital dual driver framework and a control command module.

[0064] Among them, the BCI system includes the acquisition, preprocessing and feature extraction operations of EEG signals. The features extracted by the feature extraction operation will be input into the digital dual driver framework.

[0065] In the digital dual-driver framework, physical state data acquisition (operating status and environmental information of the controlled equipment), stored historical data, and acquired real-time data are included. These data are input into the data processing unit to obtain data processing results, and the data processing results are input into the control command module.

[0066] In the control command module, the data processing results are parsed into command signals that actually control the controlled devices. Specifically, the data information representing the user's intentions (such as direction information, behavior information, or speed information, etc.) can be matched through the command library, and the matched commands can be sent to the controlled device to further control the operation of the controlled device. The controlled device also feeds back the execution results to the brain control system.

[0067] The controlled device may be a device including a power drive device, such as a car, a truck, a wheelchair, an unmanned ship, a drone, or an excavator, which is not limited in the present application.

[0068] Brain signals can be collected through the Emotiv EPOC helmet and directly input into the BCI system for processing, and the control results of the controlled equipment can be displayed through the user interface; the user interface can also provide a command interface for manual control by the driver.

[0069] For example, a specific application scenario can be to control the operation of a vehicle through a brain control system based on digital twins. Then, the entire system (here it can refer to the extended concept of the system, which includes devices that interact with the system) can be composed of the Emotiv EPOC helmet, the vehicle, and the cloud server.

[0070] The Emotiv EPOC helmet can obtain the driver's multi-channel EEG and transmit the multi-channel EEG to the cloud server through the network, and then the received multi-channel EEG is identified through service analysis in the cloud server.

[0071] The cloud server includes the virtual layer and data layer of the digital twin.

[0072] Among them, the virtual layer mainly includes the Unity 3D engine (Unity 3D) and three-dimensional computer-aided design (CAD) software (such as Solidworks) to construct virtual scenes of the geometric structure and dimensional relationship of the spatial environment and physical entities; it can also include Ansys and Matlab simulation prediction software to model and simulate the acquired signals.

[0073] The position information and motion status of the controlled vehicle are collected in real time through RS-485 communication protocol and Modbus communication protocol, and the operation status of the controlled vehicle under the control command is fed back.

[0074] The data layer mainly includes EEG signals, eye rotation signals, vehicle operation status data, vehicle location information, and the historical database collected by the Emotiv EPOC helmet.

[0075] RS-485 and Modbus protocols can be used to collect the vehicle's location information and motion status in real time, and synchronize these data to the virtual layer and data layer of the cloud server to ensure the dynamic consistency between the virtual scene and the actual vehicle.

[0076] Then, MATLAB was used for dynamic simulation, and Autodesk Revit was used to show in detail the motion trajectory of the vehicle under different EEG control commands. Based on the historical data and real-time data of the data layer, machine learning and artificial intelligence algorithms were used to build a prediction model to predict the future motion trajectory of the vehicle.

[0077] Finally, based on the predicted vehicle motion trajectory, corresponding adjustment suggestions are provided, or active intervention is made in the vehicle operation to increase the safety of vehicle operation.

[0078] The above is an example of a specific operating scenario of an embodiment of the present application. It should be understood that the cloud server can also be replaced by a PC or a personal terminal.

[0079] The following will describe in detail the implementation steps of a brain-controlled driving method based on digital twins provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0080] Please refer to Figure 2 , Figure 2 A flowchart of a brain-controlled driving method based on digital twins is provided for some embodiments of the present application. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0081] The method of the embodiment of the present application comprises the following steps:

[0082] Step 201: obtaining EEG signals, location information of a controlled device, and motion state information of the controlled device;

[0083] Step 202: Control the position and motion state of the digital twin of the controlled device to be consistent with the controlled device according to the position information and motion state information;

[0084] Step 203: predicting the motion trajectory of the digital twin of the controlled device by combining the EEG signal with the historical data of the controlled device to obtain a predicted motion trajectory;

[0085] Step 204: When it is detected that the predicted motion trajectory has a potential risk, actively intervene in the driving state of the controlled device.

[0086] In steps 201 to 204 shown in the embodiment of the present application, the operating state of the controlled device is simulated in the virtual space by controlling the position and motion state of the digital twin of the controlled device to be consistent with the controlled device; by predicting the motion trajectory of the digital twin in the virtual space based on brain control signals and historical data, an effective reference is provided for the actual motion trajectory that may be generated by the controlled device according to the current motion trend, and when there is a potential risk in the predicted motion trajectory, the driving state of the controlled device is actively intervened, thereby enhancing the control capability of mobile devices based on brain-computer interfaces, thereby improving the safety and reliability of brain control technology.

[0087] The specific implementation methods of the above steps are introduced below.

[0088] In step 201, an electroencephalogram signal, position information of a controlled device, and motion state information of the controlled device are obtained.

[0089] The EEG signal can be obtained by an electroencephalogram (EEG) device. For example, a multi-channel EEG can be obtained by Emotiv Epoc.

[0090] The EEG device may be Emotiv Epoc X, EPOC+ or EPOC Flex, which is not limited in this application.

[0091] The controlled device may be a device including a power drive device, such as a car, a truck, a wheelchair, or an excavator, a drone, an unmanned ship, etc. This application does not impose any limitation on this.

[0092] The location information and motion status information of the controlled device can be obtained through RS-485 protocol and Modbus protocol.

[0093] The position information and motion status information of the controlled equipment can be collected in real time through RS-485 protocol and Modbus protocol, and the information can be synchronized to the virtual layer and data layer of the cloud server. Among them, the virtual layer and data layer of the cloud server can provide calculations and applications in the digital twin space.

[0094] In some embodiments of the present application, visual real-time data may also be collected through sensors.

[0095] Sensors can be cameras, infrared sensors, LiDAR, etc. They can capture high-resolution images and video streams and analyze them through built-in image processing algorithms. The visual data collected by these sensors can achieve real-time perception and understanding of the environment, thereby providing the necessary information support for generating digital twin spaces.

[0096] Afterwards, digital twins of the controlled equipment and digital twins of obstacles in the environment can be generated based on the visual data.

[0097] For example, Emotiv Epoc X and cameras can be used to collect EEG signals and visual real-time data, and the collected data can be transmitted to the cloud server through 5G and IoT communication networks; Hadoop (a distributed computing platform mainly used to process massive data) and databases are used for data storage and management. Secondly, Apache Kafka (a distributed message publishing-subscription system) and machine learning algorithms are used to process and analyze the collected real-time data; Revit (a software built specifically for building information modeling (BIM) that supports sustainable design, collision detection, construction planning and construction) and MATLAB are used for modeling and simulation to create a detailed virtual model. Cloud computing and AWS edge computing are used to provide computing and storage support, and the model is optimized through artificial intelligence and machine learning algorithms. The model is intuitively displayed and interacted with based on data visualization tools and user interface technologies (for example, Tableau, VR and AR). Among them, standardized integration platforms and open APIs can ensure system interoperability and data compatibility, thereby realizing the construction of digital twin space.

[0098] The embodiment of the present application provides data support for the subsequent control command generation and the calculation of the digital twin space by acquiring EEG signals, the position information of the controlled device and the motion state information of the controlled device. By building a digital twin space, a virtual counterpart can be provided for the vehicle and its operating environment, thereby enhancing the vehicle control capability based on the brain-computer interface. Among them, this virtual space supports real-time simulation and predictive modeling, enabling the digital twin brain control system to predict and adapt to the complex dynamics of vehicle control.

[0099] In step 202, the position and motion state of the digital twin of the controlled device are controlled to be consistent with the controlled device based on the position information and motion state information.

[0100] Exemplarily, the EEG signal, the location information of the controlled device, and the motion state information of the controlled device obtained through step 201 can be transmitted to the virtual layer and data layer in the cloud server, which can be achieved through wireless communication technologies such as Wi-Fi, Bluetooth, cellular networks, etc.

[0101] In the virtual layer, the collected data is processed and analyzed. This may include steps such as data cleaning, filtering, and conversion to ensure the accuracy and consistency of the data.

[0102] According to the physical characteristics and motion laws of the controlled equipment, a mathematical model of the digital twin is established. The model should be able to accurately reflect the motion state and position of the controlled equipment.

[0103] Use the processed data to update the model of the digital twin so that its position and motion state are consistent with the controlled device. This can use real-time control systems such as PID controllers, state space controllers, etc.

[0104] In addition, a feedback mechanism can be established to feed back the status information of the digital twin to the controlled device so that necessary adjustments can be made. This helps to improve the stability and accuracy of the system.

[0105] The embodiment of the present application controls the position and motion state of the digital twin of the controlled device to be consistent with the controlled device according to the position information and motion state information, thereby providing support for the subsequent implementation of refined control of the controlled device.

[0106] In step 203, the motion trajectory of the digital twin of the controlled device is predicted by combining the EEG signal with the historical data of the controlled device to obtain a predicted motion trajectory.

[0107] For example, MATLAB can be used for dynamic simulation, and Autodesk Revit can be used to show in detail the motion trajectory of the vehicle under different EEG control commands. Based on the historical data and real-time data of the cloud server data layer, machine learning and artificial intelligence algorithms are used to build a prediction model to predict the future motion path of the vehicle.

[0108] In some implementations, a control command may be determined based on an EEG signal; the control command and historical data are input into a trajectory prediction model to obtain a predicted motion trajectory; wherein the trajectory prediction model is constructed using an artificial intelligence method.

[0109] The EEG signals may include a first group of EEG signals and a second group of EEG signals. The first group of EEG signals are input into a first model to obtain a control command to move forward or backward, wherein the first group of EEG signals are EEG signals generated by a motor imagery pattern, and the first model is a model for extracting asymmetric features of EEG signals in left and right brain regions; the second group of EEG signals are input into a second model to obtain a control command to turn left or right, wherein the second group of EEG signals are EEG signals generated by an eye blinking pattern, and the second model is a model for extracting pulse features of EEG signals.

[0110] For example, reference may be made to Figure 3 , Figure 3This is a brain control command recognition block diagram provided by some embodiments of the present application. The Emotiv Epoc can be used to obtain the driver's multi-channel EEG, which can be transmitted to the BCI system via a wireless network. In the BCI system, the EEG signal is processed without overlap for 1 second. The non-overlap processing means that there is no overlapping area between multiple sliding windows that intercept the EEG signal.

[0111] Afterwards, the common spatial pattern (CSP) and support vector machine (SVM) algorithms are used to obtain the two control commands of forward and backward movement, and the artificial neural network (ANN) based on kurtosis and amplitude is used to obtain the two control commands of left turn and right turn.

[0112] Specifically, the EEG of EEG signal windows F7, F3, FC5, T7, T8, FC6, F4 and F8 channels (the first group of EEG signals) can be used as the input of CSP to calculate the CSP filter, and the output of CSP is used as EEG features and input into SVM for offline training. Using the obtained CSP model and the trained SVM classifier, the real-time EEG of the same channel is classified to generate control commands to move the vehicle forward or backward. The forward and backward commands are generated by a combination of CSP and SVM, which uses EEG data generated by the motor imagination mode (for example, imagining the left hand movement and the right hand movement of the human body).

[0113] It is understandable that motor imagery will cause asymmetry between the left and right brain regions, and CSP can be used to extract asymmetric features well.

[0114] The kurtosis and amplitude energy features can be calculated from the EEG signals acquired using the EEG signal windows AF3 and AF4 channels, and ANN is used as a classifier. After obtaining the corresponding model and classifier, the real-time EEG of the same channel is input into the model and classifier to identify the expected control command to achieve the left or right turn of the vehicle. The left and right turn commands are generated by an artificial neural network (ANN) based on kurtosis and amplitude, which uses the EEG data generated by the eye blink pattern.

[0115] It is understandable that spike pulses appear in the EEG waveform when the eyes blink, and the use of kurtosis and amplitude combined with ANN can reveal the pulse characteristics of the waveform during blinking.

[0116] After obtaining the brain control command, the command and historical data are input into the trajectory prediction model to obtain the predicted motion trajectory.

[0117] The embodiment of the present application predicts the motion trajectory of the digital twin of the controlled device by combining EEG signals with the historical data of the controlled device to obtain a predicted motion trajectory, which provides a basis for subsequent active intervention in the controlled device through the operating status of the controlled device combined with the predicted motion trajectory.

[0118] In step 204, when it is detected that there is a potential risk in the predicted motion trajectory, the driving state of the controlled device is actively intervened.

[0119] Among them, the potential risk of the predicted motion trajectory may be that the distance between the point on the motion trajectory and the obstacle is too close, that is, less than the preset distance threshold, or the point on the motion trajectory passes through an obstacle, or the point on the motion trajectory passes through a non-drivable area. For a vehicle, its trajectory may pass through a lawn, a lake or a deep pit, etc.; for an excavator, the motion trajectory of its robotic arm may pass through a telephone pole or a high-voltage line, etc.; for a wheelchair, its trajectory may pass through a bumpy road surface that is not suitable for passing through or a multi-layered staircase road condition that cannot be passed directly.

[0120] In some embodiments, the distance between the digital twin of the controlled device and the digital twin of the obstacle can be determined based on the predicted motion trajectory; when the distance is less than a preset distance, the driving state of the controlled device is actively intervened.

[0121] For example, in the virtual space, a safety threshold between the controlled device and the obstacle can be defined. If the simulation result shows that the distance between the controlled device and the obstacle is greater than the safety threshold, the controlled device will run along the expected trajectory according to the received brain control command; if the distance is less than the safety threshold, the obstacle avoidance mechanism will be triggered.

[0122] Among them, the distance between the digital twin of the controlled device and the digital twin of the obstacle can be achieved by calculating the shortest distance between each point on the predicted motion trajectory and the digital twin of the obstacle.

[0123] Among them, the obstacle avoidance mechanism can be to send a request message when the distance is less than a preset distance, and the request message is used to request the driver to send an EEG signal again to reduce potential risks; if the potential risk is not reduced, send a deceleration command or a braking command to the controlled device.

[0124] Specifically, if the simulation result shows that the distance between the controlled device and the obstacle is less than the safety threshold, but it is not urgent enough to cause a collision, a feedback message is sent to the driver, requesting the driver to reissue the brain control command to achieve safe driving.

[0125] If no new brain control command that can effectively avoid obstacles is received within a preset time, or if it is detected that the potential risk has not been reduced, a deceleration command or a braking command is sent to the controlled device to avoid a collision with the controlled device.

[0126] In some embodiments, when no potential risk is detected in the predicted motion trajectory, the controlled device is controlled to travel based on the control command generated based on the EEG signal.

[0127] Exemplarily, when the simulation result shows that the distance between the vehicle and the obstacle is greater than a safety threshold, the vehicle runs along the expected trajectory according to the received brain control instructions.

[0128] It should be understood that after obstacle avoidance is completed, the user needs to resend the brain control command to ensure that the controlled device continues to run along the correct path.

[0129] See also Figure 4 The above process can be described as a real-time interaction between physical space and virtual space, where the physical space includes BCI, camera, controller and device, the device can be an EEG device, and the virtual space can include static and dynamic path planning, performance evaluation, position and navigation, and adaptive control. The physical space sends information including the start and end positions of the controlled device to the virtual space, and the virtual space processes and analyzes the received commands, and feeds back the speed and interrupt commands to the controlled device to actively intervene in the driving state of the controlled device.

[0130] In the embodiment provided by the present application, multi-channel EEG signals are collected by Emotiv Epoc, and control commands are issued to the controlled device by analyzing the EEG signals. During the operation of the controlled device, the motion state in the physical space is interactively mapped with the virtual space in the digital twin in real time. Since the control accuracy of the traditional brain control system cannot reach 100%, misidentification may occur and obstacles cannot be avoided. Therefore, in the digital twin virtual space, a safety threshold between the controlled device and the obstacle can be defined. When the distance is greater than the safety threshold, it is determined to be safe and the vehicle operates as expected; when the distance is less than the safety threshold, the obstacle avoidance mechanism is triggered. The obstacle avoidance mechanism can be specifically embodied as follows: when the brain-computer interface cannot change the path in time or there is still a collision trend, the digital twin layer actively intervenes in the operation of the controlled device, and then sends a signal to slow down or stop the controlled device. After that, the user needs to resend the control command, and when the command is correct, the controlled device continues to run.

[0131] In order to further illustrate the effects of the embodiments of the present application, the present application provides a set of comparative experiments, specifically a vehicle obstacle avoidance experiment.

[0132] The experiment was conducted through two platforms, one platform is the digital twin-based brain control system platform provided in this application, and the other is the traditional brain control system platform.

[0133] The obstacle avoidance experiment is as follows: 10 obstacle avoidance experiments are carried out through each platform. In the experiment, the two platforms have the same starting point and end point, and the success rate and time of the two platforms passing through three obstacles are calculated. Among them, the vehicle speed is set to 1m / s, and the safety distance threshold is set to 30cm. The experimenter is required to control the vehicle to pass through three obstacles to reach the end point under the conditions of each control platform. When the controlled vehicle contacts the obstacle, the experiment ends and is recorded as a failure of the brain control task. When the controlled vehicle passes the end point without contacting the obstacle, the brain control task is recorded as successful, and the time taken is recorded. All of the above experiments require the same experimenter to conduct driving tests and be familiar with the control process and methods of the two platforms.

[0134] For traditional brain control system testing, each command was repeated 50 times, for a total of 200 tests.

[0135] Model 1, which combines RBF-SVM with an ANN with a single hidden layer of 10 neurons, correctly identified 48, 46, 44, and 41 control intentions out of 50 attempts for each category, respectively. Therefore, the corresponding classification accuracies for the first to fourth categories (i.e., left turn, right turn, forward, and backward) are 96%, 92%, 88%, and 82%, respectively. It can be concluded that the control accuracy of the conventional brain control system is 89.5%.

[0136] In addition, the effects of different SVM kernel functions and different numbers of ANN hidden layers on control accuracy were tested in the experiment. As a comparison method, a Sigmoid-SVM (for classification using CSP-based features) with a single hidden layer of 10 neurons in the ANN (for classification using the kurtosis and amplitude of AF3 and AF4) was designed and named Model 2. It can be concluded that the control accuracy of the conventional brain control system applying Model 2 is 87%.

[0137] In addition, as another comparison method, Model 3 combines RBF-SVM with ANN with a single hidden layer of 5 neurons, and it can be concluded that the control accuracy of the traditional brain control system using Model 3 is 85.5%. After comparison, the experiment selected Model 1 as the algorithm model of the system for further study.

[0138] For traditional brain control systems, the accuracy of a single command can reach a very high level, but for a whole set of brain control tasks, the success rate still needs to be improved.

[0139] The obstacle avoidance test results are shown in Table 1:

[0140] Table 1: Obstacle avoidance test results

[0141]

[0142]

[0143] In the experiment, the task success rate and task completion time were used to evaluate the test performance of brain control on the two platforms. In the 10 tests of the traditional brain control system, 4 encountered the set obstacles and were judged as mission failures. In the 10 tests of the digital twin brain control system, although all 10 tests were successfully completed, in the 2nd, 4th, and 7th experiments, because the distance between the controlled vehicle and the obstacle exceeded the preset threshold, the digital twin layer intervened to issue an alarm and actively brake the vehicle. The fastest time for active intervention was 104.01 seconds, and the slowest time was 128.89 seconds.

[0144] In order to study whether there is a difference between the outputs of the two systems, the data of the traditional brain control system and the digital twin brain control system were tested, and the result p value was 0.0247 < 0.05. Therefore, it was concluded that there was a significant difference between the two, and the introduction of digital twins was effective in improving the system.

[0145] The experimental results show that the traditional brain control system can achieve continuous control and avoid obstacles at a faster speed, but there is a certain risk of failure. Digital twin intervention can achieve a 100% obstacle avoidance success rate, but to a certain extent at the cost of consuming more time (this problem can be overcome with higher computing resources).

[0146] In addition, the experiment also analyzed the response time from the issuance of instructions to their execution. The addition of digital twin technology only resulted in a slight increase in response time, but still maintained a high level of real-time performance. Therefore, the brain-controlled driving method and system based on digital twins proposed in this application can improve the safety and reliability of brain control.

[0147] In addition, in the implementation of this use case, the experimenter is a good operator of the brain control system. However, not all brain control users have better operating skills, so the digital twin brain control system can also enable new users to perform brain control tasks more consistently.

[0148] The above is an introduction to the embodiment of the brain-controlled driving method based on digital twin provided in this application.

[0149] The implementation scheme of the brain-controlled driving device based on digital twin provided by the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0150] For the brain-controlled driving method based on digital twin provided in the above embodiment, the embodiment of the present application also provides a brain-controlled driving device based on digital twin, which is used to implement the above method, such as Figure 5 As shown, Figure 5This is a module schematic block diagram of a brain-controlled driving device 500 based on digital twins according to an embodiment of the present application. The brain-controlled driving device 500 based on digital twins includes:

[0151] An acquisition module 501 is used to acquire EEG signals and location information and motion state information of a controlled device;

[0152] A control module 502 is used to control the position and motion state of the digital twin of the controlled device to be consistent with the controlled device according to the position information and motion state information;

[0153] The prediction module 503 is used to predict the motion trajectory of the digital twin of the controlled device by combining the EEG signal with the historical data of the controlled device to obtain the predicted motion trajectory;

[0154] The active intervention module 504 is used to actively intervene in the driving state of the controlled device when a potential risk is detected in the predicted motion trajectory.

[0155] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0156] like Figure 6 As shown, the embodiment of the present application further provides an electronic device 600, the electronic device 600 includes a memory 601, one or more processors 602 ( Figure 6 Only one is shown) and a computer program stored in the memory 601 and executable on the processor 602. Among them: the memory 601 is used to store software programs and units, and the processor 602 executes various functional applications and data processing by running the software programs and units stored in the memory to obtain the resources corresponding to the above-mentioned preset events. Optionally, the processor 602 implements the above-mentioned brain-controlled driving method based on digital twins by running the above-mentioned computer program stored in the memory 601.

[0157] The memory 601, as a non-transitory computer-readable medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 601 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 601 may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor 602 via a network.

[0158] It can be understood that the contents of the above method embodiments are all applicable to the electronic device embodiment, the functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] An embodiment of the present application also provides a mobile device, which includes a power drive device, and the operation of the mobile device can be controlled by the above-mentioned digital twin-based brain-controlled driving method, device or the above-mentioned electronic device.

[0160] Specifically, the mobile device may be a vehicle, and the vehicle may be a private vehicle, such as a sedan, SUV, MPV, or pickup truck. The vehicle may also be an operating vehicle, such as a van, bus, small truck, or large trailer. The vehicle may be a gasoline vehicle or a new energy vehicle. When the vehicle is a new energy vehicle, it may be a hybrid vehicle or a pure electric vehicle.

[0161] The mobile device may also be an electric wheelchair, an excavator, etc.

[0162] It can be understood that the contents of the above method embodiments are all applicable to the present mobile device embodiment, the functions specifically implemented by the present mobile device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0163] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it can implement the steps of the above-mentioned brain-controlled driving method based on digital twins.

[0164] It can be understood that the contents of the above method embodiments are all applicable to the present computer program product, the functions specifically implemented by the present computer program product embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0165] The digital twin-based brain-controlled driving method, apparatus, electronic device, mobile device and computer program product provided in the embodiments of the present application realize the simulation of the operating state of the controlled device in the virtual space by controlling the position and motion state of the digital twin of the controlled device to be consistent with the controlled device. By predicting the motion trajectory of the digital twin in the virtual space according to brain control signals and historical data, an effective reference is provided for the actual motion trajectory that may be generated by the controlled device according to the current motion trend. When there is a potential risk in the predicted motion trajectory, the driving state of the controlled device is actively intervened, thereby enhancing the control capability of the mobile device based on the brain-computer interface, and thus improving the safety and reliability of brain control technology.

[0166] The digital twin-based brain-controlled driving method, device, electronic device, mobile device and computer program product provided in the embodiments of the present application can provide a virtual counterpart for the mobile device and its environment by introducing digital twin technology, thereby enhancing the vehicle control capability based on the brain-computer interface. This virtual space supports real-time simulation and predictive modeling, enabling the digital twin brain control system to predict and adapt to the complex dynamics of mobile device control. By accurately modeling real-world interactions, the digital twin brain control system can bridge the gap between cognitive input and the complex control requirements of mobile device driving. This integration not only helps to achieve more accurate and sensitive control operations, but also significantly improves the overall performance and reliability of brain-controlled mobile devices in dynamic and complex situations, thereby providing a new solution for the rapid development of brain control technology.

[0167] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0168] Although specific embodiments are described herein, those of ordinary skill in the art will recognize that many other modifications or alternative embodiments are also within the scope of the present disclosure. For example, any of the functions and / or processing capabilities described in conjunction with a particular device or component may be performed by any other device or component. In addition, although various exemplary implementations and architectures have been described according to embodiments of the present disclosure, those of ordinary skill in the art will recognize that many other modifications to the exemplary implementations and architectures described herein are also within the scope of the present disclosure.

[0169] Some aspects of the present disclosure are described above with reference to the block diagram and flow chart of the system, method, system and / or computer program product according to exemplary embodiments.It should be understood that the combination of one or more blocks in the block diagram and flow chart and the blocks in the block diagram and flow chart can be realized by executing computer executable program instructions respectively.Equally, according to some embodiments, some blocks in the block diagram and flow chart may not need to be executed in the order shown, or may not need to be executed in full.In addition, additional components and / or operations beyond those components and / or operations shown in the blocks in the block diagram and flow chart may be present in certain embodiments.

[0170] Therefore, the blocks in the block diagrams and flow charts support the combination of devices for performing the specified functions, the combination of elements or steps for performing the specified functions, and the program instruction devices for performing the specified functions. It should also be understood that each block in the block diagrams and flow charts and the combination of blocks in the block diagrams and flow charts can be implemented by a special-purpose hardware computer system or a combination of special-purpose hardware and computer instructions that performs a specific function, element or step.

[0171] The program modules, applications, etc. described herein may include one or more software components, including, for example, software objects, methods, data structures, etc. Each such software component may include computer-executable instructions that, in response to execution, cause at least a portion of the functionality described herein (e.g., one or more operations of the illustrative methods described herein) to be performed.

[0172] Software components can be encoded with any of various programming languages. An exemplary programming language can be a low-level programming language, such as an assembly language associated with a specific hardware architecture and / or an operating system platform. The software component comprising assembly language instructions may need to be converted to executable machine code by an assembler before being executed by hardware architecture and / or platform. Another exemplary programming language can be a higher-level programming language, which can be transplanted across multiple architectures. The software component comprising a higher-level programming language may need to be converted to an intermediate representation by an interpreter or a compiler before execution. Other examples of programming languages ​​include but are not limited to macro languages, shells or command languages, job control languages, scripting languages, database queries or search languages ​​or report writing languages. In one or more exemplary embodiments, the software component comprising the instruction of one of the above-mentioned programming language examples can be directly executed by an operating system or other software components, without first being converted into another form.

[0173] Software components may be stored as files or other data storage structures. Software components of similar type or related functions may be stored together, such as in a specific directory, folder, or library. Software components may be static (e.g., preset or fixed) or dynamic (e.g., created or modified at execution time).

[0174] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A brain-controlled driving method based on digital twins, characterized in that: The following steps are involved: Acquiring EEG signals, location information of a controlled device, and motion state information of the controlled device; Control the position and motion state of the digital twin of the controlled device to be consistent with the controlled device according to the position information and the motion state information; Predicting the motion trajectory of the digital twin of the controlled device by combining the EEG signal with historical data of the controlled device to obtain a predicted motion trajectory; When it is detected that there is a potential risk in the predicted motion trajectory, actively intervene in the driving state of the controlled device.

2. The brain-controlled driving method based on digital twin according to claim 1 is characterized in that: Before controlling the position and motion state of the digital twin of the controlled device to be consistent with the controlled device according to the position information and the motion state information, the method further includes: Acquire visual data; A digital twin of the controlled device and digital twins of obstacles in the environment are generated based on the visual data.

3. The brain-controlled driving method based on digital twin according to claim 1 is characterized in that: The step of predicting the motion trajectory of the digital twin of the controlled device by combining the EEG signal with the historical data of the controlled device to obtain the predicted motion trajectory includes: determining a control command based on the EEG signal; Inputting the control command and the historical data into a trajectory prediction model to obtain the predicted motion trajectory; Wherein, the trajectory prediction model is constructed by artificial intelligence methods.

4. The brain-controlled driving method based on digital twins according to claim 2 is characterized in that: When detecting that the predicted motion trajectory has a potential risk, actively intervening in the driving state of the controlled device includes: Determining a distance between the digital twin of the controlled device and the digital twin of the obstacle according to the predicted motion trajectory; When the distance is less than a preset distance, actively intervene in the driving state of the controlled device.

5. The brain-controlled driving method based on digital twin according to claim 4 is characterized in that: When the distance is less than a preset distance, actively intervening in the driving state of the controlled device includes: When the distance is less than the preset distance, sending a request message, wherein the request message is used to request the driver to send the EEG signal again to reduce the potential risk; When the potential risk is not reduced, a deceleration command or a braking command is sent to the controlled device.

6. The brain-controlled driving method based on digital twin according to claim 3 is characterized in that: The EEG signals include a first group of EEG signals and a second group of EEG signals, and determining the control command based on the EEG signals includes: Inputting the first group of EEG signals into a first model to obtain a control command of moving forward or backward, wherein the first group of EEG signals are EEG signals generated by a motor imagery mode, and the first model is a model for extracting asymmetric features of EEG signals in left and right brain regions; The second group of EEG signals are input into a second model to obtain a control command of turning left or right, wherein the second group of EEG signals are EEG signals generated by an eye blinking pattern, and the second model is a model for extracting pulse features of EEG signals.

7. The brain-controlled driving method based on digital twin according to claim 1 is characterized in that: After predicting the motion trajectory of the digital twin of the controlled device by combining the EEG signal with the historical data of the controlled device and obtaining the predicted motion trajectory, the method further includes: When no potential risk is detected in the predicted motion trajectory, the controlled device is controlled to travel based on the control command generated by the electroencephalogram signal.

8. A brain-controlled driving device based on digital twins, characterized in that: The device comprises: An acquisition module is used to acquire EEG signals and the position information and motion state information of the controlled device; A control module, configured to control the position and motion state of the digital twin of the controlled device to be consistent with the controlled device according to the position information and the motion state information; A prediction module, used to predict the motion trajectory of the digital twin of the controlled device by combining the EEG signal with the historical data of the controlled device to obtain a predicted motion trajectory; The active intervention module is used to actively intervene in the driving state of the controlled device when it is detected that the predicted motion trajectory has potential risks.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the digital twin-based brain-controlled driving method as described in any one of claims 1 to 7 when executing the computer program.

10. A mobile device, characterized in that: The mobile device includes a power drive device, and the operation of the mobile device is controlled by the brain-controlled driving method based on digital twins as described in any one of claims 1 to 7, the brain-controlled driving device based on digital twins as described in claim 8, or the electronic device as described in claim 9.

Images