Image analysis processing system and method based on thought control

Abstract

The invention relates to the field of brain image processing, in particular to an image analysis processing system and method based on thought control, and the system comprises an electroencephalogram signal collection system which is used for collecting and preprocessing electroencephalogram signals of a user; the thought analysis system is used for extracting features from the preprocessed electroencephalogram signals and then carrying out thought classification and instruction mapping; the image acquisition and processing system is used for acquiring image information according to the mind control instruction of the user and performing image processing; the VR display system is used for converting the processed image into a format suitable for VR equipment display and providing interactive feedback; and the control and communication system is used for converting the mind state into a control instruction and realizing data transmission and communication among the systems. The EEG signals and the fNIRS signals are acquired through the multi-mode signals, the defect of a single signal in the aspect of reflecting the brain activity is overcome, and therefore brain activity information corresponding to the thought is more accurately interpreted.

Description

Technical Field

[0001] The present invention relates to the fields of brain image processing, virtual reality (VR) technology and artificial intelligence technology, and in particular to an image analysis and processing system and method based on mind control. Background Art

[0002] As an emerging interactive technology, brain-computer interface (BCI) technology aims to establish a direct communication channel between the brain and external devices. In recent years, brain-computer interface technology has made significant progress in signal acquisition, feature extraction, and pattern recognition, making it possible to achieve mind control.

[0003] Image processing technology is constantly developing in improving image quality and analyzing image content, while virtual reality technology provides users with an immersive experience. Combining image processing with virtual reality technology can create a more realistic and intuitive interactive environment for users. However, most existing image processing and virtual reality systems rely on traditional input devices, such as handles and keyboards, and lack direct control methods based on thoughts.

[0004] Current image analysis and processing systems have certain limitations in terms of processing speed, accuracy, and personalization. Traditional control methods are not only complex to operate, but also fail to meet the needs of some special scenarios, such as for people with disabilities or those working in special environments. In addition, when processing large amounts of image data, existing systems often have difficulty in quickly and accurately extracting information of interest to users, and cannot be adjusted in real time according to the personalized needs of users.

[0005] In view of this, the present invention provides an image analysis and processing system and method based on mind control. Summary of the invention

[0006] The purpose of the present invention is to provide an image analysis and processing system and method based on mind control in view of the deficiencies in the prior art.

[0007] In order to solve the above technical problems, the following technical solutions are adopted: The image analysis and processing system based on mind control includes: The EEG signal acquisition system is used to collect the user's EEG signals and perform pre-processing; The thought analysis system extracts features from the preprocessed EEG signals, and then performs thought classification and command mapping; Image acquisition and processing system, used to obtain image information and perform image processing according to the user's thought control instructions; VR display system, used to convert processed images into a format suitable for VR device display and provide interactive feedback; The control and communication system is used to convert the state of thought into control instructions and realize data transmission and communication between various systems.

[0008] A further improvement based on the technical solution is that the EEG signal acquisition system adopts a non-invasive EEG sensor, arranges electrodes according to the system electrode placement method, and filters, amplifies and digitally converts the collected EEG signals.

[0009] A further improvement based on the technical solution is that the thought analysis system uses wavelet transform and power spectral density analysis methods to extract EEG signal features and adopts machine learning algorithms for pattern recognition.

[0010] A further improvement based on the technical solution is that the image acquisition and processing system obtains image information through an image acquisition device, and uses an image processing algorithm to perform denoising, enhancement, segmentation, and target detection operations, and performs personalized processing based on the output of the idea analysis system.

[0011] A further improvement based on the technical solution is that the VR display system converts the image into a stereoscopic split-screen format or a panoramic format, changes the viewing angle through head movement, and provides interactive feedback such as vision, hearing or touch.

[0012] A further improvement based on the technical solution is that the control and communication system converts the state of thought into control instructions such as image acquisition device parameter adjustment, image processing algorithm selection and parameter setting, VR display effect change, etc., uses a high-speed and stable communication protocol for data transmission, and encrypts the data.

[0013] The present invention proposes another technical solution: an image analysis and processing method based on mind control, characterized in that it includes the following steps: EEG signal acquisition and preprocessing: collect the user's EEG signals and preprocess them; Thought analysis: extract features from preprocessed EEG signals, and then perform thought classification and command mapping; Image acquisition and processing: acquiring image information and performing image processing according to the user's mind control instructions; VR display and interactive feedback: convert the processed images into a format suitable for VR device display and provide interactive feedback.

[0014] A further improvement based on the technical solution is that the collected EEG signals include EEG signals and fNIRS signals, and the collected EEG signals and fNIRS signals are preprocessed using adaptive filtering algorithms respectively; A further improvement based on the technical solution is that the process of extracting features from the EEG signal is as follows: jointly extracting features from the pre-processed EEG signal and the fNIRS signal, wherein the EEG signal is subjected to a wavelet transform combined with a power spectrum density analysis method to extract features, and the fNIRS signal is subjected to a calculation of the features of the changes in the concentrations of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR), and then a feature selection algorithm based on a genetic algorithm is used to select a feature subset that best represents the information of the mind; A further improvement based on the technical solution is that the process of thought classification and instruction mapping is as follows: a hybrid classification model is constructed to classify the extracted features, the hybrid classification model combines support vector machines and deep neural networks, and maps thoughts into specific control instructions according to the classification results, and the mapping is performed according to a predefined thought instruction mapping table.

[0015] Explanation of terminology: EEG signal is electroencephalogram signal.

[0016] fNIRS signals are functional near-infrared spectroscopy signals.

[0017] The above technical solution has the following beneficial effects: 1. Improve the accuracy of signal interpretation: Through multimodal signal acquisition of EEG signals and fNIRS signals, the advantages of the two signals are fully utilized to make up for the shortcomings of a single signal in reflecting brain activity, thereby more accurately interpreting the brain activity information corresponding to thoughts.

[0018] 2. Enhanced anti-interference capability: The adaptive filtering algorithm makes adaptive adjustments to different types of signals and noise sources, effectively removing various interferences, improving the quality of the signal, and making subsequent feature extraction and classification more reliable.

[0019] 3. Optimize classification accuracy and stability: The hybrid classification model combines the advantages of support vector machines (SVM) and deep neural networks (DNN), and can achieve better classification results in different situations (small samples and large samples), thereby improving the accuracy and stability of mind classification, and thus ensuring the accuracy of mind control. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described below in conjunction with the accompanying drawings: Figure 1 Schematic diagram of the structure of the image analysis and processing system based on mind control according to an embodiment of the present invention.

[0021] Figure 2 Schematic diagram of the flow of an image analysis and processing method based on mind control according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below through the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0023] See also Figure 1 , the image analysis and processing system based on mind control includes the following: 1. EEG signal acquisition system Sensor layout: Non-invasive EEG sensors are used to arrange electrodes on the user's scalp according to the international 10-20 system electrode placement method to ensure comprehensive collection of electrical signals from different areas of the brain. These sensors can detect weak electrical signals generated by brain neuron activity, providing a data basis for subsequent thought analysis.

[0024] Signal preprocessing: Preprocess the collected EEG signals, including filtering, amplification and digital conversion. Adaptive filtering algorithm is used to remove power frequency interference, myoelectric artifacts and other noises, and the amplification factor is automatically adjusted according to the signal strength to convert the analog signal into a digital signal for subsequent processing.

[0025] 2. Thought Analysis System Feature extraction: Extract the features related to thoughts from the preprocessed EEG signals. Use wavelet transform, power spectrum density analysis and other methods to extract the energy features, rhythm features, and spatiotemporal features of specific frequency bands (such as α waves and β waves). At the same time, combine deep learning algorithms, such as convolutional neural networks (CNNs), to automatically learn complex feature patterns in EEG signals.

[0026] Pattern recognition: Use machine learning algorithms, such as support vector machines (SVM) and deep neural networks (DNN), to perform pattern recognition on the extracted features and determine the user's state of mind. By learning from a large amount of labeled data, a mapping relationship between mind features and specific operational intentions is established.

[0027] 3. Image acquisition and processing system Image acquisition: Obtain image information of the external environment through a camera or other image acquisition device. Depending on different application scenarios, image acquisition devices of different types and parameters can be selected to meet the requirements of different resolutions, frame rates and viewing angles.

[0028] Image processing: De-noising, enhancement, segmentation, target detection and other operations are performed on the collected images. Advanced image processing algorithms, such as image segmentation networks based on deep learning (such as U-Net) and target detection algorithms (such as YOLO), are used to improve the accuracy and efficiency of image analysis. At the same time, according to the output of the idea analysis system, the image is personalized, such as highlighting the area or object of interest to the user.

[0029] 4. VR display system: The VR display system is an aerial naked-eye 3D imaging interactive display system.

[0030] Principle of naked-eye 3D imaging: It uses the principle of human eye parallax to achieve 3D effect. There is a certain distance between the eyes of a person, and when observing an object, the left and right eyes see slightly different images. Naked-eye 3D technology uses special optical elements or algorithms to project slightly different images to the left and right eyes of the viewer, allowing the brain to fuse the two images to produce a 3D visual experience. For example, using cylindrical lens technology, a layer of cylindrical lenses is covered on the surface of the screen, so that light enters the left and right eyes at different angles, forming a three-dimensional vision.

[0031] Interaction principle: The interactive part of the system usually uses a variety of sensor technologies. For example, infrared sensors are used to sense the position and movement of the user's hand. When the user reaches out to touch or wave, the infrared sensor captures the position change information of the hand.

[0032] Cameras can also be used for visual recognition. By recognizing the image of user actions, such as the shape of gestures (such as making a fist, waving hands, etc.), the system can determine the user's intention. Then, according to the preset interaction rules, corresponding operations are performed on the displayed 3D content, such as rotation, scaling, selection, etc.

[0033] Immersive display: Convert the processed image into a format suitable for VR device display, such as stereo split-screen format or panoramic format. Use the display technology of VR devices to provide users with an immersive visual experience. Users can change the perspective by moving their head and observe the image content more naturally.

[0034] Interactive feedback: According to the user's mind control operation, the VR display content is updated in real time and corresponding feedback information is provided. For example, when the user selects a certain image area through mind control, the system will display a prominent logo or provide detailed information in that area. At the same time, through visual, auditory or tactile feedback, the user can feel the results of the operation.

[0035] 5. Control and communication system Instruction conversion: Convert the thought state identified by the thought analysis system into specific control instructions. These instructions may include parameter adjustment of image acquisition equipment (such as focal length, angle, resolution, etc.), selection and parameter setting of image processing algorithms, changes in VR display effects, etc.

[0036] Communication transmission: Responsible for data transmission and communication between various systems. Use high-speed and stable communication protocols to ensure real-time transmission of EEG signals, image data and control instructions. At the same time, considering the security and privacy of the system, the transmitted data is encrypted.

[0037] Image analysis and processing method based on mind control 1. EEG signal acquisition and preprocessing: The user wears an EEG sensor, which starts to collect EEG signals. During the acquisition process, the user generates specific thoughts, and electrical activity is generated in the corresponding area of ​​the brain.

[0038] The collected EEG signals include EEG signals and fNIRS signals.

[0039] This algorithm uses a multimodal brain-computer interface signal acquisition method, which not only collects traditional electroencephalogram (EEG) signals, but also combines functional near-infrared spectroscopy (fNIRS) signals. EEG signals can reflect the electrical activity of brain neurons and have high temporal resolution; fNIRS signals can detect changes in blood oxygenation in the cerebral cortex, provide information on brain metabolic activity, and have good spatial resolution. By simultaneously acquiring these two signals, more comprehensive brain activity information can be obtained.

[0040] The collected EEG signals are transmitted to the preprocessing module for filtering, amplification and digital conversion to remove noise interference, enhance signal strength and prepare for subsequent feature extraction.

[0041] Specifically, an adaptive filtering algorithm can be used for preprocessing. For the collected EEG and fNIRS signals, adaptive filtering algorithms are used for preprocessing respectively. For EEG signals, the adaptive filter automatically adjusts the filtering parameters according to the spectral characteristics and noise characteristics of the signal to remove power frequency interference (50Hz or 60Hz), electromyographic artifacts and other noise. For fNIRS signals, adaptive filtering can remove interference caused by body movement, changes in ambient light, etc., and improve the signal-to-noise ratio of the signal.

[0042] 2. Thought analysis: extract features from the preprocessed EEG signals, and then perform thought classification and command mapping.

[0043] The process of extracting features from the EEG signal is as follows: features are jointly extracted from the preprocessed EEG signal and the fNIRS signal, wherein the EEG signal is subjected to feature extraction using a method combining wavelet transform and power spectral density analysis, and the fNIRS signal is subjected to feature calculation of changes in the concentrations of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR).

[0044] In order to reduce the dimension of feature vectors and improve the efficiency and accuracy of subsequent classification, a feature selection algorithm based on genetic algorithm is adopted. This algorithm regards the selection of feature subsets as an optimization problem and selects the feature subset that best represents the idea information by simulating the biological evolution process.

[0045] The process of thought classification and instruction mapping is as follows: construct a hybrid classification model to classify the extracted features, the hybrid classification model combines support vector machine and deep neural network, and maps thoughts into specific control instructions according to the classification results. The mapping is performed according to a predefined thought instruction mapping table.

[0046] 3. Image acquisition and processing: According to the user's mind control instructions, the image acquisition device obtains the corresponding image information. For example, if the user wants to observe a scene in a certain direction, the system will adjust the angle of the camera to shoot.

[0047] The collected images are processed, including denoising, enhancement, segmentation, target detection and other operations. Areas or objects of interest are highlighted according to the user's personalized needs. For example, if the user is interested in an object, the system will separate the object from the background through image segmentation and target detection algorithms, and enhance the display.

[0048] 4. VR display and interactive feedback The processed images are converted into a format suitable for VR device display and displayed on the VR device. Users can change the viewing angle by moving their heads and immerse themselves in the virtual image environment.

[0049] When the user performs a mind control operation, the system updates the VR display content in real time and allows the user to feel the result of the operation through visual, auditory or tactile feedback. For example, when the user selects a certain image area, the system will provide feedback through highlighting, sound prompts or tactile vibrations.

[0050] The method also includes: image quality optimization and adaptive adjustment mechanism 1. Image quality evaluation indicators: Objective evaluation indicators such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), mean square error (MSE), etc. are used to measure the quality of the image. These indicators can quantitatively evaluate the changes in the image during the denoising and enhancement processes, providing a basis for image quality optimization.

[0051] Combined with subjective evaluation indicators, image quality is evaluated through the user's subjective feelings. For example, the user's evaluation of image clarity, color reproduction, realism, and comfort and immersion in the VR environment.

[0052] 2. Adaptive adjustment strategy The parameters of the thought analysis model are dynamically adjusted according to the stability and accuracy of the EEG signal. If the EEG signal is disturbed or unstable, the system will automatically reduce the sensitivity to thought control to avoid misoperation; conversely, if the EEG signal quality is good, the system will increase the sensitivity to achieve more precise control.

[0053] Based on the image quality assessment results and user feedback, the parameters of the image processing algorithm are adaptively adjusted. For example, if the image has low contrast, the system will automatically increase the parameters of the contrast enhancement algorithm; if the user is not satisfied with a certain aspect of the image, the system will adjust the corresponding processing algorithm based on the user's feedback.

[0054] Optimize VR display effects based on the performance of VR devices and user operating habits. For example, for VR devices with lower performance, reduce the image resolution and frame rate to ensure smoothness; for users who are accustomed to fast operations, provide faster response speed and simpler interactive interface.

[0055] 1. Algorithm implementation and device integration 1. Algorithm Implementation In the computer system, a programming language (such as Python) is used to implement the mind control algorithm based on the brain-computer interface. For the signal acquisition part, the software development kit (SDK) of the existing EEG and fNIRS equipment is used for data acquisition. For example, for EEG equipment, the equipment of Neuroscan and its SDK can be used, and for fNIRS equipment, the equipment of Hitachi and its SDK can be used.

[0056] According to the algorithm flow, write functions to implement adaptive filtering, feature extraction, feature selection, classification, and instruction mapping. When implementing adaptive filtering, write the filtering algorithm code according to signal processing theory. For example, use the least mean square (LMS) algorithm to implement the coefficient update of the adaptive filter. For wavelet transform in feature extraction, you can use the PyWavelets library. For the construction of DNN, you can use deep learning frameworks such as TensorFlow or PyTorch.

[0057] 2. Device Integration Integrate the EEG electrode cap and fNIRS probe into a wearable device that is easy for the subject to wear. Ensure that the EEG electrodes and fNIRS probe are positioned accurately and stably to avoid displacement during use.

[0058] The wearable device is connected to the computer system through wired (such as USB interface) or wireless (such as Bluetooth) methods to achieve data transmission. The computer system, as the center of data processing and command transmission, establishes communication connections with external controlled devices (such as wheelchairs, prostheses, smart home devices, etc.) and controls the operation of external devices according to the instructions generated by the algorithm.

[0059] (II) Algorithm training and optimization 1. Data collection and annotation A large amount of EEG and fNIRS signal data is collected for algorithm training. During the collection process, the subjects are required to perform different mental tasks, such as imagining different limb movements, thinking about different operating instructions, etc. At the same time, the collected data is accurately labeled, and the mental category corresponding to each data segment is marked.

[0060] 2. Model Training Use the labeled data to train the hybrid classification model (SVM and DNN). For SVM, select the appropriate kernel function (such as radial basis function) and parameters (such as penalty factor, kernel width, etc.), and determine the optimal parameter combination through cross-validation and other methods. For DNN, set the network structure (such as the number of layers, the number of neurons in each layer, etc.), use the back propagation algorithm and optimizer (such as Adam optimizer) to train the network, and adjust the network weights to achieve the best classification accuracy of the model.

[0061] 3. Optimization and adjustment In the actual application of the algorithm, the algorithm is optimized and adjusted according to user feedback and usage. For example, if it is found that the accuracy of a user's thought classification is low, the model can be retrained by recollecting data for the user, or the parameters of the adaptive filter and the criteria for feature selection can be adjusted to improve the applicability of the algorithm to the user.

[0062] (III) Troubleshooting and emergency measures 1. Signal acquisition failure If the EEG electrode or fNIRS probe fails during signal acquisition (such as poor electrode contact, probe damage, etc.), the system will detect signal abnormalities (such as signal interruption, signal quality degradation, etc.). At this time, the system will display a prompt message on the user interface to inform the user of possible faults. The user can check the device connection, put on the device again, or replace the faulty part.

[0063] 2. Misclassification When the hybrid classification model makes a classification error (e.g., the idea of ​​"imagining upward movement" is misclassified as "imagining downward movement"), the system will record the error information and save the error data. The model can be retrained based on the error data, or the parameters of the classification model (such as the kernel function parameters of SVM, the weights of DNN, etc.) can be adjusted to improve the accuracy of classification.

[0064] 3. Instruction mapping exception If an exception occurs during the command mapping process (for example, the command corresponding to the current classification result does not exist in the mapping table), the system will execute a safety operation command by default (such as stopping the device from moving), and display an error message on the user interface to remind the user to check the settings of the idea-command mapping table or re-train the algorithm.

[0065] The above are only specific embodiments of the present invention, but the technical features of the present invention are not limited thereto. Any simple changes, equivalent replacements or modifications made based on the present invention to solve basically the same technical problems and achieve basically the same technical effects are all included in the protection scope of the present invention.

Claims

1. An image analysis and processing system based on mind control, characterized in that: include: The EEG signal acquisition system is used to collect the user's EEG signals and perform pre-processing; The thought analysis system extracts features from the preprocessed EEG signals, and then performs thought classification and command mapping; Image acquisition and processing system, used to obtain image information and perform image processing according to the user's thought control instructions; VR display system, used to convert processed images into a format suitable for VR device display and provide interactive feedback; The control and communication system is used to convert the state of thought into control instructions and realize data transmission and communication between various systems.

2. The image analysis and processing system based on mind control according to claim 1 is characterized in that: The EEG signal acquisition system adopts a non-invasive EEG sensor, arranges electrodes according to a system electrode placement method, and filters, amplifies and digitally converts the collected EEG signals.

3. The image analysis and processing system based on mind control according to claim 1 is characterized in that: The thought analysis system uses wavelet transform and power spectrum density analysis methods to extract EEG signal features and adopts machine learning algorithms for pattern recognition.

4. The image analysis and processing system based on mind control according to claim 1 is characterized in that: The image acquisition and processing system acquires image information through an image acquisition device, and uses an image processing algorithm to perform denoising, enhancement, segmentation, and target detection operations, and performs personalized processing based on the output of the thought analysis system.

5. The image analysis and processing system based on mind control according to claim 1, characterized in that: The VR display system converts images into a stereoscopic split-screen format or a panoramic format, changes the viewing angle through head movement, and provides interactive feedback such as vision, hearing or touch.

6. The image analysis and processing system based on mind control according to claim 1, characterized in that: The control and communication system converts the state of mind into control instructions such as image acquisition device parameter adjustment, image processing algorithm selection and parameter setting, VR display effect change, etc., uses a high-speed and stable communication protocol for data transmission, and encrypts the data.

7. An image analysis and processing method based on mind control, characterized in that: The following steps are involved: EEG signal acquisition and preprocessing: collect the user's EEG signals and preprocess them; Thought analysis: extract features from preprocessed EEG signals, and then perform thought classification and command mapping; Image acquisition and processing: acquiring image information and performing image processing according to the user's mind control instructions; VR display and interactive feedback: convert the processed images into a format suitable for VR device display and provide interactive feedback.

8. The image analysis and processing method based on mind control according to claim 7, characterized in that: The collected EEG signals include EEG signals and fNIRS signals, and the collected EEG signals and fNIRS signals are preprocessed using adaptive filtering algorithms respectively.

9. The image analysis and processing method based on mind control according to claim 7, characterized in that: The process of extracting features from the EEG signal is as follows: jointly extracting features from the preprocessed EEG signal and the fNIRS signal, wherein the EEG signal is subjected to a wavelet transform combined with a power spectral density analysis method to extract features, and the fNIRS signal is subjected to a calculation of the features of the changes in the concentrations of oxyhemoglobin and deoxyhemoglobin, and then a feature selection algorithm based on a genetic algorithm is used to select a feature subset that best represents the information of the intention; 10. The image analysis and processing method based on mind control according to claim 7, characterized in that: The process of thought classification and instruction mapping is as follows: construct a hybrid classification model to classify the extracted features, the hybrid classification model combines support vector machine and deep neural network, and maps thoughts into specific control instructions according to the classification results. The mapping is performed according to a predefined thought instruction mapping table.

Images