Ontology brain wave audio auditory perception synchronous feedback method, device and system and electronic equipment

Abstract

The invention relates to the technical field of electroencephalogram signal processing, in particular to an ontology brain wave audio auditory perception synchronous feedback method, device and system and electronic equipment. The method comprises the following steps: receiving electroencephalogram signals of a collected user on line from electroencephalogram collection equipment through an upper computer, obtaining electroencephalogram signals of a specified frequency band from the electroencephalogram signals, and extracting corresponding electroencephalogram characteristics; according to the electroencephalogram features or preset rhythm parameters, the electroencephalogram signals of the specified frequency band are segmented into a plurality of electroencephalogram segments, a plurality of audio expressions corresponding to the electroencephalogram segments are generated, and feature parameters of the audio expressions are determined according to the electroencephalogram features of the corresponding electroencephalogram segments; generating brain wave audio representation data according to the audio representation corresponding to the electroencephalogram signals of the one or more designated frequency bands, and obtaining brain wave audio according to the brain wave audio representation data. Therefore, the physiological suitability of nerve regulation and control and the artistic expressivity of audio generation are met at the same time, and organic unification of nerve regulation and control and audio generation is achieved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of electroencephalogram signal processing, and in particular to an ontology brain wave audio auditory perception synchronous feedback method, device, system and electronic equipment. BACKGROUND

[0002] In recent years, the development of brain-computer interface (BCI) technology has provided new possibilities for neural regulation and audio generation. Among them, brain-wave audio (BWA) as a kind of auditory stimulation generated based on brain neural activity signals (such as electroencephalogram EEG, magnetoencephalogram MEG, near-infrared fNIRS, etc.) has shown potential application value in the fields of neuroscience, psychological treatment and music computing. Brain-wave audio realizes the mapping of brain wave features and audio features (BwA()) through a brain-computer audio interface (BCAI), and then generates personalized auditory stimulation signals. As a subclass of BWA, brain-wave music (BWM) needs to further meet the synthesis requirements of music features and instrument timbres.

[0003] At present, brain-wave audio technology is mainly applied in two aspects: neural regulation and audio generation. In the application scenario of neural regulation, brain-wave audio stimulates the auditory cortex to affect the central nervous system activity, and has been used to improve sleep quality, regulate emotional state, etc., and has certain application prospects for the regulation or treatment of neural-related diseases; in the application scenario of audio generation, as a music generation means, brain-wave audio uses BCAI technology to establish a way from neural activity to music output, and uses brain wave signals to control the generation of music, which can be used as a music creation means and an entertainment means, and applied in barrier-free music creation.

[0004] However, the existing technology has certain functional limitations in the two application scenarios of neural regulation and audio generation, mainly reflected in:

[0005] For the application scenario of neural regulation, non-real-time means are mainly used to generate brain-wave music with music features, that is, fixed audio with music features is synthesized based on offline electroencephalogram data, and then repeated playback is used for long-time and multiple stimulation for neural regulation. This method is only suitable for offline application scenarios, lacks real-time feedback capability, and cannot respond to the dynamic changes of brain waves in real time and adjust the stimulation signal according to the dynamic changes of individual neural activity, resulting in a mismatch between the stimulation signal and the current neural state.

[0006] For audio generation application scenarios, the electroencephalogram under specific tasks (such as attention or motor imagination) is generally used as a control signal to control the music generation software to generate brain wave audio, that is, by using a limited combination of selectable music or routine (such as a pre-set melody fragment, harmonic progression, rhythm pattern, etc.), the electroencephalogram signal is only used to select or adjust between these options, so as to generate the music that the operator wants to generate, rather than a direct audio expression of brain activity. Compared with the individual electroencephalogram signal of the subject, the preset parameters selected by the operator have a greater impact on the audio synthesis effect. The brain wave audio generated in this way depends on the preset template and the active intention of the operator. The final output reflects the subjective preference or intention of the operator, which is the result of the operator "selecting" through the electroencephalogram signal, rather than the objective mapping of spontaneous brain activity. The output is only for music creation and cannot realize the regulation of neural activity, which reduces the neural representation of brain wave audio.

[0007] In summary, for the application of brain wave audio, in the current brain wave audio generation path, there is a lack of effective electroencephalogram signal processing means, the generation mapping relationship and generation effect of brain wave audio depend on subjective parameter setting and subjective scale evaluation, and there is a lack of objective mapping mechanism, so that the generation effect cannot both reproduce personalized characteristics stably and establish the essential correlation between neural activity characteristics and audio characteristics. The real-time, diversity and adaptive regulation possibility of brain wave audio generation have not been fully tapped, the physiological adaptability of neural regulation and the artistic expressiveness of audio generation cannot be met at the same time, the organic unity of the neural regulation and the audio generation process has not been realized, resulting in that the existing brain wave audio technology cannot fully exert the therapeutic potential of neural regulation and is difficult to realize creative audio expression based on real neural activity, and the application effect is fundamentally restricted by artificial intervention and static methods.

[0008] How to simultaneously meet the physiological adaptability of neural regulation and the artistic expressiveness of audio generation and realize the organic unity of the neural regulation and the audio generation process is a problem to be solved. SUMMARY

[0009] To solve the problems in the related art, the embodiments of the present disclosure provide a method, device and system for auditory perception synchronous feedback of brain wave audio and an electronic device.

[0010] In a first aspect, the embodiments of the present disclosure provide a method for auditory perception synchronous feedback of brain wave audio, which is applied to a host computer, the host computer is connected to an electroencephalogram acquisition device through an electroencephalogram data transmission interface, and the method comprises the following steps.

[0011] receiving, from the EEG acquisition device via the EEG data transmission interface, an EEG signal of a user being acquired, the EEG signal being acquired by the EEG acquisition device via one or more electrode channels, wherein each electrode channel corresponds to one electrode or a combination of multiple electrodes;

[0012] acquiring one or more specified frequency band EEG signals from the EEG signal, and extracting EEG features corresponding to the specified frequency band EEG signals based on the one or more specified frequency band EEG signals, the specified frequency band EEG signals referring to EEG signals in a specified frequency band, the EEG features including EEG time domain features and / or EEG frequency domain features;

[0013] for the specified frequency band EEG signals, segmenting the specified frequency band EEG signals into a plurality of EEG segments according to the EEG features or preset rhythm parameters based on a specified audio representation form and a specified audio generation manner, and generating a plurality of audio representations corresponding to the plurality of EEG segments respectively, a characteristic parameter of the audio representation being determined according to EEG features of a corresponding EEG segment; the audio representation form includes a note form and / or a waveform form, wherein the note form corresponds to an audio representation of a note, and the waveform form corresponds to an audio representation of a waveform; the audio generation manner includes a direct mapping manner and / or an indirect control manner using control parameters;

[0014] generating brain wave audio representation data according to the audio representations corresponding to the one or more specified frequency band EEG signals;

[0015] obtaining brain wave audio according to the brain wave audio representation data.

[0016] According to an embodiment of the present disclosure, when the specified audio representation form is a note form and the specified audio generation manner is a direct mapping manner, the characteristic parameter of the audio representation includes a note duration, a pitch, and a volume; and the segmenting the specified frequency band EEG signals into a plurality of EEG segments according to preset rhythm parameters and generating a plurality of audio representations corresponding to the plurality of EEG segments respectively includes:

[0017] for a preset duration of EEG signals in the specified frequency band EEG signals of a specified electrode channel:

[0018] segmenting the EEG signals of the preset duration according to a note duration indicated by the preset rhythm parameters to obtain an EEG segment set, the EEG segment set including a plurality of EEG segments, and a single EEG segment corresponding to a note;

[0019] According to the EEG segment corresponding to the EEG segment in the EEG segment set, the pitch and intensity of the note corresponding to the EEG segment are determined according to the EEG frequency domain characteristics corresponding to the EEG segment, until the pitch and intensity of the note corresponding to each EEG segment in the EEG segment set are obtained, including: based on the power spectrum characteristics corresponding to the EEG segment, selecting the peak frequency or weighted average frequency with the largest amplitude in the power spectrum characteristics as the characteristic frequency, obtaining the corresponding specified pitch range according to the frequency range of the specified frequency band based on the preset mapping mode, and obtaining the pitch corresponding to the EEG segment according to the characteristic frequency and the specified pitch range; the intensity corresponding to the EEG segment is generated according to the total energy or peak amplitude of the power spectrum characteristics; the preset mapping mode is used to describe the mapping relationship between the frequency range of the specified frequency band and the pitch range.

[0020] According to an embodiment of the present disclosure, the brain wave audio representation data is generated according to the audio performance of the one or more specified frequency band EEG signals, including:

[0021] The notes corresponding to each EEG segment in the EEG segment set of the EEG segment of the EEG signal of the preset time length in each electrode channel are written into the music score corresponding to the specified frequency band;

[0022] The music score corresponding to each specified frequency band in each electrode channel is written into a complete music score set corresponding to the preset time length, and the complete music score set is taken as the brain wave audio representation data;

[0023] The brain wave audio is obtained according to the brain wave audio representation data, including:

[0024] According to the preset frequency band-instrument mapping table or configuration parameter-instrument mapping table, the instrument is configured for the music score corresponding to each specified frequency band in the brain wave audio representation data, the frequency band-instrument mapping table is used to describe the corresponding relationship between the frequency band and the instrument, and the configuration parameter-instrument mapping table is used to describe the corresponding relationship between the specified instrument configuration parameter and the instrument, and the specified instrument configuration parameter includes: non-brain-derived control parameter and / or non-brain-derived event occurrence frequency;

[0025] Wherein, the non-brain-derived control parameter is obtained according to the following manner: when the non-brain-derived signal and the denoised EEG signal are contained in the EEG signal of the specified electrode channel with the preset length, the non-brain-derived control parameter is obtained according to the ratio of the preset quantile of the absolute values of the non-brain-derived signal and the denoised EEG signal in the EEG signal of the specified electrode channel with the preset length;

[0026] The non-brain-derived occurrence frequency is obtained in the following manner: a non-brain-derived event threshold is obtained according to a preset quantile of the absolute value of the denoised electroencephalogram signal, positions in the non-brain-derived signal that are higher than the non-brain-derived event threshold are counted, the number of non-brain-derived events in the non-brain-derived signal is obtained, and the non-brain-derived event occurrence frequency is obtained according to the number of non-brain-derived events and the preset time length.

[0027] The brainwave audio is generated by a MIDI synthesizer or an audio synthesis library according to a musical score corresponding to each specified frequency band in the brainwave audio representation data.

[0028] According to an embodiment of the present disclosure, when the specified audio representation form is a note form and the specified audio generation manner is an indirect control manner using control parameters, the characteristic parameters of the audio representation include: note duration, pitch, and intensity; and the step of dividing the specified frequency band electroencephalogram signal into a plurality of electroencephalogram segments according to the electroencephalogram features and generating a plurality of audio representations corresponding to the plurality of electroencephalogram segments respectively comprises:

[0029] The electroencephalogram signal of the preset time length in the specified frequency band electroencephalogram signal is taken as input, and a pre-trained user state discrimination model is used to generate control parameters, including: arousal and valence.

[0030] The note duration of a single note in a note sequence corresponding to the electroencephalogram signal of the preset time length is calculated according to the arousal and / or the valence.

[0031] The electroencephalogram signal of the preset time length is divided according to the note duration to obtain an electroencephalogram segment set, which contains a plurality of electroencephalogram segments, and a single electroencephalogram segment corresponds to a note.

[0032] The occurrence probability of a note corresponding to each electroencephalogram segment in the electroencephalogram segment set is calculated according to the arousal and / or the valence, and when the occurrence probability meets an occurrence condition, the note corresponding to the corresponding electroencephalogram segment appears, otherwise the note corresponding to the corresponding electroencephalogram segment does not appear; and a note sequence corresponding to the electroencephalogram signal of the preset time length is determined according to the calculation result.

[0033] The intensity of a note in the note sequence is calculated according to the arousal, the valence, and the power spectrum energy feature of the electroencephalogram segment corresponding to the note in the note sequence.

[0034] The tonality parameter of a note in the note sequence is calculated according to the arousal and / or the valence.

[0035] The pitch of a note in the note sequence is calculated according to the tonality parameter and the power spectrum frequency feature of the electroencephalogram segment corresponding to the note in the note sequence.

[0036] configure timbres of notes in the sequence of notes according to the wakefulness and / or the valence; or

[0037] configure timbres of all notes in the sequence of notes according to a preset configuration parameter-timbre mapping table; the preset configuration parameter-timbre mapping table is used to describe a correspondence between a specified timbre configuration parameter and a timbre, and the specified timbre configuration parameter includes a non-brain-derived control parameter and / or a non-brain-derived event occurrence frequency;

[0038] The non-brain-derived control parameter is obtained according to the following manner: when the non-brain-derived signal and the denoised electroencephalogram signal in the preset length of electroencephalogram signal of the specified electrode channel are contained, a ratio of preset quantile values of absolute values of the non-brain-derived signal and the denoised electroencephalogram signal in the preset length of electroencephalogram signal of the specified electrode channel is taken as the non-brain-derived control parameter.

[0039] The non-brain-derived event occurrence frequency is obtained according to the following manner: a non-brain-derived event threshold is obtained according to a preset quantile value of an absolute value of the denoised electroencephalogram signal, positions of the non-brain-derived signal higher than the non-brain-derived event threshold are counted to obtain a number of non-brain-derived event occurrences in the non-brain-derived signal, and the non-brain-derived event occurrence frequency is obtained according to the number of non-brain-derived events and the preset time length.

[0040] According to an embodiment of the present disclosure, the brain wave audio representation data is generated according to audio performances corresponding to the one or more specified frequency bands of electroencephalogram signals, and includes:

[0041] generate a musical score corresponding to each specified frequency band based on a sequence of notes in the specified frequency band of each electrode channel, a note intensity, a note pitch and a note timbre in the sequence of notes;

[0042] write the musical score corresponding to each specified frequency band in each electrode channel into a complete musical score set corresponding to the preset time length, and take the complete musical score set as the brain wave audio representation data;

[0043] The brain wave audio is obtained according to the brain wave audio representation data, and includes:

[0044] generate the brain wave audio according to the musical score corresponding to each specified frequency band in the brain wave audio representation data through a MIDI synthesizer or an audio synthesis library.

[0045] According to an embodiment of the present disclosure, when the specified audio representation form is a waveform form and the specified audio generation manner is a direct mapping manner, the characteristic parameters of the audio representation include: a fundamental frequency and a harmonic frequency, and an intensity of the fundamental frequency and an intensity of the harmonic frequency; the specified frequency band electroencephalogram signal is segmented into a plurality of electroencephalogram segments according to the electroencephalogram characteristics, and a plurality of audio representations corresponding to the plurality of electroencephalogram segments are generated, including:

[0046] The electroencephalogram signal of the specified frequency band electroencephalogram signal of the specified electrode channel for a preset time length:

[0047] Based on the time domain envelope corresponding to the electroencephalogram signal of the preset time length, the electroencephalogram signal of the preset time length is segmented to obtain an electroencephalogram segment set, including: determining an effective amplitude threshold according to the mean of the time domain envelope; determining a trough set of the time domain envelope according to the effective amplitude threshold, the trough amplitude in the trough set is not greater than the effective amplitude threshold; segmenting the electroencephalogram signal of the preset time length according to the preset effective time threshold and the trough set of the time domain envelope to obtain the electroencephalogram segment set, the electroencephalogram segment set contains one or more electroencephalogram segments, and the time value of each electroencephalogram segment is greater than the effective time threshold;

[0048] For any electroencephalogram segment in the electroencephalogram segment set, the characteristic parameters of the waveform corresponding to the any electroencephalogram segment are determined according to the electroencephalogram frequency domain characteristics corresponding to the any electroencephalogram segment, until the characteristic parameters of the waveform corresponding to each electroencephalogram segment in the electroencephalogram segment set are obtained, including: based on the power spectrum characteristics corresponding to the any electroencephalogram segment, the fundamental frequency and the harmonic frequency of the waveform corresponding to the any electroencephalogram segment and the intensity of the fundamental frequency and the intensity of the harmonic frequency are determined according to the peak frequency and amplitude of the first preset number of peak values in the power spectrum characteristics in the order of amplitude size.

[0049] According to an embodiment of the present disclosure, the brain wave audio representation data is generated according to the audio representation corresponding to the one or more specified frequency band electroencephalogram signals, including:

[0050] For any electroencephalogram segment in the electroencephalogram segment set, the audio waveform corresponding to the any electroencephalogram segment is determined according to the characteristic parameters of the waveform corresponding to the any electroencephalogram segment and the time domain envelope, until the audio waveform corresponding to each electroencephalogram segment in the electroencephalogram segment set is obtained, including: generating a sine wave of a corresponding frequency and mixing to obtain a mixed sine wave according to the fundamental frequency and the harmonic frequency of the waveform corresponding to the any electroencephalogram segment and the intensity of the fundamental frequency and the intensity of the harmonic frequency; modulating the mixed sine wave by applying the time domain envelope corresponding to the any electroencephalogram segment to obtain the audio waveform corresponding to the any electroencephalogram segment;

[0051] corresponding to each brain electrical segment in the brain electrical segment set of each specified frequency band in the specified electrode channel as the audio waveform corresponding to the brain electrical signal of the preset time length of the specified electrode channel;

[0052] corresponding to each brain electrical segment in the brain electrical segment set of each specified frequency band in the specified electrode channel as the audio waveform corresponding to the brain electrical signal of the preset time length of the specified electrode channel;

[0053] The brain wave audio is obtained according to the brain wave audio representation data.

[0054] The brain wave audio is obtained according to the brain wave audio representation data.

[0055] In a second aspect, the present disclosure provides a brain wave audio auditory perception synchronization feedback device, which is arranged in a host computer, and the host computer is connected with an electroencephalogram acquisition device through an electroencephalogram data transmission interface. The device comprises an electroencephalogram signal receiving module, an electroencephalogram feature extraction module, and an electroencephalogram-audio generation module, wherein

[0056] The electroencephalogram signal receiving module is configured to receive the electroencephalogram signal of a user being collected from the electroencephalogram acquisition device online through the electroencephalogram data transmission interface, and the electroencephalogram signal is acquired by the electroencephalogram acquisition device through one or more electrode channels, wherein each electrode channel corresponds to one electrode or a combination of multiple electrodes.

[0057] The electroencephalogram feature extraction module is configured to acquire one or more specified frequency band electroencephalogram signals in the electroencephalogram signal, and extract electroencephalogram features corresponding to the specified frequency band electroencephalogram signal based on the one or more specified frequency band electroencephalogram signals, wherein the specified frequency band electroencephalogram signal refers to the electroencephalogram signal in a specified frequency band, and the electroencephalogram features include electroencephalogram time domain features and / or electroencephalogram frequency domain features.

[0058] The electroencephalogram-audio generation module is configured to, for the specified frequency band electroencephalogram signal, based on a specified audio representation form and a specified audio generation method, segment the specified frequency band electroencephalogram signal into multiple electroencephalogram segments according to the electroencephalogram features or preset rhythm parameters, and generate multiple audio representations corresponding to the multiple electroencephalogram segments respectively, wherein the feature parameters of the audio representations are determined according to the electroencephalogram features of the corresponding electroencephalogram segments; the audio representation form includes a note form and / or a waveform form, wherein the audio representation corresponding to the note form is a note, and the audio representation corresponding to the waveform form is a waveform; the audio generation method includes a direct mapping method and / or an indirect control method using control parameters; brain wave audio representation data is generated according to the audio representations corresponding to the one or more specified frequency band electroencephalogram signals; and brain wave audio is obtained according to the brain wave audio representation data.

[0059] In a third aspect, the embodiments of the present disclosure provide an electronic device, including the apparatus of any one of the second aspect, or including a memory and a processor; wherein the memory is configured to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the method of any one of the first aspect.

[0060] In a fourth aspect, the embodiments of the present disclosure provide a body brain wave audio auditory perception synchronization feedback system, the system comprising: an electroencephalogram acquisition device and the electronic device of the third aspect, the electroencephalogram acquisition device is connected with the electronic device through an electroencephalogram data transmission interface, wherein:

[0061] The electroencephalogram acquisition device is configured to acquire the electroencephalogram signal of the collected user through one or more electrode channels, and transmit the electroencephalogram signal of the one or more electrode channels to the electronic device online based on the electroencephalogram data transmission interface.

[0062] In a fifth aspect, the embodiments of the present disclosure provide a computer readable storage medium, having stored thereon computer instructions, the computer instructions being executed by a processor to implement the method of any one of the first aspect.

[0063] In a sixth aspect, the embodiments of the present disclosure provide a computer program product, comprising a computer program, characterized in that the computer program is executed by a processor to implement the method of any one of the first aspect.

[0064] According to the technical scheme provided by the embodiment of the present disclosure, the host computer receives the brain electrical signals of the collected user from the brain electrical signal collection device online, obtains one or more specified frequency band brain electrical signals in the brain electrical signals, and extracts brain electrical features corresponding to the specified frequency band brain electrical signals. Based on the specified audio form and the specified audio generation method, the specified frequency band brain electrical signals are segmented into multiple brain electrical segments according to the brain electrical features or the preset rhythm parameters, and multiple audio performances corresponding to the multiple brain electrical segments are generated. The characteristic parameters of the audio performances are determined according to the brain electrical features of the corresponding brain electrical segments. The brain wave audio representation data is generated according to the audio performances corresponding to the one or more specified frequency band brain electrical signals. The brain wave audio is obtained according to the brain wave audio representation data. When the brain wave audio is fed back to the collected user synchronously, the nervous system activity of the collected user can be dynamically regulated in real time through auditory perception, and the precise auditoryization of physiological signals is realized. In addition, by determining the characteristic parameters of the corresponding audio performances according to the brain electrical features of each brain electrical segment, the brain wave audio generated based on the audio performances can accurately match the brain electrical features of the collected user, thereby enhancing the physiological adaptability of neural feedback and overcoming the limitations of traditional preset music. At the same time, the brain electrical features are generated based on the online real-time acquired brain electrical signals, so that through the closed-loop brain electrical-audio real-time feedback interaction, the high synchronization of neural activity and auditory feedback is realized, and the physiological adaptability of neural regulation and the artistic expressiveness of audio generation are simultaneously satisfied, and the organic unity of neural regulation and audio generation is realized.

[0065] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0066] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of the non-limiting embodiments in conjunction with the accompanying drawings. In the drawings:

[0067] Figure 1 A flowchart of a method for synchronously feeding back a body brain wave audio auditory perception according to an embodiment of the present disclosure is shown;

[0068] Figure 2 A flowchart of another method for synchronously feeding back a body brain wave audio auditory perception according to an embodiment of the present disclosure is shown;

[0069] Figure 3 A structural schematic diagram of a brain-derived signal and non-brain-derived signal separation network model according to an embodiment of the present disclosure is shown;

[0070] Figure 4 A flowchart of a mapping-symbol audio synthesis method according to an embodiment of the present disclosure is shown;

[0071] Figure 5A flow chart showing a control-type-symbolic audio synthesis method according to an embodiment of the present disclosure;

[0072] Figure 6 A flow chart showing a mapping-type-waveform audio synthesis method according to an embodiment of the present disclosure;

[0073] Figure 7 A schematic diagram showing a division result of an electroencephalogram segment in a mapping-type-waveform audio synthesis method according to an embodiment of the present disclosure;

[0074] Figure 8 A flow chart showing yet another embodiment of the body brainwave audio auditory perception synchronous feedback method according to the present disclosure;

[0075] Figure 9 A schematic diagram showing a sound intensity position display of a brainwave audio of a specified frequency band according to an embodiment of the present disclosure;

[0076] Figure 10 A schematic diagram showing a structure of a body brainwave audio auditory perception synchronous feedback device according to an embodiment of the present disclosure;

[0077] Figure 11 A schematic diagram showing a structure of another body brainwave audio auditory perception synchronous feedback device according to an embodiment of the present disclosure;

[0078] Figure 12 A schematic diagram showing a structure of yet another body brainwave audio auditory perception synchronous feedback device according to an embodiment of the present disclosure;

[0079] Figure 13 A structure block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0080] Figure 14 A structure block diagram of another electronic device according to an embodiment of the present disclosure is shown;

[0081] Figure 15 A structure block diagram of yet another electronic device according to an embodiment of the present disclosure is shown;

[0082] Figure 16 A structure block diagram of still another electronic device according to an embodiment of the present disclosure is shown;

[0083] Figure 17 A structure block diagram of a body brainwave audio auditory perception synchronous feedback system according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0084] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so as to be easily implemented by those skilled in the art. Also, parts irrelevant to the description of the exemplary embodiments are omitted in the accompanying drawings for the sake of clarity.

[0085] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate that there are features, numbers, steps, actions, parts or combinations thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, actions, parts or combinations thereof exist or are added.

[0086] It should also be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0087] As described above, in the conventional technology, the application of brainwave audio is often divided into two independent directions: one is offline neuroregulation based on a fixed template, which lacks real-time and individual adaptability; the other is music generation relying on subjective preset, which is difficult to truly reflect spontaneous brain activity. Thus, it is impossible to simultaneously meet the physiological adaptability of neuroregulation and the artistic expressiveness of audio generation, and the organic unification of the neuroregulation and audio generation process is not achieved, resulting in that the existing brainwave audio technology cannot fully exert the therapeutic potential of neuroregulation, nor can it realize the creative audio expression based on real neural activity.

[0088] In order to overcome the functional limitations of the existing technology in the field of brainwave audio application, the present disclosure proposes an organic brainwave audio auditory perception synchronous feedback method, which realizes the organic unification of neuroregulation and audio generation through real-time online electroencephalogram signal acquisition, electroencephalogram feature extraction, dynamic brainwave audio generation and feedback. The "organic" in the present disclosure refers to the collected user's body, emphasizing the direct collection of the user's own electroencephalogram signal, rather than an analog signal or someone else's electroencephalogram signal, and aiming at the unique neural activity pattern of the collected user, generating corresponding dynamic brainwave audio based on the real-time electroencephalogram features of the collected user, rather than standardized processing "one size fits all", and then through real-time "synchronous feedback" and "auditory perception", the brainwave audio feedback is directly applied to the signal provider (the collected user himself), helping the collected user to perceive and adjust his own brain state (such as concentration or relaxation), forming a "self-regulation" closed loop, rather than one-way output (such as brain-controlled composition), and strengthening the role of "organic sensation" in neuroregulation.

[0089] The real-time brain wave audio of a specific individual in the present disclosure directly stimulates the auditory cortex located in the body brain through the auditory system, triggers auditory perception and auditory cognitive response of the central nervous system through the auditory neural circuit, and forms the body brain wave audio auditory perception synchronous feedback. Compared with non-body brain wave audio, the difference of the body brain wave audio is that the change of brain network connection relationship triggered by auditory perception and cognition will cause the change of the body brain wave audio through the brain wave-audio feature mapping relationship, forming a brain wave-audio-stimulation-brain wave closed loop process, and further meeting the demand of brain wave audio closed loop regulation in real-time neural regulation application scenarios.

[0090] Figure 1 A flow chart of a body brain wave audio auditory perception synchronous feedback method according to an embodiment of the present disclosure is shown. The method is applied to a host computer connected with an electroencephalogram acquisition device through an electroencephalogram data transmission interface.

[0091] As shown in Figure 1 The body brain wave audio auditory perception synchronous feedback method includes the following steps S110-S150:

[0092] In step S110, the electroencephalogram signal of the collected user is received online from the electroencephalogram acquisition device through the electroencephalogram data transmission interface, and the electroencephalogram signal is obtained by the electroencephalogram acquisition device through one or more electrode channels, wherein each electrode channel corresponds to one electrode or a combination of multiple electrodes.

[0093] The electroencephalogram acquisition device in the present disclosure is a special device for receiving neuron electrical activity on the scalp. The weak electroencephalogram signal is collected from the scalp surface through the electrode, and then the original signal is amplified, filtered, analog-digital converted and processed. After processing, the processed electroencephalogram signal is transmitted to the host computer through the electroencephalogram data transmission interface for analysis and processing.

[0094] The EEG acquisition device in the present disclosure can be a portable / wearable EEG acquisition device (such as a dry electrode headset), or a large fixed device, depending on the specific design and application scenario. In a specific example, the EEG acquisition device in the present disclosure adopts a portable EEG acquisition device and a real-time EEG data transmission interface to realize real-time acquisition and data acquisition of scalp EEG. The portable EEG acquisition device is a multi-modal human factor signal acquisition system specially designed for real-time and lightweight neural signal monitoring, which can support simultaneous acquisition of multiple modalities of human electrical signals, and adopts wireless transmission (such as Bluetooth / Wi-Fi) or lightweight wired design. The EEG signal uses frontal lobe single electrode 5 electrode channels, and the specific electrode positions are {F7, Fp1, Fpz, Fp2, F8}. The electrode positions meet the international standard 10 / 20 electrode arrangement, and the sampling rate is 1000 Hz. Based on the device supporting software development kit (SDK), low delay and high reliability transmission of EEG signal from the acquisition device to the host computer is realized, and real-time transmission of EEG acquisition device signal to the host computer is realized. When transmitting the EEG signal, the transmission interval is set to 30 ms.

[0095] The "electrode channel" in the present disclosure refers to an independent signal acquisition path in the EEG acquisition system. Each electrode channel (such as F7, Fp1, etc.) records the integrated electrical signal of the neuron activity at that position. In the corresponding relationship with the electrode, it can be a one-to-one mode, that is, a single electrode corresponds to an independent electrode channel (such as Fpz electrode → Channel1), which directly records the potential at that point (relative to the reference electrode). It can also be a many-to-one mode, that is, multiple electrodes are combined to form an electrode channel (such as F7+F8→ bipolar lead Channel X), which records the potential difference between the two points. In the specific example as above, each recording electrode is an independent channel (a total of 5 electrode channels), and the reference electrode is usually the earlobe (A1 / A2) or the average reference. Although electrodes at different positions may be more sensitive to certain frequency bands, in general, the EEG signal acquired by each electrode channel covers all frequency bands (Delta, Theta, Alpha, Beta, Gamma).

[0096] The host computer in the present disclosure can be directly connected with the electroencephalogram acquisition device through the electroencephalogram data transmission interface when the host computer is connected with the electroencephalogram acquisition device through the electroencephalogram data transmission interface, at which time the host computer is the corresponding matching device of the electroencephalogram acquisition device and has the processing capability of generating electroencephalogram audio according to the collected electroencephalogram signals by executing steps S110-S150; the host computer can also be indirectly connected with the electroencephalogram acquisition device through the electroencephalogram data transmission interface based on the corresponding matching device of the electroencephalogram acquisition device, that is, the electroencephalogram acquisition device is connected with its matching device through the electroencephalogram data transmission interface, and then the matching device is connected with the host computer through wireless or wired communication connection mode, so as to transmit the collected electroencephalogram signals to the host computer for analysis and processing, in which case the host computer corresponds to an electroencephalogram signal processing terminal.

[0097] When the electroencephalogram acquisition device in the specific example above is used to collect the electroencephalogram signals of a user, the electroencephalogram acquisition device collects the electroencephalogram signals through 5 electrode channels at a frequency of 1000 data points per second, and then sends a data packet to the host computer every 30 ms, that is, each transmission contains 30 ms x 1000 Hz = 30 continuous signals of time points, and the data format can be a numerical matrix, such as a 5 x 30 matrix, each row corresponding to a voltage sequence of an electrode channel. When the host computer obtains the electroencephalogram signals, it extracts the electroencephalogram signals of each electrode channel by analyzing the received numerical matrix.

[0098] Generally, the electroencephalogram signals collected by the electroencephalogram acquisition device are easily contaminated by eye movement artifacts (EOG) and muscle artifacts (EMG), and if the eye movement artifacts are not removed, blinking may cause sudden high noise points in the audio, and if the muscle artifacts are not removed, the rhythm of the generated audio may be disturbed by the activity of the jaw muscles. Therefore, in the case that the electroencephalogram acquisition device and its SDK do not contain real-time eye movement artifact and muscle artifact removal function modules, in order to ensure that the electroencephalogram signals reflect the true neural activity rather than physiological artifacts, avoid misjudgment or misoperation in medical diagnosis, BCI control and other scenarios, and ensure the effect of neural regulation and audio generation, the electroencephalogram signals need to be denoised before subsequent analysis and processing to obtain brain-derived signals that do not contain eye movement artifacts and muscle artifacts.

[0099] Figure 2 A flowchart of another method for synchronously feeding back auditory perception of body brain waves according to an embodiment of the present disclosure is shown. As shown in Figure 2 Before step S120, that is, before obtaining one or more specified frequency band electroencephalogram signals in the electroencephalogram signals, the following step S111 is also included:

[0100] In step S111, the pre-trained brain-derived signal and non-brain-derived signal separation network model is used to preprocess the electroencephalogram signals of the one or more electrode channels to obtain denoised electroencephalogram signals.

[0101] Existing techniques typically use ICA (Independent Component Analysis) based on blind source separation to decompose mixed signals into independent components. The electrooculogram (EOG) and electromyography (EMG) components are then manually or automatically removed. While ICA can separate complex mixed signals, it requires offline analysis and high computational complexity. It also has requirements for the number of independent electrodes and signal length (typically requiring data segments of 1-5 minutes), resulting in high latency and difficulty meeting the requirements of real-time EEG processing, making it unsuitable for real-time automated applications.

[0102] The present disclosure uses deep learning technology to provide a network model for separating brain-derived signals from non-brain-derived signals. The pre-trained network model can automatically and effectively remove noise components such as electrooculogram and electromyography in EEG signals in real time.

[0103] Figure 3 FIG. 1 is a schematic diagram showing a structural diagram of a network model for separating brain-derived signals from non-brain-derived signals according to an embodiment of the present disclosure. Figure 3 As shown, the brain-derived signal and non-brain-derived signal separation network model includes: a denoising module and a jump connection module, and the denoising module includes: a decomposition module, a channel spatiotemporal attention processing module and a reconstruction module.

[0104] According to an embodiment of the present disclosure, the preprocessing of the EEG signals of the one or more electrode channels using the pretrained brain-derived signal and non-brain-derived signal separation network model to obtain the denoised EEG signals includes:

[0105] The denoising module removes non-brain-derived signals from the EEG signals of the one or more electrode channels, and the specific process includes the following steps:

[0106] First, the decomposition module decomposes the EEG signals of the one or more electrode channels into multi-dimensional embedding vectors corresponding to multiple signal channels, and the multiple signal channels include brain-derived signal channels and non-brain-derived signal channels.

[0107] The non-brain-derived signals include but are not limited to electrooculographic signals and / or electromyographic signals.

[0108] Specifically, the following formula is first used to map the EEG signals of one or more electrode channels into a multidimensional space through a set of learnable convolution kernels:

[0109] V = Conv1D(X, W d )∈R D×T ;

[0110] Where X is the EEG signal of one or more electrode channels, X∈R C×T, C is the number of electrode channels, T is the number of time points, W d is the convolution kernel weight matrix, W d ∈R D×C×K , D is the number of signal channels, and K is the convolution kernel time length. The convolution operation stride defaults to 1, and padding is set to "same" to keep the time dimension T unchanged. Conv1D is a one-dimensional convolution operation, which extracts local features from time series signals. It is used to map EEG signals into a multidimensional embedding space (D dimensions), implicitly separating different signal components (e.g., brain-derived / non-brain-derived). V is the multidimensional embedding vector.

[0111] Then the multi-dimensional embedding vector is separated into the embedding vector V of the brain-derived signal channel according to a preset ratio. b and the embedding vector V of the non-brain-derived signal channel n ,in, D b is the number of brain-derived signal channels, D n is the number of non-brain-derived signal channels, D = D b +D n , V=V b +V n .

[0112] Afterwards, the channel-spatiotemporal attention processing module performs channel attention processing and spatiotemporal attention processing on the multidimensional embedding vector, respectively, and generates channel attention weights and temporal attention weights, respectively. The generated channel attention weights and temporal attention weights are fused with the multidimensional embedding vectors corresponding to multiple signal channels to obtain the processed multidimensional embedding vector.

[0113] In the present disclosure, the channel spatiotemporal attention processing module enhances the weight of brain-derived signal channels through the attention mechanism and captures temporal dependencies. Specifically, it includes channel attention processing and spatiotemporal attention processing.

[0114] When performing channel attention processing, the channel attention weight α is calculated using the following formula:

[0115] α=σ(W2·ReLU(W1·GAP(V)));

[0116] where GAP(·) denotes global average pooling, W1 and W2 denote the first and second layer fully connected weights, respectively, the first layer fully connected weight is used to reduce the dimension of the channel descriptor after global average pooling (GAP) to a low-dimensional space, the design purpose is to reduce the amount of calculation and extract the nonlinear interaction information between channels; the second layer fully connected weight is used to map the reduced features back to the original dimension, the design purpose is to reconstruct the dependence relationship between signal channels, generate signal channel by signal channel attention weight, W1∈R D / r×D , W2∈R D×D / r , r denotes the reduction ratio, ReLU is the rectified linear unit used to output the nonlinear features after dimension reduction, and σ denotes the Sigmoid activation function.

[0117] The role of introducing ReLU in channel attention includes:

[0118] 1. Introducing nonlinearity: enabling the model to learn complex interactions between signal channels rather than simple linear combinations.

[0119] 2. Sparse activation: suppress negative features and highlight the contributions of important signal channels.

[0120] 3. Preventing gradient vanishing: compared to Sigmoid / Tanh, ReLU has a constant gradient of 1 in the positive interval, which alleviates the training problems of deep networks.

[0121] When performing spatio-temporal attention processing, first perform linear projection to generate query (Query) matrix Q, key (Key) matrix K, and value (Value) matrix V value , which is specifically performed using the following formula:

[0122] Q=VW q ;

[0123] K=VW k ;

[0124] V value =VW v ;

[0125] where W q , W k , and W v are projection weights, which are automatically learned through the training process (such as backpropagation and gradient descent), d k is the attention dimension, usually d k =D / 2.

[0126] Then, the time attention weight β is calculated using the following formula:

[0127]

[0128] Then, the signal weight and the time attention weight of each generated signal channel are fused with the multi-dimensional embedding vector of the corresponding signal channel by the following formula to obtain a processed multi-dimensional embedding vector V out :

[0129] V channel = α ⊙ V;

[0130] V time = βV value ∈ R D×T ;

[0131] V out = V + V channel + V time ;

[0132] Where ⊙ represents element-wise multiplication.

[0133] Next, the reconstruction module generates a reconstructed electroencephalogram signal X rec :

[0134] X rec = ConvTranspose1D (V out , W r ) ∈ R C×T ;

[0135] Where ConvTranspose1D is a one-dimensional deconvolution operation used to reconstruct the original electroencephalogram signal space from the multi-dimensional embedding vector, W r is the deconvolution kernel, and W r ∈ R C×D×K .

[0136] After obtaining the reconstructed electroencephalogram signal through the above steps, the processed multi-dimensional embedding vector is obtained through the skip connection module, and based on the processed multi-dimensional embedding vector and the electroencephalogram signal of the one or more electrode channels, a dynamic fusion parameter is generated by the gate linear unit GLU and the noise perception module NAM. The dynamic fusion parameter includes a preliminary weight and a noise level.

[0137] Specifically, the preliminary weight γ is first generated by the gate linear unit GLU by the following formula:

[0138] γ = σ (W g · [V out ; X]) + b g ;

[0139] Where W g is the weight matrix of GLU, and W g ∈ RC×(D+C) ; [V out ; X] is used to splice V out and the brain electrical signals X of the one or more electrode channels, [V out ; X] ∈ R (D+C)×T , b g is a bias term, b g ∈ R C , and σ is a Sigmoid function used to compress the preliminary weight to [0, 1].

[0140] Then, the noise perception module NAM analyzes the residual signal to estimate the noise level η by using the following formula:

[0141] η = MLP (Std (X-X rec ));

[0142] wherein η ∈ R G×T , Std(·) represents calculating the noise standard deviation of each signal channel, and MLP is a multi-layer perception machine. The greater the value of η quantifying the noise level, the more significant the noise of the signal channel, and the need to reduce the fusion ratio of the brain electrical signals of the one or more electrode channels.

[0143] Finally, the one or more electrode channels of the brain electrical signals are fused with the reconstructed brain electrical signals according to the dynamic fusion parameter through the jump connection module, to obtain the denoised brain electrical signals, and the dynamic fusion parameter is used to control the fusion ratio of the one or more electrode channels of the brain electrical signals and the reconstructed brain electrical signals.

[0144] Specifically, the denoised brain electrical signals X final can be obtained according to the preliminary weight γ and the noise level η by using the following formula:

[0145] X final = γ ⊙ X rec + (1-γ-ρη) ⊙ X;

[0146] wherein ρ represents a noise suppression coefficient, used to control the strength of noise perception.

[0147] The brain-derived signal and non-brain-derived signal separation network model provided by the present disclosure has a synergistic mechanism of deep decomposition, channel attention mechanism and dynamic fusion, can automatically identify noise / clean signal segments, does not need to pre-label noise segments, realizes end-to-end real-time adaptive processing, has certain advantages compared with traditional ICA in terms of calculation efficiency (real-time), adaptive ability (automation) and signal-to-noise ratio improvement (effectiveness), and is particularly suitable for online electroencephalogram processing systems. In addition, the skip connection module in the present disclosure uses the technology of GLU (Gated Linear Unit) and NAM (Noise Perception Module) to cooperatively generate dynamic fusion parameters, which not only allows the electroencephalogram signal segment without noise to pass through the network model directly, avoiding distortion caused by unnecessary processing, but also shows significant advantages in multiple dimensions compared with traditional methods based on average amplitude and fully connected layers. In terms of dynamic adaptability, GLU generates weights point by point by fusing spatiotemporal attention features and original signals, accurately dealing with transient noise (such as blink artifacts), while traditional methods only output static channel weights and cannot handle time-varying interference. In addition, the traditional scheme only relies on the amplitude mean and cannot distinguish between effective signals and noise, while in the technical solution of the present disclosure, NAM quantifies the noise intensity by analyzing the local standard deviation of the residual signal, specifically suppresses high-noise regions, and improves the noise perception ability. In addition, the traditional method is easily affected by amplitude mutation and fails, while in the technical solution of the present disclosure, the residual analysis can effectively resist sudden disturbances such as electrode loosening, and improves the anti-interference robustness.

[0148] After obtaining the denoised electroencephalogram signal based on the present step S111, on the one hand, the denoised electroencephalogram signal is analyzed in subsequent steps to extract various features (such as electroencephalogram features, brain network features, and source space features, etc.), and on the other hand, the waveform of the denoised electroencephalogram signal is dynamically displayed.

[0149] According to an embodiment of the present disclosure, the method further comprises: dynamically displaying the waveform of the denoised electroencephalogram signal in real time on the visualization interface, so as to provide an interactive feedback interface for the tester or expert to visually observe the changes of different frequency band brain waves, and facilitate the tester to observe the changes of different frequency band brain waves of the subject, thereby providing a basis for generating a targeted stimulation scheme.

[0150] In a specific example, the waveform of the electroencephalogram signal after separation of noise such as electrooculogram and electromyogram signals is dynamically displayed using a web page. The waveform refresh rate f r The range can be [2Hz, 8Hz] as determined by the electroencephalogram acquisition device, and the display front-end feedback delay t delay The range is

[0151] In step S120, one or more specified frequency band EEG signals in the EEG signals are acquired, and EEG features corresponding to the specified frequency band EEG signals are extracted based on the one or more specified frequency band EEG signals, the specified frequency band EEG signals refer to EEG signals in a specified frequency band, and the EEG features include EEG time domain features and / or EEG frequency domain features.

[0152] Generally, EEG signals contain signals of five typical frequency bands: Delta / [1-4 Hz), Theta / [4-8 Hz), Alpha / [8-13 Hz), Beta / [13-30 Hz), and Gamma / [30-48 Hz). In the present disclosure, the EEG signals corresponding to each electrode channel can cover mixed signals of multiple or all frequency bands. When the one or more specified frequency band EEG signals in the EEG signals are acquired, each frequency band signal can be separated by a band-pass filter to obtain the specified frequency band EEG signals.

[0153] The EEG time domain features include but are not limited to time domain envelopes. Unlike traditional EEG signal time domain feature extraction methods, the present disclosure does not focus on the relevant EEG amplitude changes, but focuses on the EEG signals themselves, extracts EEG time domain envelopes corresponding to the specified frequency band EEG signals, and since the EEG time domain envelope reflects the change of the EEG signal intensity of the frequency band over time, the brain wave audio generated by the EEG features has more "brain wave timbre".

[0154] When the time domain envelope corresponding to the specified frequency band EEG signals is extracted based on the one or more specified frequency band EEG signals, an envelope extraction algorithm (such as Hilbert transform) can be used to extract the time domain envelope corresponding to the specified frequency band EEG signals.

[0155] The EEG frequency domain features include but are not limited to EEG power spectrum features. In the present disclosure, when the EEG power spectrum features corresponding to the specified frequency band EEG signals are extracted based on the one or more specified frequency band EEG signals, a band-selective spectrum transform algorithm is used to extract refined EEG power spectrum features corresponding to the specified frequency band EEG signals, and the band-selective spectrum transform algorithm includes but is not limited to any one of the following transform algorithms: Chirp-Z transform, Zoom-FFT, Goertzel algorithm, continuous wavelet transform CWT.

[0156] In the present disclosure, considering that the effective frequency band range of the scalp EEG signal is 0-50 Hz, while the frequency range of the specified frequency band is much narrower, far less than the sampling rate of the device (e.g., 1000 Hz), using uniform sampling DFT will result in a large number of out-of-band results, causing waste of calculation and reduction of frequency domain resolution. Therefore, in the present disclosure, a frequency band selective spectrum transform algorithm (e.g., Chirp-z transform) is used instead of the DFT commonly used in the prior art to obtain the refined power spectrum density in the specified frequency band, thereby focusing on the target frequency band and improving the frequency resolution, which is more suitable for analyzing the narrowband oscillation characteristics of the EEG. Among them, the refined EEG power spectrum feature refers to high-resolution spectrum estimation for the specified frequency band (e.g., Alpha band 8-13 Hz) based on traditional power spectrum analysis, so as to more accurately capture the subtle frequency components of the EEG signal.

[0157] According to an embodiment of the present disclosure, the refined EEG power spectrum feature corresponding to the specified frequency band EEG signal is extracted by the frequency band selective spectrum transform algorithm, comprising:

[0158] First, the rotation factor and the starting point are calculated according to the start frequency and the end frequency of the specified frequency band and the target resolution.

[0159] Specifically, the rotation factor W is calculated by the following formula:

[0160]

[0161] The starting point A is calculated by the following formula:

[0162]

[0163] Where f s is the sampling rate, f1 is the start frequency of the specified frequency band, f2 is the end frequency of the specified frequency band, M is the number of output points in the specified frequency band, and the target resolution is represented as:

[0164] Then, the frequency band selective spectrum transform algorithm is performed on the specified frequency band EEG signal according to the rotation factor and the starting point, to obtain the spectrum corresponding to the specified frequency band EEG signal.

[0165] Taking Chirp-z transform (CZT) as an example, specifically, the following steps are implemented:

[0166] 1. Construct a pre-processing sequence g[n]:

[0167]

[0168] 2. Construct a convolution kernel h[n]:

[0169]

[0170] 3. Perform FFT on g[n] and h[n] respectively:

[0171] G[k] = FFT(g[n]);

[0172] H[k] = FFT(h[n]);

[0173] 4. Multiply in frequency domain:

[0174] Y[k] = G[k] · H[k];

[0175] 5. Get convolution result y[n] by IFFT:

[0176] y[n] = IFFT(Y[k]);

[0177] 6. Modify output, take the first M points:

[0178]

[0179] where s[n] is the specified frequency band EEG signal, and N is the length of the specified frequency band EEG signal.

[0180] At this point, the CZT transform result of the specified frequency band EEG signal s[n] is obtained, that is, the spectrum S[k], S[k] is a complex number, representing the frequency domain representation of the EEG signal at the refined frequency point k.

[0181] Finally, the following formula is used to calculate the refined EEG power spectrum feature P[k] corresponding to the specified frequency band EEG signal based on the spectrum:

[0182] P[k] = A[k] 2 ;

[0183]

[0184] where A[k] is the amplitude spectrum corresponding to the specified frequency band EEG signal, and the refined EEG power spectrum feature P[k] can be directly used for: frequency band energy statistics (such as calculating the total energy of the frequency band), feature classification (such as extracting the power of a specific frequency point as a classification feature), and time-frequency analysis (such as combining a sliding window to realize dynamic power spectrum tracking).

[0185] In step S130, for the specified frequency band electroencephalogram signal, based on the specified audio form and the specified audio generation mode, the specified frequency band electroencephalogram signal is segmented into a plurality of electroencephalogram segments according to the electroencephalogram features or preset rhythm parameters, and a plurality of audio performances corresponding to the plurality of electroencephalogram segments are generated, wherein the characteristic parameters of the audio performances are determined according to the electroencephalogram features of the corresponding electroencephalogram segments; the audio form includes a note form and / or a waveform form, wherein the audio performance corresponding to the note form is a note, and the audio performance corresponding to the waveform form is a waveform, and the audio generation mode includes a direct mapping mode and / or an indirect control mode using control parameters.

[0186] In step S140, the brain wave audio representation data is generated according to the audio performances corresponding to the one or more specified frequency band electroencephalogram signals.

[0187] In step S150, the brain wave audio is obtained according to the brain wave audio representation data.

[0188] After the brain electrical characteristics corresponding to the specified frequency band brain electrical signals are obtained based on the above steps S110-S120, the present disclosure generates the corresponding brain wave audio based on the brain electrical characteristics according to the above steps S130-S150. For the brain wave audio generation process, the present disclosure implements two real-time brain electrical-audio generation paths, namely a symbolic generation path and a waveform generation path. For the symbolic generation path, the final generated audio is in the form of notes, and the characteristic parameters of the audio corresponding to the note form include but are not limited to note duration, pitch, and intensity. For the waveform generation path, the final generated audio is in the form of a waveform, and the characteristic parameters of the audio corresponding to the waveform form include but are not limited to fundamental frequency and harmonic frequency, as well as the intensity of the fundamental frequency and the intensity of the harmonic frequency. At the same time, the present disclosure also implements two different audio parameter generation methods: one is a control type generation, that is, using the results of the brain electrical related discrimination model to control the selection of audio generation parameters, which is called "control parameter indirect control method". By indirectly controlling the note parameters (pitch, rhythm, etc.) through the discrimination model (such as classifying attention state), the target-oriented neural feedback is realized, thereby improving the neural regulation accuracy. The second is a mapping type generation, that is, directly mapping the brain electrical characteristics to the audio characteristics in a certain mapping relationship, without using other intermediate calculation results to control the mapping parameters, which is called "direct mapping method". By directly mapping the brain electrical characteristics, the specific neural electrical activity state of different individuals is intuitively reflected in the audio, meeting the individual difference generation and adjustment needs. At the same time, the parameter generation process of this method is shorter in time than the discrimination model, thereby avoiding model calculation delay and being suitable for ultra-real-time response scenarios. Among them, the output in the form of notes is more in line with traditional audio cognition, which is convenient for users to understand the association between neural state and audio, and waveform synthesis can generate complex soundscapes, which is suitable for emotion regulation. For the symbolic generation path, the audio parameter generation method is divided into control type generation and mapping type generation. For the waveform generation path, the parameter generation method is mapping type generation.

[0189] Therefore, the present disclosure provides three audio synthesis paths, namely a mapping type-symbolic synthesis path, a control type-symbolic synthesis path, and a mapping type-waveform synthesis path. Specifically, when the specified audio representation form is in the form of notes and the specified audio generation method is the control parameter indirect control method, the corresponding audio synthesis path is the control type-symbolic synthesis path. When the specified audio representation form is in the form of notes and the specified audio generation method is the direct mapping method, the corresponding audio synthesis path is the mapping type-symbolic synthesis path. When the specified audio representation form is in the form of a waveform and the specified audio generation method is the direct mapping method, the corresponding audio synthesis path is the mapping type-waveform synthesis path.

[0190] In a specific implementation, the audio representation form and the audio generation mode can be selected based on a selection parameter input by a user or a selection parameter automatically generated by the system within a preset period, and then a corresponding audio synthesis path is determined. The selection parameter input by the user can be set according to an application scenario and / or user preference. For example, for a medical rehabilitation (such as ADHD training) scenario, a control-symbol synthesis path can be selected to control note parameters through a classification model closed loop; for an artistic creation scenario, a mapping-symbol path can be selected to directly map electroencephalogram into MIDI notes; when the selection parameter is automatically generated by the system, the selection parameter can be generated according to the current neural state and / or signal quality evaluation parameter. For example, in a focused state, a control-symbol generation path can be selected to increase note density, and in a relaxed state, a mapping-waveform generation path can be selected to generate audio with a soothing tone. In addition, the audio representation form and the audio generation mode can also be selected based on a specified frequency band in the system, and then an audio synthesis path used by default by the system is determined. For example, the electroencephalogram signals in the Delta / δ and Theta / θ frequency bands use a mapping-symbol synthesis path, and the electroencephalogram signals in the remaining frequency bands use a mapping-waveform synthesis path.

[0191] For the electroencephalogram signals of multiple electrode channels, after the specified frequency band electroencephalogram signals in each electrode channel or multiple specified electrode channels are obtained based on the above step S120, the corresponding audio representation (notes or audio waveform) is generated based on each specified frequency band electroencephalogram signal, and finally the brain wave audio is generated according to the audio representation corresponding to the specified frequency band electroencephalogram signals in each electrode channel.

[0192] Each audio synthesis path will be described in detail below.

[0193] Suppose that the electroencephalogram signals EEG(t) of multiple electrode channels have N electrode channels, and the specified frequency band set Bands = {δ [1-4 Hz), θ [4-8 Hz), α [8-12 Hz), β [12-30 Hz), γ [30-48 Hz]}.

[0194] First, the electroencephalogram signals of each specified frequency band of each electrode channel are segmented non-overlappingly according to a preset time length (such as 4 seconds), and then the note sequence (for the mapping-symbol synthesis path and the control-symbol synthesis path) or the audio waveform (for the mapping-waveform synthesis path) corresponding to each segment is generated. Each note sequence or audio waveform corresponds to a single specified frequency band electroencephalogram signal in a single electrode channel, and these note sequences or audio waveforms constitute the brain wave audio corresponding to the preset time length. The preset time length is the processing window length of the electroencephalogram signals, which is usually an integer number of measures.

[0195] For the mapping-symbol synthesis path:

[0196] The path maps the electroencephalogram features corresponding to the electroencephalogram signals to audio features, thereby generating corresponding musical notes and scores, and finally performing playing using musical instrument samples. When performing electroencephalogram-audio mapping, in order to improve real-time performance, the electroencephalogram signals in each specified frequency band in each electrode channel can be processed in parallel.

[0197] Figure 4 A flowchart of a mapping-type-symbolic audio synthesis method according to an embodiment of the present disclosure is shown. For the electroencephalogram signals of a preset time length in each specified frequency band of each electrode channel, the following steps S410-S460 are performed as shown: Figure 4 The following steps S410-S460 are performed as shown:

[0198] For the electroencephalogram signals of a preset time length in each specified frequency band of each electrode channel, the following steps S410-S420 are performed:

[0199] In step S410, the electroencephalogram signals of the preset time length are segmented according to the note length indicated by the preset rhythm parameter, to obtain an electroencephalogram segment set, which contains a plurality of electroencephalogram segments, and a single electroencephalogram segment corresponds to a musical note.

[0200] The rhythm parameter is a set of rules for controlling the time structure and rhythm pattern in the process of converting electroencephalogram signals to musical notes, to ensure that the generated audio has reasonable beats, length and rhythm, including but not limited to: BPM (beats per minute), rhythm template, minimum note length and note segmentation number. For example: BPM = 120, minimum note length = 16 eighth notes (0.125s), rhythm template = 4 / 4 beats.

[0201] For example: for electrode channel i and specified frequency band δ, the corresponding electroencephalogram signal is EEG_i_δ(t), after segmentation according to the preset rhythm parameter, the obtained electroencephalogram segment set notelist = {s note1 ,s note2 ,…,s noteN}.

[0202] In step S420, for any electroencephalogram segment in the electroencephalogram segment set, the pitch and intensity of the musical note corresponding to the any electroencephalogram segment are determined according to the electroencephalogram frequency domain features corresponding to the any electroencephalogram segment, until the pitch and intensity of the musical note corresponding to each electroencephalogram segment in the electroencephalogram segment set are obtained.

[0203] In determining the pitch and intensity of the musical note corresponding to the any electroencephalogram segment according to the electroencephalogram frequency domain features corresponding to the any electroencephalogram segment, the following steps S421-S422 are included:

[0204] In step S421, based on the power spectrum feature corresponding to any one of the EEG segments, the peak frequency or weighted average frequency with the largest amplitude in the power spectrum feature is selected as the characteristic frequency.

[0205] The power spectrum features corresponding to any EEG segment include: the refined power spectrum features obtained by the frequency band selective spectrum transformation algorithm, such as: CZT power spectrum features sp note , whose peak frequency is f peak , weighted average frequency f weight It can be calculated by the following formula:

[0206]

[0207] Among them, f k is the kth frequency point (Hz) in the power spectrum corresponding to any EEG segment, sp note (f k ) is the power spectrum amplitude at the kth frequency point.

[0208] In step S422, based on the preset mapping method, the corresponding specified pitch range is obtained according to the frequency range of the specified frequency band, and the pitch corresponding to any one of the EEG segments is obtained according to the characteristic frequency and the specified pitch range; the sound intensity corresponding to any one of the EEG segments is generated according to the total energy or peak amplitude of the power spectrum characteristics; the preset mapping method is used to describe the mapping relationship between the frequency range of the specified frequency band and the pitch range.

[0209] When determining the pitch of a note, the pitch range to which it is mapped can first be divided according to the specified frequency band. Taking the mapping to the MIDI pitch range as an example, the MIDI note numbers are 0-127, corresponding to C-1 to G9. Then, the MIDI pitch range corresponding to the EEG signal in the Delta & Theta frequency band is [48, 59], the MIDI pitch range corresponding to the EEG signal in the Alpha frequency band is [56, 64], the MIDI pitch range corresponding to the EEG signal in the Beta frequency band is [65, 88], and the MIDI pitch range corresponding to the EEG signal in the Gamma frequency band is [88, 96].

[0210] When determining the preset mapping method, you can choose to use a linear or logarithmic method based on the application scenario and specific needs. Taking the linear method as an example, the pitch mapping can be performed using the following formula:

[0211]

[0212] Among them, MIDI 音高 The pitch of the note corresponding to any EEG segment, MIDI min The minimum value of the MIDI pitch range, MIDImax is the maximum value of the MIDI pitch range, f min is the minimum value of the specified frequency band, f max is the maximum value of the specified frequency band, and f is the characteristic frequency.

[0213] In special cases, if the power spectrum amplitude corresponding to any of the EEG segments is lower than the threshold, the pitch is forced to be 0 (rest).

[0214] When determining the intensity of a note, first determine the total energy E of the power spectrum feature. total or peak amplitude a peak , where E total =∑ k sp note (f k ), A peak =max(sp note (f k )), and then generates a sound intensity corresponding to any EEG segment based on the total energy or peak amplitude of the power spectrum feature. The choice of using the total energy or peak amplitude of the power spectrum feature can be made according to the following conditions: if the EEG signal is stable (such as alpha waves), the total energy is preferred; if transient features (such as gamma wave bursts) need to be captured, the peak amplitude is used.

[0215] When generating the sound intensity corresponding to any EEG segment based on the total energy or peak amplitude of the power spectrum feature, first determine a preset sound intensity range, for example, setting the sound intensity range to a continuous value of normalized values ​​0 to 1; then determine a preset mapping method, for example:

[0216] When the sound intensity S corresponding to any EEG segment is generated according to the total energy of the power spectrum feature energy When , the following formula can be used to calculate:

[0217]

[0218] When the sound intensity S corresponding to any EEG segment is generated according to the peak amplitude of the power spectrum feature peak When , the following formula can be used to calculate:

[0219]

[0220] Among them, E min and E max are the minimum energy threshold and maximum energy threshold calibrated by historical data or experiments, A min and A max are the minimum and maximum calibration thresholds of the peak amplitude, respectively.

[0221] When the pitch and intensity of the notes corresponding to each EEG segment in the EEG segment set are determined, the notes corresponding to each EEG segment in the EEG segment set of the EEG signal of the preset time length in each specified electrode channel are determined.

[0222] The steps S410-S420 are repeatedly performed to obtain the notes corresponding to each EEG segment in the EEG segment set of the EEG signal of the same preset time length in the EEG signal of the other specified frequency band of each specified electrode channel.

[0223] In step S430, the notes corresponding to each EEG segment in the EEG segment set of the EEG signal of the preset time length in the EEG signal of each electrode channel are written into the corresponding music score of the specified frequency band.

[0224] The purpose of this step S430 is to organize all the notes of each electrode channel in a single frequency band in chronological order into a structured music score. The music score of each frequency band is composed of a series of notes, which contains the starting time, pitch, intensity, note duration, electrode channel number, and frequency band identifier of each note.

[0225] In a specific example, the synthesized music score is stored in the format of MIDI (Music Instrument Digital Interfacing).

[0226] The steps S410-S430 are repeatedly performed to obtain the music score corresponding to each specified frequency band of each specified electrode channel.

[0227] In step S440, the music score corresponding to each specified frequency band in each electrode channel is written into the complete music score set corresponding to the preset time length, and the complete music score set is taken as the brain wave audio representation data.

[0228] The brain wave audio representation data is an intermediate data format that converts the time / frequency domain features of the EEG signal into audible audio.

[0229] The purpose of this step S440 is to merge the music scores of all electrode channels and frequency bands to generate a time-synchronized complete music score set. The complete music score set contains the music scores corresponding to all specified frequency bands in all electrode channels.

[0230] In step S450, the timbre of the music score corresponding to each specified frequency band in the brain wave audio representation data is configured according to the preset frequency band-timbre mapping table or the configuration parameter-timbre mapping table.

[0231] The frequency band-tone mapping table is used to describe the correspondence between the frequency band and the tone, and the configuration parameter-tone mapping table is used to describe the correspondence between the specified tone configuration parameter and the tone. The specified tone configuration parameter includes a non-brain-derived control parameter and / or a non-brain-derived event occurrence frequency.

[0232] The configured tone includes a MIDI instrument tone. As described above, the tone can be configured according to the tone mapping table. The present disclosure provides two tone mapping tables for the mapping-type-symbol synthesis path, which are a frequency band-tone mapping table and a configuration parameter-tone mapping table. The frequency band-tone mapping table defines the frequency band-tone (instrument) correspondence relationship, and different tones can be configured for different specified frequency bands, for example, a bass tone can be configured for the score corresponding to the delta frequency band, a violin tone can be configured for the score corresponding to the beta frequency band, and the like. The configuration parameter-tone mapping table defines the specified tone configuration parameter-tone (instrument) correspondence relationship, and the tone used for each frequency band can be obtained according to different tone configuration parameters. In specific applications, one of the tone mapping tables can be selected to configure the tone according to the current scene needs.

[0233] Specifically, the non-brain-derived control parameter is obtained according to the following manner:

[0234] When the non-brain-derived signal and the denoised electroencephalogram signal are contained in the preset length of the electroencephalogram signal of the specified electrode channel, the non-brain-derived control parameter is obtained according to the ratio of the preset percentile of the absolute values of the non-brain-derived signal and the denoised electroencephalogram signal in the preset length of the electroencephalogram signal of the specified electrode channel.

[0235] The specified electrode channel can be all electrode channels, or any one or several electrode channels in all electrode channels. In this way, when the non-brain-derived control parameter is obtained according to the ratio of the preset percentile of the absolute values of the non-brain-derived signal and the denoised electroencephalogram signal in the preset length of the electroencephalogram signal of the specified electrode channel, the non-brain-derived control parameter can be the average of the ratio of the preset percentile of the absolute values of the non-brain-derived signal and the denoised electroencephalogram signal in the preset length of the electroencephalogram signal of all electrode channels, or the maximum value of all preset percentile ratios, or the preset percentile ratio of a specific electrode channel of interest. The preset percentile ratio can be a 90% percentile ratio, which can be set according to requirements. The non-brain-derived signal can include, but is not limited to, an electrooculogram signal.

[0236] The calculation process of the non-brain-derived control parameter is described in detail as follows:

[0237] For the preset length of electroencephalogram signal of the specified electrode channel, if it contains non-brain-derived signal and de-noised electroencephalogram signal, first, the pre-trained brain-derived signal and non-brain-derived signal separation network model is used to preprocess the preset length of electroencephalogram signal of the specified electrode channel, and the de-noised electroencephalogram signal EEG brain is obtained. Then, the non-brain-derived signal EOG is obtained according to the difference between the preset length of electroencephalogram signal of the specified electrode channel and the de-noised electroencephalogram signal EEG brain . Taking the ratio of the 90th percentile of the specific electrode channel of interest as the non-brain-derived control parameter as an example, the non-brain-derived control parameter AR can be calculated using the following formula:

[0238] AR = EOG 90 / EEG brain90 ;

[0239] Wherein, AR represents the ratio of the 90th percentile of the absolute value of the non-brain-derived signal EOG and the de-noised electroencephalogram signal EEG brain , the 90th percentile is defined as the value at the 90th position after the signal data is arranged in ascending order (i.e. the threshold of the first 10% high amplitude), EOG 90 represents the 90th percentile of the absolute value of the non-brain-derived signal EOG, and EEG brain90 represents the 90th percentile of the absolute value of the de-noised electroencephalogram signal EEG brain .

[0240] The non-brain-derived event frequency is obtained according to the following method: a non-brain-derived event threshold is obtained according to a preset ratio of the absolute value of the de-noised electroencephalogram signal, the positions of the non-brain-derived signal higher than the non-brain-derived event threshold are counted, the number of non-brain-derived events in the non-brain-derived signal is obtained, and the non-brain-derived event frequency is obtained according to the number of non-brain-derived events and the preset time length.

[0241] Also taking the ratio of the 90th percentile of the specific electrode channel of interest as the non-brain-derived control parameter as an example:

[0242] For example, first, the non-brain-derived event threshold Th can be calculated using the following formula:

[0243] Th = a * EEG braih90 ;

[0244] Wherein, a is a preset constant value, for example, a = 3.

[0245] Then, the positions in the non-brain-derived signal EOG that are above the non-brain-derived event threshold are counted. The non-brain-derived parts that exceed the threshold are counted and distinguished as independent events to obtain the number N of non-brain-derived events that occur in the non-brain-derived signal.

[0246] Finally, the frequency D of non-brain-related events is obtained based on the ratio of the number of non-brain-related events N to the preset duration T. For example, if there are 4 non-brain-related events exceeding the threshold in a signal of a preset duration of 4 seconds, then D = 4 / 4 = 1 (times / s).

[0247] After obtaining the non-brain-derived control parameter AR and the non-brain-derived event occurrence frequency D, different frequency band timbre combinations can be determined based on one or a combination of the two to form a corresponding configuration parameter-timbre mapping table.

[0248] For example, using only the AR parameter to determine the timbre used in each frequency band (each note belonging to this frequency band uses this timbre):

[0249] First, pre-set the mapping parameters c1, c2 (c2>c1) and rangeConfig, where c1 is the relaxation threshold, c2 is the excitement threshold, and rangeConfig is the transition parameter (which can be preset to 0.25). c1 and c2 can be manually adjusted parameters, and their preset values ​​can be c1=1 and c2=2.

[0250] Then, the timbre used in each frequency band corresponding to the AR parameter is determined based on the following mapping rules:

[0251] The transition zone width midrange is calculated using the following formula:

[0252] midrange=c1-c2;

[0253] When AR <c1-rangeConfig*midrange时,使用音色组合1。

[0254] When c1-rangeConfig*midrange≤AR <c1+rangeConfig*midrange时,使用音色组合2。

[0255] When c1+rangeConfig*midrange≤AR <c2-rangeConfig*midrange时,使用音色组合3。

[0256] When c2-rangeConfig*midrange≤AR <c2+rangeConfig*midrange时,使用音色组合4。

[0257] When c2+rangeConfig*midrange≤AR, timbre combination 5 is used.

[0258] In a specific example, the timbre combinations are as shown in Table 1 (only for example, actual can be adjusted according to needs):

[0259] Table 1: Corresponding table of timbre combinations of each frequency band

[0260]

[0261] As shown in Table 1, timbre combinations 1 to 5 correspond to five emotions from relaxed to intense, corresponding to the intensity of non-brain-derived signal changes in different emotional states. Using D or a combination of D and AR, similar correspondence can be used to achieve the above effects. The specific correspondence can be determined according to specific needs.

[0262] In step S460, the brain wave audio is generated by a MIDI synthesizer or an audio synthesis library according to the corresponding music score of each specified frequency band in the brain wave audio representation data.

[0263] The purpose of this step S460 is to render the music score into a playable audio file, so as to be played by an audio player for feedback to the user subsequently.

[0264] For the control-type-symbol synthesis path:

[0265] This path first generates control signals using electroencephalogram signals and combining a discriminant model, for example, based on emotion discriminant scores to determine control signals for generating audio: Arosal (abbreviated as aro) and Valence (abbreviated as val), then using the control signals and electroencephalogram features corresponding to the electroencephalogram signals to generate notes corresponding to the electroencephalogram signals, and using instrument samples to play based on the notes.

[0266] Figure 5 A flowchart of a control-type-symbol audio synthesis method according to an embodiment of the present disclosure is shown. For the electroencephalogram signals of a preset time length in each specified frequency band of the electroencephalogram signals of each electrode channel, the following steps S510-S580 are performed as shown in Figure 5

[0267] For the electroencephalogram signals of a preset time length in the specified frequency band of the specified electrode channel, the following steps S510-S550 are performed:

[0268] In step S510, the electroencephalogram signals of a preset time length in the specified frequency band are input, and a pre-trained user state discriminant model is used to generate control parameters, including: arousal and valence.

[0269] ​In a specific example, the user state discrimination model includes a multi-layer perception emotion discrimination model. In generating the control parameter according to the emotion discrimination model, the control parameter is respectively: arousal aro and valence val, wherein the arousal aro represents the activation intensity of the physiology or emotion, describes the continuous state from "calm" to "excited", and can be divided into high arousal, medium arousal and low arousal; the valence val represents the positive or negative tendency of the emotion, describes the emotional polarity from "negative" to "positive", and can be divided into positive, neutral and negative. The arousal aro and the valence val are the core dimensions of the emotion quantification model, which are mapped to the audio parameter through the electroencephalogram feature, realizing the closed loop of "physiological signal → emotional expression → audio generation". The arousal aro and the valence val can be calculated according to the following formula:

[0270] aro=aro base (MotionInst,MotionLong)+meanConf×C0+(1-consistency)×

[0271] variation×C1;

[0272]

[0273] val=val base (MotionInst,MotionLong,meanConf)+meanConf×C2;

[0274] wherein, aro base is a function of the real-time emotion discrimination result MotionInst and the long-time emotion discrimination result MotionLong, in a specific example, MotionLong is the classification result of the 10s electroencephalogram signal inputting the user state discrimination model, MotionInst is the result of the last 2s electroencephalogram segment in the 10s electroencephalogram signal inputting the user state discrimination model, if the preset time length is 4s, MotionInst is the result of the last 2s electroencephalogram segment in the 4s electroencephalogram signal inputting the user state discrimination model, and MotionLong is the classification result of the 4s electroencephalogram signal combined with the past 6s electroencephalogram signal inputting the user state discrimination model; meanConf, consistency and variation are statistical results output by the emotion discrimination model, which respectively represent the average confidence of the real-time emotion discrimination result of the preset number of times in the past preset time, the consistency of the discrimination result and the average variation rate, val base is a function of the real-time emotion discrimination result, the long-time emotion discrimination result and the average confidence, and C0, C1 and C2 are constants.

[0275] In the specific implementation of steps S520-S550, the specific generation mode of the occurrence probability, note duration, pitch, intensity and timbre of the note can be set in combination with the stimulation task requirements, so as to flexibly serve the objective monitoring or active intervention and meet the diversified needs of medical treatment, entertainment and the like. For example, if the current stimulation task (such as a neurofeedback training task or a state monitoring task) needs to accurately map the electroencephalogram features to the audio parameters and accurately present the neural activity state of the user, for the electroencephalogram signals expressing high arousal and positive state (aro=8, val=8), fast rhythm (short note), dense, strong sound, high pitch, major bright brain wave audio is generated, and for the electroencephalogram signals expressing low arousal and negative state (aro=2, val=3), slow rhythm (long note), sparse, weak sound, low pitch, minor melancholy melody is generated; if the current stimulation task (such as a clinical intervention task) needs to reversely regulate the neural activity of the collected user and guide the user to adjust the neural state to the target direction (such as inhibiting excessive excitement or activating the depression state) through reverse audio feedback, for the electroencephalogram signals expressing excessive excitement (aro=9, val=6), slow rhythm (long note), sparse, weak sound, low pitch, minor soft melody is generated to induce relaxation, and for the electroencephalogram signals expressing the depression state (aro=1, val=2), fast rhythm (short note), dense, strong sound, high pitch, major bright melody is generated to stimulate the mood.

[0276] In step S520, the note duration of a single note in the note sequence corresponding to the electroencephalogram signal of the preset time length is calculated according to the arousal and / or the valence.

[0277] In a specific example, the note duration is obtained by calculating the rhythm feature tempo, where:

[0278] tempo:note dur (t)=F tempo (aro,val);

[0279] Wherein, note dur (t) represents the duration of the note at time t, determines the rhythm feature of the audio, and determines note dur (t) after which the length of all notes in the segment generated at time t can be note dur (t), F tempo is a function with aro and val as parameters. For example:

[0280] note dur (t)=[1-(aro+val)×0.4] norm ;

[0281] Wherein, [] normAn operation is defined, for example, to have a time length of 2s as a whole note, [a] norm represents that a is changed to the nearest multiple of 0.25 (8th note).

[0282] In step S530, the electroencephalogram of the preset time length is divided according to the note time length, to obtain an electroencephalogram segment set containing a plurality of electroencephalogram segments, and a single electroencephalogram segment corresponds to a note.

[0283] In a specific example, each note has the same note time length. In this way, when the electroencephalogram of the preset time length is divided according to the note time length, a series of electroencephalogram segments with the same time length are obtained.

[0284] In step S540, the occurrence probability of the note corresponding to each electroencephalogram segment in the electroencephalogram segment set is calculated according to the arousal and / or the valence, and when the occurrence probability meets the occurrence condition, the note corresponding to the corresponding electroencephalogram segment appears, otherwise the note corresponding to the corresponding electroencephalogram segment does not appear; the note sequence corresponding to the electroencephalogram of the preset time length is determined according to the calculation result.

[0285] In a specific example, the occurrence probability of the note is obtained by calculating the rhythm feature rhythm, where:

[0286] rhythm: p(note = 1) = F rhythm (aro, val);

[0287] This parameter determines the complexity of the generated audio, p(note = 1) represents the probability of the occurrence of the note within the audio segment of the fixed length, the higher the probability, the higher the frequency of the occurrence of the note, and the more complex the melody.

[0288] For example, the i-th note note i at time t with a length of Δt will be determined by the following formula:

[0289]

[0290] Where roughness is a preset constant or a function of aro and val, for example:

[0291] roughness = (aro + val) / 2;

[0292] activate is a calculation result with a random number, for example:

[0293] activate = random x aro;

[0294] Where random is a random number uniformly distributed in [0, 1].

[0295] In step S550, the loudness of the notes in the note sequence is calculated according to the arousal, the valence, and the power spectrum energy features of the EEG segments corresponding to the notes in the note sequence; the tonality parameters of the notes in the note sequence are calculated according to the arousal and / or the valence; the pitch of the notes in the note sequence is calculated according to the tonality parameters and the power spectrum frequency features of the EEG segments corresponding to the notes in the note sequence; the timbre of the notes in the note sequence is configured according to the arousal and / or the valence; or the timbres of all the notes in the note sequence are configured according to a preset configuration parameter-timbre mapping table, which is used to describe the correspondence between specified timbre configuration parameters and timbres.

[0296] The specified timbre configuration parameters include non-brain-derived control parameters and / or non-brain-derived event occurrence frequency, and the non-brain-derived control parameters are obtained in the following manner: when the EEG signal of the preset length of the specified electrode channel contains non-brain-derived signals and denoised EEG signals, the ratio of the preset percentile of the absolute values of the non-brain-derived signals to the preset percentile of the absolute values of the denoised EEG signals in the EEG signal of the preset length of the specified electrode channel is taken as the non-brain-derived control parameter.

[0297] The non-brain-derived event occurrence frequency is obtained in the following manner: a non-brain-derived event threshold is obtained according to the preset percentile of the absolute values of the denoised EEG signals, the positions of the non-brain-derived signals that are higher than the non-brain-derived event threshold are counted to obtain the number of non-brain-derived events in the non-brain-derived signals, and the non-brain-derived event occurrence frequency is obtained according to the number of non-brain-derived events and the preset time length.

[0298] The loudness of the notes can be calculated by the following formula:

[0299] loudness:note vel (t)=F loudness (aro,val)*P eeg (t);

[0300] The loudness of the notes in the note sequence is calculated according to the arousal, the valence, and the power spectrum energy features of the EEG segments corresponding to the notes in the note sequence; the tonality parameters of the notes in the note sequence are calculated according to the arousal and / or the valence; the pitch of the notes in the note sequence is calculated according to the tonality parameters and the power spectrum frequency features of the EEG segments corresponding to the notes in the note sequence; the timbre of the notes in the note sequence is configured according to the arousal and / or the valence; or the timbres of all the notes in the note sequence are configured according to a preset configuration parameter-timbre mapping table, which is used to describe the correspondence between specified timbre configuration parameters and timbres. vel (t) indicates the loudness of the notes at time t, F loudness is a function with aro and val as parameters, P eeg (t) is the EEG power spectrum energy feature of the EEG segment extracted at time t.

[0301] The pitch of the notes can be calculated by the following formula:

[0302] pitch:note=F pitch (f eeg(t))+mode(i-1);

[0303] The parameter determines the pitch of the note, i.e., the number of the note under the MIDI (Musical Instrument Digital Interface) representation; F pitch is a function of f eeg (t) is a function of t, representing the mapping of the electroencephalogram frequency feature to the note pitch feature, f eeg (t) is the electroencephalogram frequency feature at time t, for example: energy weighted average frequency f weight mode is the mode, and i represents the order of the note in the note sequence, and the calculation method is as follows:

[0304] mode=F mode (aro,val);

[0305] For example:

[0306]

[0307] The generation of the first note starts from mode[0], and the i-th note is different from the previous note by mode[i-1] in MIDI number.

[0308] In configuring the timbre of the notes in the note sequence according to the arousal and / or the valence, a mapping relationship between the arousal and / or the valence and the timbre can be set in advance, and then the corresponding timbre to be configured is obtained according to the arousal and / or the valence and based on the mapping relationship. The timbre can be set based on the characteristics of a specific musical instrument, for example: the timbre of the notes in the note sequence can be configured through the mapping relationship table between the arousal and the valence and the timbre as shown in Table 2 below:

[0309] Table 2: Mapping relationship table between arousal and valence and timbre

[0310] Emotional quadrant Awakening degree Valence Instrument selection High awakening positive 7-10 7-10 Electric guitar High awakening neutral 7-10 4-7 Trumpet High awakening negative 7-10 0-4 Electric bass distortion Medium awakening positive 4-7 7-10 Piano Medium awakening neutral 4-7 4-7 Harp Medium awakening negative 4-7 0-4 Bass drum Low awakening positive 0-4 7-10 Steel pan Low awakening neutral 0-4 4-7 Cello Low awakening negative 0-4 0-4 Bass flute

[0311] Based on Table 2 as shown above, when the arousal is 8 and the valence is 4, the corresponding configured timbre is trumpet.

[0312] In configuring the timbre of all notes in the note sequence according to the preset configuration parameter-timbre mapping table, the description as described above and in combination with Table 1 can be used, which will not be repeated here.

[0313] When the pitch, intensity and timbre of the notes in the note sequence are determined, the corresponding notes of each electroencephalogram segment in the set of electroencephalogram segments are determined.

[0314] The steps S510-S550 are repeatedly executed to obtain the notes corresponding to each of the brain electrical segments in the brain electrical segment set of the brain electrical signals of the preset time length in the brain electrical signals of the other specified frequency bands of each of the specified electrode channels.

[0315] In step S560, the musical scores corresponding to the specified frequency bands are generated based on the note sequences of the specified frequency bands in each of the electrode channels, the sound intensity, the pitch and the timbre of the notes in the note sequences.

[0316] The steps S510-S560 are repeatedly executed to obtain the musical scores corresponding to the other specified frequency bands of each of the specified electrode channels.

[0317] In step S570, the musical scores corresponding to each of the specified frequency bands in each of the electrode channels are written into a complete musical score set corresponding to the preset time length, and the complete musical score set is taken as the brain wave audio representation data.

[0318] In step S580, the brain wave audio is generated according to the musical scores corresponding to each of the specified frequency bands in the brain wave audio representation data by a MIDI synthesizer or an audio synthesis library.

[0319] In the control-type-symbol synthesis path and the mapping-type-symbol synthesis path, when determining the pitch of the note corresponding to the EEG segment, the pitch is calculated according to the power spectrum frequency feature in the EEG feature of the corresponding EEG segment, and when determining the intensity of the note corresponding to the EEG segment, the intensity is calculated according to the power spectrum energy feature in the EEG feature of the corresponding EEG segment. On the one hand, the pitch of the note is determined by using the power spectrum frequency feature in the EEG signal instead of the time domain feature amplitude of the EEG waveform, which has the following beneficial technical effects: since the frequency components (such as alpha wave 8-13 Hz and beta wave 14-30 Hz) of the EEG signal are less affected by environmental noise and electrode contact impedance, and the amplitude is prone to fluctuation due to scalp impedance changes, motion artifacts and the like, frequency analysis can effectively filter out non-target frequency band interference, thereby improving the stability of pitch recognition; and specific frequency bands are directly related to cognitive states (such as gamma wave and auditory processing), and through frequency-pitch mapping, the music intention of the user can be more naturally matched, while the amplitude only reflects the intensity of neural activity and has no clear physiological correlation with the pitch. In addition, the frequency feature can be accurately mapped to the pitch through frequency band segmentation (such as 1 Hz step), while the amplitude is limited by the dynamic range of the signal and it is difficult to distinguish small pitch differences. On the other hand, the power spectrum energy feature (instead of the average power) in the EEG signal is used to determine the intensity of the note, which can realize more accurate and anti-interference intensity mapping by distinguishing the energy distribution of different frequency bands, and the specific performance is as follows: ① Frequency specificity makes the intensity adjustment conform to the physiological intention (such as beta wave enhancement corresponding to strong playing); ② Dynamic range expansion supports delicate dynamics changes (such as crescendo and tremolo); ③ High computational efficiency reduces the delay through selective band analysis. Compared with the single scalar output of the average power, the power spectrum energy feature significantly improves the expressiveness and robustness in multi-modal music interaction.

[0320] In addition, when determining the timbre of the note corresponding to the EEG segment, the present disclosure provides three different timbre mapping schemes, which are respectively configuring the timbre of the note sequence in different frequency bands according to a preset frequency band-timbre mapping table, a preset configuration parameter-timbre mapping table and the arousal and / or the valence. In specific applications, the corresponding timbre mapping scheme can be flexibly selected according to different application scenarios or requirements.

[0321] For the mapping-type-waveform synthesis path:

[0322] This path maps the EEG feature (such as EEG frequency domain feature and time domain envelope) corresponding to the EEG signal to the audio feature, and modulates the audio signal using the EEG feature (such as time domain envelope) to obtain an audio waveform with a special EEG timbre.

[0323] Figure 6 A flowchart of a mapping-type-waveform audio synthesis method according to an embodiment of the present disclosure is shown. For the EEG signal of a preset time length in each electrode channel of the EEG signal in the specified frequency band, the EEG signal is processed as follows: Figure 6The following steps S610-S650 are shown:

[0324] For the preset time length of the brain electrical signal in the specified frequency band of the specified electrode channel, the following steps S610-S630 are performed:

[0325] In step S610, the preset time length of the brain electrical signal is segmented based on the time domain envelope corresponding to the preset time length of the brain electrical signal, to obtain a set of brain electrical segments.

[0326] When segmenting, the following steps S611-S613 are included:

[0327] In step S611, the effective amplitude threshold is determined according to the mean of the time domain envelope.

[0328] The effective amplitude threshold is used to filter low-amplitude noise, and setting the effective amplitude threshold based on the envelope mean can avoid the sensitivity problem of fixed threshold to signal amplitude variation.

[0329] When determining the effective amplitude threshold, it can be determined based on the product of a preset scale factor and the mean of the time domain envelope. In a specific example, the scale factor can range from 1.2 to 1.5.

[0330] In step S612, a set of troughs of the time domain envelope is determined according to the effective amplitude threshold, and the trough amplitude in the set of troughs is not greater than the effective amplitude threshold.

[0331] Specifically, a sliding window (such as 100 ms) is used to detect the local minimum value of the time domain envelope Envelope(t), and the troughs with amplitude ≤th amp are retained, and false troughs caused by high-frequency noise are removed, and the set of troughs V is represented as:

[0332] V = {v0, v1, … v n |v i ≤th amp};

[0333] Where th amp is the effective amplitude threshold.

[0334] In step S613, the preset time length of the brain electrical signal is segmented according to the preset effective time threshold and the set of troughs of the time domain envelope, to obtain the set of brain electrical segments, which contains one or more brain electrical segments, and the time value of each brain electrical segment is greater than the effective time threshold.

[0335] Wherein, each brain electrical segment corresponds to an audio waveform; the effective time threshold is used to ensure the minimum duration of a note, which can avoid short noise being misjudged as a note.

[0336] The set of wave troughs of the time domain envelope determines each possible electroencephalogram segment seg i , the left boundary of seg i is v i , the right boundary is v i+1 , and a set of electroencephalogram segments Seg = {seg i | v i+1 -v i > th time} is obtained, where th time is the effective time threshold.

[0337] Figure 7 A schematic diagram showing the division result of an electroencephalogram segment in a mapping-type waveform audio synthesis method according to an embodiment of the present disclosure is shown. The blue part is an electroencephalogram signal, the red part is a time domain envelope of the electroencephalogram signal, and the black part is an electroencephalogram segment division result.

[0338] In step S620, for any electroencephalogram segment in the set of electroencephalogram segments, a characteristic parameter of a waveform corresponding to the electroencephalogram segment is determined according to an electroencephalogram frequency domain feature corresponding to the electroencephalogram segment, until the characteristic parameters of the waveforms corresponding to the electroencephalogram segments in the set of electroencephalogram segments are obtained.

[0339] The characteristic parameter of the waveform corresponding to the electroencephalogram segment is determined according to the electroencephalogram frequency domain feature corresponding to the electroencephalogram segment, including: based on a power spectrum feature corresponding to the electroencephalogram segment, determining a fundamental frequency and a harmonic frequency of the waveform corresponding to the electroencephalogram segment, and an intensity of the fundamental frequency and an intensity of the harmonic frequency according to a preset number of peak frequencies and amplitudes ranked by amplitude in the power spectrum feature.

[0340] In a specific example, the harmonic frequency includes a second harmonic frequency and a third harmonic frequency, and then: the first 3 peaks with the largest amplitudes in the power spectrum feature corresponding to the electroencephalogram segment are obtained, the peak frequencies {f0, f1, f2} and the corresponding amplitudes {A0, A1, A2} are recorded, the largest peak frequency f0 and the corresponding amplitude A0 can be taken as the fundamental frequency and the intensity, f1 and the corresponding amplitude A1 can be taken as the second harmonic frequency and the intensity, and f2 and the corresponding amplitude A2 can be taken as the third harmonic frequency and the intensity.

[0341] In step S630, for any electroencephalogram segment in the set of electroencephalogram segments, an audio waveform corresponding to the electroencephalogram segment is determined according to the characteristic parameter of the waveform corresponding to the electroencephalogram segment and the time domain envelope, until the audio waveforms corresponding to the electroencephalogram segments in the set of electroencephalogram segments are obtained.

[0342] The step S631-S632 comprises:

[0343] In step S631, a sine wave of corresponding frequency is generated according to the fundamental frequency and harmonic frequency of the waveform corresponding to the any EEG segment and the intensity of the fundamental frequency and the intensity of the harmonic frequency, and mixed to obtain a mixed sine wave.

[0344] Specifically, the fundamental frequency sine wave S0(t) is generated using the following formula:

[0345] S0(t)=A0·sin(2πf0t);

[0346] The second harmonic sine wave S1(t) is generated using the following formula:

[0347] S1(t)=A1·sin(2πf1t);

[0348] The third harmonic sine wave S2(t) is generated using the following formula:

[0349] S2(t)=A2·sin(2πf2t);

[0350] The mixed sine wave s mix (t) is generated using the following formula:

[0351] s mix (t)=S0(t)+S1(t)+S2(t);

[0352] In step S632, the mixed sine wave is modulated by the time domain envelope corresponding to the any EEG segment to obtain the audio waveform corresponding to the any EEG segment.

[0353] Specifically, the audio waveform a note corresponding to the any EEG segment can be obtained using the following formula:

[0354] a note (t)=s mix (t)*e nite (t);

[0355] Wherein, e note (t) is the time domain envelope corresponding to the any EEG segment.

[0356] The steps S610-S630 are repeatedly executed to obtain the audio waveform corresponding to each EEG segment in the EEG segment set of the EEG signal of the preset time length in the other specified frequency band EEG signal of each specified electrode channel.

[0357] In step S640, the audio waveform corresponding to each EEG segment in the set of EEG segments of each specified frequency band in the specified electrode channel is used as the audio waveform corresponding to the EEG signal of the specified electrode channel with the preset duration.

[0358] In step S650, the audio waveform corresponding to the electroencephalogram signal of the preset duration of each electrode channel is used as the brainwave audio representation data; and the brainwave audio representation data is directly used as the brainwave audio.

[0359] In the Mapping-Waveform Synthesis path, a time envelope is used to determine the audio's rhythm, while the spectral characteristics of the EEG clip determine the audio's fundamental pitch, harmonics, and the intensity of each component. A sine wave of the corresponding frequency is then synthesized. Applying a time envelope to the resulting harmonic sine wave creates audio with a "brainwave-like" quality.

[0360] The real-time brainwave audio synthesis feedback loop implemented in the present disclosure realizes a path for converting real-time EEG signals into personalized audio stimulation signals, supports three different audio synthesis paths, can use a variety of instrument timbres or brainwave envelope timbres, supports multi-band, multi-channel audio synthesis and playback, and has significant application value in personalized brainwave music therapy, real-time brainwave audio feedback stimulation, and other fields.

[0361] Figure 8 FIG. 1 is a flow chart showing another method for synchronous feedback of proprioceptive brainwave audio and auditory perception according to an embodiment of the present disclosure. Figure 8 As shown, after executing step S150, the following step S160 is also included:

[0362] In step S160, the brainwave audio is synchronously fed back to the collected user, so as to dynamically regulate the nervous system activity of the collected user through auditory perception.

[0363] By synchronously playing the generated brainwave audio to the user being collected, real-time audio stimulation can be achieved. At the same time, the synchronously collected EEG signal after stimulation will continue to execute the above audio synthesis process to control the synthesis of subsequent audio stimulation signals, thus realizing a closed loop from EEG collection to audio stimulation. In this closed loop, the default frame length of brainwave audio generation is 4s, and the brainwave audio feedback delay is less than 5s. In addition, users can interact with the front-end visual interface to determine the frequency band used to generate brainwave audio, thereby realizing a variety of audio matching options corresponding to each brainwave frequency band.

[0364] In addition, with the rapid development of virtual reality (VR) technology, users increasingly demand personalized and interactive content in immersive experiences. In the application scenario of VR technology, the real-time electroencephalogram acquisition and brainwave audio generation scheme provided by the embodiments of the present disclosure can be used to provide personalized background audio based on individual electroencephalogram features for users, thereby significantly improving the immersion and engagement in VR interactive scenarios.

[0365] For this specific application scenario, a portable electroencephalogram acquisition device is used to acquire real-time electroencephalogram signals in the VR device wearing scenario, and a personalized brainwave audio is generated and processed synchronously, which is fed back to the user as background audio. This background audio will fully use the user's electroencephalogram features to generate brainwave audio with individual characteristics of the user's electroencephalogram, and can analyze the user's emotional state in real time according to the changes in the user's electroencephalogram signals (based on the changes in the electroencephalogram features corresponding to the electroencephalogram signals), adjust the feature parameters of the generated brainwave audio according to the user's emotional state and scene, and realize closed-loop adjustment of personalized background audio. For example, when the user's electroencephalogram features are identified to be consistent with emotions such as anxiety and tension, the feature parameters are adjusted to generate calming brainwave audio; when the scene requires the user to concentrate, brainwave audio that helps to concentrate is generated.

[0366] Through the implementation of the embodiments of the present disclosure, users can experience a highly personalized audio environment in the VR scenario. This audio not only perfectly blends with the virtual scene, but also intelligently adjusts according to the user's real-time emotional state, which not only significantly improves the user's immersion and satisfaction, but also has broad application prospects in the fields of psychological health treatment, rehabilitation training, entertainment experience, etc. In the future, with the further development of technology, this system is expected to achieve breakthrough applications in more fields, providing users with more rich and personalized virtual reality experiences.

[0367] According to an embodiment of the present disclosure, the brainwave audio is fed back to the collected user in synchronization, including:

[0368] According to the combination relationship between the spatial positions of the electrodes in the one or more electrode channels and the one or more specified frequency bands, the brainwave audio corresponding to a plurality of sound channels is obtained based on the brainwave audio.

[0369] In a specific example, for more than two sound channels, a spatial region can be defined according to the electrode coordinates in one or more electrode channels (e.g., Fp1 corresponds to the left side of the forehead, Cz corresponds to the central top region, and O1 corresponds to the left occipital lobe), and the scalp can be divided into forehead, central, occipital, and temporal regions, each of which corresponds to a specific sound channel group for the brain wave audio of a frequency band, for example: the brain wave audio of the Theta frequency band in the forehead region (Fp1, Fp2) can be mapped to the left front / right front sound channel, the brain wave audio of the Beta frequency band in the central region (C3, C4) can be mapped to the central sound channel, and the brain wave audio of the Alpha frequency band in the occipital region (O1, O2) can be mapped to the left rear / right rear sound channel.

[0370] In another specific example, for left and right sound channels, left and right brain regions and a central region can be defined according to the electrode coordinates in one or more electrode channels, for example: the left brain region corresponds to F3 (left forehead), C3 (left central), T3 (left temporal lobe), P3 (left parietal lobe), and O1 (left occipital lobe), the right brain region corresponds to F4 (right forehead), C4 (right central), T4 (right temporal lobe), P4 (right parietal lobe), and O2 (right occipital lobe), and the central region corresponds to Cz (central top region), then the brain wave audio of a frequency band corresponding to the left brain region is mapped to the left sound channel, the brain wave audio of a frequency band corresponding to the right brain region is mapped to the right sound channel, and the brain wave audio of the central region is balanced and distributed to the left and right sound channels according to a preset ratio (e.g., 50:50).

[0371] Next, the brain wave audio of the multiple sound channels is played through an audio device, for example: the left front sound channel is played through a left front speaker, and the right rear sound channel is played through a right rear speaker, so as to synchronously feed back the brain wave audio to the collected user.

[0372] The present disclosure can make the collected user perceive the spatial distribution of brain activity (e.g., when the forehead Theta is enhanced, the left front sound channel volume is highlighted) through the auditory perception of the brain wave audio of multiple sound channels, enhance the spatial perception effect, provide more abundant information than the traditional single channel through the multi-dimensional feedback of the multi-channel brain wave audio, and thus effectively assist the neurofeedback training.

[0373] In addition to generating and feeding back the brain wave audio of multiple sound channels to the collected user to form auditory feedback, the present disclosure also provides a scheme of forming visual feedback through dynamic display of sound image positions.

[0374] According to the principle of binaural hearing, the time and intensity of sound sources played at different positions reaching the two ears are different, and the nervous system can judge the direction of the sound through these differences, and the sound has a sense of direction. Using a playback device with independent double sound channels, the strength and phase of the left and right sound channels can be adjusted to simulate directional sound, that is, stereo effect, or more than two channels of sound arrays can be used to synchronize the playback of multi-channel, multi-space position brainwave audio, forming a more rich spatial surround effect. Directional sound can effectively attract the attention of the listener, and changes in sound image position and intensity can be used to adjust the attention state of the listener.

[0375] In the present disclosure, the sound image position calculation of real-time brainwave audio is realized, and a dynamic sound image display interface is realized. The brainwave audio is played, and the change of the sound image position of the brainwave audio is dynamically visualized and displayed.

[0376] According to an embodiment of the present disclosure, the method further comprises:

[0377] Obtaining sound image position parameters of the brainwave audio corresponding to the plurality of sound channels, and dynamically displaying the sound image position of the brainwave audio on the visual interface according to the sound image position parameters.

[0378] According to an embodiment of the present disclosure, the obtaining of the sound image position parameters of the brainwave audio corresponding to the plurality of sound channels comprises:

[0379] Obtaining the brainwave audio corresponding to each specified frequency band in the plurality of sound channels, calculating the intensity of the brainwave audio corresponding to each specified frequency band in each sound channel within a specified time length, obtaining the sound image direction indication parameter corresponding to the brainwave audio of the specified frequency band based on the intensity difference of the brainwave audio of the same specified frequency band in different sound channels, and taking the sound image direction indication parameter and the intensity of the brainwave audio of each specified frequency band in the plurality of sound channels as the sound image position parameters.

[0380] According to an embodiment of the present disclosure, the method further comprises:

[0381] The sound image position of the brainwave audio is dynamically and synchronously fed back to the collected user on the visual interface, so as to dynamically regulate the neural activity of the collected user through visual perception synchronization.

[0382] In a specific example, taking left and right channels as an example, for brain wave audio signals a e A corresponding to different frequency bands, the waveform of a is monitored and independent sound image positions are calculated. The audio analysis tool in Tone.js can be used to monitor the brain wave audio waveform played in the left and right channels corresponding to the specified frequency band in real time, calculate the respective intensities of the brain wave audio corresponding to the specified frequency band in the left and right channels within the specified time length, obtain the intensity difference of the brain wave audio corresponding to the specified frequency band in the left and right channels, and then obtain the sound image direction indication parameter based on the intensity difference.

[0383] Specifically, the sound image direction indication parameter c can be calculated by the following formula:

[0384]

[0385] wherein v right represents the intensity of the brain wave audio of the specified frequency band in the right channel, v left represents the intensity of the brain wave audio of the specified frequency band in the left channel, v right -v left is the intensity difference.

[0386] In a specific example, c e [-1, 1], c = -1 indicates that the sound image position is at the leftmost position, c = 1 indicates that the sound image position is at the rightmost position, and c = 0 indicates that the sound image position is at the front (middle position).

[0387] Figure 9 A sound intensity position display schematic diagram of the brain wave audio of the specified frequency band according to an embodiment of the present disclosure is shown. As Figure 9 shown, when the sound intensity position is dynamically displayed in real time, the front-end visual dynamic feedback interface displays the sound image position using a plurality of small balls at different positions. The small balls are divided into multiple rows, and each row represents the brain wave audio corresponding to one frequency band. The size and color of the small balls at different positions are determined based on the sound image position parameter and the intensity parameter calculated in real time, which are used to represent the sound image position. The sound image direction indication parameter can be normalized, and the normalized sound image direction indication parameter is used to represent the offset of the sound image position relative to the center point. The intensity is used to represent the size of the small ball, so as to realize real-time calculation and dynamic visual display of the sound image position of the brain wave audio.

[0388] The refresh rate of the front-end feedback animation can be set to [5Hz, 10Hz], the sound image display is performed simultaneously with the audio playback feedback, and the delay is less than 200ms. Different frequency bands can be selected for visualization of the sound image position, and there are four selectable frequency bands, which are δ&θ, α, β, and γ, corresponding to the specified frequency bands used in the generation of the brain wave audio.

[0389] The embodiment provides an interactive feedback interface for users to visually observe the changes of the brain wave audio video, which is beneficial for the tester to observe the changes of the brain waves of the collected user in different frequency bands and the movement of the brain wave audio video; the dynamic visual feedback of the video position also has potential use in the attention regulation scenario of the collected user, has the potential to stimulate and regulate in combination with the brain wave audio, promotes psychological relaxation and cognitive regulation, and can be applied in multiple fields such as medicine, entertainment, and education.

[0390] In the present disclosure, in addition to extracting the brain electrical feature corresponding to the specified frequency band brain electrical signal based on the one or more specified frequency band brain electrical signals for directly generating the corresponding brain wave audio without considering the dependence relationship between the electrodes of the brain electrical collection device, other related features such as brain network features and source space features are also extracted in real time under the condition of considering the relationship between multiple brain electrical electrode leads, and these features are visualized and fed back to doctors or researchers, which is helpful for the doctors or researchers to make real-time assessment on the current state of the patient and make flexible and personalized treatment plan adjustment. In view of the problems such as difficulty in real-time feature calculation and single feedback channel in the traditional brain electrical collection-feedback process, the method for extracting brain network features and source space features is improved in the present disclosure, and a set of real-time automatic brain electrical processing and feedback process is proposed, realizing a low-delay real-time link of brain electrical signal collection-analysis-feature visualization feedback.

[0391] According to an embodiment of the present disclosure, the method further comprises:

[0392] extracting brain network features corresponding to the brain electrical signals of the plurality of electrode channels, the brain network features comprising node-node functional connection features and / or edge-edge interaction features, the node-node functional connection features being used to describe the statistical dependence relationship between two node signals, and the edge-edge interaction features being used to describe the time dependence relationship between two edges; wherein the node corresponds to a single electrode channel, the node signal corresponds to the brain electrical signal of a single electrode channel, and the edge corresponds to the connection relationship between two nodes.

[0393] The node-node functional connection feature corresponding to the electroencephalogram signals of the plurality of electrode channels is extracted by calculating a weighted phase lag index (WPLI) between each two node signals. The WPLI is used to quantify the phase lag relationship between two signals, and the influence of zero-lag pseudo-correlation is reduced by weighting the sign of the cross-spectral imaginary part. Compared with the PLI (Weighted Phase Lag Index) which only counts the expectation of the imaginary part sign and ignores the amplitude information, the WPLI can more accurately quantify the statistical significance of the phase lag, reduce the influence of noise or zero-lag pseudo-correlation, and improve the sensitivity to real phase lag. In addition, the length of the electroencephalogram signal segment required for calculating the WPLI is short, and the WPLI can be used for real-time functional connection calculation.

[0394] According to an embodiment of the present disclosure, when calculating the weighted phase lag index between any two node signals, the following steps are included: performing Hilbert transform on the first node signal and the second node signal to obtain a first node analytic signal and a second node analytic signal corresponding to the first node signal and the second node signal respectively; obtaining the instantaneous cross power spectrum of the first node signal and the second node signal based on the first node analytic signal and the second node analytic signal; averaging the absolute value of the imaginary part of the instantaneous cross power spectrum at each time point in a preset time window within the preset time window to obtain an imaginary part absolute value mean; calculating the weight of the imaginary part of the instantaneous cross power spectrum at each time point in the preset time window, and obtaining a weighted complex covariance based on the instantaneous cross power spectrum at each time point in the preset time window and the weight of the imaginary part corresponding to the instantaneous cross power spectrum; and calculating the weighted phase lag index of the first node signal and the second node signal by using the absolute value of the imaginary part of the weighted complex covariance and the imaginary part absolute value mean.

[0395] The first node analytic signal and the second node analytic signal corresponding to the first node signal and the second node signal respectively are obtained by the following formula:

[0396] z I (t)=x i (t)+j·H(x I (t));

[0397] z j (t)=x j (t)+j·H(x j (t));

[0398] The instantaneous cross power spectrum is obtained by the following formula:

[0399]

[0400] The weight of the imaginary part of the instantaneous cross power spectrum at each time point in the preset time window is calculated by the following formula:

[0401]

[0402] c = median (|Im (P ij (t)) |) ;

[0403] The weighted complex covariance is obtained by the following formula:

[0404]

[0405] The weighted phase-lag coefficient is calculated by the following formula:

[0406]

[0407] Wherein, x i (t) is the first node signal, x j (t) is the second node signal, z i (t) is the first node analytic signal, z j (t) is the second node analytic signal, j is the imaginary unit, H () is the Hilbert transform operator; P ij (t) represents the instantaneous cross power spectrum, * represents the complex conjugate, median () is the median operation, is the weighted complex covariance, and T is the number of time points in the preset time window.

[0408] In the traditional method of calculating the weighted phase-lag coefficient, although the weighting reduces the influence of zero-lag noise, high-frequency noise or non-stationary noise may still interfere with the result, showing sensitivity to outliers (such as burst noise). In the present disclosure, when calculating the weighted phase-lag coefficient between any two node signals, a dynamic weight mechanism is introduced. On the one hand, the weight is calculated by the absolute value of the imaginary part, giving higher weight to time points with high signal-to-noise ratio, suppressing the influence of low signal-to-noise ratio points, making the phase-lag estimation more robust in the electroencephalogram noise environment (such as electrooculogram artifact, electromyographic interference); on the other hand, the complex covariance is weighted, retaining the joint information of phase difference amplitude and direction, improving the phase synchronization feature resolution; in addition, the noise connection is suppressed by the weight, the small-world properties of brain network (such as clustering coefficient, global efficiency) and the correlation with behavior / disease are significantly enhanced, improving the reliability of the graph theory analysis results.

[0409] In extracting the edge-edge interaction features corresponding to the brain electrical signals of the plurality of electrode channels, the present disclosure uses instantaneous phase coherence as the measurement of the connection (edge) between nodes to ensure that there is enough edge time sequence length in each window for measuring the correlation, and then calculates the Pearson correlation coefficient of the two edge time sequences. The Pearson correlation coefficient reflects the orthogonal relationship between the signals of the electrode channels. The larger the eigenvalue is, the weaker the connection between the two electrode channels is, and the more independent the information is. In addition, the length of the brain electrical signal segment required for calculating the Pearson correlation coefficient is short, which can be used for real-time functional connection calculation.

[0410] According to an embodiment of the present disclosure, edge-edge interaction features corresponding to the brain electrical signals of the plurality of electrode channels are extracted by calculating the Pearson correlation coefficient between each two edges in a candidate edge set, which is composed of the first preset number of edges after sorting all edge average phase synchronization indexes from large to small and retaining the first preset number of edges. The edge average phase synchronization index is used to indicate the phase synchronization strength of the edge. When the edge average phase synchronization index is 0 or approximately 0, it means that the phase difference between the two nodes is randomly distributed and has no synchronization. When the edge average phase synchronization index is 1 or approximately 1, it means that the phase difference between the two nodes is constant and completely synchronized. By pre-screening all edges based on the edge average phase synchronization index, only the high-synchronization functional connection edges are retained, and then the instantaneous linear correlation of the high-synchronization edges in the local window is analyzed in the sliding window. By calculating the sparse Pearson correlation coefficient between the high-synchronization edges, dynamic interaction and rapidly changing connections can be captured, and full connection calculation is avoided, thereby significantly reducing the computational complexity. Therefore, by combining pre-screening and sparse Pearson calculation, the calculation efficiency can be significantly improved while ensuring the accuracy.

[0411] According to an embodiment of the present disclosure, in calculating the Pearson correlation coefficient between any two edges in the candidate edge set, the instantaneous phase difference of the first edge and the second edge at each time point in the preset time window is calculated respectively, wherein the instantaneous phase difference is the difference between the instantaneous phases of one node and another node in the edge. Based on the instantaneous phase difference of the first edge and the instantaneous phase difference of the second edge at each time point in the preset time window, the sparse Pearson correlation coefficient between the first edge and the second edge is calculated by using a sliding window.

[0412] The edge average phase synchronization index is calculated by the following formula:

[0413]

[0414] The instantaneous phase of the node is calculated by the following formula:

[0415] z(t) = x(t) + j H(x(t));

[0416]

[0417] The instantaneous phase difference of the first edge and the instantaneous phase difference of the second edge are obtained by the following formula:

[0418]

[0419]

[0420] The Pearson correlation coefficient between the first edge and the second edge is calculated by the following formula:

[0421]

[0422] wherein x(t) is a node signal, z(t) is a node analytic signal, j is an imaginary unit, and H() is a Hilbert transform operator; denotes the instantaneous phase of a node at time point t, and arg() represents taking a phase angle, denotes the instantaneous phase of a node in the first edge at time point t, denotes the instantaneous phase of another node in the first edge at time point t, denotes the instantaneous phase of a node in the second edge at time point t, denotes the instantaneous phase of another node in the second edge at time point t, denotes the instantaneous phase difference of the first edge at time point t, denotes the instantaneous phase difference of the second edge at time point t, T is the number of time points in the preset time window, W is the length of a sliding window, W < T, μ ij is the mean of the instantaneous phase difference of the first edge in the sliding window, μ uv is the mean of the instantaneous phase difference of the second edge in the sliding window.

[0423] According to an embodiment of the present disclosure, the method further comprises:

[0424] dynamically drawing electrodes corresponding to the plurality of electrode channels on the individualized three-dimensional head model of the visualization interface, the electrodes being distributed on a scalp surface portion of the individualized three-dimensional head model; dynamically displaying the connection relationship and the connection strength between the electrodes and / or between the electrode connections in real time according to the brain network features, electrode display parameters input by a user, and feature display parameters, the electrode display parameters being used to indicate display of one or more of the electrodes corresponding to the plurality of electrode channels, and the feature display parameters being used to indicate display of node-node functional connection features and / or edge-edge interaction features in the brain network features.​

[0425] Specifically, a connection line can be used to represent the connection relationship between different electrodes, the thickness and color of the connection line represent the connection strength, and the electrodes that need to be displayed can be selected, and the electrodes that are not selected are not displayed. The refresh rate and delay of the functional connection feedback are the same as the brain wave form display parameters.

[0426] Electroencephalogram tracing / source localization refers to a technology for calculating neural electrical activity in the brain by using scalp electroencephalogram signals and a head electromagnetic model. By using this technology, signals in the electrode space on the scalp surface can be mapped to the cerebral cortex (source space), thereby directly reflecting the distribution of neural electrical activity in the brain. In the present disclosure, for different individuals, an individual head conductivity model is calculated using individual T1 image MRI data, and then the individual head conductivity model is used as a parameter to realize the MNE (Minimum Norm Estimation) method of electroencephalogram tracing in real time, thereby realizing real-time mapping of electroencephalogram features from the electrode space to the source space.

[0427] Specifically, according to an embodiment of the present disclosure, the method further comprises: extracting source space features corresponding to the electroencephalogram signals of the plurality of electrode channels, comprising: obtaining a pre-calculated lead field pseudo-inverse matrix, and obtaining the source space features based on the pre-calculated lead field pseudo-inverse matrix and the electroencephalogram signals of the plurality of electrode channels; wherein the lead field pseudo-inverse matrix is determined based on a lead field matrix and an offline-optimized regularization parameter, the lead field matrix is used to describe the electrical signal transmission relationship from the source space to the electrode space, and the lead field matrix is calculated by using a pre-constructed individualized three-dimensional head model corresponding to the collected user as input; the offline-optimized regularization parameter is obtained by drawing a relationship curve between a residual norm and a solution norm, and selecting a regularization parameter corresponding to a maximum curvature point of the relationship curve.

[0428] wherein the source space features are calculated by the following formula:

[0429]

[0430] or, the source space features are calculated by the following formula:

[0431]

[0432] L + =L T (LL T +λI) -1 ;

[0433] R=L T (LL T +λI)-1 L;

[0434] wherein, is the source space feature at time t, x(t) is the electroencephalogram signal of the plurality of electrode channels corresponding to time t, L + is the lead field pseudo-inverse matrix, L is the lead field matrix, L∈R C×D , C is the number of scalp electrodes, D is the source space dimension, T is the transpose operation of the matrix, I is the unit matrix, λ is the regularization parameter optimized offline, diag() represents extracting the diagonal elements of the matrix.

[0435] In the present disclosure, when extracting the source space feature corresponding to the electroencephalogram signal of the plurality of electrode channels, on the one hand, the complexity of source space feature calculation is reduced by pre-computing the lead field pseudo-inverse matrix, the calculation efficiency is improved, and the real-time performance of source space feature extraction is ensured; on the other hand, when calculating the lead field pseudo-inverse matrix, the lead field matrix used is calculated based on the individualized three-dimensional head model corresponding to the user being collected, thereby improving the source localization accuracy; in addition, when calculating the lead field pseudo-inverse matrix, the regularization parameter optimized offline is used, and when optimizing the regularization parameter offline, the regularization parameter corresponding to the maximum curvature point of the relationship curve between the residual norm and the solution norm is determined by drawing the relationship curve between the residual norm and the solution norm, the residual norm is used to measure the difference between the electrode signal predicted by the estimated source activity through the forward model and the actual observed signal, the smaller the value, the more the solution of the inverse problem fits the observation data; the solution norm is used to quantify the overall amplitude (energy) of the source activity estimation, the larger the value, the more unstable or sparse the solution is. Through the balance of the two parameters, overfitting noise or excessively smoothed solution can be avoided, thereby improving the reliability of source localization. Among them, the maximum curvature point corresponds to the best trade-off between residual and solution complexity, ensuring that the source localization is stable and accurate.

[0436] According to an embodiment of the present disclosure, the method further comprises:

[0437] dynamically drawing the source space activation map of the corresponding part of the individualized three-dimensional head model on the visualization interface according to the model display parameter input by the user, the individualized three-dimensional head model comprising: a cerebral surface part, a skull surface part and a scalp surface part; and dynamically displaying the source space electrical activity intensity in the source space activation map of the corresponding part of the displayed individualized three-dimensional head model in real time according to the source space feature.

[0438] In a specific example, a web page is used to dynamically display the source space electrical activity distribution of real-time tracing of five-channel EEG signals. Based on individual MRI data, the individual's head geometry boundary element model (BEM) is calculated, and the web page displays the individual's head geometry model according to user selection, including brain surface modeling, skull surface modeling and scalp surface modeling. The user can choose whether to display the geometry model of each part on the web page and adjust its transparency. The observation angle can be adjusted by dragging the model. The skull is distributed with the individual's source space, and the source space electrical activity intensity is dynamically displayed according to the real-time calculation of the source space feature (i.e., the source space feature). The refresh rate of the dynamic feedback of the source space feature is the same as the delay of the EEG waveform display, and the delay is the same as the delay of the EEG waveform display.

[0439] According to an embodiment of the present disclosure, the method further comprises: obtaining a dynamic regulation parameter; wherein the dynamic regulation parameter is generated according to the stimulation task requirement, or is generated according to the brain network feature corresponding to the brain electrical signals of the plurality of electrode channels; and adjusting the characteristic parameter of the audio performance according to the dynamic regulation parameter.

[0440] When obtaining the dynamic regulation parameter, the present disclosure has two implementation modes:

[0441] Mode 1: manual intervention mode. When performing neural regulation, experts observe the waveform of the collected brain electrical signals of the user, the dynamically displayed brain network feature or the source space feature through the visualization interface, and then form a stimulation task for the current brain electrical signal according to prior knowledge. The dynamic regulation parameter is generated according to the stimulation task requirement, and the characteristic parameter of the audio performance is manually input to adjust the current brain wave audio. For example: the expert observes that the user's frontal lobe theta wave (4-8Hz) energy is insufficient, and wants to enhance the low-frequency note feedback to improve attention. The theta band gain can be adjusted from 1.0 to 1.5 to change the pitch of the theta band note (lower), increase the volume, and adjust the note density from 5 to 8 to affect the note frequency to increase the number of notes per unit time and make the rhythm more dense. By adjusting the brain wave audio in this way, the effect of neural regulation to improve attention is achieved.

[0442] Mode 2: automatic analysis mode. The dynamic regulation parameter can be generated according to the brain network feature (such as node-node functional connection feature) corresponding to the brain electrical signals of the plurality of electrode channels, that is, for a brain electrical signal corresponding to a period of time T, when the corresponding brain wave audio is generated according to the brain electrical frequency domain feature and / or brain electrical time domain feature of the specified frequency band brain electrical signal in the brain electrical signal, the dynamic regulation parameter is generated according to the functional connection feature between the electrode channel to which the brain electrical signal belongs and other electrode channels, and then the modulation of the cross-channel audio characteristic parameter is triggered.

[0443] In a specific example, for a brain electrical signal corresponding to a time period T, the brain electrical signal is represented as: EEG(t), t∈T, first, the node-node functional connection feature between the electrode channel i where the brain electrical signal is located and other electrode channels in the plurality of electrode channels is obtained, then the electrode channel j with the strongest connection relationship is selected, that is, the maximum FC ij is selected from the obtained node-node functional connection features, in order to avoid noise interference, it is judged whether the maximum satisfies a preset condition (such as: FC ij is greater than a preset threshold th), if yes, the node-node functional connection feature with the maximum value FC ij is taken as the dynamic regulation parameter, then the characteristic parameter of the audio performance is adjusted according to the dynamic regulation parameter; if no, the modulation of the cross-channel audio characteristic parameter is not triggered.

[0444] When adjusting the characteristic parameter of the audio performance according to the dynamic regulation parameter, the characteristic parameter of the audio performance that can be adjusted includes but is not limited to: envelope and pitch.

[0445] Wherein, the envelope of the audio performance generated based on the specified frequency band brain electrical signal in the brain electrical signal can be adjusted according to the dynamic regulation parameter by using the following formula:

[0446]

[0447] The pitch of the audio performance generated based on the specified frequency band brain electrical signal in the brain electrical signal can be adjusted according to the dynamic regulation parameter by using the following formula:

[0448]

[0449] Wherein, envelope is the adjusted envelope of the audio performance generated based on the specified frequency band brain electrical signal in the brain electrical signal of the electrode channel i, envelope i is the original envelope of the audio performance generated based on the specified frequency band brain electrical signal in the brain electrical signal of the electrode channel i, envelope j is the envelope of the audio performance generated based on the specified frequency band brain electrical signal in the brain electrical signal of the electrode channel j, FC ij is the node-node functional connection feature between the electrode channel i and the electrode channel j, pitch is the adjusted pitch of the audio performance generated based on the specified frequency band brain electrical signal in the brain electrical signal of the electrode channel i, pitch i is the original pitch of the audio performance generated based on the specified frequency band brain electrical signal in the brain electrical signal of the electrode channel i, pitch jThe pitch of the audio performance corresponding to the generated audio performance of the specified frequency band of the electroencephalogram signal based on the electrode channel j is C1, the cross-channel envelope influence degree coefficient is C2, and the cross-channel pitch influence degree coefficient is C2.

[0450] In specific applications, one of mode 1 and mode 2 can be selected for use according to the application scenario and task requirements, or mode 1 and mode 2 can be combined for use.

[0451] For mode 1:

[0452] The present disclosure dynamically displays the waveform of the electroencephalogram signal, the brain network feature, and the source space feature on the visualization interface, so that experts can observe and understand the current neural activity of the collected user in real time, then form a targeted stimulation scheme, and dynamically adjust the audio parameters through the interactive interface, which can target intervention for specific neural oscillation defects, and realize precise neural feedback reinforcement. For example: by increasing the gain of the theta band to enhance the low-frequency note feedback, thereby promoting the synchronization of the frontal lobe theta wave and improving attention; by reducing the volume of the alpha band to reduce excessive excitement, thereby relieving anxiety. In addition, the strategy can be flexibly adjusted according to the real-time response of the user (such as brain wave changes, behavior performance), avoiding the limitations of the "one-size-fits-all" algorithm. For example: increasing the rhythm complexity to maintain interest for children with ADHD, and simplifying the melody for anxious adults to induce relaxation. In addition, by artificially identifying abnormal rhythms (such as epileptiform discharges), the audio parameters (such as inserting a silent segment or a low-frequency pulse) can be modified in real time to interrupt abnormal neural activity and reduce the risk of seizures, thereby correcting abnormal electroencephalogram patterns.

[0453] For mode 2:

[0454] The present disclosure automatically generates dynamic regulation parameters based on brain network features, which can respond to electroencephalogram changes at the millisecond level, form a closed-loop regulation, and improve the real-time closed-loop feedback efficiency. For example: when the high connection between the motor cortex and the auditory cortex is detected, the rhythm is automatically enhanced to strengthen the motor imagination training effect. In addition, by driving audio modulation through node-node functional connections, the functional integration of remote brain areas (such as the default mode network and the executive control network) is promoted, thereby more accurately capturing the functional association between brain areas and improving the neural relevance of the synthesized audio. For example: when the parietal lobe-frontal lobe has high connection, the two areas are fused to generate a composite melody, enhancing working memory.

[0455] In summary, the present disclosure improves the neural regulation accuracy and artistic expressiveness of the brain wave audio system from the perspectives of human intelligence and computational intelligence through the above two modes.

[0456] Figure 10A structural schematic diagram of a body brain wave audio auditory perception synchronous feedback device according to an embodiment of the present disclosure is shown, the device 1000 is arranged in a host computer, the host computer is connected with an electroencephalogram acquisition device through an electroencephalogram data transmission interface, the device 1000 comprises: an electroencephalogram signal receiving module, an electroencephalogram feature extraction module and an electroencephalogram-audio generation module, wherein the electroencephalogram signal receiving module is configured to receive the electroencephalogram signal of the collected user online from the electroencephalogram acquisition device through the electroencephalogram data transmission interface, the electroencephalogram signal is obtained by the electroencephalogram acquisition device through one or more electrode channels, wherein each electrode channel corresponds to one electrode or a combination of multiple electrodes; the electroencephalogram feature extraction module is configured to obtain one or more specified frequency band electroencephalogram signals in the electroencephalogram signal, extract electroencephalogram features corresponding to the specified frequency band electroencephalogram signal based on the one or more specified frequency band electroencephalogram signals, the electroencephalogram features include electroencephalogram time domain features and / or electroencephalogram frequency domain features; the electroencephalogram-audio generation module is configured to generate audio performance corresponding to the audio performance form according to the electroencephalogram features based on the specified audio performance form and the specified audio generation mode for the specified frequency band electroencephalogram signal, the characteristic parameters of the audio performance are determined according to the electroencephalogram features; the audio performance form includes: note form and / or waveform form, wherein the audio performance corresponding to the note form is a note, the audio performance corresponding to the waveform form is a waveform, the audio generation mode includes: direct mapping mode and / or indirect control mode using control parameters; generate brain wave audio representation data according to the audio performance corresponding to the one or more specified frequency band electroencephalogram signals; obtain brain wave audio according to the brain wave audio representation data.

[0457] Figure 11 A structural schematic diagram of another body brain wave audio auditory perception synchronous feedback device according to an embodiment of the present disclosure is shown, the electroencephalogram signal includes a denoised electroencephalogram signal, the device 1100 further comprises: an electroencephalogram preprocessing module, the electroencephalogram preprocessing module is connected with the electroencephalogram signal receiving module and the electroencephalogram feature extraction module respectively; the electroencephalogram preprocessing module is configured to: before obtaining the one or more specified frequency band electroencephalogram signals in the electroencephalogram signal through the electroencephalogram feature extraction module, pre-process the electroencephalogram signals of the one or more electrode channels by using a pre-trained brain-derived signal and non-brain-derived signal separation network model, obtain a denoised electroencephalogram signal, and transmit the denoised electroencephalogram signal to the electroencephalogram feature extraction module.

[0458] Figure 12Fig. 12 shows a structural schematic diagram of still another brain wave audio hearing perception synchronization feedback device according to an embodiment of the present disclosure. The device 1200 further comprises a feedback module, which comprises an audio feedback module; the feedback module is configured to feed back the brain wave audio to the collected user through the audio feedback module, so as to dynamically regulate the nervous system activity of the collected user through hearing perception.

[0459] According to an embodiment of the present disclosure, the device further comprises a dynamic regulation module connected with the EEG feature extraction module, configured to obtain a dynamic regulation parameter; wherein the dynamic regulation parameter is generated according to stimulation task requirements, or is generated according to brain network features corresponding to the EEG signals of the multiple electrode channels; and the characteristic parameters of the audio performance are adjusted according to the dynamic regulation parameter.

[0460] According to an embodiment of the present disclosure, the EEG preprocessing module is connected with the feedback module, and the feedback module further comprises an EEG waveform feedback module; the feedback module is further configured to dynamically display the waveform of the denoised EEG signal on a visual interface in real time through the EEG waveform feedback module.

[0461] According to an embodiment of the present disclosure, the EEG feature extraction module is further configured to:

[0462] extract brain network features corresponding to the brain electrical signals of the plurality of electrode channels, the brain network features comprising: node-node functional connectivity features and edge-edge interaction features, the node-node functional connectivity features being used to describe statistical dependence between two node signals, and the edge-edge interaction features being used to describe temporal dependence between two edges; wherein the node corresponds to a single electrode channel, the node signal corresponds to the brain electrical signal of a single electrode channel, and the edge corresponds to a connection relationship between two nodes; wherein the node-node functional connectivity features corresponding to the brain electrical signals of the plurality of electrode channels are extracted by calculating a weighted phase-lag coefficient between each two node signals, and in calculating the weighted phase-lag coefficient between any two node signals, the following steps are included: performing Hilbert transform on a first node signal and a second node signal to obtain a first node analytic signal and a second node analytic signal corresponding to the first node signal and the second node signal respectively; obtaining an instantaneous cross power spectrum of the first node signal and the second node signal based on the first node analytic signal and the second node analytic signal; averaging the absolute value of the imaginary part of the instantaneous cross power spectrum at each time point in a preset time window to obtain an average imaginary part absolute value; calculating the weight of the imaginary part of the instantaneous cross power spectrum at each time point in the preset time window, and obtaining a weighted complex covariance based on the instantaneous cross power spectrum at each time point in the preset time window and the weight of the imaginary part corresponding to the instantaneous cross power spectrum; and calculating the weighted phase-lag coefficient of the first node signal and the second node signal by using the absolute value of the imaginary part of the weighted complex covariance and the average imaginary part absolute value; and the edge-edge interaction features corresponding to the brain electrical signals of the plurality of electrode channels are extracted by calculating a Pearson correlation coefficient between each two edges in a candidate edge set, the candidate edge set being formed by calculating an average phase synchronization index of all edges, then sorting the average phase synchronization indexes of all edges from large to small and retaining the first preset number of edges, and in calculating the Pearson correlation coefficient between any two edges in the candidate edge set, the following steps are included: calculating the instantaneous phase difference of a first edge and a second edge at each time point in a preset time window respectively, wherein the instantaneous phase difference is the difference between the instantaneous phase of one node in the edge and the instantaneous phase of another node in the edge; and calculating the sparse Pearson correlation coefficient between the first edge and the second edge by using a sliding window based on the instantaneous phase difference of the first edge and the instantaneous phase difference of the second edge at each time point in the preset time window.

[0463] According to an embodiment of the present disclosure, the electroencephalogram feature extraction module is connected with the feedback module, and the feedback module further comprises a brain network functional connection feedback module; the feedback module is further configured to: draw electrodes corresponding to the plurality of electrode channels on an individualized three-dimensional head model of a visual interface dynamically through the brain network functional connection feedback module, the electrodes being distributed on a scalp surface part of the individualized three-dimensional head model; and display the connection relationship and the connection strength between the electrodes and / or the connection between the electrode connections in real time dynamically according to the brain network features, electrode display parameters input by a user, and feature display parameters; the electrode display parameters are used to indicate display of one or more of the electrodes corresponding to the plurality of electrode channels, and the feature display parameters are used to indicate display of node-node functional connection features and / or edge-edge interaction features in the brain network features.

[0464] According to an embodiment of the present disclosure, the electroencephalogram feature extraction module is further configured to:

[0465] extracting source space features corresponding to the electroencephalogram signals of the plurality of electrode channels comprises: obtaining a pre-calculated lead field pseudo-inverse matrix, and obtaining the source space features based on the pre-calculated lead field pseudo-inverse matrix and the electroencephalogram signals of the plurality of electrode channels; wherein the lead field pseudo-inverse matrix is determined based on a lead field matrix and a regularization parameter optimized offline, the lead field matrix is used to describe an electrical signal transmission relationship from a source space to an electrode space, and the lead field matrix is calculated by taking T1-weighted magnetic resonance imaging data of the collected user as input and using a pre-constructed individualized three-dimensional head model corresponding to the collected user; and the regularization parameter optimized offline is obtained by drawing a relationship curve of a residual norm and a solution norm, and selecting a regularization parameter corresponding to a maximum curvature point of the relationship curve.

[0466] According to an embodiment of the present disclosure, the electroencephalogram feature extraction module is connected with the feedback module, and the feedback module further comprises a brain network functional connection feedback module; the feedback module is further configured to: draw electrodes corresponding to the plurality of electrode channels on an individualized three-dimensional head model of a visual interface dynamically through the brain network functional connection feedback module, the electrodes being distributed on a scalp surface part of the individualized three-dimensional head model; and display the connection relationship and the connection strength between the electrodes and / or the connection between the electrode connections in real time dynamically according to the brain network features, electrode display parameters input by a user, and feature display parameters; the electrode display parameters are used to indicate display of one or more of the electrodes corresponding to the plurality of electrode channels, and the feature display parameters are used to indicate display of node-node functional connection features and / or edge-edge interaction features in the brain network features.

[0467] According to an embodiment of the present disclosure, the feedback module further comprises: a sound image position feedback module; the feedback module is further configured to: acquire, by the sound image position feedback module, sound image position parameters of the brainwave audio corresponding to a plurality of sound channels, and dynamically display the sound image position of the brainwave audio on a visual interface according to the sound image position parameters.

[0468] The present disclosure also discloses an electronic device, Figure 13 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. As Figure 13 shown, the electronic device 1300 comprises the apparatus according to an embodiment of the present disclosure.

[0469] Those skilled in the art can understand that the apparatus can be arranged on one electronic device, or on multiple electronic devices.

[0470] Taking the apparatus arranged on two electronic devices as an example, specifically, the electronic device comprises: a first electronic device and a second electronic device, in a specific embodiment, as Figure 14 shown, the EEG signal receiving module, the EEG preprocessing module and the feedback module in the apparatus are arranged on the first electronic device, and the EEG feature extraction module and the EEG-audio generation module in the apparatus are arranged on the second electronic device; in another specific embodiment, as Figure 15 shown, the EEG signal receiving module and the feedback module in the apparatus are arranged on the first electronic device, and the EEG preprocessing module, the EEG feature extraction module and the EEG-audio generation module in the apparatus are arranged on the second electronic device.

[0471] Figure 16 A structural block diagram of another electronic device according to an embodiment of the present disclosure is shown. As Figure 16 shown, the electronic device comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to an embodiment of the present disclosure.

[0472] Figure 17 A structural block diagram of a body brainwave audio auditory perception synchronization feedback system according to an embodiment of the present disclosure is shown. As Figure 17 shown, the system 1700 comprises: an EEG acquisition device and an electronic device according to an embodiment of the present disclosure, the EEG acquisition device is connected with the electronic device through an EEG data transmission interface, wherein:

[0473] The EEG acquisition device is configured to acquire the EEG signal of a user to be collected through one or more electrode channels, and transmit the EEG signal of the one or more electrode channels to the electronic device online based on the EEG data transmission interface.

[0474] In a specific embodiment, the body brain wave audio auditory perception synchronous feedback system is implemented based on an end-edge-cloud collaborative system. Specifically, the electroencephalogram acquisition device is the end, the computer (upper computer) matched with the acquisition device is the edge, the end-edge contact is established, the electroencephalogram signal is sent from the end side to the edge side at a transmission interval of 30 ms, the electroencephalogram signal is sent from the edge side to the cloud in real time, the functions of electroencephalogram feature extraction, electroencephalogram-audio generation and the like are implemented in the cloud, and the audio generation result is returned to the edge side in real time. The edge side performs real-time dynamic feedback of multiple electroencephalogram feature calculation results, audio synthesis, real-time playing and the like based on web development.

[0475] The present disclosure also provides a computer readable storage medium, which can be a computer readable storage medium contained in the electronic device or computer system in the above embodiments; or can exist separately and not be assembled into the device. The computer readable storage medium stores one or more programs, which are executed by one or more processors to perform the method described in the present disclosure.

[0476] The present disclosure also provides a computer program product, which includes a computer program executed by a processor to implement the method described in any one of the present disclosure.

[0477] According to the technical scheme provided by the embodiments of the present disclosure, the upper computer receives the electroencephalogram signal of the collected user from the electroencephalogram acquisition device online, obtains one or more specified frequency band electroencephalogram signals in the electroencephalogram signal, extracts the electroencephalogram features corresponding to the specified frequency band electroencephalogram signal, generates the audio performance corresponding to the specified audio performance form according to the electroencephalogram features based on the specified audio performance form and the specified audio generation method, generates the brain wave audio representation data according to the audio performance corresponding to the one or more specified frequency band electroencephalogram signals, and obtains the brain wave audio according to the brain wave audio representation data. When the brain wave audio is fed back to the collected user synchronously, the nervous system activity of the collected user can be dynamically regulated in real time through auditory perception, the brain wave audio obtained based on the technical scheme can accurately match the electroencephalogram features of the collected user, the electroencephalogram features are generated based on the electroencephalogram signal obtained online in real time, and thus the closed-loop electroencephalogram-audio real-time feedback interaction is realized, the physiological adaptability of neural regulation and the artistic expressiveness of audio generation are simultaneously satisfied, and the organic unification of neural regulation and audio generation is realized.

[0478] The above description is merely that of the preferred embodiments of the present disclosure and a description of the technical principles of the present disclosure. It should be understood by those skilled in the art that the inventive scope involved in the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the present disclosure (but not limited to) without departing from the inventive concept.

Claims

1. A method for synchronous feedback of proprioceptive brainwave audio and auditory perception, characterized in that: The method is applied to a host computer, which is connected to an EEG acquisition device via an EEG data transmission interface, and includes: Receiving an EEG signal of a user being collected online from the EEG collection device via the EEG data transmission interface, wherein the EEG signal is obtained by the EEG collection device through one or more electrode channels, wherein each electrode channel corresponds to one electrode or a combination of multiple electrodes; Acquiring one or more designated frequency band EEG signals from the EEG signals, and extracting EEG features corresponding to the designated frequency band EEG signals based on the one or more designated frequency band EEG signals, wherein the designated frequency band EEG signals refer to EEG signals in the designated frequency band, and the EEG features include EEG time domain features and / or EEG frequency domain features; For the EEG signal of the specified frequency band, based on the specified audio representation form and the specified audio generation method, the EEG signal of the specified frequency band is divided into multiple EEG segments according to the EEG characteristics or preset rhythm parameters, and multiple audio representations corresponding to the multiple EEG segments are generated, wherein the characteristic parameters of the audio representation are determined according to the EEG characteristics of the corresponding EEG segments; the audio representation form includes: a note form and / or a waveform form, wherein the audio representation corresponding to the note form is a note, and the audio representation corresponding to the waveform form is a waveform; the audio generation method includes: a direct mapping method and / or an indirect control method using control parameters; generating brainwave audio representation data according to the audio representation corresponding to the one or more designated frequency band electroencephalogram signals; The brainwave audio is obtained according to the brainwave audio representation data.

2. The method according to claim 1, characterized in that The method further comprises: The brainwave audio is synchronously fed back to the collected user to dynamically regulate the nervous system activity of the collected user through auditory perception.

3. The method according to claim 2, characterized in that The step of synchronously feeding back the brainwave audio to the collected user includes: Obtaining brainwave audio corresponding to a plurality of channels based on the brainwave audio according to a combination relationship between the spatial positions of the electrodes in the one or more electrode channels and the one or more designated frequency bands; The brainwave audio of the multiple channels is played through an audio device so as to synchronously feed back the brainwave audio to the collected user.

4. The method according to claim 1, wherein The EEG signal includes a denoised EEG signal. Before acquiring one or more EEG signals in a specified frequency band from the EEG signal, the method further includes: Preprocessing the EEG signals of the one or more electrode channels using a pretrained brain-derived signal and non-brain-derived signal separation network model to obtain denoised EEG signals; The brain-derived signal and non-brain-derived signal separation network model includes: a denoising module and a skip connection module. The denoising module includes: a decomposition module, a channel spatiotemporal attention processing module and a reconstruction module. The method of preprocessing the EEG signals of the one or more electrode channels using a pretrained brain-derived signal and non-brain-derived signal separation network model to obtain a denoised EEG signal includes: The denoising module removes non-brain-derived signals from the EEG signals of the one or more electrode channels, including: decomposing the EEG signals of the one or more electrode channels into multidimensional embedding vectors corresponding to multiple signal channels through the decomposition module, wherein the multiple signal channels include brain-derived signal channels and non-brain-derived signal channels; performing channel attention processing and spatiotemporal attention processing on the multidimensional embedding vectors through the channel spatiotemporal attention processing module, respectively generating channel attention weights and time attention weights, and fusing the generated channel attention weights and time attention weights with the multidimensional embedding vectors corresponding to the multiple signal channels to obtain a processed multidimensional embedding vector; and generating a reconstructed EEG signal according to the processed multidimensional embedding vector through the reconstruction module; The processed multidimensional embedding vector is obtained through the jump connection module, and based on the processed multidimensional embedding vector and the EEG signals of the one or more electrode channels, the gated linear unit GLU and the noise perception module NAM are used to collaboratively generate dynamic fusion parameters, and the EEG signals of the one or more electrode channels are fused with the reconstructed EEG signals according to the dynamic fusion parameters to obtain a denoised EEG signal. The dynamic fusion parameters are used to control the fusion ratio of the EEG signals of the one or more electrode channels and the reconstructed EEG signals.

5. The method according to claim 4, characterized in that The method further comprises: The waveform of the denoised EEG signal is dynamically displayed in real time on a visual interface.

6. The method according to claim 1, characterized in that in, The EEG frequency domain features include: EEG power spectrum features, and the EEG time domain features include: time domain envelope; When extracting the EEG power spectrum features corresponding to the EEG signals of the specified frequency band based on the one or more EEG signals of the specified frequency band, extracting the refined EEG power spectrum features corresponding to the EEG signals of the specified frequency band through a frequency band selective spectrum transformation algorithm, including: calculating the rotation factor and the starting point according to the starting frequency and the ending frequency of the specified frequency band and the target resolution; executing the frequency band selective spectrum transformation algorithm on the EEG signals of the specified frequency band according to the rotation factor and the starting point to obtain the spectrum corresponding to the EEG signals of the specified frequency band; calculating the refined EEG power spectrum features corresponding to the EEG signals of the specified frequency band based on the spectrum; the frequency band selective spectrum transformation algorithm includes any one of the following transformation algorithms: Chirp-Z transform, Zoom-FFT, Goertzel algorithm, continuous wavelet transform CWT.

7. The method according to claim 1, characterized in that When the specified audio representation is in the form of musical notes and the specified audio generation method is a direct mapping method, the characteristic parameters of the audio representation include: note duration, pitch, and intensity; the segmenting of the specified frequency band EEG signal into multiple EEG segments according to the preset rhythm parameter, and the generation of multiple audio representations corresponding to the multiple EEG segments, respectively, include: For the EEG signal of the specified frequency band of the specified electrode channel with a preset duration: Segmenting the EEG signal of the preset duration according to the note duration indicated by the preset rhythm parameter to obtain an EEG segment set, wherein the EEG segment set includes multiple EEG segments, and each EEG segment corresponds to one note; For any EEG segment in the set of EEG segments, the pitch and intensity of the note corresponding to any EEG segment are determined according to the EEG frequency domain characteristics corresponding to the any EEG segment, until the pitch and intensity of the note corresponding to each EEG segment in the set of EEG segments are obtained, including: based on the power spectrum characteristics corresponding to any EEG segment, selecting the peak frequency or weighted average frequency with the largest amplitude in the power spectrum characteristics as the characteristic frequency, based on a preset mapping method, obtaining the corresponding specified pitch range according to the frequency range of the specified frequency band, and obtaining the pitch corresponding to any EEG segment according to the characteristic frequency and the specified pitch range; generating the intensity corresponding to any EEG segment according to the total energy or peak amplitude of the power spectrum characteristics; the preset mapping method is used to describe the mapping relationship between the frequency range of the specified frequency band and the pitch range.

8. The method according to claim 7, characterized in that Generating brainwave audio representation data according to the audio representation corresponding to the one or more designated frequency band electroencephalogram signals includes: Writing the musical notes corresponding to the respective EEG segments in the set of EEG segments of the EEG signals of the predetermined duration in the EEG signals of the designated frequency band of each electrode channel into the musical score corresponding to the designated frequency band; Writing the music score corresponding to each designated frequency band in each electrode channel into a complete music score set corresponding to the preset duration, and using the complete music score set as the brain wave audio representation data; The obtaining of brainwave audio according to the brainwave audio representation data includes: Configuring a timbre for the musical score corresponding to each specified frequency band in the brainwave audio representation data according to a preset frequency band-timbre mapping table or a configuration parameter-timbre mapping table; the frequency band-timbre mapping table is used to describe the correspondence between frequency bands and timbre, and the configuration parameter-timbre mapping table is used to describe the correspondence between specified timbre configuration parameters and timbre, wherein the specified timbre configuration parameters include: non-brain-derived control parameters and / or non-brain-derived event occurrence frequencies; The non-brain-derived control parameter is obtained as follows: when the EEG signal of the preset length of the designated electrode channel contains a non-brain-derived signal and a denoised EEG signal, the non-brain-derived control parameter is obtained by taking a preset quantile ratio of the absolute values ​​of the non-brain-derived signal and the denoised EEG signal in the EEG signal of the preset length of the designated electrode channel; The non-brain-source event frequency is obtained as follows: a non-brain-source event threshold is obtained based on a preset fractional value of the absolute value of the denoised EEG signal; positions in the non-brain-source signal that are above the non-brain-source event threshold are counted to obtain the number of non-brain-source events in the non-brain-source signal; and the non-brain-source event frequency is obtained based on the number of non-brain-source events and the preset duration. The brainwave audio is generated by a MIDI synthesizer or an audio synthesis library according to the musical scores corresponding to each designated frequency band in the brainwave audio representation data.

9. The method according to claim 1, characterized in that When the designated audio representation is in the form of musical notes and the designated audio generation method is an indirect control method using control parameters, the characteristic parameters of the audio representation include: note duration, pitch, and intensity; the segmenting of the designated frequency band EEG signal into multiple EEG segments based on the EEG characteristics and the generation of multiple audio representations corresponding to the multiple EEG segments respectively include: Taking the EEG signal of a preset duration in the designated frequency band as input, generating control parameters using a pre-trained user state discrimination model, the control parameters including arousal and valence; Calculating the note duration of a single note in the note sequence corresponding to the EEG signal of the preset duration according to the arousal level and / or the valence; Segmenting the EEG signal of the preset duration according to the note duration to obtain an EEG segment set, wherein the EEG segment set includes multiple EEG segments, and each EEG segment corresponds to one note; Calculating the occurrence probability of the musical note corresponding to each EEG segment in the EEG segment set based on the arousal degree and / or the valence, and when the occurrence probability satisfies an occurrence condition, the musical note corresponding to the corresponding EEG segment appears; otherwise, the musical note corresponding to the corresponding EEG segment does not appear; and determining a musical note sequence corresponding to the EEG signal of the preset duration based on the calculation result; Calculating the intensity of the notes in the note sequence based on the arousal level, the valence, and power spectrum energy characteristics of the EEG segments corresponding to the notes in the note sequence; Calculating tonality parameters corresponding to notes in the note sequence according to the arousal and / or the valence; Calculating the pitches of the notes in the note sequence based on the tonality parameters and the power spectrum frequency characteristics of the EEG segments corresponding to the notes in the note sequence; configuring the timbre of the notes in the note sequence according to the arousal and / or the valence; or, Configuring the timbre of all notes in the note sequence according to a preset configuration parameter-timbre mapping table; the configuration parameter-timbre mapping table is used to describe the correspondence between specified timbre configuration parameters and timbre, the specified timbre configuration parameters including: non-brain-derived control parameters and / or non-brain-derived event occurrence frequencies; The non-brain-derived control parameter is obtained as follows: when the EEG signal of the preset length of the designated electrode channel contains a non-brain-derived signal and a denoised EEG signal, the ratio of the absolute values ​​of the non-brain-derived signal and the denoised EEG signal in the EEG signal of the preset length of the designated electrode channel to a preset quantile is used as the non-brain-derived control parameter; The frequency of non-brain-source events is obtained as follows: a non-brain-source event threshold is obtained based on a preset fractional quantile after taking the absolute value of the denoised EEG signal, positions in the non-brain-source signal that are above the non-brain-source event threshold are counted to obtain the number of non-brain-source events occurring in the non-brain-source signal, and the frequency of non-brain-source events is obtained based on the number of non-brain-source events and the preset duration.

10. The method according to claim 9, characterized in that Generating brainwave audio representation data according to the audio representation corresponding to the one or more designated frequency band electroencephalogram signals includes: generating a musical score corresponding to the designated frequency band based on the note sequence of the designated frequency band in each electrode channel, and the intensity, pitch and timbre of the notes in the note sequence; Writing the music score corresponding to each designated frequency band in each electrode channel into a complete music score set corresponding to the preset duration, and using the complete music score set as the brain wave audio representation data; The obtaining of brainwave audio according to the brainwave audio representation data includes: The brainwave audio is generated by a MIDI synthesizer or an audio synthesis library according to the musical scores corresponding to each designated frequency band in the brainwave audio representation data.

11. The method according to claim 1, wherein When the specified audio representation is in waveform form and the specified audio generation method is a direct mapping method, the characteristic parameters of the audio representation include: a fundamental frequency and a harmonic frequency, and the intensity of the fundamental frequency and the intensity of the harmonic frequency; dividing the EEG signal of the specified frequency band into a plurality of EEG segments according to the EEG characteristics, and generating a plurality of audio representations corresponding to the plurality of EEG segments, respectively, includes: For the EEG signal of the specified frequency band of the specified electrode channel with a preset duration: Segmenting the EEG signal of the preset duration based on the time domain envelope corresponding to the EEG signal of the preset duration to obtain a set of EEG segments, including: determining an effective amplitude threshold according to the mean value of the time domain envelope; determining a set of troughs of the time domain envelope according to the effective amplitude threshold, wherein the amplitudes of the troughs in the set of troughs are not greater than the effective amplitude threshold; segmenting the EEG signal of the preset duration according to the preset effective time threshold and the set of troughs of the time domain envelope to obtain the set of EEG segments, wherein the set of EEG segments includes one or more EEG segments, and the duration of each EEG segment is greater than the effective time threshold; For any EEG segment in the set of EEG segments, characteristic parameters of the waveform corresponding to any EEG segment are determined according to the EEG frequency domain characteristics corresponding to any EEG segment, until characteristic parameters of the waveform corresponding to each EEG segment in the set of EEG segments are obtained, including: based on the power spectrum characteristics corresponding to any EEG segment, determining the fundamental frequency and harmonic frequencies of the waveform corresponding to any EEG segment and the intensity of the fundamental frequency and the intensity of the harmonic frequencies according to the peak frequencies and amplitudes of the top preset number ranked by amplitude in the power spectrum characteristics.

12. The method according to claim 11, characterized in that Generating brainwave audio representation data according to the audio representation corresponding to the one or more designated frequency band electroencephalogram signals includes: For any EEG segment in the set of EEG segments, determining an audio waveform corresponding to the any EEG segment based on characteristic parameters and a time domain envelope of the waveform corresponding to the any EEG segment until an audio waveform corresponding to each EEG segment in the set of EEG segments is obtained, including: generating a sine wave of a corresponding frequency based on the fundamental frequency and the harmonic frequency of the waveform corresponding to the any EEG segment and the intensity of the fundamental frequency and the intensity of the harmonic frequency, and mixing the generated sine waves to obtain a mixed sine wave; and applying the time domain envelope corresponding to the any EEG segment to modulate the mixed sine wave to obtain the audio waveform corresponding to the any EEG segment; Using the audio waveform corresponding to each EEG segment in the set of EEG segments of each specified frequency band in the specified electrode channel as the audio waveform corresponding to the EEG signal of the specified electrode channel with the preset duration; The audio waveform corresponding to the EEG signal of the preset duration of each electrode channel is used as the EEG audio representation data; The obtaining of brainwave audio according to the brainwave audio representation data includes: The brainwave audio representation data is directly used as the brainwave audio.

13. The method according to claim 1, wherein The method further comprises: Extracting brain network features corresponding to the EEG signals of the multiple electrode channels, the brain network features including: node-node functional connectivity features and / or edge-edge interaction features, the node-node functional connectivity features being used to describe the statistical dependency between two node signals, and the edge-edge interaction features being used to describe the temporal dependency between two edges; wherein the node corresponds to a single electrode channel, the node signal corresponds to the EEG signal of a single electrode channel, and the edge corresponds to the connection relationship between two nodes; Among them, the node-node functional connection characteristics corresponding to the EEG signals of the multiple electrode channels are extracted by calculating the weighted phase lag coefficient between every two node signals. When calculating the weighted phase lag coefficient between any two node signals, it includes: performing Hilbert transform on the first node signal and the second node signal to obtain a first node analytic signal and a second node analytic signal corresponding to the first node signal and the second node signal respectively; obtaining the instantaneous cross power spectrum of the first node signal and the second node signal based on the first node analytic signal and the second node analytic signal; averaging the absolute value of the imaginary part of the instantaneous cross power spectrum at each time point in a preset time window within the preset time window to obtain the mean of the absolute value of the imaginary part; calculating the weight of the imaginary part of the instantaneous cross power spectrum at each time point in the preset time window, and obtaining a weighted complex covariance based on the instantaneous cross power spectrum at each time point in the preset time window and the weight of the imaginary part corresponding to the instantaneous cross power spectrum; and calculating the weighted phase lag coefficient of the first node signal and the second node signal using the absolute value of the imaginary part of the weighted complex covariance and the mean of the absolute value of the imaginary part; The edge-edge interaction features corresponding to the electroencephalogram signals of the multiple electrode channels are extracted by calculating the Pearson correlation coefficient between every two edges in the candidate edge set, wherein the candidate edge set is formed by calculating the average phase synchronization index of all edges, and then sorting the average phase synchronization index of all edges from large to small and retaining a preset number of edges. When calculating the Pearson correlation coefficient between any two edges in the candidate edge set, it includes: respectively calculating the instantaneous phase difference between the first edge and the second edge at each time point in a preset time window, wherein the instantaneous phase difference is the difference between the instantaneous phases of one node and another node in the edge; based on the instantaneous phase difference of the first edge and the instantaneous phase difference of the second edge at each time point in the preset time window, using a sliding window to calculate the sparse Pearson correlation coefficient between the first edge and the second edge.

14. The method according to claim 13, wherein: in, The first node resolution signal and the second node resolution signal corresponding to the first node signal and the second node signal, respectively, are obtained by the following formula: z i (t)=x i (t)+j·H(x i (t)); z j (t)=x j (t)+j·H(x j (t)); The instantaneous cross power spectrum is obtained by the following formula: The weight of the imaginary part of the instantaneous cross power spectrum at each time point in the preset time window is calculated by the following formula: The weighted complex covariance is obtained by the following formula: The weighted phase lag coefficient is calculated using the following formula: Among them, x i (t) is the first node signal, x j (t) is the second node signal, z i (t) is the first node analytical signal, z j (t) is the second node analytical signal, j is the imaginary unit, H() is the Hilbert transform operator; P ij (t) represents the instantaneous cross power spectrum, * represents the complex conjugate, median() is the median operation, is the weighted complex covariance, and T is the number of time points in the preset time window.

15. The method according to claim 13, characterized in that in, The average phase synchronization index of the edge is calculated by the following formula: The instantaneous phase of the node is calculated using the following formula: z(t)=x(t)+j·H(x(t)); The instantaneous phase difference of the first side and the instantaneous phase difference of the second side are obtained by the following formula: The Pearson correlation coefficient between the first edge and the second edge is calculated using the following formula: Where x(t) is the node signal, z(t) is the node analytical signal, j is the imaginary unit, and H() is the Hilbert transform operator; Represents the instantaneous phase of the node at time point t, arg() represents the phase angle, represents the instantaneous phase of a node in the first edge at time point t, represents the instantaneous phase of the other node in the first edge at time point t, represents the instantaneous phase of a node in the second edge at time point t, represents the instantaneous phase of the other node in the second edge at time point t, represents the instantaneous phase difference of the first edge at time point t, represents the instantaneous phase difference of the second edge at time point t, T is the number of time points in the preset time window, W is the length of the sliding window, μ ij is the mean instantaneous phase difference of the first edge in the sliding window, μ uv is the mean instantaneous phase difference of the second edge within the sliding window.

16. The method according to claim 13, characterized in that The method further comprises: Electrodes corresponding to the multiple electrode channels are dynamically drawn on a personalized three-dimensional head model of a visualization interface, and the electrodes are distributed on the scalp surface portion of the personalized three-dimensional head model; the connection relationship and connection strength between the electrodes and / or between the electrode connections are dynamically displayed in real time based on the brain network characteristics, the electrode display parameters input by the user, and the feature display parameters, wherein the electrode display parameters are used to indicate the display of one or more of the electrodes corresponding to the multiple electrode channels, and the feature display parameters are used to indicate the display of the node-node functional connection characteristics and / or the edge-edge interaction characteristics in the brain network characteristics.

17. The method according to claim 1, wherein The method further comprises: Extracting source space features corresponding to the EEG signals of the multiple electrode channels includes: obtaining a pre-calculated lead field pseudo-inverse matrix, and obtaining the source space features based on the pre-calculated lead field pseudo-inverse matrix and the EEG signals of the multiple electrode channels; wherein the lead field pseudo-inverse matrix is ​​determined based on the lead field matrix and a regularization parameter after offline optimization, the lead field matrix is ​​used to describe the electrical signal transmission relationship from the source space to the electrode space, the lead field matrix is ​​calculated using the T1-weighted magnetic resonance imaging data of the collected user as input and using a pre-constructed individualized three-dimensional head model corresponding to the collected user; the regularization parameter after offline optimization is obtained by drawing a relationship curve between the residual norm and the solution norm, and selecting the regularization parameter corresponding to the maximum curvature point of the relationship curve.

18. The method according to claim 17, characterized in that in, The source space feature is calculated by the following formula: Alternatively, the source space feature is calculated using the following formula: L + =L T (LL T +λI) -1 ; R=L T (LL T +λI) -1 L; in, is the source space feature at time t, x(t) is the EEG signal of the multiple electrode channels corresponding to time t, L + is the lead field pseudo-inverse matrix, L is the lead field matrix, L∈R C×D , C is the number of scalp electrodes, D is the dimension of the source space, [] T is the transpose operation of the matrix, I is the identity matrix, λ is the regularization parameter after offline optimization, and diag() represents the extraction of matrix diagonal elements.

19. The method according to claim 17, wherein The method further comprises: A source space activation map of a corresponding portion of the personalized three-dimensional head model is dynamically drawn on a visualization interface based on model display parameters input by a user, wherein the personalized three-dimensional head model includes: a brain surface portion, a skull surface portion, and a scalp surface portion; and the source space electrical activity intensity is dynamically displayed in real time in the source space activation map of the corresponding portion of the displayed personalized three-dimensional head model based on the source space characteristics.

20. The method according to claim 1, wherein The method further comprises: Acquiring sound and image position parameters of brainwave audio corresponding to a plurality of channels, and dynamically displaying the sound and image position of the brainwave audio on a visual interface according to the sound and image position parameters.

21. The method according to claim 20, characterized in that The step of obtaining the sound image position parameters of the brainwave audio corresponding to the multiple channels includes: Obtaining brainwave audio corresponding to each designated frequency band in the brainwave audio of the multiple channels, calculating the intensity of the brainwave audio of each designated frequency band in each channel within a designated duration; obtaining an audio-visual orientation indication parameter corresponding to the brainwave audio of the designated frequency band based on an intensity difference between brainwave audio of the same designated frequency band in different channels; The sound image position indication parameter and intensity corresponding to the brain wave audio of each designated frequency band in the multiple channels are used as the sound image position parameter.

22. The method according to claim 20, characterized in that The method further comprises: The sound and image position of the brainwave audio is dynamically and synchronously fed back to the collected user on a visual interface, so as to dynamically regulate the neural activity of the collected user through visual perception synchronization.

23. The method according to claim 1, wherein The method further comprises: Obtaining dynamic control parameters; wherein the dynamic control parameters are generated according to stimulation task requirements, or generated according to brain network characteristics corresponding to the electroencephalogram signals of the multiple electrode channels; The characteristic parameters of the audio performance are adjusted according to the dynamic control parameters.

24. A device for synchronous feedback of brainwave, audio and auditory perception, characterized in that: The device is set on a host computer, which is connected to the EEG acquisition device through an EEG data transmission interface. The device includes: an EEG signal receiving module, an EEG feature extraction module and an EEG-audio generation module, wherein: The EEG signal receiving module is configured to receive EEG signals of the user being collected online from the EEG acquisition device through the EEG data transmission interface, wherein the EEG signals are acquired by the EEG acquisition device through one or more electrode channels, wherein each electrode channel corresponds to one electrode or a combination of multiple electrodes; The EEG feature extraction module is configured to obtain one or more designated frequency band EEG signals from the EEG signals, and extract EEG features corresponding to the designated frequency band EEG signals based on the one or more designated frequency band EEG signals, wherein the designated frequency band EEG signals refer to EEG signals in the designated frequency band, and the EEG features include EEG time domain features and / or EEG frequency domain features; The EEG-audio generation module is configured to, for the EEG signal of the specified frequency band, based on the specified audio expression form and the specified audio generation method, divide the EEG signal of the specified frequency band into multiple EEG segments according to the EEG characteristics or preset rhythm parameters, and generate multiple audio expressions corresponding to the multiple EEG segments, wherein the characteristic parameters of the audio expression are determined according to the EEG characteristics of the corresponding EEG segments; the audio expression form includes: note form and / or waveform form, wherein the audio expression corresponding to the note form is a note, and the audio expression corresponding to the waveform form is a waveform, and the audio generation method includes: direct mapping method and / or indirect control method using control parameters; generate brainwave audio representation data according to the audio expression corresponding to the one or more EEG signals of the specified frequency band; obtain brainwave audio according to the brainwave audio representation data.

25. The device according to claim 24, characterized in that The device further includes: a feedback module, wherein the feedback module includes: an audio feedback module; The feedback module is configured to synchronously feed back the brainwave audio to the collected user through the audio feedback module, so as to dynamically regulate the nervous system activity of the collected user through auditory perception.

26. The device according to claim 25, characterized in that The EEG signal includes a denoised EEG signal, and the device further includes: an EEG preprocessing module, the EEG preprocessing module being connected to the EEG signal receiving module and the EEG feature extraction module respectively; The EEG preprocessing module is configured to: before obtaining one or more EEG signals of specified frequency bands in the EEG signal through the EEG feature extraction module, preprocess the EEG signals of the one or more electrode channels using a pre-trained brain-derived signal and non-brain-derived signal separation network model to obtain denoised EEG signals, and transmit the denoised EEG signals to the EEG feature extraction module.

27. The device according to claim 26, characterized in that The EEG preprocessing module is connected to the feedback module, and the feedback module further includes: an EEG waveform feedback module; The feedback module is further configured to dynamically display the waveform of the denoised EEG signal in real time on a visual interface through the EEG waveform feedback module.

28. The device according to claim 25, characterized in that The EEG feature extraction module is further configured to: Extracting brain network features corresponding to the EEG signals of the multiple electrode channels, the brain network features including: node-node functional connectivity features and edge-edge interaction features, the node-node functional connectivity features being used to describe the statistical dependency between two node signals, and the edge-edge interaction features being used to describe the temporal dependency between two edges; wherein the node corresponds to a single electrode channel, the node signal corresponds to the EEG signal of a single electrode channel, and the edge corresponds to the connection relationship between two nodes; Among them, the node-node functional connection characteristics corresponding to the EEG signals of the multiple electrode channels are extracted by calculating the weighted phase lag coefficient between every two node signals. When calculating the weighted phase lag coefficient between any two node signals, it includes: performing Hilbert transform on the first node signal and the second node signal to obtain a first node analytic signal and a second node analytic signal corresponding to the first node signal and the second node signal respectively; obtaining the instantaneous cross power spectrum of the first node signal and the second node signal based on the first node analytic signal and the second node analytic signal; averaging the absolute value of the imaginary part of the instantaneous cross power spectrum at each time point in a preset time window within the preset time window to obtain the mean of the absolute value of the imaginary part; calculating the weight of the imaginary part of the instantaneous cross power spectrum at each time point in the preset time window, and obtaining a weighted complex covariance based on the instantaneous cross power spectrum at each time point in the preset time window and the weight of the imaginary part corresponding to the instantaneous cross power spectrum; and calculating the weighted phase lag coefficient of the first node signal and the second node signal using the absolute value of the imaginary part of the weighted complex covariance and the mean of the absolute value of the imaginary part; The edge-edge interaction features corresponding to the electroencephalogram signals of the multiple electrode channels are extracted by calculating the Pearson correlation coefficient between every two edges in the candidate edge set, wherein the candidate edge set is formed by calculating the average phase synchronization index of all edges, and then sorting the average phase synchronization index of all edges from large to small and retaining a preset number of edges. When calculating the Pearson correlation coefficient between any two edges in the candidate edge set, it includes: respectively calculating the instantaneous phase difference between the first edge and the second edge at each time point in a preset time window, wherein the instantaneous phase difference is the difference between the instantaneous phases of one node and another node in the edge; based on the instantaneous phase difference of the first edge and the instantaneous phase difference of the second edge at each time point in the preset time window, using a sliding window to calculate the sparse Pearson correlation coefficient between the first edge and the second edge.

29. The device according to claim 28, characterized in that The EEG feature extraction module is connected to the feedback module, and the feedback module further includes: a brain network functional connection feedback module; The feedback module is further configured to dynamically draw electrodes corresponding to the multiple electrode channels on the individualized three-dimensional head model of the visualization interface through the brain network functional connection feedback module, wherein the electrodes are distributed on the scalp surface portion of the individualized three-dimensional head model; dynamically display the connection relationship and connection strength between the electrodes and / or between the electrode connections in real time according to the brain network characteristics, the electrode display parameters input by the user, and the feature display parameters; the electrode display parameters are used to indicate the display of one or more of the electrodes corresponding to the multiple electrode channels, and the feature display parameters are used to indicate the display of the node-node functional connection characteristics and / or the edge-edge interaction characteristics in the brain network characteristics.

30. The device according to claim 25, wherein The EEG feature extraction module is further configured to: Extracting source space features corresponding to the EEG signals of the multiple electrode channels includes: obtaining a pre-calculated lead field pseudo-inverse matrix, and obtaining the source space features based on the pre-calculated lead field pseudo-inverse matrix and the EEG signals of the multiple electrode channels; wherein the lead field pseudo-inverse matrix is ​​determined based on the lead field matrix and a regularization parameter after offline optimization, the lead field matrix is ​​used to describe the electrical signal transmission relationship from the source space to the electrode space, the lead field matrix is ​​calculated using the T1-weighted magnetic resonance imaging data of the collected user as input and using a pre-constructed individualized three-dimensional head model corresponding to the collected user; the regularization parameter after offline optimization is obtained by drawing a relationship curve between the residual norm and the solution norm, and selecting the regularization parameter corresponding to the maximum curvature point of the relationship curve.

31. The device according to claim 30, characterized in that The EEG feature extraction module is connected to the feedback module, and the feedback module further includes: an EEG traceability result feedback module; The feedback module is further configured to dynamically draw a source space activation map of the corresponding part of the personalized three-dimensional head model on a visualization interface according to the model display parameters input by the user through the EEG tracing result feedback module, wherein the personalized three-dimensional head model includes: a brain surface part, a skull surface part, and a scalp surface part; and dynamically display the source space electrical activity intensity in real time in the source space activation map of the corresponding part of the displayed personalized three-dimensional head model according to the source space characteristics.

32. The device according to claim 25, characterized in that The feedback module further comprises: an audio and video position feedback module; The feedback module is further configured to obtain sound image position parameters of brainwave audio corresponding to multiple channels through the sound image position feedback module, and dynamically display the sound image position of the brainwave audio on a visual interface according to the sound image position parameters.

33. An electronic device, characterized in that: A device comprising any one of claims 24 to 32, or comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 23.

34. A brainwave audio and auditory perception synchronous feedback system, characterized in that: The system comprises: an EEG acquisition device and the electronic device according to claim 33, wherein the EEG acquisition device is connected to the electronic device via an EEG data transmission interface, wherein: The EEG acquisition device is configured to acquire EEG signals of the user being acquired through one or more electrode channels, and transmit the EEG signals of the one or more electrode channels online to the electronic device based on the EEG data transmission interface.

35. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 23 is implemented.

36. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 23 is implemented.

Images