Quantitative analysis method and system for auditory and visual functional cross-compensation effect

Abstract

The application discloses a method and system for quantitatively analyzing auditory and visual function interaction compensation effects, and relates to the technical field of brain-computer interfaces. A specific embodiment of the method comprises: receiving reaction behavior data of a subject for a plurality of preset audio-visual stimulation tasks; the audio-visual stimulation tasks belong to at least one of the following audio-visual stimulation paradigms: an auditory stimulation paradigm, a visual stimulation paradigm, an auditory and visual consistent paradigm, and an auditory and visual inconsistent paradigm; obtaining index data of the subject in a preset audio-visual performance index according to the reaction behavior data; and determining an audio-visual compensation degree of the subject by using the index data. The embodiment can accurately measure the compensation degree of the auditory and visual functions by a computer device.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and in particular to a quantitative analysis method and system for the interactive compensation effect of auditory and visual functions. Background Technology

[0002] Hearing and vision are the two main senses by which humans perceive the external environment. When hearing is impaired, the brain undergoes cross-modal reorganization through neuroplasticity to compensate for auditory function, with the most significant compensation occurring in the visual system. Studies have shown that deaf patients outperform hearing individuals in visual tasks such as peripheral motion detection and lip reading comprehension. This phenomenon suggests that after the loss of auditory input, some cortical areas that were originally primarily involved in auditory processing become more involved in visual information processing, especially in tasks such as motion perception and retrieval of communication cues. However, this compensatory effect is not always fully effective. When auditory dysfunction is insufficiently compensated, subjects may subsequently experience speech impairment, and elderly subjects may even further develop cognitive impairment, manifesting as "slurred speech, decreased pitch, and reduced speech recognition ability," as well as a higher incidence of Alzheimer's disease in elderly deaf patients.

[0003] Studies have shown that cross-modal remodeling can produce compensatory benefits (such as enhanced vision or hearing) or uncompensatory costs (such as sensory deprivation impairing the function of other channels). This dynamic process is greatly influenced by the duration (congenital vs. acquired) and degree (complete vs. partial deprivation) of deprivation. This suggests that the clinical effect of cross-modal remodeling is a double-edged sword – moderate remodeling promotes compensation, while excessive remodeling may hinder rehabilitation, which in turn directly affects the timing and effectiveness of hearing aid or cochlear implantation in deaf patients.

[0004] Studies show that the degree of visual cross-modal reorganization before cochlear implantation is negatively correlated with postoperative speech comprehension in patients with age-related hearing loss. Cochlear implants often fail to effectively activate this area for auditory processing. Therefore, preoperative assessment of the patient's visual compensation level helps predict rehabilitation outcomes and optimize intervention timing. For patients with high visual dependence, preoperative transcranial magnetic stimulation or transcranial direct current stimulation can modulate cortical excitability, reducing the degree of visual over-recruitment of the auditory cortex and creating better neural conditions for post-implantation auditory integration. For patients with mild to moderate hearing loss, early intervention (such as hearing aid fitting) may help prevent the formation of excessive cross-modal reorganization.

[0005] Therefore, developing a system capable of objectively, dynamically, and quantitatively assessing the visual-auditory interaction compensation effect in deaf patients could provide a neuroscientific basis for guiding personalized treatment and optimizing rehabilitation outcomes. However, currently, there is no systematic, objective, and quantifiable assessment method in clinical practice for cross-modal visual and auditory remodeling.

[0006] Current audiological testing methods primarily assess auditory function. Specifically, with the development of modern audiology, clinical testing methods for the auditory system are relatively comprehensive, including pure-tone audiometry, acoustic impedance, speech recognition rate testing, auditory brainstem response, auditory steady-state response, and distortion product otoacoustic emissions. However, existing assessment systems mainly focus on the structural and functional state of the auditory system itself. There is still no systematic, objective, and quantifiable assessment method for how the brain utilizes other senses for cross-modal reorganization after hearing loss. Audiovisual cross-modal reorganization remains at the basic research level and has not yet been translated into mature clinical assessment tools. Existing research on audiovisual integration or visual compensation is mostly at the basic research level, primarily using single-modal brain function testing methods, and has not yet formed a standardized tool system that can be used for clinical assessment, efficacy monitoring, and rehabilitation guidance. Especially in actual communication environments, patients often do not solely rely on residual hearing to complete speech comprehension, but simultaneously utilize lip movements, facial expressions, and other visual information for multi-sensory integration. Therefore, how to objectively measure the compensatory effect of vision on hearing has become an unresolved issue in the functional assessment of hearing impairment.

[0007] Even with the same degree of hearing loss, patients exhibit significant individual differences in their ability to compensate for auditory loss using visual information. Furthermore, the effectiveness of visual compensation is not directly correlated with the degree of hearing loss; its manifestation depends more on the individual's multisensory integration ability. Studies have shown that in the mild to moderate hearing loss stage (such as mild to moderate age-related hearing loss), patients often exhibit "overcompensation," with visual information significantly contributing to speech recognition, and the McGurk effect becoming more pronounced, creating a stronger illusion of fusion. However, current clinical assessments of this phenomenon primarily rely on subjective descriptions or simple questionnaires (such as the SSQ spatial hearing questionnaire), lacking objective, standardized quantitative indicators. Although experimental methods such as the McGurk paradigm have been widely used in research, they have not yet been translated into mature clinical assessment tools. The lack of quantitative parameters makes it difficult for physicians to objectively determine whether a subject is in a state of "effective compensation" or is experiencing "perceptual disorder" due to cross-modal reorganization imbalance, thus affecting the accuracy of treatment decisions. Summary of the Invention

[0008] In view of this, embodiments of the present invention provide a method and system for quantitative analysis of the interactive compensation effect of auditory and visual functions, which can accurately measure the degree of auditory and visual compensation by computer equipment.

[0009] To achieve the above objectives, according to one aspect of the present invention, a method for quantitative analysis of the interactive compensation effect of auditory and visual functions is provided.

[0010] The quantitative analysis method for the auditory and visual function interaction compensation effect of this invention is executed by a computer device, and includes: receiving response behavior data of a subject to multiple preset audiovisual stimulus tasks; wherein the audiovisual stimulus tasks belong to at least one of the following audiovisual stimulus paradigms: auditory stimulus paradigm, visual stimulus paradigm, auditory-visual congruent paradigm, and auditory-visual incongruent paradigm; obtaining index data of the subject in preset audiovisual efficacy indicators based on the response behavior data; and determining the degree of audiovisual compensation of the subject using the index data.

[0011] Optionally, when the audiovisual stimulation paradigm includes an auditory stimulation paradigm, a visual stimulation paradigm, or an auditory-visual congruent paradigm, the audiovisual performance index includes at least one of the following: perceptual accuracy and average reaction time; or, when the audiovisual stimulation paradigm includes an auditory-visual incongruent paradigm, the audiovisual performance index includes at least two of the following: auditory dominance ratio, audiovisual fusion ratio, and visual dominance ratio.

[0012] Optionally, the auditory stimulation paradigm includes: emitting a monosyllabic speech signal; or, the visual stimulation paradigm includes: displaying a lip-sync video of the monosyllabic speech signal; or, the auditory-visual congruent paradigm includes: displaying a congruent lip-sync video while emitting a monosyllabic speech signal; or, the auditory-visual incongruent paradigm includes: displaying an incongruent lip-sync video while emitting a monosyllabic speech signal; or, if the audiovisual stimulation task belongs to at least two audiovisual stimulation paradigms, the execution order of each audiovisual stimulation task is pre-shuffled according to its respective audiovisual stimulation paradigm.

[0013] Optionally, determining the degree of audiovisual compensation of the subject using the indicator data includes: determining the audiovisual compensation level of the subject in each audiovisual efficacy indicator based on the subject's indicator data for each audiovisual efficacy indicator and a predetermined range of values ​​for the healthy population for that audiovisual efficacy indicator; and determining the degree of audiovisual compensation based on the subject's audiovisual compensation level in each audiovisual efficacy indicator.

[0014] Optionally, the method further includes: before determining the subject's audiovisual compensation level using the index data: receiving oxygenation level-dependent signals from multiple brain regions output by the subject performing functional magnetic resonance imaging (fMRI), and identifying multiple audiovisual response-related brain regions from the multiple brain regions based on the received oxygenation level-dependent signals; the subject is performing fMRI while receiving the audiovisual stimulation paradigm; receiving EEG signals from multiple leads output by the subject performing EEG operations, and identifying the leads corresponding to the audiovisual response-related brain regions as key leads; the subject is... The process involves performing the electroencephalogram (EEG) operation while receiving the audiovisual stimulation paradigm; extracting feature data of the audiovisual response-related brain regions in response to the audiovisual stimulation paradigm from the EEG data of the key leads; determining the device-side audiovisual compensation level of the subject based on the feature data and the blood oxygen level-dependent signal of the audiovisual response-related brain regions; and determining the degree of audiovisual compensation based on the subject's audiovisual compensation level for each audiovisual efficacy index, which includes determining the degree of audiovisual compensation based on the subject's audiovisual compensation level for each audiovisual efficacy index and the device-side audiovisual compensation level.

[0015] Optionally, determining multiple audiovisual response-related brain regions from the multiple brain regions based on the received blood oxygen level-dependent signals includes: acquiring the time-series function of the audiovisual stimulus paradigm and the hemodynamic response function of the multiple brain regions; performing convolution on the hemodynamic response function and the time-series function to obtain the basic signal change curve of each brain region; subtracting the blood oxygen level-dependent signal of each brain region from the basic signal change curve of the corresponding brain region to obtain the magnetic resonance intensity data of each brain region; and determining a preset number of brain regions with the highest magnetic resonance intensity as the audiovisual response-related brain regions based on the magnetic resonance intensity data.

[0016] Optionally, the preset features include at least one of the following: event-related potential component features, time-frequency features, and brain region functional connectivity features; the event-related potential components include at least one of the following: mismatched negative wave MMN, P1 / N1 wave, P300 wave, N400 wave, and anterior negative wave audFN; the time-frequency features include: short-time Fourier transform features of the EEG signals in the key leads, and the short-time Fourier transform features include at least one of the following: time-frequency power features, time-frequency resolution trade-off features, spectral entropy features, time-frequency peak frequency features, and time-frequency ridge features; the brain region functional connectivity features include: phase-locking value features of any two visual-auditory response-related brain regions in the preset frequency band.

[0017] Optionally, determining the device-side audiovisual compensation level of the subject based on the feature data and the blood oxygen level-dependent signal of the brain region related to audiovisual response includes: inputting the feature data and the blood oxygen level-dependent signal of the brain region related to audiovisual response into a pre-trained feature fusion and calculation model to obtain the device-side audiovisual compensation level; or, determining the degree of audiovisual compensation based on the subject's audiovisual compensation level for each audiovisual performance indicator and the device-side audiovisual compensation level includes: determining the subject's behavioral-side audiovisual compensation level based on the subject's audiovisual compensation level for each audiovisual performance indicator, and adjusting the behavioral-side audiovisual compensation level using the device-side audiovisual compensation level to obtain the degree of audiovisual compensation.

[0018] To achieve the above objectives, according to another aspect of the present invention, a quantitative analysis system for the interactive compensation effect of auditory and visual functions is provided.

[0019] The quantitative analysis system for the interactive compensation effect of auditory and visual functions according to an embodiment of the present invention is installed in a computer device and includes: a behavioral data receiving unit for receiving behavioral data of a subject's response to multiple preset audiovisual stimulation tasks; wherein the audiovisual stimulation tasks belong to at least one of the following audiovisual stimulation paradigms: auditory stimulation paradigm, visual stimulation paradigm, auditory-visual congruence paradigm, and auditory-visual incongruence paradigm; an index data acquisition unit for acquiring index data of the subject's audiovisual efficacy index based on the behavioral data; and an audiovisual compensation calculation unit for determining the degree of audiovisual compensation of the subject using the index data.

[0020] To achieve the above objectives, according to another aspect of the present invention, a brain-computer interface control system is provided.

[0021] The brain-computer interface control system of this invention is installed in a computer device and includes: an MRI receiving unit, used to receive blood oxygen level-dependent signals from multiple brain regions output by a subject performing functional magnetic resonance imaging (fMRI), and to determine multiple visually and auditorily active brain regions from the multiple brain regions based on the received blood oxygen level-dependent signals; the subject performs fMRI while receiving a preset visual and auditory stimulation paradigm, the visual and auditory stimulation paradigm including at least one of the following: auditory stimulation paradigm, visual stimulation paradigm, auditory-visual congruent paradigm, and auditory-visual incongruent paradigm; and an EEG receiving unit, used for... The system receives EEG signals from multiple leads output by the subject performing an EEG operation, identifies the leads corresponding to the visually and aurally active brain regions as key leads; the subject performs the EEG operation while receiving the visually and aurally stimulating paradigm; and a control unit extracts feature data of the visually and aurally active brain regions in response to the visually and aurally stimulating paradigm from the EEG data of the key leads, determines the subject's execution intention based on the feature data and the blood oxygen level-dependent signal of the visually and aurally active brain regions, and sends the determined execution intention to a pre-connected electronic device.

[0022] To achieve the above objectives, according to another aspect of the present invention, an electronic device is provided.

[0023] An electronic device according to the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the quantitative analysis method for the interactive compensation effect of auditory and visual functions provided by the present invention.

[0024] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is provided.

[0025] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the quantitative analysis method for the interactive compensation effect of auditory and visual functions provided by the present invention.

[0026] To achieve the above objectives, according to another aspect of the present invention, a computer program product is provided.

[0027] One computer program product of the present invention includes a computer program that, when executed by a processor, implements the quantitative analysis method for the interactive compensation effect of auditory and visual functions provided by the present invention.

[0028] According to the technical solution of the present invention, the embodiments described above have the following advantages or beneficial effects: The computer device first receives behavioral data on the subject's responses to multiple pre-set audiovisual stimulation tasks, which belong to at least one of the following audiovisual stimulation paradigms: auditory stimulation paradigm, visual stimulation paradigm, auditory-visual congruence paradigm, and auditory-visual incongruence paradigm. Subsequently, the computer device obtains the subject's index data on pre-set audiovisual efficacy indicators based on the behavioral data, and uses the above index data to determine the subject's degree of audiovisual compensation. In this way, based on specific audiovisual stimulation paradigms and data processing steps, the accurate quantification of auditory and visual interaction compensation is achieved from the behavioral perspective.

[0029] Optionally, embodiments of the present invention may combine MRI and EEG manipulation to provide an auxiliary device-side quantification scheme. Specifically, the computer device receives oxygenation level-dependent signals from multiple brain regions output by the subject undergoing functional MRI, and identifies multiple visual-auditory response-related brain regions based on the received oxygenation level-dependent signals; the subject undergoes MRI while receiving the aforementioned visual-auditory stimulation paradigm. The computer device also receives EEG signals from multiple leads output by the subject undergoing EEG manipulation, identifying the leads corresponding to the visual-auditory response-related brain regions as key leads; the subject undergoes EEG manipulation while receiving the aforementioned visual-auditory stimulation paradigm. Next, the computer device extracts feature data of the visual-auditory response-related brain regions in response to the preset features of the visual-auditory stimulation paradigm from the EEG data of the key leads, and determines the subject's device-side visual-auditory compensation level based on the feature data and the oxygenation level-dependent signals from the visual-auditory response-related brain regions. Subsequently, the computer device determines the subject's degree of visual-auditory compensation based on the above indicator data and the device-side visual-auditory compensation level. In this way, computer equipment, by combining the subject's response to visual and auditory stimulus paradigms with brain magnetic resonance imaging and electroencephalography, can accurately quantify the degree of compensation for auditory and visual interaction, which helps to achieve reliable and quantifiable functional assessment of hearing impairment.

[0030] Furthermore, in this embodiment of the invention, when subjects simultaneously view the same audiovisual stimulation paradigm during brain magnetic resonance imaging (MRI) and electroencephalography (EEG) operations, it generates highly consistent and easily collectable activation data on both the MRI and EEG sides. This facilitates the localization of brain regions related to audiovisual responses and the extraction of feature data, while also aiding in the fusion of MRI and EEG feature data, enabling accurate assessment and quantification of the degree of visual and auditory interaction compensation. Further, this embodiment of the invention can deeply integrate high spatial resolution MRI with high temporal resolution EEG operations based on the same audiovisual stimulation paradigm, forming a complementary spatiotemporal fusion framework to comprehensively characterize the degree of audiovisual interaction compensation in subjects. The above methods of this embodiment of the invention can be applied to brain-computer interface scenarios, using the fusion of MRI and EEG operations based on the same audiovisual stimulation paradigm to collect brain signals from subjects to accurately infer their execution intentions, thereby controlling electronic devices.

[0031] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0032] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main steps of the quantitative analysis method for the interactive compensation effect of auditory and visual functions in an embodiment of the present invention; Figure 2 This is a schematic diagram of the components of the quantitative analysis device for the interactive compensation effect of auditory and visual functions in an embodiment of the present invention. Figure 3 This is an exemplary system architecture diagram that can be applied thereto according to embodiments of the present invention; Figure 4 This is a schematic diagram of an electronic device structure used to implement the quantitative analysis method for the interactive compensation effect of auditory and visual functions in the embodiments of the present invention. Detailed Implementation

[0033] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0034] It should be noted that, unless otherwise specified, the embodiments of the present invention and the technical features thereof can be combined with each other.

[0035] Figure 1 This is a schematic diagram of the main steps of the quantitative analysis method for the interactive compensation effect of auditory and visual functions in an embodiment of the present invention.

[0036] like Figure 1 As shown, the quantitative analysis method for the interactive compensation effect of auditory and visual functions in this embodiment of the invention can be executed by computer equipment (such as a server, personal computer, mobile smart terminal, etc.), and the specific execution steps are as follows: Step S101: The computer device receives the subject's response behavior data to multiple preset audiovisual stimulus tasks.

[0037] The subjects can be healthy individuals, deaf patients, or patients with other visual and auditory impairments. The audiovisual stimulation tasks described above belong to at least one of the following audiovisual stimulation paradigms: auditory stimulation paradigm, visual stimulation paradigm, auditory-visual congruent paradigm, and auditory-visual incongruent paradigm. For example, the auditory stimulation paradigm may include: emitting monosyllabic speech signals (without visual information), such as the speech signals "ga," "ba," "da," etc.; the visual stimulation paradigm may include: displaying a lip-sync video of a monosyllabic speech signal (without sound); the auditory-visual congruent paradigm may include: displaying a congruent lip-sync video while emitting a monosyllabic speech signal; the auditory-visual incongruent paradigm may include: displaying an incongruent lip-sync video while emitting a monosyllabic speech signal. For example, the auditory-visual incongruent paradigm may include the McGurk paradigm or a non-McGurk paradigm. In the McGurk paradigm, for example, playing the speech "ba" and displaying a silent lip-sync video of "ga" would result in the recognition result being "da" if audiovisual fusion occurs. For non-McGurk paradigms, for example, playing the speech "ga" and displaying a silent lip-sync video of "ba," if audiovisual fusion occurs, the recognition result would be "da." Of course, the above speech signal can also be a disyllabic, trisyllabic, or other signal. The response behavior data in this step refers to the subject's recognition results after receiving an audiovisual stimulus task. In one embodiment, a preset duration can be set between adjacent audiovisual stimulus tasks.

[0038] As an alternative, when the audiovisual stimulus task belongs to at least two audiovisual stimulus paradigms, the execution order of each audiovisual stimulus task is shuffled in advance according to its respective audiovisual stimulus paradigm to reflect irregularity or low regularity and improve the accuracy of assessment. For example, the auditory stimulus paradigm, visual stimulus paradigm, auditory-visual congruent paradigm, and auditory-visual incongruent paradigm are represented by a, b, c, and d, respectively. The execution order of the audiovisual stimulus task can be shuffled as: b, d, a, a, c, b, b, d, a, c, c, c.

[0039] Step S102: The computer device acquires the subject's performance data based on the response behavior data and preset audiovisual efficacy indicators.

[0040] In this step, the computer device can acquire the subject's response behavior data to the audiovisual stimulus paradigm (i.e., the identified results such as ba, da, ga), and then statistically extract the preset audiovisual efficacy index data from the response behavior data.

[0041] For example, when the audiovisual stimulus paradigm includes an auditory stimulus paradigm, a visual stimulus paradigm, or an auditory-visual congruent paradigm, the audiovisual efficacy index includes at least one of the following: perceptual accuracy (e.g., accuracy of pure visual recognition, accuracy of pure auditory recognition, accuracy of audiovisual congruent recognition), and average reaction time; when the audiovisual stimulus paradigm includes an auditory-visual incongruent paradigm, the audiovisual efficacy index includes at least two of the following: auditory dominance ratio (i.e., the ratio of the number of times auditory dominance is equal to the total number of times, where auditory dominance refers to the recognition result being the emitted auditory signal), audiovisual fusion ratio (i.e., the ratio of the number of times visual dominance is equal to the total number of times, where visual dominance refers to the recognition result being the emitted visual signal), and visual dominance ratio (i.e., the ratio of the number of times audiovisual fusion effect is equal to the total number of times, where audiovisual fusion effect refers to the recognition result being an audiovisual fusion result). The sum of the auditory dominance ratio, the visual dominance ratio, and the audiovisual fusion ratio is 1, so two independent indicators can be selected.

[0042] Step S103: The computer equipment uses the index data to determine the degree of audiovisual compensation of the subject.

[0043] In one embodiment, the computer device first determines the subject's audiovisual compensation level for each audiovisual efficacy index based on the subject's index data for each index and a pre-determined range of values ​​for the healthy population for that index. Then, it determines the degree of audiovisual compensation based on the subject's audiovisual compensation level for each index. For example, the audiovisual compensation levels and degrees for each audiovisual efficacy index are preset to 1, 2, 3, 4, and 5 (with the degree of compensation increasing sequentially), where 1 represents the healthy population and 5 represents the degree of audiovisual compensation for severe hearing loss. First, the range of values ​​for the healthy population for a certain audiovisual efficacy index is determined through big data analysis. Then, based on the range of values ​​for the healthy population and automatic reasoning by big data analysis or a machine learning model, the value ranges for other levels (2-5) are determined. Subsequently, the audiovisual compensation level for each index is determined based on these value ranges. Finally, a weighted calculation is performed on the audiovisual compensation levels of each index to obtain the subject's degree of audiovisual compensation (the weights can be set based on experience or obtained through reasoning by a trained machine learning model). Of course, the audiovisual compensation levels and degrees mentioned above can also be numerical values ​​of other granularities, such as percentage values ​​from 1%, 2%, 3% to 100%. The audiovisual compensation degrees mentioned above can be the degree of visual compensation for auditory compensation, or the degree of auditory compensation for visual compensation.

[0044] In another embodiment, the computer device first calculates the subject's deviation data (characterizing the degree to which the subject deviates from the healthy population's numerical range) for each audiovisual efficacy index based on the subject's index data and a predetermined healthy population numerical range for that audiovisual efficacy index. Then, it calculates the subject's audiovisual compensation level for that audiovisual efficacy index based on the deviation data. Finally, it performs a weighted calculation on the audiovisual compensation levels of each audiovisual efficacy index to obtain the subject's degree of audiovisual compensation.

[0045] In another embodiment, the above audiovisual stimulation tasks belong to the following audiovisual stimulation paradigms: auditory stimulation paradigm, visual stimulation paradigm, auditory-visual congruent paradigm, and auditory-visual incongruent paradigm. Audiovisual performance indicators include: accuracy and average reaction time for pure visual recognition, accuracy and average reaction time for pure auditory recognition, accuracy and average reaction time for auditory-visual congruent recognition, and the auditory dominance and audiovisual fusion ratio in the auditory-visual incongruent paradigm. The computer device first fuses the indicator data for accuracy and average reaction time of pure visual recognition, and calculates the audiovisual compensation level on the pure visual side (which also represents the subject's visual level) based on the fused data and the numerical range of healthy individuals. The system integrates the accuracy and average reaction time data for pure auditory recognition, and calculates the audiovisual compensation level for the pure auditory side (which also represents the subject's auditory level) based on the integrated data and the numerical range of healthy individuals. It also integrates the accuracy and average reaction time data for audiovisual consistency recognition, and calculates the audiovisual compensation level for the audiovisual consistency side based on the integrated data and the numerical range of healthy individuals. Finally, it integrates the auditory dominance ratio and the audiovisual fusion ratio data, and calculates the audiovisual non-consistent side based on the integrated data and the numerical range of healthy individuals. The computer then performs a weighted calculation of the four audiovisual compensation levels to obtain the subject's degree of audiovisual compensation.

[0046] In other embodiments, the computer device may also calculate the degree of audiovisual compensation based on a pre-trained machine learning model and preset functions. The machine learning model may employ known algorithms such as Convolutional Neural Networks (CNN) and Transformer, and be trained using existing training methods.

[0047] The following describes another embodiment of the present invention. As an optional solution, this embodiment uses MRI and EEG data to assist in the calculation of the aforementioned behavioral scheme from the device side. It can combine fMRI equipment, EEG equipment and subject response behavior data to comprehensively and accurately quantify the subject's degree of visual and auditory compensation. The specific execution steps are as follows.

[0048] Step 1: Before using index data to determine the subject's degree of visual and auditory compensation, the computer device receives oxygenation level-dependent BOLD signals from multiple brain regions output by the subject's functional magnetic resonance imaging (fMRI). Based on the received oxygenation level-dependent signals, multiple brain regions related to visual and auditory responses are identified from the multiple brain regions. The subject undergoes fMRI while receiving visual and auditory stimulation paradigms.

[0049] The brain regions mentioned above can be pre-divided into brain regions according to any applicable granularity. Optionally, after acquiring the BOLD signal, the computer device can perform known preprocessing such as format conversion, temporal correction, head motion correction, spatial normalization, and spatial smoothing to obtain the required data.

[0050] In one embodiment, a computer device identifies multiple brain regions related to audiovisual responses by: first, acquiring a time-series function of the audiovisual stimulus paradigm and a hemodynamic response function (HRF) for multiple brain regions, where the time-series function represents the duration of each test and the termination time in each audiovisual stimulus paradigm; next, convolving the HRF with the time-series function to obtain the baseline signal change curves for each brain region; then, subtracting the oxygenation level-dependent signal of each brain region from the baseline signal change curves for that region to obtain magnetic resonance imaging (MRI) data for each brain region; subsequently, based on the MRI data, the computer device identifies a predetermined number of brain regions with the highest MRI intensity as the audiovisual response-related brain regions, where the MRI intensity can be the average MRI intensity within the brain region calculated based on the MRI data.

[0051] Step 2: The computer device receives the EEG signals from multiple leads output by the subject performing the EEG operation, and identifies the leads corresponding to the brain regions related to visual and auditory responses as key leads; the subject performs the EEG operation while receiving the aforementioned visual and auditory stimulation paradigm.

[0052] This step can be performed in parallel with step 1. In this step, after acquiring the EEG signal, the computer device can perform known preprocessing techniques such as filtering, bad lead removal, independent component analysis, and block segmentation. While receiving the aforementioned audiovisual stimulation paradigm, the subject undergoes EEG manipulation. The computer device identifies the leads corresponding to the previously determined audiovisual response-related brain regions as key leads, and then analyzes the EEG data from these key leads. Thus, the key brain region information provided by MRI serves as a spatial prior for EEG analysis. These key brain region leads are analyzed in detail, thereby limiting the EEG analysis to the audiovisual-related target network. The high spatial resolution of MRI enhances the specificity and interpretability of EEG feature extraction.

[0053] Step 3: The computer device extracts feature data of the brain regions related to visual and auditory response to the visual and auditory stimulus paradigm in the key leads from the EEG data. Based on the feature data and the blood oxygen level dependent signal of the brain regions related to visual and auditory response, the device-side visual and auditory compensation level of the subject is determined.

[0054] For example, the above preset features may include at least one of the following: event-related potential component features, time-frequency features, and brain region functional connectivity features. For example, event-related potential components may include at least one of the following: mismatched negative wave MMN (lasting approximately 150-250 milliseconds, located in the anterior central or occipital lobe), P1 / N1 waves (appearing in pairs, P1 approximately 100 milliseconds, N1 approximately 150-200 milliseconds), P300 wave (lasting approximately 300 milliseconds, located in the parietal or frontal central region), N400 wave (approximately 400 milliseconds, located in the central parietal lobe), and anterior negative wave audFN (approximately 180-280 milliseconds, located in the anterior scalp); time-frequency features may include: short-time Fourier transform (STFT) features of EEG signals in the key leads, and STFT features may include at least one of the following: time-frequency power features, resolution trade-off features, spectral entropy features, time-frequency peak frequency (frequency value at the point of strongest energy), and time-frequency ridge (trajectory of concentrated energy in the time-frequency graph). For example, brain region functional connectivity features may include the phase locking value (PLV) features of any two visual and auditory response-related brain regions in a preset frequency band. PLV reflects the degree of synchronization between brain regions in a certain frequency band by characterizing the consistency of the instantaneous phase difference between two EEG signals.

[0055] In this step, the computer device can input the feature data of the above-mentioned preset features and the blood oxygenation level-dependent signals of brain regions related to audiovisual responses into a pre-trained feature fusion and calculation model to obtain the device-side audiovisual compensation level. The above feature fusion and calculation model can be established using known deep neural network (DNN), Transformer, and other machine learning algorithms, and can be trained using known training methods.

[0056] Step 4: The computer device determines the degree of audiovisual compensation based on the subject's audiovisual compensation level for each audiovisual performance index and the audiovisual compensation level on the device side.

[0057] In this step, the computer device can determine the subject's behavioral-side audiovisual compensation level based on the subject's audiovisual compensation level for each audiovisual efficacy index (which can be calculated according to the methods described in steps S101-S103 above). Subsequently, the behavioral-side audiovisual compensation level is adjusted using the device-side audiovisual compensation level to obtain the subject's degree of audiovisual compensation. The adjustment method can be weighted calculation or other methods, such as multiplying the adjustment coefficient corresponding to the device-side audiovisual compensation level by the behavioral-side audiovisual compensation level to obtain the degree of audiovisual compensation.

[0058] In the technical solution of this invention, a dynamic monitoring tool executed by computer equipment is established, applicable to the entire process of hearing aid fitting, cochlear implantation, and postoperative rehabilitation after ear surgery. It is not only used for a one-time assessment of auditory and visual interaction compensation, but can be used throughout multiple stages such as screening of deaf patients, preoperative assessment, treatment plan formulation, postoperative rehabilitation follow-up, and post-operative monitoring. By measuring the changes in the degree of auditory and visual compensation and brain function indicators of subjects at different time points over a long period of time, this invention can dynamically judge the process of neural reorganization and rehabilitation trend, providing a continuous basis for the selection of hearing aid wearing time, cochlear implantation decision-making, and adjustment of postoperative rehabilitation training plan, and has practical value.

[0059] Furthermore, this invention fills a gap in existing technology, improving clinical interpretation efficiency and scalability. Most current research on audiovisual integration, visual compensation, and the McGurk effect remains at the laboratory research stage, lacking a standardized, scalable, and reportable practical application system. This invention employs a unified audiovisual stimulation paradigm, combining fMRI and EEG operations to jointly quantify the audiovisual interaction compensation effect in deaf patients. It possesses triple functions: scientific research, clinical assessment, and rehabilitation guidance, filling the gap in standardized assessment tools for the audiovisual interaction compensation effect in deaf patients. This invention can automatically output individualized quantitative analysis reports based on preset algorithms executed by computer equipment. Compared to traditional methods relying on physician experience, this invention offers advantages such as high standardization, good repeatability, traceability and comparison, and ease of multi-center promotion, facilitating the formation of unified assessment standards in clinical settings.

[0060] It should be noted that the technical solutions of this invention, including the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0061] For the foregoing method embodiments, they are described as a series of actions for ease of description. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, and some steps may actually be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential for implementing the present invention.

[0062] To facilitate better implementation of the above-described solutions of the embodiments of the present invention, a related system for implementing the above-described solutions is also provided below.

[0063] Please see Figure 2 As shown, the quantitative analysis system 200 for the interactive compensation effect of auditory and visual functions provided in this embodiment of the invention is installed in a computer device and may include: a behavior data receiving unit 201, used to receive the response behavior data of a subject to multiple preset audiovisual stimulation tasks; wherein the audiovisual stimulation tasks belong to at least one of the following audiovisual stimulation paradigms: auditory stimulation paradigm, visual stimulation paradigm, auditory-visual congruence paradigm, and auditory-visual incongruence paradigm; an index data acquisition unit 202, used to acquire the index data of the subject in preset audiovisual efficacy indicators based on the response behavior data; and an audiovisual compensation calculation unit 203, used to determine the degree of audiovisual compensation of the subject using the index data.

[0064] In one embodiment, when the audiovisual stimulation paradigm includes an auditory stimulation paradigm, a visual stimulation paradigm, or an auditory-visual congruent paradigm, the audiovisual performance index includes at least one of the following: perceptual accuracy and average reaction time; or, when the audiovisual stimulation paradigm includes an auditory-visual incongruent paradigm, the audiovisual performance index includes at least two of the following: auditory dominance ratio, audiovisual fusion ratio, and visual dominance ratio.

[0065] In one embodiment, the auditory stimulation paradigm includes: emitting a monosyllabic speech signal; or, the visual stimulation paradigm includes: displaying a lip-sync video of the monosyllabic speech signal; or, the auditory-visual congruent paradigm includes: displaying a congruent lip-sync video while emitting a monosyllabic speech signal; or, the auditory-visual incongruent paradigm includes: displaying an incongruent lip-sync video while emitting a monosyllabic speech signal; or, if the audiovisual stimulation task belongs to at least two audiovisual stimulation paradigms, the execution order of each audiovisual stimulation task is pre-shuffled according to its respective audiovisual stimulation paradigm.

[0066] In one embodiment, the audiovisual compensation calculation unit 203 is further configured to: determine the audiovisual compensation level of the subject in each audiovisual performance index based on the subject's index data for each audiovisual performance index and a predetermined range of healthy population values ​​for the audiovisual performance index; and determine the degree of audiovisual compensation based on the subject's audiovisual compensation level in each audiovisual performance index.

[0067] In one embodiment, the device 200 further includes a device-side computing unit configured to: receive oxygenation level-dependent signals from multiple brain regions output by the subject performing functional magnetic resonance imaging (fMRI); determine multiple audiovisual response-related brain regions from the multiple brain regions based on the received oxygenation level-dependent signals; wherein the subject performs fMRI while receiving the audiovisual stimulation paradigm; and receive multiple leads of electroencephalogram (EEG) signals output by the subject performing an EEG operation, and correlate the audiovisual response-related brain regions with... The corresponding leads are identified as key leads; the subject performs the EEG operation while receiving the audiovisual stimulation paradigm; feature data of the audiovisual response-related brain regions in response to the audiovisual stimulation paradigm are extracted from the EEG data of the key leads; the device-side audiovisual compensation level of the subject is determined based on the feature data and the blood oxygen level-dependent signal of the audiovisual response-related brain regions; and the audiovisual compensation calculation unit 203 is further configured to: determine the degree of audiovisual compensation based on the audiovisual compensation level of the subject in each audiovisual efficacy index and the device-side audiovisual compensation level.

[0068] In one embodiment, the device-side computing unit is further configured to: acquire the time-series function of the audiovisual stimulation paradigm and the hemodynamic response function of the plurality of brain regions; perform convolution on the hemodynamic response function and the time-series function to obtain the basic signal change curve of each brain region; subtract the blood oxygen level dependent signal of each brain region from the basic signal change curve of the corresponding brain region to obtain the magnetic resonance intensity data of each brain region; and, based on the magnetic resonance intensity data, determine the brain regions with the highest magnetic resonance intensity and a preset number of brain regions as the audiovisual response-related brain regions.

[0069] In one embodiment, the preset features include at least one of the following: event-related potential component features, time-frequency features, and brain region functional connectivity features; the event-related potential components include at least one of the following: mismatched negative wave MMN, P1 / N1 wave, P300 wave, N400 wave, and anterior negative wave audFN; the time-frequency features include: short-time Fourier transform features of the EEG signals in the key leads, and the short-time Fourier transform features include at least one of the following: time-frequency power features, time-frequency resolution trade-off features, spectral entropy features, time-frequency peak frequency features, and time-frequency ridge features; the brain region functional connectivity features include: phase-locking value features of any two visual-auditory response-related brain regions in the preset frequency band.

[0070] In one embodiment, the device-side computing unit is further configured to: input the feature data and the blood oxygen level-dependent signal of the brain region related to the audiovisual response into a pre-trained feature fusion and computing model to obtain the device-side audiovisual compensation level; determine the subject's behavioral-side audiovisual compensation level based on the subject's audiovisual compensation level for each audiovisual performance index; and adjust the behavioral-side audiovisual compensation level using the device-side audiovisual compensation level to obtain the degree of audiovisual compensation.

[0071] This invention can assess the compensatory state of auditory and visual interactions. For various types of hearing loss patients, it uses precise quantification of the degree of auditory and visual compensation to determine whether the subject's brain has established effective cross-modal reorganization, which has predictive value for assessing the subject's communication ability in real life. This invention can also develop individualized rehabilitation strategies. By identifying the subject's degree of compensation, clinicians and rehabilitation physicians can adjust the proportion of visual aid training accordingly. Desensitization training can be implemented for those with excessive visual dependence, while visual guidance can be strengthened for those with insufficient compensation, thereby avoiding blind training and shortening the rehabilitation period.

[0072] This invention can predict the prognosis of cochlear implants and hearing aids. If the auditory cortex of the brain is overcompensated for auditory function by visual function, it will affect the outcome of cochlear implantation. Multiple studies have shown that the preoperative activation level of the auditory cortex to visual-speech stimuli is negatively correlated with postoperative language comprehension ability, indicating that the cochlear implant cannot effectively activate this area for auditory processing. This invention monitors the level of cortical remodeling, identifies potentially high-benefit individuals, and provides a basis for assessing postoperative speech rehabilitation potential. For subjects expected to face greater rehabilitation difficulties, this invention can support the development of more targeted individualized intervention plans, thereby maximizing treatment effectiveness and minimizing the risk of ineffective intervention.

[0073] This invention also provides a brain-computer interface control system, which is installed in a computer device. The system includes: an MRI receiving unit, configured to receive oxygenation level-dependent signals from multiple brain regions output by a subject performing functional magnetic resonance imaging (fMRI), and to identify multiple visually and aurally active brain regions from the multiple brain regions based on the received oxygenation level-dependent signals; the subject performs fMRI while receiving a preset visually and aurally stimulating paradigm, the visually and aurally stimulating paradigm including at least one of the following: auditory stimulation paradigm, visual stimulation paradigm, auditory-visual congruence paradigm, and auditory-visual non-congruence paradigm; an EEG receiving unit, configured to receive EEG signals from multiple leads output by the subject performing an EEG operation, and to identify the leads corresponding to the visually and aurally active brain regions as key leads; the subject performs the EEG operation while receiving the visually and aurally stimulating paradigm; and a control unit, configured to extract feature data of the visually and aurally active brain regions in relation to the preset features of the visually and aurally stimulating paradigm from the EEG data of the key leads, determine the subject's execution intention based on the feature data and the oxygenation level-dependent signals from the visually and aurally active brain regions, and send the determined execution intention to a pre-connected electronic device.

[0074] In one embodiment, the MRI receiving unit is further configured to: acquire the time-series function of the audiovisual stimulation paradigm and the hemodynamic response function of the plurality of brain regions; perform convolution on the hemodynamic response function and the time-series function to obtain the basic signal change curve of each brain region; subtract the blood oxygen level dependent signal of each brain region from the basic signal change curve of the corresponding brain region to obtain the magnetic resonance intensity data of each brain region; and, based on the magnetic resonance intensity data, determine a preset number of brain regions with the highest magnetic resonance intensity as the audiovisual-responsive brain regions.

[0075] In one embodiment, the auditory stimulation paradigm includes: emitting a monosyllabic speech signal; or, the visual stimulation paradigm includes: displaying a lip-sync video of the monosyllabic speech signal; or, the auditory-visual congruent paradigm includes: displaying a congruent lip-sync video while emitting a monosyllabic speech signal; or, the auditory-visual incongruent paradigm includes: displaying an incongruent lip-sync video while emitting a monosyllabic speech signal.

[0076] In one embodiment, the preset features include at least one of the following: event-related potential component features, time-frequency features, and brain region functional connectivity features; the event-related potential components include at least one of the following: mismatched negative wave MMN, P1 / N1 wave, P300 wave, N400 wave, and anterior negative wave audFN; the time-frequency features include: short-time Fourier transform features of the EEG signals in the key leads, and the short-time Fourier transform features include at least one of the following: time-frequency power features, time-frequency resolution trade-off features, spectral entropy features, time-frequency peak frequency features, and time-frequency ridge features; the brain region functional connectivity features include: phase-locking value features of any two visual-auditory response-related brain regions in the preset frequency band.

[0077] In one embodiment, the control unit is further configured to: input the feature data of the visually and aurally active brain region and the magnetic resonance intensity data into a pre-trained feature fusion and calculation model to obtain an execution intent ID output by the feature fusion and calculation model, wherein the execution intent may be one of a number of preset intents.

[0078] Thus, the method of this invention can be applied to brain-computer interface scenarios, using the same audiovisual stimulation paradigm to fuse functional magnetic resonance imaging and electroencephalography to collect brain signals from subjects in order to accurately infer their execution intentions and thereby control electronic devices.

[0079] Figure 3 An exemplary system architecture 300 is shown, which can be applied to the quantitative analysis method or system for the quantitative analysis of the interactive compensation effect of auditory and visual functions according to embodiments of the present invention.

[0080] like Figure 3 As shown, system architecture 300 may include terminal devices 301, 302, and 303, network 304, and server 305 (this architecture is merely an example; the components included in a specific architecture may be adjusted according to the specific application). Network 304 serves as the medium for providing a communication link between terminal devices 301, 302, and 303 and server 305. Network 304 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0081] Users can use terminal devices 301, 302, and 303 to interact with server 305 via network 304 to receive or send information, etc. Various client applications can be installed on terminal devices 301, 302, and 303, such as quantitative calculation applications (for example only).

[0082] Terminal devices 301, 302, and 303 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0083] Server 305 can be a server that provides various services, such as a backend server that supports the quantitative calculation applications operated by users using terminal devices 301, 302, and 303 (for example only). The backend server can process received quantitative analysis requests and feed back the processing results (such as the degree of audiovisual compensation, for example only) to terminal devices 301, 302, and 303.

[0084] It should be noted that the quantitative analysis method for the interactive compensation effect of auditory and visual functions provided in the embodiments of the present invention is generally executed by the server 305, and the quantitative analysis system for the interactive compensation effect of auditory and visual functions is also generally set in the server 305.

[0085] It should be understood that Figure 3 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0086] The present invention also provides an electronic device. The electronic device of this invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the quantitative analysis method for the interactive compensation effect of auditory and visual functions provided by the present invention.

[0087] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a computer system 400 suitable for implementing an electronic device according to embodiments of the present invention. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0088] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the computer system 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0089] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0090] In particular, according to the embodiments disclosed in this invention, the processes described in the above main step diagrams can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit 401, it performs the functions defined in the system of this invention.

[0091] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0093] The units described in the embodiments of the present invention can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including a behavior data receiving unit, an indicator data acquisition unit, and an audiovisual compensation calculation unit. The names of these units do not necessarily limit the specific unit; for example, the data receiving unit can also be described as "a unit that provides reaction behavior data to the indicator data acquisition unit."

[0094] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, and when the device executes the one or more programs, the steps performed by the device include: receiving behavioral data of a subject's response to multiple preset audiovisual stimulus tasks; obtaining index data of the subject's performance on preset audiovisual efficacy indicators based on the behavioral data; and determining the subject's degree of audiovisual compensation using the index data.

[0095] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the quantitative analysis method for the interactive compensation effect of auditory and visual functions provided by the present invention.

[0096] In the technical solution of this invention embodiment, the computer device first receives the subject's response behavior data to multiple preset audiovisual stimulation tasks, wherein the audiovisual stimulation tasks belong to at least one of the following audiovisual stimulation paradigms: auditory stimulation paradigm, visual stimulation paradigm, auditory-visual congruence paradigm, and auditory-visual incongruence paradigm; thereafter, the computer device obtains the subject's index data on preset audiovisual efficacy indicators based on the response behavior data, and uses the above index data to determine the subject's degree of audiovisual compensation. In this way, based on the specific audiovisual stimulation paradigm and data processing steps, the accurate quantification of auditory and visual interaction compensation is achieved from the behavioral side.

[0097] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for quantitatively analyzing the interactive compensation effect of auditory and visual functions, executed by a computer device; characterized in that, The method includes: Receive behavioral data on the subjects' responses to multiple pre-set audiovisual stimulation tasks; wherein the audiovisual stimulation tasks belong to at least one of the following audiovisual stimulation paradigms: auditory stimulation paradigm, visual stimulation paradigm, auditory-visual congruent paradigm, and auditory-visual incongruent paradigm. Based on the aforementioned behavioral data, the subject's performance data in the preset audiovisual efficacy index is obtained; and, The degree of audiovisual compensation for the subject is determined using the aforementioned indicator data.

2. The method according to claim 1, characterized in that, When the audiovisual stimulation paradigm includes an auditory stimulation paradigm, a visual stimulation paradigm, or an auditory-visual congruent paradigm, the audiovisual performance index includes at least one of the following: perceptual accuracy, average reaction time; or, When the audiovisual stimulation paradigm includes an auditory and visual non-consistent paradigm, the audiovisual efficacy index includes at least two of the following: auditory dominance ratio, audiovisual fusion ratio, and visual dominance ratio.

3. The method according to claim 1, characterized in that, The auditory stimulation paradigm includes: emitting monosyllabic speech signals; or... The visual stimulation paradigm includes: lip-syncing videos displaying monosyllabic speech signals; or, The auditory-visual consistency paradigm includes: displaying consistent lip-sync video while emitting monosyllabic speech signals; or... The auditory-visual inconsistency paradigm includes: displaying inconsistent lip-sync video while emitting monosyllabic speech signals; or... When the audiovisual stimulus task belongs to at least two audiovisual stimulus paradigms, the execution order of each audiovisual stimulus task is shuffled in advance according to its respective audiovisual stimulus paradigm.

4. The method according to claim 1, characterized in that, The step of determining the degree of audiovisual compensation of the subject using the index data includes: The audiovisual compensation level of the subject for each audiovisual efficacy index is determined based on the subject's index data for each index and a pre-determined range of values ​​for that index in a healthy population; and, The degree of audiovisual compensation is determined based on the audiovisual compensation level of the subject for each audiovisual efficacy index.

5. The method according to claim 4, characterized in that, The method further includes: before determining the degree of audiovisual compensation of the subject using the index data: The system receives oxygenation level-dependent signals from multiple brain regions output by a subject undergoing functional magnetic resonance imaging (fMRI), and identifies multiple visual-auditory response-related brain regions from these regions based on the received oxygenation level-dependent signals; the subject undergoes fMRI while receiving the visual-auditory stimulation paradigm. The system receives EEG signals from multiple leads output by the subject performing an EEG operation, and identifies the leads corresponding to the brain regions related to the audiovisual response as key leads; the subject performs the EEG operation while receiving the audiovisual stimulation paradigm; From the EEG data of the key leads, feature data of the audiovisual response-related brain regions in response to the audiovisual stimulation paradigm in a preset manner are extracted; based on the feature data and the blood oxygenation level-dependent signal of the audiovisual response-related brain regions, the device-side audiovisual compensation level of the subject is determined; and... The step of determining the degree of audiovisual compensation based on the audiovisual compensation level of the subject in each audiovisual performance index includes: determining the degree of audiovisual compensation based on the audiovisual compensation level of the subject in each audiovisual performance index and the audiovisual compensation level on the device side.

6. The method according to claim 5, characterized in that, The determination of multiple visual-auditory response-related brain regions from the multiple brain regions based on received blood oxygen level-dependent signals includes: Obtain the time-series function of the audiovisual stimulation paradigm and the hemodynamic response function of the multiple brain regions, and perform convolution on the hemodynamic response function and the time-series function to obtain the basic signal change curve of each brain region; Subtracting the blood oxygenation level-dependent signal of each brain region from the baseline signal change curve of the corresponding brain region yields the magnetic resonance imaging (MRI) intensity data for each brain region; and... Based on the magnetic resonance intensity data, the brain regions with the highest magnetic resonance intensity and a predetermined number of regions are identified as the brain regions related to visual and auditory responses.

7. The method according to claim 5, characterized in that, The preset features include at least one of the following: event-related potential component features, time-frequency features, and brain region functional connectivity features; The event-related potential components include at least one of the following: mismatch negative wave MMN, P1 / N1 wave, P300 wave, N400 wave, and front negative wave audFN; The time-frequency features include: the short-time Fourier transform features of the EEG signals of the key leads, and the short-time Fourier transform features include at least one of the following: time-frequency power features, time-frequency resolution trade-off features, spectral entropy features, time-frequency peak frequency features, and time-frequency ridge features; The brain region functional connectivity features include: the phase-locking value features of any two visual-auditory response-related brain regions in a preset frequency band.

8. The method according to claim 5, characterized in that, Determining the device-side audiovisual compensation level of the subject based on the feature data and the blood oxygen level-dependent signal of the brain regions related to audiovisual response includes: inputting the feature data and the blood oxygen level-dependent signal of the brain regions related to audiovisual response into a pre-trained feature fusion and calculation model to obtain the device-side audiovisual compensation level; or... Determining the degree of audiovisual compensation based on the audiovisual compensation level of the subject in each audiovisual performance index and the audiovisual compensation level on the device side includes: determining the subject's behavioral audiovisual compensation level based on the subject's audiovisual compensation level in each audiovisual performance index, and adjusting the behavioral audiovisual compensation level using the device side audiovisual compensation level to obtain the degree of audiovisual compensation.

9. A quantitative analysis system for the interactive compensation effect of auditory and visual functions, set in a computer device; characterized in that, The system includes: The behavioral data receiving unit is used to receive behavioral data of the subject's response to multiple preset audiovisual stimulus tasks; wherein the audiovisual stimulus tasks belong to at least one of the following audiovisual stimulus paradigms: auditory stimulus paradigm, visual stimulus paradigm, auditory-visual congruent paradigm, and auditory-visual incongruent paradigm. The indicator data acquisition unit is used to acquire indicator data of the subject in a preset audiovisual efficacy index based on the reaction behavior data; and, The audiovisual compensation calculation unit is used to determine the degree of audiovisual compensation for the subject using the index data.

10. A brain-computer interface control system, installed in a computer device; characterized in that, The system includes: The MRI receiving unit is used to receive blood oxygen level-dependent signals from multiple brain regions output by a subject undergoing functional magnetic resonance imaging (fMRI), and to identify multiple brain regions with active audiovisual responses from the multiple brain regions based on the received blood oxygen level-dependent signals; the subject undergoes the fMRI while receiving a preset audiovisual stimulation paradigm, the audiovisual stimulation paradigm including at least one of the following: auditory stimulation paradigm, visual stimulation paradigm, auditory-visual congruent paradigm, and auditory-visual incongruent paradigm; An EEG receiving unit is configured to receive EEG signals from multiple leads output by the subject performing an EEG operation, and to identify the leads corresponding to the brain regions with active visual and auditory responses as key leads; the subject performs the EEG operation while receiving the visual and auditory stimulation paradigm; and, The control unit is configured to extract feature data of the visually and aurally active brain regions in response to the visually and aurally stimulating paradigm from the EEG data of the key leads, determine the subject's execution intention based on the feature data and the blood oxygen level-dependent signal of the visually and aurally active brain regions, and send the determined execution intention to a pre-connected electronic device.

Images