Audio-visual integration analysis method and device, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of data processing, and provides an audio-visual integration analysis method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining to-be-analyzed data which comprises the data of a tested person for word discrimination in different stimulation environments, the stimulus environment comprises at least one gesture audiovisual stimulus and a reference audiovisual stimulus, and the reference audiovisual stimulus comprises one or more audiovisual stimulus except the gesture audiovisual stimulus; according to the to-be-analyzed data, determining a response duration corresponding to each stimulation environment, wherein the response duration reflects a duration of completing word discrimination by the tested person in the stimulation environment; and determining an analysis result based on each reaction duration, wherein the analysis result reflects the influence degree of the audio-visual stimulation based on the gesture on the audio-visual integration capability of the tested person. The accuracy of audio-visual integration analysis can be improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular relates to an audio-visual integrated analysis method, device, electronic device, and computer-readable storage medium. Background Art

[0002] The audiovisual integration effect refers to the process in which the visual and auditory systems do not work in isolation when receiving external information, but rather interact and collaborate to process information. Because audiovisual integration enables the brain to interpret environmental stimuli more efficiently and accurately, people's audiovisual integration ability reflects their brain's ability to process cross-modal information. Declining auditory and visual perception in the elderly can lead to difficulties understanding speech, increasing the likelihood of developing mental disorders and cognitive risks. Therefore, analyzing the elderly's audiovisual integration ability is often used to help identify mental disorders and cognitive risks.

[0003] Currently, audiovisual integration ability is usually analyzed simply based on vision, hearing, and audiovisual stimulation. However, in fact, as people age, their auditory function gradually declines, which may make it more difficult for them to accurately perceive and understand verbal information, resulting in less accurate audiovisual integration ability analysis. Summary of the Invention

[0004] The embodiments of the present application provide an audio-visual integrated analysis method, device, electronic device, and storage medium, which can improve the accuracy of audio-visual integrated analysis.

[0005] In a first aspect, an embodiment of the present application provides an audio-visual integrated analysis method, comprising:

[0006] Acquiring data to be analyzed, the data to be analyzed comprising data of word discrimination performed by a test subject under different stimulation environments, the stimulation environments comprising at least one gesture audiovisual stimulus and a reference audiovisual stimulus, the reference audiovisual stimulus comprising one or more audiovisual stimuli other than the gesture audiovisual stimulus;

[0007] Determining the reaction time corresponding to each of the stimulation environments according to the data to be analyzed, wherein the reaction time reflects the time it takes for the tested person to complete word recognition in the stimulation environment;

[0008] An analysis result is determined based on each of the reaction times, and the analysis result reflects the degree of influence of the gesture-based audiovisual stimulation on the audiovisual integration ability of the tested person.

[0009] In a second aspect, an embodiment of the present application provides an audio-visual integrated analysis device, comprising:

[0010] a module for acquiring data to be analyzed, configured to acquire data to be analyzed, the data to be analyzed comprising data of word discrimination performed by a test subject under different stimulation environments, the stimulation environments comprising at least one gesture audiovisual stimulus and a reference audiovisual stimulus, the reference audiovisual stimulus being a stimulus other than the gesture audiovisual stimulus, and the reference audiovisual stimulus comprising one or more audiovisual stimuli other than the gesture audiovisual stimulus;

[0011] A reaction time determination module, configured to determine the reaction time corresponding to each of the stimulus environments based on the data to be analyzed, wherein the reaction time reflects the time it takes for the test subject to complete word recognition in the stimulus environment;

[0012] An analysis module is configured to determine an analysis result based on each of the reaction times, wherein the analysis result reflects the degree of influence of the gesture-based audiovisual stimulation on the audiovisual integration ability of the tested person.

[0013] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the audio-visual integration analysis method described in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the audio-visual integration analysis method described in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the audio-visual integration analysis method described in the first aspect above.

[0016] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0017] In an embodiment of the present application, data of word recognition performed by the test person in different stimulation environments is obtained as data to be analyzed, wherein the stimulation environment includes at least one gesture audiovisual stimulation and a reference audiovisual stimulation, and the reference audiovisual stimulation is one or more audiovisual stimulations other than gesture audiovisual stimulation. Since gestures can provide hearing-impaired elderly people with visual clues and additional predictive information that cannot be expressed in words, therefore, analyzing the impact of gesture-based audiovisual stimulation on the audiovisual integration ability of the test person based on the data to be analyzed can reduce the impact of hearing impairment on the elderly in the word recognition process, and obtain a more accurate analysis result. Furthermore, based on the more accurate analysis result, it can provide an effective basis for the identification and prevention of mental disorders and cognitive risks in the elderly. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art.

[0019] Figure 1 This is a flowchart of an audio-visual integrated analysis method provided in one embodiment of the present application;

[0020] Figure 2 This is a flow chart of determining analysis results based on the reaction times provided in an embodiment of the present application;

[0021] Figure 3 This is another flow chart of determining analysis results based on the reaction times provided in an embodiment of the present application;

[0022] Figure 4 Schematic diagram of the structure of the audio-visual integrated analysis device provided in an embodiment of the present application;

[0023] Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0025] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0026] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0027] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized.

[0029] Example 1:

[0030] Figure 1 A flowchart of an audiovisual integrated analysis method provided in an embodiment of the present application is shown, and is described in detail as follows:

[0031] Step S101, obtaining data to be analyzed, wherein the data to be analyzed includes data of word recognition performed by the test person in different stimulation environments, wherein the stimulation environment includes at least one gesture audio-visual stimulation and a reference audio-visual stimulation, and the reference audio-visual stimulation includes one or more audio-visual stimulations other than the gesture audio-visual stimulation.

[0032] The above-mentioned gesture audiovisual stimulation is gesture-based audiovisual stimulation, such as rhythmic gesture auditory stimulation or rhythmic gesture visual auditory stimulation (that is, a stimulation environment that includes rhythmic gesture stimulation, auditory stimulation and visual stimulation at the same time).

[0033] Optionally, the data to be analyzed may be in the form of video data, which reflects data related to the actions and speech of the test subject when performing word recognition in different stimulation environments.

[0034] In some embodiments, the data to be analyzed may be data of word recognition performed by the test subject in a noise scene under different stimulation environments. The noise scene may be a white noise scene, a crowd noise scene, or a cocktail party scene containing both white noise and crowd noise.

[0035] In some embodiments, the above-mentioned tested persons may include non-elderly people (such as young people) and elderly people, so as to compare with the corresponding analysis results of young people and analyze the degree of influence of gesture-based audiovisual stimulation on the audiovisual integration ability of the elderly, so as to better identify mental disorders and cognitive risks of the elderly.

[0036] It should be noted that when the tested person performs word identification in any stimulation environment, the identification of multiple words is usually performed continuously. For example, for auditory stimulation, the word identification of the auditory stimulation will be performed within a continuous time period, wherein the audio of a word can be played every preset time length (such as every 20 seconds), and a total of 50 words are played. Then the data corresponding to the auditory stimulation in the data to be analyzed is 1000 seconds of continuous data.

[0037] Step S102 : determining the reaction time corresponding to each of the stimulus environments based on the data to be analyzed, wherein the reaction time reflects the time it takes for the test subject to complete word recognition in the stimulus environment.

[0038] The above reaction time is the time it takes for the test subject to complete word discrimination in the stimulus environment, that is, the time it takes for the test subject to give a word discrimination answer after the word is displayed and / or played in the stimulus environment.

[0039] Since people are affected by their multi-sensory interaction and conflict resolution abilities during word recognition, the time it takes them to complete word recognition in different stimulation environments usually varies. Therefore, the time it takes for the person being tested to complete word recognition in a stimulation environment can, to a certain extent, reflect the person being tested's multi-sensory interaction abilities and conflict resolution abilities in that stimulation environment. In other words, it reflects the person being tested's audio-visual integration abilities in that stimulation environment. Therefore, when analyzing the data to be analyzed, the embodiment of the present application can determine the time it takes for the person being tested to complete word recognition in the corresponding stimulation environment based on the data corresponding to different stimulation environments in the data to be analyzed, and obtain the reaction time corresponding to each stimulation environment.

[0040] Step S103 : determining an analysis result based on each of the reaction times, wherein the analysis result reflects the degree of influence of the gesture-based audiovisual stimulation on the audiovisual integration ability of the test subject.

[0041] Since each reaction time can reflect the audio-visual integration ability of the test person under at least one gesture-based audio-visual stimulus and one or more audio-visual stimuli other than gesture-based audio-visual stimulation (i.e., reference audio-visual stimulation), in an embodiment of the present application, the audio-visual integration ability of the test person under gesture-based audio-visual stimulation and reference audio-visual stimulation can be first analyzed based on each reaction time.

[0042] Since gestures can provide visual cues and additional predictive information to hearing-impaired elderly people, thereby helping them understand verbal information and reducing the impact of decreased auditory function, the difference between the audio-visual integration ability under gesture-based audio-visual stimulation and the audio-visual integration ability under reference audio-visual stimulation can better analyze the degree of influence of gesture-based audio-visual stimulation on the audio-visual integration of the test subjects and obtain more accurate analysis results.

[0043] Furthermore, based on the analysis results, the ability of the test subjects to perceive gestures and verbal information under gesture-based audiovisual stimulation can be better analyzed, providing an effective basis for the identification and prevention of mental disorders and cognitive risks in the elderly.

[0044] In an embodiment of the present application, data of word recognition performed by the test person in different stimulation environments is obtained as data to be analyzed, wherein the stimulation environment includes at least one gesture audiovisual stimulation and a reference audiovisual stimulation, and the reference audiovisual stimulation is one or more audiovisual stimulations other than gesture audiovisual stimulation. Since gestures can provide hearing-impaired elderly people with visual clues and additional predictive information that cannot be expressed in words, therefore, analyzing the impact of gesture-based audiovisual stimulation on the audiovisual integration ability of the test person based on the data to be analyzed can reduce the impact of hearing impairment on the elderly in the word recognition process, and obtain a more accurate analysis result. Furthermore, based on the more accurate analysis result, it can provide an effective basis for the identification and prevention of mental disorders and cognitive risks in the elderly.

[0045] In some embodiments, the reference audiovisual stimulation includes visual stimulation, auditory stimulation, and multimodal audiovisual stimulation, and the step S103 includes:

[0046] S201 : Determine the unimodal probability distribution functions corresponding to the visual stimulus and the auditory stimulus respectively according to the reaction duration corresponding to the reference audiovisual stimulus.

[0047] S202 : Process each of the above single-modal probability distribution functions based on a competition model to obtain a predicted cumulative distribution function.

[0048] S203. Determine the integration time window corresponding to the multimodal audiovisual stimulation based on the difference between the predicted cumulative distribution function and the first cumulative distribution function. The first cumulative distribution function is determined based on the reaction duration corresponding to the multimodal audiovisual stimulation. The integration time window is a time window in which the audiovisual integration effect exists.

[0049] S204. Determine the integration time window corresponding to the gesture audiovisual stimulation based on the difference between the predicted cumulative distribution function and the second cumulative distribution function, wherein the second cumulative distribution function is determined based on the reaction time corresponding to the gesture audiovisual stimulation.

[0050] S205: Determine the analysis results according to each of the integrated time windows.

[0051] The competition model is a framework for describing and predicting how an individual processes information from different senses (such as vision and hearing) that arrive simultaneously or successively. In an embodiment of the present application, the competition model is used to predict the cumulative distribution under visual and auditory stimulation based on the unimodal distribution function corresponding to the visual stimulation and the unimodal distribution function corresponding to the auditory stimulation, and obtain a predicted cumulative distribution function. Since the two unimodal distribution functions are determined based on real data to be analyzed, they can more accurately reflect the actual reaction time of the test person when performing word discrimination under the two stimulation environments of visual stimulation and auditory stimulation. Therefore, based on these two unimodal distribution functions, the reaction time of the test person for word discrimination under visual and auditory stimulation can be more accurately predicted, and a predicted cumulative distribution function with higher accuracy can be obtained.

[0052] In order to improve the accuracy of subsequent analysis results, in an embodiment of the present application, reference audio-visual stimulation is set to include two unimodal audio-visual stimulations, namely visual stimulation and auditory stimulation, as well as multimodal audio-visual stimulation, so as to analyze the duration of audio-visual integration of the tested person under multimodal audio-visual stimulation and gesture audio-visual stimulation in combination with the reaction time of the tested person under unimodal audio-visual stimulation. In an embodiment of the present application, the above-mentioned multimodal audio-visual stimulation includes visual and auditory stimulation, and the above-mentioned gesture audio-visual stimulation includes rhythmic gesture visual and auditory stimulation and / or meaningful gesture visual and auditory stimulation.

[0053] During the analysis process, the reaction time corresponding to the visual stimulus in the reference audio-visual stimulus can be first determined from each reaction time, and then the unimodal probability distribution function corresponding to the visual stimulus can be calculated based on the reaction time corresponding to the visual stimulus; and the reaction time corresponding to the auditory stimulus in the reference audio-visual stimulus can be determined from each reaction time, and the unimodal probability distribution function corresponding to the auditory stimulus can be calculated based on the reaction time corresponding to the auditory stimulus.

[0054] It can be understood that the calculated unimodal probability distribution function can reflect the probability distribution of the reaction time corresponding to each word discrimination under the corresponding unimodal audiovisual stimulation. After obtaining the unimodal probability distribution functions corresponding to the two unimodal audiovisual stimulations, the cumulative distribution function of the test subject's reaction time when performing word discrimination in the visual and auditory stimulation environment can be predicted based on these two unimodal probability distribution functions to obtain a predicted cumulative distribution function.

[0055] As an example, the above-mentioned predicted cumulative distribution function can be expressed in the following form:

[0056] P(RTRacemodel < t) = P(R TA < t) + P(R TV < t) - P(R TA < t) * P(R TV < t)

[0057] RTRacemodel represents the reaction time corresponding to visual-auditory correspondence. P(RTRacemodel < t) represents the probability that the reaction time corresponding to visual-auditory stimulation is less than or equal to a specific value. t represents a time window of a preset duration (such as 10 ms) (referred to as the base time window). R TA represents the reaction time corresponding to auditory stimulation. P(R TA [[ID=…]]< t) represents the reaction probability of auditory stimulation within a given time window t. R TV represents the reaction time corresponding to visual stimulation. P(R TV < t) represents the reaction probability of visual stimulation within a given time window t. <…]]

[0058] Meanwhile, according to the reaction time corresponding to multi-modal visual-auditory stimulation among each reaction time, calculate the actual cumulative distribution of its reaction time to obtain the first cumulative distribution function; and, according to the reaction time corresponding to gesture-auditory stimulation among each reaction time, calculate the actual cumulative distribution of its reaction time to obtain the second cumulative distribution function. It should be noted that when the tested person performs word discrimination under various gesture-auditory stimulations, multiple second cumulative distribution functions will be calculated, that is, one gesture-auditory stimulation corresponds to one second cumulative distribution function.

[0059] Optionally, when determining the integration time window (the first integration time window) corresponding to the above-mentioned multi-modal visual-auditory stimulation according to the difference between the above-mentioned predicted cumulative distribution function and the first cumulative distribution function, the predicted cumulative distribution function can be subtracted from the first cumulative distribution function to obtain a first calculation result, and this calculation result includes the probability difference corresponding to each base time window. After obtaining the first calculation result, the probability difference corresponding to each base time window in this first calculation result can be subjected to an independent sample test with a first threshold (such as 0) to judge whether there is a significant difference between the probability difference corresponding to this base time window and the first threshold.

[0060] Based on the result of the independent test, determine the base time window where there is a significant difference between the probability difference and the first threshold to obtain the target base time window, and this target base time window is the window where there is an auditory-visual integration effect; finally, according to each target base time window, the integration time window corresponding to the multi-modal visual-auditory stimulation can be obtained.

[0061] It can be understood that the method for determining the integration time window corresponding to the gesture audiovisual stimulation based on the difference between the predicted cumulative distribution function and the second cumulative distribution function is the same as above and will not be repeated here.

[0062] like Figure 2 As shown, after determining the integration time windows corresponding to the multimodal audio-visual stimulation and the gesture audio-visual stimulation, the audio-visual integration ability of the tested user in the environment of gesture-based audio-visual stimulation and other audio-visual stimulation can be compared according to the determined integration time windows, thereby determining the degree of influence of the gesture-based audio-visual stimulation on the audio-visual integration ability of the tested person.

[0063] In an embodiment of the present application, the cumulative distribution function of the reaction time of the test person when performing word discrimination under visual and auditory stimulation is predicted based on two unimodal audiovisual stimulations, namely visual stimulation and auditory stimulation, to obtain a predicted cumulative distribution function, which is then compared with the actual cumulative distribution function of the test person corresponding to visual and auditory stimulation and gesture audiovisual stimulation, respectively, to obtain the integration time window corresponding to the multimodal audiovisual stimulation and gesture audiovisual stimulation. Since the integration time window is a time window in which the audiovisual integration effect exists, based on each integration time window, it is possible to better analyze the difference in the audiovisual integration ability of the test user under gesture-based audiovisual stimulation and visual and auditory stimulation, and more accurately determine the degree of influence of gesture-based audiovisual stimulation on the audiovisual integration ability of the test person.

[0064] In some embodiments, the above step S205 includes:

[0065] Based on the types of the above-mentioned tested persons and the different above-mentioned stimulation environments, a binary mixed variance analysis was performed on the duration corresponding to each of the above-mentioned integrated time windows, the peak value corresponding to the above-mentioned integrated time windows, and the area under the curve corresponding to the above-mentioned integrated time windows to obtain the above-mentioned analysis results. The above-mentioned area under the curve was determined according to the above-mentioned integrated time window and the corresponding cumulative distribution function. The types of the above-mentioned tested persons include the elderly and non-elderly.

[0066] It can be understood that the duration corresponding to each integration time window can reflect the duration of the corresponding audio-visual integration effect; the peak value corresponding to the integration time window can reflect the efficient collaborative state of the tested person in processing cross-sensory information in the corresponding audio-visual integration effect; the area under the above curve can reflect the cumulative influence or intensity of audio-visual integration in the corresponding audio-visual integration effect, that is, it can show the overall effect of audio-visual integration from a quantitative perspective.

[0067] The above-mentioned two-way mixed ANOVA is a statistical analysis method used to study how two factors affect a continuous variable, and one of the two factors is a fixed effect and the other is a random effect.

[0068] It should be noted that the above-mentioned elderly people are those whose age reaches or exceeds the age threshold (such as 60 years old), and non-elderly people are those whose age does not reach the age threshold, such as adults whose age does not reach the age threshold.

[0069] In order to improve the reliability of the obtained analysis results, in an embodiment of the present application, the tested persons may include the elderly and non-elderly. When determining the analysis results based on the determined integration time windows, the binary mixed variance analysis method can be used to analyze the degree of influence of different types of tested persons and different stimulation environments on the duration corresponding to the integration time window, the peak value corresponding to the integration time window, and the area under the curve corresponding to the integration time window. Then, according to the degree of each influence, the degree of influence of gesture audio-visual stimulation on the audio-visual integration effect of the tested persons is analyzed to obtain the required analysis results.

[0070] Since the analysis results obtained include the degree of influence of gesture and audio-visual stimulation on the audio-visual integration of the elderly, as well as the degree of influence of gesture and audio-visual stimulation on the audio-visual integration of non-elderly people, when it is necessary to identify and prevent mental disorders and cognitive risks in the elderly based on the analysis results, the degree of influence of gesture and audio-visual stimulation on the audio-visual integration of non-elderly people can be used to understand the degree of influence of gesture and audio-visual stimulation on the audio-visual integration of people under normal circumstances, and then, use this as a reference to evaluate mental disorders and cognitive risks in the elderly, that is, to provide a more reliable and accurate basis for the identification and prevention of mental disorders and cognitive risks in the elderly.

[0071] In some embodiments, the reference audiovisual stimulation includes at least one of a single-modal audiovisual stimulation and a multimodal audiovisual stimulation, and the step S103 includes:

[0072] B1. Determine the degree of enhancement of the first reaction corresponding to the multimodal audiovisual stimulation based on the reaction time and the target reaction time corresponding to the multimodal audiovisual stimulation, wherein the target reaction time is the reaction time with the smallest median value among the reaction times corresponding to the unimodal audiovisual stimulation.

[0073] B2. Determine the degree of enhancement of the second reaction corresponding to the gesture audiovisual stimulation according to the reaction duration corresponding to the gesture audiovisual stimulation and the target reaction duration.

[0074] B3. Determine the above analysis result based on the above first reaction enhancement degree and the above second reaction enhancement degree.

[0075] The above-mentioned unimodal audiovisual stimulation refers to an audiovisual stimulation including a single stimulation environment, such as visual stimulation or auditory stimulation.

[0076] The multimodal audiovisual stimulation refers to an audiovisual stimulation including multiple stimuli. Optionally, the multimodal audiovisual stimulation may be an audiovisual stimulation including visual stimulation and auditory stimulation (ie, visual-auditory stimulation).

[0077] In order to improve the accuracy of the analysis results, when analyzing according to each reaction time, calculation can be performed based on the reaction time corresponding to the multimodal audio-visual stimulation and the determined target audio-visual time to obtain the first reaction enhancement degree corresponding to the multimodal audio-visual stimulation. The first reaction enhancement degree can reflect the degree of improvement in the reaction ability of the tested person when receiving the multimodal audio-visual stimulation, relative to the single-modal audio-visual stimulation corresponding to the target reaction time.

[0078] Similarly, for gesture audio-visual stimulation, the corresponding second reaction enhancement degree is calculated based on its corresponding reaction time and target reaction time. The second reaction enhancement degree reflects the degree of improvement in the reaction ability of the test person when receiving gesture audio-visual stimulation, compared with the unimodal audio-visual stimulation corresponding to the target reaction time.

[0079] In some embodiments, the first reaction enhancement degree can be expressed as follows:

[0080]

[0081] rMRE AV Indicates the degree of enhancement of the first reaction, RT A Indicates the reaction time corresponding to auditory stimulation, RT A Indicates the reaction time corresponding to visual stimulation, RT AV Indicates the reaction time corresponding to visual and auditory stimuli;

[0082] median(RT A ) represents the median value of each reaction time corresponding to the auditory stimulus, median(RT V ) means taking the median value of each reaction time corresponding to the visual stimulus, min(median(RT A ) means taking the minimum value between the median value corresponding to the auditory stimulus and the median value corresponding to the visual stimulus.

[0083] The above-mentioned second reaction enhancement degree can be expressed as follows:

[0084]

[0085] rMRE GAV Indicates the degree of enhancement of the second reaction, RT GAV Indicates the reaction time corresponding to the gesture audiovisual stimulation, median (RT GAV ) represents the median value of each reaction time corresponding to the gesture audio-visual stimulation.

[0086] Finally, the first reaction enhancement degree and the second reaction enhancement degree can be compared to analyze the influence of the gesture audiovisual stimulation on the audiovisual integration ability of the tested person and obtain the required analysis results.

[0087] In other embodiments, when determining the analysis results based on the degree of enhancement of the first reaction and the degree of enhancement of the second reaction, a binary mixed variance analysis can be performed on the degree of enhancement of the first reaction and the degree of enhancement of the second reaction respectively based on the type of the test person and the above-mentioned stimulation environment. The binary mixed variance analysis is used to evaluate the degree of difference in the audio-visual integration ability of the test person under different conditions (i.e., different types of test persons and different stimulation environments). Furthermore, the degree of difference in the audio-visual integration ability of the test person under different conditions can better determine the degree of influence of gesture audio-visual stimulation on the audio-visual integration ability of the test person.

[0088] In an embodiment of the present application, the first reaction enhancement degree corresponding to the multimodal audiovisual stimulation and the second reaction enhancement degree corresponding to the gesture audiovisual stimulation are calculated respectively based on the target reaction time determined by the unimodal audiovisual stimulation. Since the first reaction enhancement degree and the second reaction enhancement degree obtained by the above processing respectively reflect the degree of improvement in the reaction ability of the person being tested relative to the unimodal audiovisual stimulation corresponding to the target reaction time when receiving the multimodal audiovisual stimulation and the gesture audiovisual stimulation, according to the difference between the first reaction enhancement degree and the second reaction enhancement degree, the degree of improvement in the reaction ability of the person being tested due to the gesture audiovisual stimulation can be better analyzed, that is, the degree of improvement in the audiovisual integration ability of the person being tested due to the gesture audiovisual stimulation can be analyzed, and a more accurate analysis result can be obtained.

[0089] In some embodiments, before the above step B3, the method further includes:

[0090] Determine whether the first reaction enhancement degree and the second reaction enhancement degree are credible.

[0091] Correspondingly, determining the analysis result according to the first reaction enhancement degree and the second reaction enhancement degree includes:

[0092] In the case where it is determined that the first reaction enhancement degree and the second reaction enhancement degree are credible, the analysis result is determined according to the first reaction enhancement degree and the second reaction enhancement degree.

[0093] Because in the process of word discrimination tasks based on different stimulus environments, although a single stimulus may not be sufficient to trigger a reaction, the statistical characteristics such as the average intensity of these stimulus environments may be integrated by the nervous system and trigger a reaction, so that the improvement of the test person's reaction ability is promoted by statistical characteristics rather than audio-visual integration. Therefore, in order to improve the accuracy of the analysis results, after calculating the first reaction enhancement degree and the second reaction enhancement degree, we can first determine whether the first reaction enhancement degree and the second reaction enhancement degree are the result of statistical characteristics promotion, and thus determine whether the first reaction enhancement degree and the second reaction enhancement degree are credible.

[0094] like Figure 3 As shown, only when it is determined that both the first reaction enhancement degree and the second reaction enhancement degree are credible, the final analysis result is determined based on the first reaction enhancement degree and the second reaction enhancement degree.

[0095] If it is determined that the first reaction enhancement degree and / or the second reaction enhancement degree is not credible, at this time, the corresponding data to be analyzed can be re-acquired and calculations can be performed based on the newly acquired data to be analyzed until the new first reaction enhancement degree and / or the second reaction enhancement degree obtained are credible.

[0096] For example, if any reaction enhancement degree is determined to be unreliable, the data of the test subject's word discrimination under unimodal audiovisual stimulation, multimodal audiovisual stimulation, and gesture audiovisual stimulation can be re-acquired as new data to be analyzed. Based on the new data to be analyzed, calculations are performed to obtain new first reaction enhancement degrees and new second reaction enhancement degrees. Alternatively, an error notification can be sent to the target terminal (the terminal corresponding to the user), indicating that the reliability of the data to be analyzed does not meet the requirements.

[0097] In some embodiments, each reaction enhancement degree can be independently sampled and verified with a second threshold value (such as 0.01), and the verification results of the independent sample verification can be used to determine whether the improvement in the reaction ability of the tested person is the result of the promotion of audio-visual integration.

[0098] For any reaction enhancement degree, if the verification result indicates that the difference between the reaction enhancement degree and the second threshold is significant, it can be determined that the reaction enhancement degree is the result of audio-visual integration promotion. At this time, the reaction enhancement degree is credible; if the verification result indicates that the difference between the reaction enhancement degree and the second threshold is not significant, it can be determined that the first reaction enhancement degree is the result of statistical characteristic promotion, rather than the result of audio-visual integration promotion. At this time, the reaction enhancement degree is not credible.

[0099] In other embodiments, whether the first reaction enhancement degree and the second reaction enhancement degree are credible may also be determined based on a competition model.

[0100] In the embodiment of the present application, considering that the reaction enhancement degree of the tested person may be the result of the promotion of statistical characteristics, before determining the analysis result based on the first reaction enhancement degree and the second reaction enhancement degree, it is first determined whether the first reaction enhancement degree and the second reaction enhancement degree are the result of the promotion of statistical characteristics, that is, whether the first reaction enhancement degree and the second reaction enhancement degree are credible. When it is determined that they are credible, the analysis result is obtained by further analysis to ensure the accuracy and reliability of the analysis result.

[0101] In some embodiments, before the above step S103, the method further includes:

[0102] The word recognition ability corresponding to each of the above-mentioned stimulation environments is determined, and the above-mentioned word recognition ability reflects the ability of the above-mentioned test subject to correctly recognize words in the above-mentioned stimulation environment.

[0103] Correspondingly, the above analysis results are determined based on each of the above reaction times, including:

[0104] Based on the types of the test subjects and the different stimulation environments, a binary mixed variance analysis was conducted on the word discrimination ability and the reaction time to obtain the above analysis results.

[0105] In order to further improve the accuracy of the analysis results, in the embodiment of the present application, the ability of the tested person to correctly distinguish words in various stimulation environments is analyzed based on the data to be analyzed to obtain the word recognition ability corresponding to each stimulation environment.

[0106] Optionally, when analyzing the ability of the test subjects to correctly distinguish words in any stimulation environment, the hit rate and false alarm rate of the test subjects in the word discrimination task under the stimulation environment can be calculated. The hit rate can reflect the proportion of words that the test subjects correctly distinguish when the words appear, and the false alarm rate can reflect the proportion of non-words (such as noise) that the test subjects distinguish as words when they appear. Therefore, by analyzing the ability of the test subjects to correctly distinguish words in the stimulation environment based on the hit rate and false alarm rate, a more accurate word discrimination ability can be obtained.

[0107] In some embodiments, the above word recognition capability can be expressed as follows:

[0108] d ′ =Z 击中 -Z 虚报

[0109] d ′ refers to the ability to distinguish words, Z 击中 is the hit rate, Z 虚报 It refers to the false reporting rate.

[0110] When analyzing the effect of gesture-based audiovisual stimulation on the audiovisual integration ability of the testee, a binary mixed variance analysis can be performed on the word discrimination ability and reaction time based on the type of the testee and different stimulation environments, so as to analyze the specific effects of different stimulation environments on the audiovisual integration abilities of different types of testees, thereby being able to more accurately analyze the degree of influence of gesture-based audiovisual stimulation on the audiovisual integration abilities of the testees. Moreover, in the embodiment of the present application, the types of testees include the elderly and non-elderly people, that is, the degree of influence of gesture-based audiovisual stimulation on the audiovisual integration abilities of the elderly and the degree of influence on the audiovisual integration abilities of non-elderly people can be analyzed, thereby being able to analyze the degree of influence of gesture-based audiovisual stimulation on the word discrimination process of hearing-impaired elderly people, thereby being able to provide a more reliable and effective basis for the identification and prevention of mental disorders and cognitive risks in the elderly.

[0111] In the embodiments of the present application, the ability of the test persons to correctly distinguish words in various stimulation environments is also analyzed, and then the specific effects of different stimulation environments on the audiovisual integration abilities of different types of test persons are comprehensively analyzed in combination with their reaction time for distinguishing words. Furthermore, the degree of influence of gesture-based audiovisual stimulation on the word recognition process of hearing-impaired elderly people can be better analyzed, providing a more reliable and effective basis for the identification and prevention of mental disorders and cognitive risks in the elderly.

[0112] In some embodiments, the data to be analyzed further includes EEG data of the test subject performing word recognition under different stimulation environments, and the reference audiovisual stimulation includes visual stimulation, auditory stimulation, and multimodal audiovisual stimulation. Before step S103, the following steps may also be included:

[0113] The potential waveform corresponding to each of the stimulation environments is determined based on the data to be analyzed, and the potential waveform reflects the event-related potential changes of the test subject during the word discrimination process under the stimulation environment.

[0114] A first difference waveform is determined based on the difference between the potential waveform corresponding to the multimodal audiovisual stimulation and the unimodal potential waveform, where the unimodal potential waveform includes the potential waveforms corresponding to the visual stimulation and the auditory stimulation.

[0115] A second difference waveform is determined according to the difference between the potential waveform corresponding to the gesture audiovisual stimulation and the unimodal potential waveform.

[0116] Correspondingly, the above analysis results are determined based on each of the above reaction times, including:

[0117] The analysis result is determined according to the first difference waveform, the second difference waveform, and the reaction times.

[0118] Event-Related Potential (ERP) refers to the potential signal obtained by measuring the brain's immediate electrophysiological response to specific external stimuli or internal psychological events. It can reflect the brain's immediate response to specific stimuli, and the potential waveform is the manifestation of the event-related potential in time and amplitude.

[0119] To further improve the accuracy of audiovisual integrated analysis, in this embodiment of the application, EEG data of the test subjects performing word discrimination in various stimulation environments is also obtained as the data to be analyzed. Among them, the reference audiovisual stimulation in this embodiment of the application includes visual stimulation, auditory stimulation, and multimodal audiovisual stimulation.

[0120] Optionally, when determining the potential waveform corresponding to each stimulation environment based on the data to be analyzed, the EEG data can be time-locked and averaged in combination with the timestamps of word discrimination in each stimulation environment to highlight ERP components related to audiovisual integration ability, such as early components (C1, P1, or N1, etc.) and late components (such as P300 or N400, etc.). Optionally, embodiments of the present application can analyze the N400 component in the EEG data to obtain the potential waveform corresponding to each stimulation environment.

[0121] Since the potential waveform can reflect the event-related potential changes in the test subjects during word discrimination in the stimulation environment, that is, reflect the cognitive response of the test subjects when performing word discrimination in the stimulation environment, etc., therefore, according to the difference waveform between the potential waveforms corresponding to different stimulation environments, it is possible to better analyze the differences in cognitive responses of the test subjects when performing word discrimination in different stimulation environments and whether there is an audio-visual integration effect. Furthermore, combined with the first difference waveform and reaction time corresponding to the multimodal audio-visual stimulation, and the second difference waveform and reaction time corresponding to the gesture audio-visual stimulation, the degree of influence of the gesture audio-visual stimulation on the audio-visual integration ability of the test subjects is analyzed, and a more accurate analysis result is obtained.

[0122] Optionally, when analyzing the degree of influence of gesture audio-visual stimulation on the audio-visual integration ability of the tested person in combination with the first difference waveform and reaction time corresponding to the multimodal audio-visual stimulation, and the second difference waveform and reaction time corresponding to the gesture audio-visual stimulation, the method described in the above embodiment can be used to first analyze each reaction time (such as calculating the first reaction enhancement degree and the second reaction enhancement degree) to obtain a behavioral analysis result; and based on the first difference waveform and the second difference waveform, analyze the time window in which the audio-visual integration effect exists in the corresponding word discrimination process to obtain the second integration time window corresponding to the multimodal audio-visual stimulation and the gesture audio-visual stimulation, and then analyze the degree of influence of the gesture audio-visual stimulation on the audio-visual integration needs of the tested person based on each second integration time window to obtain an EEG analysis result; finally, the final analysis result is determined comprehensively based on the behavioral analysis results and EEG analysis results obtained from different data analyses.

[0123] It is understood that the behavioral analysis results and EEG analysis results are based on different forms of data analysis when the test subjects perform word discrimination in various stimulation environments, and both can reflect the degree of influence of the gesture-based stimulation environment on the test subjects' audio-visual integration ability. In the embodiment of the present application, the degree of influence of the gesture-based stimulation environment on the test subjects' audio-visual integration ability is comprehensively analyzed based on the two different forms of data, further ensuring the accuracy of the final analysis results.

[0124] Optionally, when analyzing the time window in which the audiovisual integration effect exists in the corresponding word recognition process based on the first difference waveform and the second difference waveform, independent sample verification can be performed on each data point in each difference waveform and the third threshold (such as 0) to determine whether there is a significant difference between each data point in the difference waveform and the third threshold, that is, to determine whether there is an audiovisual integration effect for each data point, and then, the corresponding second integration time window is obtained based on the time corresponding to at least N (such as 12) consecutive data points that are significantly different from the third threshold.

[0125] When analyzing the degree of influence of gesture audiovisual stimulation on the audiovisual integration needs of the test person according to each second integration time window, the time window of interest can be determined based on each difference waveform, and then based on the type of the test person and the different stimulation environments, a binary mixed variance analysis can be performed on the average amplitude of the time window of interest and the second integration time window to obtain the required EEG analysis results.

[0126] In an embodiment of the present application, the EEG data and behavioral data of the test person when performing word discrimination in different stimulation environments are analyzed separately to obtain corresponding EEG analysis results and behavioral analysis results, and then the final analysis results are determined based on the EEG analysis results and behavioral analysis results. That is, the degree of influence of the gesture-based stimulation environment on the audio-visual integration ability of the test person is comprehensively analyzed based on data of different forms, thereby further ensuring the accuracy and reliability of the obtained analysis results.

[0127] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0128] Example 2:

[0129] Corresponding to the audio-visual integrated analysis method described in the above embodiment, Figure 4 The structural block diagram of the audio-visual integrated analysis device provided in an embodiment of the present application is shown. For the sake of convenience, only the parts related to the embodiment of the present application are shown.

[0130] Reference Figure 4 The device includes: a data acquisition module 41 for analysis, a reaction time determination module 42 and an analysis module 43.

[0131] The data acquisition module 41 for analysis is used to acquire the data to be analyzed. The data to be analyzed include data of word recognition performed by the test person under different stimulation environments. The stimulation environment includes at least one gesture audio-visual stimulation and a reference audio-visual stimulation. The reference audio-visual stimulation is a stimulation other than the gesture audio-visual stimulation. The reference audio-visual stimulation includes one or more audio-visual stimulations other than the gesture audio-visual stimulation.

[0132] The reaction time determination module 42 is used to determine the reaction time corresponding to each of the above-mentioned stimulation environments based on the above-mentioned data to be analyzed. The above-mentioned reaction time reflects the time it takes for the above-mentioned test subject to complete word recognition in the above-mentioned stimulation environment.

[0133] The analysis module 43 is configured to determine an analysis result based on each of the reaction times, wherein the analysis result reflects the degree of influence of the gesture-based audiovisual stimulation on the audiovisual integration ability of the test subject.

[0134] In an embodiment of the present application, data of word recognition performed by the test person in different stimulation environments is obtained as data to be analyzed, wherein the stimulation environment includes at least one gesture audiovisual stimulation and a reference audiovisual stimulation, and the reference audiovisual stimulation is one or more audiovisual stimulations other than gesture audiovisual stimulation. Since gestures can provide hearing-impaired elderly people with visual clues and additional predictive information that cannot be expressed in words, therefore, analyzing the impact of gesture-based audiovisual stimulation on the audiovisual integration ability of the test person based on the data to be analyzed can reduce the impact of hearing impairment on the elderly in the word recognition process, and obtain a more accurate analysis result. Furthermore, based on the more accurate analysis result, it can provide an effective basis for the identification and prevention of mental disorders and cognitive risks in the elderly.

[0135] In some embodiments, the reference audiovisual stimulation includes visual stimulation, auditory stimulation, and multimodal audiovisual stimulation, and the analysis module 43 includes:

[0136] The unimodal probability distribution function calculation unit is used to determine the unimodal probability distribution functions corresponding to the visual stimulation and the auditory stimulation respectively according to the reaction time corresponding to the reference audiovisual stimulation.

[0137] The prediction unit is used to process each of the above-mentioned single-modal probability distribution functions based on the competition model to obtain a predicted cumulative distribution function.

[0138] The first function processing unit is used to determine the integration time window corresponding to the above-mentioned multimodal audio-visual stimulation based on the difference between the above-mentioned predicted cumulative distribution function and the first cumulative distribution function. The above-mentioned first cumulative distribution function is determined based on the above-mentioned reaction time corresponding to the above-mentioned multimodal audio-visual stimulation. The above-mentioned integration time window is a time window in which there is an audio-visual integration effect.

[0139] The second function processing unit is used to determine the above-mentioned integration time window corresponding to the above-mentioned gesture audio-visual stimulation based on the difference between the above-mentioned predicted cumulative distribution function and the second cumulative distribution function, and the above-mentioned second cumulative distribution function is determined according to the above-mentioned reaction time corresponding to the above-mentioned gesture audio-visual stimulation.

[0140] The first analysis unit is configured to determine the analysis result according to each of the integration time windows.

[0141] In some embodiments, the analysis module 43 further includes:

[0142] The comparative analysis unit is used to perform a binary mixed variance analysis on the duration corresponding to each of the above-mentioned integrated time windows, the peak value corresponding to the above-mentioned integrated time window, and the area under the curve corresponding to the above-mentioned integrated time window based on the type of the above-mentioned tested persons and the different above-mentioned stimulation environments, to obtain the above-mentioned analysis results. The above-mentioned area under the curve is determined according to the above-mentioned integrated time window and the corresponding cumulative distribution function. The types of the above-mentioned tested persons include the elderly and non-elderly.

[0143] In some embodiments, the reference audiovisual stimulus includes at least one of a single-modal audiovisual stimulus and a multimodal audiovisual stimulus, and the analysis module 43 includes:

[0144] The first reaction enhancement degree calculation unit is used to determine the first reaction enhancement degree corresponding to the above-mentioned multimodal audio-visual stimulation based on the above-mentioned reaction time and target reaction time corresponding to the above-mentioned multimodal audio-visual stimulation, and the above-mentioned target reaction time is the above-mentioned reaction time with the smallest median value of the reaction time corresponding to the above-mentioned unimodal audio-visual stimulation.

[0145] The second reaction enhancement degree calculation unit is used to determine the second reaction enhancement degree corresponding to the gesture audio-visual stimulation according to the reaction duration corresponding to the gesture audio-visual stimulation and the target reaction duration.

[0146] The second analysis unit is used to determine the analysis result according to the first reaction enhancement degree and the second reaction enhancement degree.

[0147] In some embodiments, the audio-visual integrated analysis device further includes:

[0148] The judgment module is used to judge whether the first reaction enhancement degree and the second reaction enhancement degree are credible.

[0149] Correspondingly, the analysis module 43 is specifically configured to: when it is determined that the first reaction enhancement degree and the second reaction enhancement degree are credible, determine the analysis result according to the first reaction enhancement degree and the second reaction enhancement degree.

[0150] In some embodiments, the audio-visual integrated analysis device further includes:

[0151] The word recognition ability determination module is used to determine the word recognition ability corresponding to each of the above-mentioned stimulation environments. The above-mentioned word recognition ability reflects the ability of the above-mentioned test person to correctly recognize words in the above-mentioned stimulation environment.

[0152] Correspondingly, the analysis module 43 is specifically configured to perform a binary mixed variance analysis on the word recognition ability and the reaction time based on the type of the test person and the different stimulation environments, to obtain the analysis results.

[0153] In some embodiments, the data to be analyzed further includes EEG data of the test subject performing word recognition under different stimulation environments, the reference audiovisual stimulation includes visual stimulation, auditory stimulation, and multimodal audiovisual stimulation, and the audiovisual integrated analysis device further includes:

[0154] The potential waveform determination module is used to determine the potential waveform corresponding to each of the above stimulation environments based on the above data to be analyzed, and the above potential waveform reflects the event-related potential changes of the above test person during the word discrimination process in the above stimulation environment.

[0155] The first difference waveform determination module is used to determine a first difference waveform based on the difference between the potential waveform corresponding to the multimodal audiovisual stimulation and the unimodal potential waveform, where the unimodal potential waveform includes the potential waveform corresponding to the visual stimulation and the auditory stimulation.

[0156] The second difference waveform determination module is configured to determine a second difference waveform according to the difference between the potential waveform corresponding to the gesture audiovisual stimulation and the unimodal potential waveform.

[0157] Correspondingly, the analysis module 43 is specifically configured to determine the analysis result according to the first difference waveform, the second difference waveform, and the reaction times.

[0158] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0159] Example 3:

[0160] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. Figure 5 As shown, the electronic device 5 of this embodiment includes: at least one processor 50 ( Figure 5 Only one processor is shown in the figure), a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, wherein the processor 50 implements the steps of any of the above-mentioned method embodiments when executing the computer program 52.

[0161] The electronic device 5 can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device can include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will understand that Figure 5This is merely an example of the electronic device 5 and does not constitute a limitation on the electronic device 5 . The electronic device 5 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0162] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0163] In some embodiments, the memory 51 may be an internal storage unit of the electronic device 5, such as a hard disk or memory of the electronic device 5. In other embodiments, the memory 51 may also be an external storage device of the electronic device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. Furthermore, the memory 51 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 51 is used to store an operating system, application programs, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 51 may also be used to temporarily store data that has been output or is about to be output.

[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0165] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0166] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0167] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps of the above-mentioned method embodiments when executing the computer program product.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0169] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0170] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0171] In the embodiments provided in this application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0172] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0173] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An audio-visual integrated analysis method, characterized in that: include: Acquiring data to be analyzed, the data to be analyzed comprising data of word recognition performed by the test person in different stimulation environments, the stimulation environments comprising at least one gesture audiovisual stimulation and a reference audiovisual stimulation, the reference audiovisual stimulation comprising one or more audiovisual stimulations other than the gesture audiovisual stimulation; Determine the reaction time corresponding to each of the stimulation environments according to the data to be analyzed, wherein the reaction time reflects the time it takes for the tested person to complete word recognition in the stimulation environment; An analysis result is determined based on each of the reaction times, and the analysis result reflects the degree of influence of the gesture-based audiovisual stimulation on the audiovisual integration ability of the tested person.

2. The audio-visual integrated analysis method according to claim 1, characterized in that: The reference audiovisual stimulation includes visual stimulation, auditory stimulation and multimodal audiovisual stimulation, and the analysis result is determined based on each of the reaction times, including: Determining the unimodal probability distribution functions corresponding to the visual stimulus and the auditory stimulus respectively according to the reaction duration corresponding to the reference audiovisual stimulus; Processing each of the unimodal probability distribution functions based on a competition model to obtain a predicted cumulative distribution function; Determining an integration time window corresponding to the multimodal audiovisual stimulation according to a difference between the predicted cumulative distribution function and a first cumulative distribution function, wherein the first cumulative distribution function is determined according to the reaction duration corresponding to the multimodal audiovisual stimulation, and the integration time window is a time window in which an audiovisual integration effect exists; determining the integration time window corresponding to the gesture audiovisual stimulation according to a difference between the predicted cumulative distribution function and a second cumulative distribution function, wherein the second cumulative distribution function is determined according to the reaction time corresponding to the gesture audiovisual stimulation; The analysis results are determined according to each of the integration time windows.

3. The audio-visual integrated analysis method according to claim 2, characterized in that: The determining the analysis result according to each of the integrated time windows includes: Based on the type of the tested persons and the different stimulation environments, a binary mixed analysis of variance is performed on the duration corresponding to each integrated time window, the peak value corresponding to the integrated time window, and the area under the curve corresponding to the integrated time window to obtain the analysis result. The area under the curve is determined according to the integrated time window and the corresponding cumulative distribution function. The types of the tested persons include the elderly and non-elderly.

4. The audio-visual integrated analysis method according to claim 1, characterized in that: The reference audiovisual stimulus includes at least one of a single-modal audiovisual stimulus and a multi-modal audiovisual stimulus, and the determining of the analysis result based on each of the reaction times includes: Determining a first reaction enhancement degree corresponding to the multimodal audiovisual stimulation according to the reaction duration corresponding to the multimodal audiovisual stimulation and a target reaction duration, wherein the target reaction duration is the reaction duration with the smallest median value of the reaction durations corresponding to the unimodal audiovisual stimulation; Determining a second reaction enhancement degree corresponding to the gesture audiovisual stimulation according to the reaction duration corresponding to the gesture audiovisual stimulation and the target reaction duration; The analysis result is determined according to the first reaction enhancement degree and the second reaction enhancement degree.

5. The audio-visual integrated analysis method according to claim 4, characterized in that: Before determining the analysis result according to the first reaction enhancement degree and the second reaction enhancement degree, the method further includes: determining whether the first reaction enhancement degree and the second reaction enhancement degree are credible; Correspondingly, determining the analysis result according to the first reaction enhancement degree and the second reaction enhancement degree includes: In the case where it is determined that the first reaction enhancement degree and the second reaction enhancement degree are credible, the analysis result is determined according to the first reaction enhancement degree and the second reaction enhancement degree.

6. The audio-visual integrated analysis method according to claim 1, characterized in that: Before determining the analysis result based on each of the reaction times, the method further includes: Determining the word recognition ability corresponding to each of the stimulation environments, wherein the word recognition ability reflects the ability of the tested person to correctly recognize words in the stimulation environment; Correspondingly, determining the analysis result based on each of the reaction times includes: Based on the type of the tested person and the different stimulation environments, a binary mixed variance analysis is performed on the word discrimination ability and the reaction time to obtain the analysis result.

7. The audio-visual integrated analysis method according to any one of claims 1 to 6, characterized in that: The data to be analyzed also includes EEG data of the tested person performing word recognition under different stimulation environments, the reference audiovisual stimulation includes visual stimulation, auditory stimulation and multimodal audiovisual stimulation, and before determining the analysis result based on each reaction time, it also includes: Determine the potential waveform corresponding to each of the stimulation environments according to the data to be analyzed, wherein the potential waveform reflects the event-related potential change of the tested person in the process of word discrimination in the stimulation environment; Determine a first difference waveform according to a difference between the potential waveform corresponding to the multimodal audiovisual stimulation and a unimodal potential waveform, wherein the unimodal potential waveform includes the potential waveforms corresponding to the visual stimulation and the auditory stimulation; determining a second difference waveform according to a difference between the potential waveform corresponding to the gesture audiovisual stimulation and the unimodal potential waveform; Correspondingly, determining the analysis result based on each of the reaction times includes: The analysis result is determined according to the first difference waveform, the second difference waveform and the respective reaction times.

8. An audio-visual integrated analysis device, characterized in that: include: A module for acquiring data to be analyzed, used for acquiring data to be analyzed, wherein the data to be analyzed includes data of word recognition performed by the test person under different stimulation environments, wherein the stimulation environment includes at least one gesture audiovisual stimulation and a reference audiovisual stimulation, wherein the reference audiovisual stimulation is a stimulation other than the gesture audiovisual stimulation, and the reference audiovisual stimulation includes one or more audiovisual stimulations other than the gesture audiovisual stimulation; A reaction time determination module, used to determine the reaction time corresponding to each of the stimulation environments according to the data to be analyzed, wherein the reaction time reflects the time it takes for the tested person to complete word recognition in the stimulation environment; The analysis module is used to determine an analysis result based on each of the reaction times, wherein the analysis result reflects the influence of the gesture-based audiovisual stimulation on the audiovisual integration ability of the tested person.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images