Parkinson's disease auxiliary diagnosis method and system based on multi-stage interactive audio-visual fusion

Abstract

The invention provides a Parkinson's disease auxiliary diagnosis method and system based on multi-stage interactive audio-visual fusion, and belongs to the technical field of auxiliary diagnos.The Parkinson's disease auxiliary diagnosis method comprises the steps that image data and audio data of a patient to be diagnosed are obtained, and the image data comprise a dynamic optical flow image and a static apparent image; a multi-scale dynamic and static feature extraction module is adopted to extract dynamic features and static features of the dynamic optical flow image and the static apparent image; inputting the dynamic features and the static features of each scale into a dynamic and static feature fusion module for fusion to obtain video modal features of each scale; a time-frequency attention speech feature extraction module is adopted to extract speech modal features from the audio data; fusing the video modal features of each scale and the voice modal features of the corresponding scale by using a global and local attention fusion module to obtain audio-visual fusion features; and inputting the audio-visual fusion features into a full-connection layer classification network, and outputting a classification result to assist diagnosis.

Description

Technical Field

[0001] The present invention belongs to the field of auxiliary diagnosis technology, and in particular relates to a Parkinson's disease auxiliary diagnosis method and system based on multi-stage interactive audio-visual fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Parkinson's disease (PD) is a common neurodegenerative disease. Early symptoms such as facial stiffness, bradykinesia, and speech difficulties are often subtle and difficult to detect, making early diagnosis challenging. Early diagnosis is crucial for delaying disease progression and improving treatment outcomes, but current clinical methods rely on patient history and neuroimaging, which are insufficient for a definitive diagnosis. Research based on non-invasive intelligent assisted diagnostic technology can reduce diagnostic costs while improving doctors' diagnostic efficiency, providing a more convenient solution for early screening. However, existing research still has some limitations, including the difficulty of accurately characterizing the pathological characteristics of Parkinson's disease with single modality information and the insufficient utilization of feature information by deep learning methods.

[0004] Existing technologies use artificial intelligence to analyze facial and voice features, and multimodal data fusion can provide a comprehensive understanding of the pathological characteristics of Parkinson's disease. This method has the advantages of being non-invasive, low-cost, and having high diagnostic accuracy, making it suitable for early screening and large-scale population studies. In addition, it can enable timely intervention, improve quality of life, and lay the foundation for personalized treatment. However, existing audio-visual fusion methods still have some limitations. First, many current studies rely on manually extracted features such as acoustic parameters and facial geometry, and fluctuations in data quality can easily lead to information loss. In addition, the heterogeneity of speech and facial expressions increases the difficulty of feature alignment and model optimization, and sample size and data annotation limitations also affect the generalization performance of the model. Summary of the Invention

[0005] To overcome the shortcomings of the above-mentioned prior art, the present invention proposes a Parkinson's disease auxiliary diagnosis method and system based on multi-stage interactive audio-visual fusion. First, a dual-stream network architecture for audio-visual feature extraction is constructed to extract dynamic, static and temporal audio features of Parkinson's patients from optical flow, image sequence and speech data respectively; secondly, a dynamic-static feature fusion module and a global-local attention audio-visual feature fusion module are proposed to achieve full interaction and fusion between modalities; in addition, a multi-stage progressive interactive audio-visual fusion strategy is introduced to achieve deep interaction within and between modalities at different semantic levels, generating audio-visual fusion features with clear structure and complementary semantics to achieve the final Parkinson's disease classification task.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] In a first aspect, a method for assisting diagnosis of Parkinson's disease based on multi-stage interactive audio-visual fusion is disclosed, comprising:

[0008] Acquiring image data and audio data of a patient to be diagnosed, wherein the image data includes a dynamic optical flow image and a static appearance image;

[0009] A multi-scale dynamic and static feature extraction module is used to extract dynamic features and static features from the dynamic optical flow image and the static appearance image; the dynamic features and static features of each scale are input into a dynamic and static feature fusion module for fusion to obtain video modality features of each scale;

[0010] Extracting speech modal features from the audio data using a time-frequency attention speech feature extraction module;

[0011] The video modality features of each scale are fused with the speech modality features of the corresponding scale using a global and local attention fusion module to obtain audio-visual fusion features;

[0012] The audio-visual fusion features are input into a fully connected layer classification network, and a classification result is output to assist diagnosis.

[0013] In the second aspect, a Parkinson's disease auxiliary diagnosis system based on multi-stage interactive audio-visual fusion is disclosed, including:

[0014] A data acquisition module is configured to: acquire image data and audio data of a patient to be diagnosed, wherein the image data includes a dynamic optical flow image and a static appearance image;

[0015] An image feature extraction module is configured to: extract dynamic features and static features from the dynamic optical flow image and the static appearance image using a multi-scale dynamic and static feature extraction module; input the dynamic features and static features of each scale into a dynamic and static feature fusion module to fuse them to obtain video modality features of each scale;

[0016] A speech modality feature extraction module is configured to: extract speech modality features from the audio data using a time-frequency attention speech feature extraction module;

[0017] A feature fusion module is configured to: fuse the video modality features of each scale with the speech modality features of the corresponding scale using a global and local attention fusion module to obtain an audio-visual fusion feature;

[0018] The auxiliary diagnosis module is configured to: input the audio-visual fusion features into a fully connected layer classification network and output a classification result to assist diagnosis.

[0019] In a third aspect, an electronic device is disclosed, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps of the above-mentioned Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion are completed.

[0020] In a fourth aspect, a computer-readable storage medium is disclosed for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion are completed.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention optimizes the extraction and fusion strategy of multimodal features. Through the synergistic effect of the multi-scale spatiotemporal feature extraction module and the global-local attention fusion module, it more effectively captures the complementary information between different modalities and improves the ability to recognize micro-expressions and speech abnormalities of Parkinson's patients.

[0023] The present invention adopts a multi-stage interactive recursive fusion strategy to further enhance the interactive modeling capability of multimodal features and overcome the limitations of existing technologies in feature fusion depth; it introduces a dynamic-static feature interaction mechanism and a global-local attention audio-visual feature fusion module, and through a progressive step-by-step interactive audio-visual fusion strategy, completes deep interaction within and between modalities at different semantic levels, generates audio-visual fusion features with clear structure and semantic complementarity, and improves the interactive modeling capability between modalities.

[0024] The present invention builds a dual-pathway feature representation framework for speech and visual modalities, and designs a multi-scale dynamic-static feature extraction module to simultaneously process facial image sequences and optical flow information, capturing the multi-level pathological characteristics of Parkinson's patients in static expressions and dynamic movements.

[0025] The time-frequency attention speech feature extraction module integrates the channel-spatial attention mechanism and the temporal modeling module to mine key change patterns in speech signals.

[0026] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0028] Figure 1 This is an overall block diagram of the Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion described in Example 1 of the present invention.

[0029] Figure 2 Schematic diagram of the multi-scale dynamic and static feature extraction module described in Example 1 of the present invention.

[0030] Figure 3 Schematic diagram of the dynamic and static feature fusion module described in Example 1 of the present invention.

[0031] Figure 4 This is a structural diagram of the time-frequency attention speech feature extraction module described in Example 1 of the present invention.

[0032] Figure 5 Schematic diagram of the global and local attention fusion module described in Example 1 of the present invention. DETAILED DESCRIPTION

[0033] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0034] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0035] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0036] Example 1

[0037] In one or more embodiments, a method for assisting diagnosis of Parkinson's disease based on multi-stage interactive audio-visual fusion is disclosed, such as Figure 1 As shown, the following steps are included:

[0038] Step S1: Acquire image data and audio data of a patient to be diagnosed, where the image data includes a dynamic optical flow image and a static appearance image.

[0039] In this embodiment, the dynamic optical flow image is obtained by calculating the Farneback dense optical flow algorithm. This method assumes that the grayscale of adjacent video frames is invariant and establishes a second-order polynomial motion model. The grayscale value of a pixel in the image is represented as I(x, y, t) and satisfies the following equation:

[0040] I(x+u1,y+u2,t+1)=I(x,y,t)

[0041] Where u1 and u2 represent the horizontal and vertical motion components of the pixel, respectively. After Taylor expansion of the above formula and ignoring high-order terms, the optical flow constraint equation can be obtained as follows:

[0042] I x u1+I y u2+I t =0

[0043] Among them, I x and I y Represent the spatial gradient of the pixel point, I t is the temporal gradient. This equation describes the constraint relationship between the pixel motion components u1 and u2.

[0044] To address the underdetermined nature of the equation, the Farneback algorithm estimates the optical flow vector field layer by layer in a multi-scale pyramid structure by fitting a quadratic polynomial to adjacent frames, effectively capturing pixel motion information. The optical flow vector field is calculated using the following formula:

[0045] E(u1,u2)=∫ω(x,y)[I t (x+u1,y+u2)-I t-1 (x,y)] 2 dxdy

[0046] Where ω(x,y) is a Gaussian weighting function used to emphasize the contribution of local regions.

[0047] The optical flow vector in the Cartesian coordinate system is converted to the polar coordinate system to enhance the physical interpretability of the motion features, and the magnitude and angle of the optical flow are obtained as shown in the following formula:

[0048]

[0049] The optical flow information is further mapped into the HSV color space to visualize the motion pattern and obtain a dynamic optical flow image. The angle of the optical flow is used as the hue channel H in the HSV color space, the amplitude is normalized as the brightness channel V, and the saturation channel S is fixed to 255. The above mapping process is as follows:

[0050] S=255,V=Normalize(magnitude,0,255)

[0051] The static apparent image is the original video frame sequence.

[0052] The preprocessing of video data includes data segmentation, key frame selection, face detection and alignment, and data formatting. First, the original video is segmented into fixed lengths, and the length of each segment is set to 3s. During the segmentation process, segments with insufficient lighting, jittery images, or complex backgrounds are eliminated to reduce the impact of low-quality data on model performance. For multiple data segments of the same subject, this method can fully reflect their facial dynamic features and voice changes in the experimental task. After the data segmentation is completed, in order to reduce the computational overhead and extract representative feature information, 30 frames are extracted from each video as key frames. The selection of key frames follows a uniform distribution strategy, that is, the index position of each frame in the total number of frames is calculated so that the selected key frames can cover the information of the entire segment.

[0053] During the video data formatting phase, all video frames were adjusted to 60 frames per second to unify timing information. Subsequently, a multi-task convolutional neural network (MTCNN) was used to detect and align facial regions in each frame. The MTCNN consists of a three-stage cascaded network (P-Net, R-Net, and O-Net). The P-Net first screens candidate regions, the R-Net refines the detection results, and finally the O-Net accurately locates five key points (eyes, nose tip, and mouth corners) for high-precision face alignment. This approach effectively reduces feature drift caused by head pose changes, shooting angle deviations, or complex backgrounds, while highlighting key facial features and improving the stability and accuracy of subsequent expression analysis. To adapt to the input requirements of the deep learning model, the detected facial regions were cropped and resized to 224×224 pixels and stored in a unified format to further ensure data standardization and consistency.

[0054] Voice data is extracted directly from the video and fully synchronized with the video clips on the timeline. The extracted voice data is sampled at a uniform 44.1kHz rate, and adaptive filtering technology is used to reduce background noise, eliminating environmental interference and improving the signal-to-noise ratio. This preserves voice characteristics while suppressing noise interference, ensuring that the processed voice data truly reflects the subject's voice characteristics.

[0055] This embodiment uses a logarithmic Mel spectrum conversion method to convert the original waveform signal into a feature representation that can reflect the time-frequency distribution. The method mainly includes the steps of signal normalization, short-time Fourier transform, Mel frequency mapping, and logarithmic compression. The original speech signal is stored in a 16-bit encoding format with an amplitude range of [-32768, +32767]. To simplify the calculation and ensure numerical stability, the signal amplitude is normalized to the interval [-1, 1], as shown below:

[0056]

[0057] Among them, x raw is the original speech sample, x norm is the normalized signal. In addition, the voice data is converted into a mono signal by using the channel-by-channel averaging method to reduce the interference of redundant information on subsequent feature extraction.

[0058] The normalized signal is divided into frames of fixed length. Each frame of data is windowed to reduce the spectrum leakage effect. The window function is a periodic Hanning window, which is defined as follows:

[0059]

[0060] Among them, n w Represents the time index in the window function, L w Indicates the length of the window function, w[n w ] represents the window function.

[0061] Perform a short-time Fourier transform on each frame of signal to decompose the time domain signal into the frequency domain signal and calculate the spectrum amplitude matrix. The Fourier transform is as follows:

[0062]

[0063] Among them, k1 represents the frequency index, k2 represents the frame index, L FFT Indicates the number of Fourier transform points, L mov represents frame shift, x norm [·] represents the normalized signal, STFT mag represents the amplitude spectrum after short-time Fourier transform.

[0064] The Fourier transform projects the signal's time domain information into the frequency domain, forming a two-dimensional matrix that jointly represents time and frequency. To further adapt to the human auditory system's frequency perception characteristics, the frequency scale is converted from a linear scale to a nonlinear Mel scale. The conversion relationship between Mel frequency and Hertz frequency is shown below:

[0065]

[0066] Where f is the Hertz frequency, f mel is the Mel frequency.

[0067] The amplitude spectrum obtained by short-time Fourier transform is weighted mapped through 64 Mel filter banks. Mel filter banks are a group of triangular filters that simulate the human auditory perception. Each filter covers a specific frequency range. The filter group H m (k) is defined as follows:

[0068]

[0069] Where k represents the actual frequency corresponding to the current frequency point, f(m) represents the center frequency of the mth Mel filter, and f(m-1) and f(m+1) represent the left and right boundary frequencies of the mth filter, respectively.

[0070] The distribution of spectral energy in different Mel frequency bands is calculated by the following formula to obtain the Mel spectrum.

[0071] Mel spec =STFT mag ·H m (k)

[0072] Finally, the following formula is applied to perform logarithmic compression on the Mel spectrum to enhance the dynamic range and reduce the difference in feature ratios between different frequencies, and the logarithmic Mel spectrum Log-Mel is obtained. spec The audio data is used as input for subsequent models to extract audio features.

[0073]

[0074] Among them, Mel spec represents the weighted energy of each frame on each Mel filter, and ζ represents a small positive offset.

[0075] Step S2: Use a multi-scale dynamic and static feature extraction module to extract dynamic features and static features from the dynamic optical flow image and the static appearance image; input the dynamic features and static features of each scale into the dynamic and static feature fusion module to obtain the video modality features of each scale.

[0076] Step S2-1: Construct a multi-scale dynamic-static dual-stream feature collaborative learning framework, in which the dynamic branch, i.e., the optical flow branch, is used to capture dynamic features such as weakening of motion amplitude and abnormal frequency changes, while the static branch is used to extract static texture and expression features of the face from static image sequences.

[0077] The dynamic optical flow image and the static appearance image are input into the multi-scale dynamic and static feature extraction module respectively. The module architecture consists of three core components: multi-scale convolution layer (PSConv), lightweight convolution layer (MV2), and MobileViT hybrid layer, which are connected in sequence to form four stages, such as Figure 2 As shown in the figure, the first stage includes a multi-scale convolution layer (PSConv) and a lightweight convolution layer (MV2) connected in sequence; the second stage includes a lightweight convolution layer (MV2), a downsampling layer with a stride of 2, and a lightweight convolution layer (MV2) connected in sequence; the third stage includes a lightweight convolution layer (MV2), a downsampling layer with a stride of 2, and a MobileViT hybrid layer connected in sequence; the fourth stage is a 1×1 convolution layer.

[0078] Among them, the PSConv multi-scale convolution layer uses convolution kernels with different expansion rates to perform multi-scale weighted calculations on the input features, generate intermediate feature maps, and integrate multi-scale information through subsequent convolution operations. The calculation process is as follows:

[0079]

[0080] Among them, H c,x,y Represents the eigenvalue of the output feature map at the output channel c and spatial position (x, y); C in Indicates the number of channels of the input feature map; G c,k,i,j represents the weight coefficient of the convolution kernel at position (i, j) corresponding to the c-th output channel and the k-th input channel; F k,x+iD(c,k),y+jD(c,k) represents the pixel value of the kth channel of the input feature map at the expanded position (x+iD(c,k),y+jD(c,k)); D(c,k) represents the dilation rate of the convolution kernel between the cth output channel and the kth input channel, which controls the size of the receptive field; K represents the size of the convolution kernel. By periodically assigning the dilation rate, PSConv can achieve multi-receptive field feature fusion within a single convolution layer, avoiding the complexity of traditional multi-scale network structures.

[0081] The MV2 layer is based on the optimized architecture of depthwise separable convolution, which realizes efficient feature learning by building a synergistic mechanism of channel expansion and compression. , the module first expands the channel dimension to (In this embodiment, ), forming a high-dimensional latent space , to enhance the nonlinear ability of feature dimension expression. Subsequently, 3×3 depthwise separable convolution is used to extract spatial features channel by channel, effectively retaining spatial information. Finally, point-by-point convolution is used to integrate information between different channels, while restoring the channel dimension to the original C, and introducing a linear activation function to construct a lightweight bottleneck structure. This design improves the feature retention rate by suppressing the information truncation problem of the traditional ReLU activation function in the process of feature dimensionality reduction. The MV2 layer improves the feature expression ability while ensuring computational efficiency by simultaneously modeling channel correlation and spatial correlation, laying the foundation for dynamic-static feature collaborative modeling.

[0082] MobileViT combines the local perception of CNNs with the global modeling of Transformers. First, a spatial partitioning operation decomposes the input feature map into multiple non-overlapping local blocks, converting the two-dimensional spatial structure into a sequential representation adapted to the Transformer processing paradigm. During the global modeling phase, a multi-head self-attention mechanism is used to establish long-range dependencies across regions, capturing the spatiotemporal correlations of facial micro-expressions. A feedforward neural network is then used to enhance the nonlinear expressiveness of features. After spatial reconstruction to restore the original dimensionality of the sequential features processed by the Transformer, a residual connection is used with the original convolutional features to achieve a deep fusion of local details and global context.

[0083] In the entire feature extraction module, refined modeling of facial motion is achieved through the parallel extraction and processing of optical flow dynamic features and static appearance features, combined with a hybrid architecture of multi-scale convolution and visual Transformer.

[0084] Feature extraction of optical flow and static images is completed through a shared multi-scale feature extraction architecture, but in the specific processing process, dynamic and static features still retain independence, thereby reducing the impact of mutual interference between modalities on feature representation capabilities.

[0085] Step S2-2: Use the dynamic and static feature fusion module to fuse the dynamic features and static features, such as Figure 3 As shown in the figure, the motion-static features fusion module (MSF) consists of global attention weight generation, cross-modal mapping, and fused convolution operations. Through cross-modal mapping and feature weighting, it achieves deep interaction between optical flow features and static image sequence features, improving the model's ability to perceive the facial features of Parkinson's disease.

[0086] This module first uses global average pooling to transform the input dynamic features F m With the static feature F s Global feature aggregation is performed so that the features of each channel can better represent the overall motion amplitude and static information. Subsequently, a bidirectional attention weight generator is constructed through a cascade of 1×1 convolutional layers and Sigmoid activation functions, as shown below:

[0087] α m =σ(Conv[V m ; V s ])

[0088] α s =σ(Conv[V s ; V m ])

[0089] Among them, [·;·] represents the channel splicing operation, V m 、V s represents the dynamic and static feature vectors with global representation capabilities, σ(·) represents the Sigmoid activation function, α m and α s Represent the weights corresponding to dynamic and static features respectively.

[0090] This mechanism enables static features to focus on areas sensitive to facial expression changes under the guidance of motion features, while optical flow features focus on key facial motion areas under the constraints of static features. The feature recalibration process is as follows:

[0091]

[0092] in, represents the channel weighted operation, F m ′、F s ′ represents the weighted dynamic and static features of optical flow. Through this soft attention mechanism, the model can adaptively enhance key features and suppress irrelevant noise.

[0093] In order to further deepen the cross-modal interaction, the model introduces a cross-mapping correction mechanism and uses the global statistics μ of motion features m The static features are recalibrated in the spatial domain, and the distribution parameter γ of the static features is s Dynamic range constraints on motion features:

[0094] F″ s =LN(F′ s )×μ m +β m

[0095] F″ m =LN(F′ m )×γ s +η s

[0096] Among them, LN represents the layer normalization operation, β m and η s represents the learnable bias term, F″ s 、F″ m Represents the corrected dynamic and static features of optical flow.

[0097] The bidirectional feature correction mechanism can effectively establish a complementary dependency between dynamic and static features, ensuring a coordinated distribution of features between modalities. To further enhance the effectiveness of fused features, the MSF module employs a fused convolution operation to perform channel-wise splicing of the corrected optical flow features, static features, and fused features from the previous stage. This fusion process uses a 3×3 convolution operation to enable local interaction between optical flow and static features in the spatial domain. Combined with the nonlinear ReLU activation function, this enhances feature expression capabilities. Furthermore, to ensure the stability of fused features across different batches of data, the MSF module introduces batch normalization after the fused convolution layer to suppress vanishing and exploding gradients, ensuring the stability of the training process.

[0098] Step S3: Use the time-frequency attention speech feature extraction module to extract speech modal features from the audio data.

[0099] The architecture of the time-frequency attention speech feature extraction module is as follows: Figure 4 As shown, it includes the first-stage, second-stage, third-stage, and fourth-stage speech feature extraction modules connected in sequence. The first-stage speech feature extraction module is a 2-dimensional convolutional layer and an LSTM layer connected in sequence; the second-stage speech feature extraction module is an attention mechanism layer, the third-stage speech feature extraction module is a ResNet layer, and the fourth-stage speech feature extraction module is an attention mechanism layer.

[0100] First, a 2D-LSTM architecture is introduced, capturing the time-frequency features of the speech spectrum through the collaborative work of a 2D-CNN and LSTM. Subsequently, a dual channel-spatial attention mechanism is combined to dynamically focus on pathological features. Finally, a residual network is used to progressively extract multi-level semantic features. The entire feature extraction process is divided into four stages: primary feature extraction, a residual network layer, and two channel-spatial attention feature enhancement layers.

[0101] The two-dimensional feature maps generated by the convolutional layers exhibit sparse distribution in the feature space and lack sufficient contextual relevance in the temporal dimension. Therefore, the LSTM module is introduced after the two-dimensional convolutional layer in the initial stage. Its internal gating mechanism effectively captures long-term dependencies in the time series, compensating for the convolution operation's limited ability to model global temporal context and extracting more discriminative temporal features.

[0102] These features are then fed into the second-stage channel-spatial attention layer. By introducing an attention mechanism, the weight of local key features in the temporal representation is further emphasized, improving the model's classification performance. The channel-spatial attention layer is deployed in the second and fourth stages of feature extraction to perform weighted optimization on both primary and deep features, enhancing the model's expressive power.

[0103] First, global feature aggregation is performed on each channel through global average pooling to extract the background information of the channel. Subsequently, the channel compression convolution layer is used to compress the channel dimension, thereby reducing redundant dimensions and highlighting key features. In order to enhance the model's ability to model sparse features and accelerate the convergence of the network, the LeakyReLU function is introduced into the activation function of the compression convolution layer for nonlinear transformation. Subsequently, the channel dimension is restored using the channel expansion convolution layer to obtain the channel attention weight based on average pooling. In addition, the maximum pooling operation is introduced, and the channel attention weight based on the maximum pooling is generated through the same channel compression and expansion process. Finally, and The final channel attention feature is formed by element-by-element addition. The calculation process of channel attention is as follows:

[0104]

[0105] The spatial attention module focuses on capturing the significant areas of the feature map in the spatial dimension. Different from channel attention, spatial attention pays more attention to the local features and positional relationships of the feature map. When calculating spatial attention, we first perform a spatial attention on the input feature map F. t+1 Global average pooling and maximum pooling are performed to generate two two-dimensional spatial feature maps. These two feature maps are then concatenated in the channel dimension to form a fused feature. Finally, a two-dimensional convolution operation is performed on the fused feature to generate the final spatial attention weights. The process is as follows:

[0106]

[0107] in, In order to enhance the nonlinear learning ability of the model, the Sigmoid activation function is introduced in the output layer of the spatial attention mechanism.

[0108] During the primary feature extraction phase, the 2D-LSTM module performs preliminary temporal and spatial feature modeling on the input speech signal. By extracting shallow features, it captures the signal's local information and temporal dependencies. However, the expressive power of shallow features is still limited in terms of semantic hierarchy and global information modeling. This makes it difficult to fully reveal the complex characteristic patterns in Parkinson's disease patient speech signals, especially when it comes to extracting multi-layer semantic information and capturing contextual associations.

[0109] Based on this, the third stage of the time-frequency attention speech feature extraction module introduces a ResNet-based residual network structure to extract deep semantic features. The residual network effectively alleviates the gradient vanishing problem in deep networks through deep structural design and skip connection mechanism, and realizes layer-by-layer optimization and expression enhancement of speech pathology features. In terms of structural design, the residual network can not only retain the detailed information of low-level features, but also extract more abstract high-level semantic features through hierarchical stacking. The skip connection mechanism allows input features to be directly passed to subsequent layers in the network and fused with deep features to achieve cross-layer reuse of information, further improving the expression efficiency and discrimination ability of high-dimensional features.

[0110] Step S4: The video modality features of each scale are fused with the speech modality features of the corresponding scale using the global and local attention fusion module to obtain the audio-visual fusion features.

[0111] Global and local attention fusion module, such as Figure 5 As shown in the figure, it is mainly composed of two symmetrical multi-scale channel attention mechanisms (Multi-Scale Channel Attention Module, MS-CAM), which are used to improve the interaction ability between speech and visual features, construct audio-visual fusion features, and optimize the model's ability to capture the characteristic patterns of Parkinson's patients.

[0112] Two symmetrical multi-scale channel attention mechanisms act on the speech modality features X a and video modality features X v .

[0113] The global and local attention fusion module (GLAF module) is as follows Figure 5 As shown, the dotted line part is the pathway when the video modality is the dominant feature.

[0114] Taking speech modality features as the dominant input features as an example, first, X a Input the MS-CAM module, which uses global average pooling between channels to extract the global feature distribution and obtain the global representation G in the channel dimension. a , the calculation process is as follows:

[0115] G a =GlobalAvgPool(X a )

[0116] Global pooling effectively captures the statistical information of the two modal features in the channel dimension, allowing the model to focus more on global cross-modal features during the fusion process. However, relying solely on global features may overlook local details. Therefore, a local attention mechanism is further introduced in the MS-CAM module to compensate for the lack of global information. Local attention compresses, nonlinearly transforms, and restores the scale of speech features through point-by-point convolution and nonlinear activation functions in the following formula, achieving weighted adjustment of the input features:

[0117]

[0118] Among them, W1, W2, W3, and W4 are the trainable parameters of channel compression and expansion respectively, and δ(·) represents the ReLU activation function. represents the element-by-element addition operation, and λ represents the local attention weight, which is used to measure the contribution ratio of speech features.

[0119] The function of this module is to strengthen the local response of the feature space, amplify the key features, and thus improve the distinguishability of the features.

[0120] After obtaining global features through global pooling operations and then calculating local attention weights by combining global information with input features, the GLAF module uses a channel-by-channel weighted fusion strategy to fuse the audio and video features that have undergone the multi-scale channel attention mechanism, as shown below:

[0121] X f ′=λX a +(1-λ)X v

[0122] like Figure 5 As shown, a two-step fusion strategy is adopted to obtain the initial fusion feature X f ′, then X f ′ is used as the input of the MS-CAM module, and the above operation is repeated to obtain the fusion feature with the speech modality as the dominant feature.

[0123] After the two sets of modal features are input into the model, they are calculated as the dominant features respectively. The same is true when the video modality is the dominant feature, and the fusion feature is obtained. After obtaining the fusion features of the two modalities as the dominant ones, the element-by-element addition operation is used to obtain the final audio-visual fusion features. Since Parkinson's disease patients have large individual differences in speech and facial expressions, the adaptive weight mechanism can automatically adjust the fusion ratio of audio-visual features according to the feature distribution of different samples, thereby improving the generalization ability of the model.

[0124] In addition, a multi-stage interactive progressive feature fusion strategy is introduced in the GLAF module, that is, the audio-visual features are gradually interacted in each feature extraction stage, rather than fusing all features at once. As the input of the current stage, it is combined with the audio-visual fusion features extracted in the current stage Perform weighted optimization as follows:

[0125]

[0126] Here, θ is the fusion weight, which is automatically learned during training. This multi-stage interactive progressive fusion mechanism helps gradually enhance the interaction of audiovisual features at shallow, mid, and deep layers, avoiding information loss that can result from a one-time fusion, and effectively improving the model's classification performance.

[0127] Step S5: Input the audio-visual fusion features into the fully connected layer classification network and output the classification results to assist diagnosis.

[0128] The audio-visual fusion features of the final stage are input into a fully connected layer classification network consisting of two fully connected layers to output the diseased or healthy category.

[0129] This embodiment achieves full interaction and fusion between modalities through a dynamic-static feature fusion module and a global-local attention audio-visual feature fusion module. Furthermore, a multi-stage progressive interactive audio-visual fusion strategy is introduced to achieve deep interaction within and between modalities at different semantic levels, generating audio-visual fusion features with clear structure and complementary semantics to achieve the final Parkinson's disease classification task.

[0130] Example 2

[0131] In one or more embodiments, a Parkinson's disease auxiliary diagnosis system based on multi-stage interactive audio-visual fusion is disclosed, specifically including:

[0132] A data acquisition module is configured to: acquire image data and audio data of a patient to be diagnosed, wherein the image data includes a dynamic optical flow image and a static appearance image;

[0133] An image feature extraction module is configured to: extract dynamic features and static features from the dynamic optical flow image and the static appearance image using a multi-scale dynamic and static feature extraction module; input the dynamic features and static features of each scale into a dynamic and static feature fusion module to fuse them to obtain video modality features of each scale;

[0134] A speech modality feature extraction module is configured to: extract speech modality features from the audio data using a time-frequency attention speech feature extraction module;

[0135] A feature fusion module is configured to: fuse the video modality features of each scale with the speech modality features of the corresponding scale using a global and local attention fusion module to obtain an audio-visual fusion feature;

[0136] The auxiliary diagnosis module is configured to: input the audio-visual fusion features into a fully connected layer classification network and output a classification result to assist diagnosis.

[0137] Example 3

[0138] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion are completed.

[0139] Example 4

[0140] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned Parkinson's disease auxiliary diagnosis method based on multi-stage interaction and audio-visual fusion are completed.

[0141] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0144] The descriptions of the various embodiments in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0145] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion, characterized by: include: Acquiring image data and audio data of a patient to be diagnosed, wherein the image data includes a dynamic optical flow image and a static appearance image; A multi-scale dynamic and static feature extraction module is used to extract dynamic features and static features from the dynamic optical flow image and the static appearance image; the dynamic features and static features of each scale are input into a dynamic and static feature fusion module for fusion to obtain video modality features of each scale; Extracting speech modal features from the audio data using a time-frequency attention speech feature extraction module; The video modality features of each scale are fused with the speech modality features of the corresponding scale using a global and local attention fusion module to obtain audio-visual fusion features; The audio-visual fusion features are input into a fully connected layer classification network, and a classification result is output to assist diagnosis.

2. The Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion according to claim 1 is characterized in that: The multi-scale dynamic and static feature extraction module includes four stages connected in sequence. The first stage includes a multi-scale convolution layer and a lightweight convolution layer connected in sequence. The second stage includes a lightweight convolution layer, a downsampling layer, and a lightweight convolution layer connected in sequence. The third stage includes a lightweight convolution layer, a downsampling layer, and a MobileViT hybrid layer connected in sequence. The fourth stage is a 1×1 convolution layer. The multi-scale convolution layer uses convolution kernels with different expansion rates to perform multi-scale weighted calculations on the input features, generates an intermediate feature map, and integrates the multi-scale information through subsequent convolution operations: Among them, H c,x,y Represents the eigenvalue of the output feature map at the output channel c and spatial position (x, y); C in Indicates the number of channels of the input feature map; G c,k,i,j represents the weight coefficient of the convolution kernel at position (i, j) corresponding to the c-th output channel and the k-th input channel; F k,x+iD(c,k),y+jD(c,k) represents the pixel value of the k-th channel of the input feature map at the expanded position (x+iD(c,k),y+jD(c,k)); D(c,k) represents the expansion rate of the convolution kernel between the c-th output channel and the k-th input channel; The lightweight convolution layer expands the channel dimension through 1×1 convolution, and then uses 3×3 depth-wise separable convolution to extract spatial features channel by channel. The spatial features integrate information between different channels through point-by-point convolution, while restoring the channel dimension to the original C, and introducing a linear activation function to construct a lightweight bottleneck structure.

3. The Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion according to claim 1 is characterized in that: The dynamic and static feature fusion module consists of global attention weight generation, cross-modal mapping and fusion convolution operations; The global attention weight generation adopts global average pooling to aggregate the dynamic features and static features of the input globally, and generates bidirectional attention weights through cascaded 1×1 convolutional layers and Sigmoid activation functions; The cross-modal mapping uses the global statistics of motion features to recalibrate the static features in the spatial domain after weighting the dynamic and static features. At the same time, the distribution parameters of the static features constrain the dynamic range of the motion features: F s ″=LN(F s ′)×μ m +b m F m ″=LN(F m ′)×γ s +n s Among them, μ m is the global statistics of motion features, γ s is the distribution parameter of the static feature; LN represents the layer normalization operation, β m and η s represents the learnable bias term, F s ″、F m ″ represents the corrected dynamic and static features of optical flow, F m ′、F s ′ represents the weighted dynamic and static features of optical flow; The fused convolution operation locally fuses dynamic features with static features through convolution layers and activation function layers.

4. The Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion according to claim 1, characterized in that: The time-frequency attention speech feature extraction module includes the first-stage, second-stage, third-stage, and fourth-stage speech feature extraction modules connected in sequence. The first-stage speech feature extraction module is a 2-convolutional layer and an LSTM layer connected in sequence; the second-stage speech feature extraction module is an attention mechanism layer, the third-stage speech feature extraction module is a ResNet layer, and the fourth-stage speech feature extraction module is an attention mechanism layer.

5. The Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion according to claim 1 is characterized in that: The global and local attention fusion module includes two symmetrical multi-scale channel attention mechanisms, which act on speech modality features and video modality features respectively. First, with speech modality features as the dominant feature, a channel-by-channel weighted fusion strategy is adopted to fuse the speech modality features and video modality features after the first multi-scale channel attention mechanism module to obtain the preliminary fusion features: X f ′=λX a +(1-λ)X v Where, X f ′ is the initial fusion feature; X a is the speech modality feature; X v is the video modality feature; λ represents the local attention weight; Then, the preliminary fusion features are input into the second multi-scale channel attention mechanism module to obtain the speech-dominant fusion features; the above calculation is also performed with the video modality features as the dominant features to obtain the video-dominant fusion features, and the element-by-element addition operation is used to obtain the audio-visual fusion features.

6. The Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion according to claim 5 is characterized in that: The multi-scale channel attention mechanism uses global average pooling between channels to extract the global feature distribution of its input features and obtain a global representation in the channel dimension. Based on the global representation, the local attention of point-by-point convolution and nonlinear activation function is used to weight the input features: λ=σ(W3δ(W1X a )⊕W4δ(W2G a )) Among them, W1, W2, W3, and W4 are trainable parameters for channel compression and expansion, respectively, δ(·) represents the ReLU activation function, ⊕ represents the element-wise addition operation, and λ represents the local attention weight.

7. The Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion according to claim 5 is characterized in that: The global and local attention fusion module adopts a multi-stage interactive progressive feature fusion strategy, which gradually interacts with the audio-visual features in each feature extraction stage. Specifically, the features fused in the previous stage are used as the input of the current stage and weighted optimized with the audio-visual fusion features extracted in the current stage, as shown below: Among them, θ is the fusion weight; It is the feature obtained by fusion in the previous stage; It is the audio-visual fusion feature extracted in the current stage.

8. A Parkinson's disease auxiliary diagnosis system based on multi-stage interactive audio-visual fusion, characterized by: include: A data acquisition module is configured to: acquire image data and audio data of a patient to be diagnosed, wherein the image data includes a dynamic optical flow image and a static appearance image; An image feature extraction module is configured to: extract dynamic features and static features from the dynamic optical flow image and the static appearance image using a multi-scale dynamic and static feature extraction module; input the dynamic features and static features of each scale into a dynamic and static feature fusion module to fuse them to obtain video modality features of each scale; A speech modality feature extraction module is configured to: extract speech modality features from the audio data using a time-frequency attention speech feature extraction module; A feature fusion module is configured to: fuse the video modality features of each scale with the speech modality features of the corresponding scale using a global and local attention fusion module to obtain an audio-visual fusion feature; The auxiliary diagnosis module is configured to: input the audio-visual fusion features into a fully connected layer classification network and output a classification result to assist diagnosis.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the auxiliary diagnosis method for Parkinson's disease based on multi-stage interactive audio-visual fusion according to any one of claims 1 to 7 is completed.

10. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the Parkinson's disease auxiliary diagnosis method based on multi-stage interactive audio-visual fusion as described in any one of claims 1 to 7.

Images