A Deep Learning Method for Health Detection Based on Multimodal Spectral Rank-Guided Attention

Abstract

This invention discloses a deep learning method for health detection based on multimodal spectral rank guided attention, comprising: collecting user facial images, audio, and tabular data to form multimodal health monitoring data, and feeding them into the corresponding multimodal feature encoding and unified projection module to complete modality-specific feature extraction and dimension alignment; feeding facial image features and audio features into the spectral rank guided attention module, calculating noise perception bias based on local low-rank spectral entropy, and dynamically injecting attention calculation process to obtain enhanced facial image features and enhanced audio features; feeding the enhanced facial image features, enhanced audio features, and tabular features together into the multimodal feature fusion and classification module, performing global pooling and deep fusion, feeding them into a fully connected layer for classification decision, and outputting the user's fatigue detection result. This invention can effectively achieve high-precision and robust multimodal health status detection.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and smart healthcare, and in particular to a deep learning method for health detection based on multimodal spectral rank-guided attention. Background Technology

[0002] Sudden illnesses and accidents caused by abnormal health conditions have always been urgent problems to be solved in the fields of modern public health safety and personal life protection. Intelligent health monitoring systems are a major tool for improving health management efficiency and reducing potential risks. By collecting and analyzing users' multidimensional behavioral characteristics and physiological states, these systems predict abnormal health conditions and assess potential dangers, thereby preventing serious health accidents from occurring.

[0003] With the rapid development of technologies such as artificial intelligence, big data, and the Internet of Things, non-contact, intelligent health monitoring based on sensing technology has gradually become a research hotspot. Sensing technology integrates physiological data from subjects, such as exercise data, physiological indicators, and nutritional information, to comprehensively capture human health status, providing a scientific basis for physical fitness assessment, health analysis, and personalized health management. However, despite significant progress in this field, existing deep learning solutions still face two major technological bottlenecks in practical applications: 1. Difficulty in Deep Fusion of Multimodal Heterogeneous Data: Health detection involves data with vastly different formats. For example, facial images are high-dimensional spatial-temporal data, audio signals are high-dimensional frequency-temporal data, while height, weight, and age are low-dimensional static scalar data. Existing mainstream methods mostly employ simple feature concatenation or late-delay decision weighting strategies, which struggle to effectively align the differences between different modalities at both the temporal scale and semantic level. This shallow fusion approach makes it difficult for models to uncover deep complementary relationships between modalities, limiting further improvements in detection accuracy.

[0004] 2. Severe noise interference in complex environments: In real-world applications, sudden changes in lighting, background noise, and equipment noise inevitably contaminate the original signal. Most existing attention mechanisms focus only on the weighting of features in the temporal or spatial dimensions, lacking robustness to noisy data. This leads to the model mistakenly identifying high-energy noise signals as key features and assigning them high weights when faced with strong noise interference, resulting in focusing bias and a significantly increased false positive rate in detection results.

[0005] To address the above problems, this invention proposes a deep learning method for health detection that can deeply integrate images, audio, and structured heterogeneous data, and utilize spectral rank characteristics to guide an attention mechanism for precise noise reduction. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a deep learning method for health detection based on multimodal spectral rank guided attention, addressing the shortcomings of existing technologies.

[0007] The technical solution adopted by this invention to solve its technical problem is: This invention provides a deep learning method for health detection based on multimodal spectral rank guided attention, the method comprising the following steps: Step 1: Collect users' facial images, audio, and tabular data to form multimodal health monitoring data, and send them to the multimodal feature encoding and unified projection module corresponding to each type of data to complete modality-specific feature extraction and dimension alignment. Step 2: Feed the facial image features and audio features into the spectral rank guided attention module, calculate the noise perception bias based on the local low-rank spectral entropy, and dynamically inject it into the attention calculation process to automatically suppress high-entropy noise regions, enhance low-entropy effective features, and preserve spatial topological relationships to obtain enhanced facial image features and enhanced audio features. Step 3: The enhanced facial image features, enhanced audio features, and table features are fed into the multimodal feature fusion and classification module for global pooling and deep fusion. Then, they are fed into the fully connected layer for classification decision and output the user's fatigue detection results.

[0008] Furthermore, the multimodal feature encoding and unified projection module in step 1 of the present invention includes: For the multi-scale orthogonal feature extraction module of facial image data, a multi-scale orthogonal convolution branch is deployed in parallel. Different sizes of horizontal and vertical convolution kernels are used to extract local and contextual features in the horizontal and vertical directions of the image, respectively, constructing a feature set sensitive to texture direction. Subsequently, the extracted multi-scale and multi-directional features are concatenated along the channel dimension, and a scale-direction attention mechanism is used to perform spatial and channel-adaptive weight calculations to achieve feature selection and weighted fusion. Next, convolution is used to adjust the number of channels and perform non-linear mapping on the weighted fused features. Finally, residual connections are used to add the output of this module to the original input features after appropriate projection to form the final output. For audio data, an audio feature downsampling module based on an adaptive spectrum anti-aliasing mechanism is used. The original audio data is converted into time-frequency domain spectral features by a spectrum analysis module and then fed into an adaptive anti-aliasing filter. This filter dynamically adjusts the convolutional filter weights according to the energy distribution and frequency structure characteristics of the current frequency band, actively suppressing high-frequency aliasing components while spatially downsampling and preserving key harmonics and details. The filtered features enter the feature extractor, which further extracts time-frequency structure features through multi-layer convolution and activation mapping. The features after dimensionality reduction are then fed into the downsampling output module to complete the efficient compression and encoding of audio features. For the residual table data encoding module based on identity mapping initialization, the original high-dimensional table data first enters the main processing path through a linear projection layer. A multi-layer feedforward network, including batch normalization, activation functions, and random deactivation operations, performs nonlinear transformation and dimensional adjustment on the features, outputting transformed features. Simultaneously, the original input table data is dimension-adapted through a residual mapping path. When the input and output dimensions are consistent, the path is an identity mapping; otherwise, an additional linear layer is used for dimension alignment. The weights and biases of the last layer of the main processing path network are initialized to zero to ensure that the initial state satisfies the identity mapping relationship, i.e., the output approaches zero. This module combines the transformed features with the dimension-adapted original features through additive fusion to form the output result of the residual connection.

[0009] Furthermore, the specific implementation method of the multi-scale orthogonal feature extraction module of the present invention is as follows: Let the feature map of the input facial image be... Define the scale set as , direction set Representing the horizontal and vertical directions respectively; firstly, for each scale The module executes two asymmetric convolutional branches in parallel to decouple the directional features of facial texture, thus enabling the lateral branch features to be processed. Calculate according to formula (1): (1) in, This indicates that a core size of [size] is used. Convolutional processing captures horizontal features such as forehead lines and lip lines; vertical branch features... Calculate according to formula (2): (2) in, This indicates that a core size of [size] is used. The convolution captures vertical features; the outputs of all branches are concatenated along the channel dimension to form an anisotropic feature map according to formula (3). : (3) in, This indicates channel concatenation, and then features are processed through... Convolution compresses back to the target number of channels And according to formula (4), residual connections are added to retain the original information to obtain the final output. : (4) in, This represents a mapping function that ensures the input channel equals the output channel.

[0010] Furthermore, the specific implementation method of the audio feature downsampling module based on the adaptive spectrum anti-aliasing mechanism of the present invention is as follows: Let the Mel spectrogram features of the input audio data be... , For the input tensor, the first The sample, the first Channel, frequency index Time Index The core setting of the audio feature downsampling module based on adaptive spectrum anti-aliasing mechanism includes a two-dimensional convolution operation with a stride of 2. The convolution kernel is learned end-to-end through the backpropagation algorithm. Let the kernel size be... The convolution kernel weights are , bias is Output feature map The calculation formula (5) is: (5) in, For output channel index, , For the frequency and time index of the output space, These are learnable parameters, representing the anti-aliasing filter coefficients that the network automatically learns. To pad the offset and ensure center alignment; to accelerate convergence and stabilize training, batch normalization is performed immediately after the convolution output, for each channel. Calculate the mean value within this batch according to formula (6). and variance : (6) Calculate the normalized output according to formula (7). for: (7) in, , For learnable scaling and translation parameters, To prevent the small constant from dividing by zero, nonlinearity is introduced by modifying the ReLU linear unit, thus enhancing the model's ability to express complex audio patterns and obtaining the final output. This is the high-fidelity feature map after anti-aliasing downsampling.

[0011] Furthermore, the specific implementation method of the residual table data encoding module based on identity mapping initialization of the present invention is as follows: Let the input table data be ,in, For batch size, Enter dimensions for the table data. Indicates the first The tabular feature vector of each sample; for each sample First, a nonlinear mapping is performed through the main network path; let the weights and biases of the first fully connected layer be respectively... and The weights and biases of the second fully connected layer are respectively and The hidden layer dimension is The output embedding dimension is The calculation process of the main network path is shown in formulas (8)-(12): (8) (9) (10) (11) (12) in, This indicates a batch normalization operation. This indicates a modified linear activation function. This indicates a random deactivation operation, output by the main network path. This represents the amount of compensation for deep nonlinear features learned from the original tabular data; Simultaneously, residual paths are set to preserve the basic health semantic information in the original table data; when the input dimension and the output embedding dimension are consistent, the residual path directly adopts the identity mapping; when the input dimension and the output embedding dimension are inconsistent, the linear projection matrix PPP is used for dimension alignment, as shown in formula (13): (13) in, The residual projection matrix is ​​used; finally, the output of the table data encoding module is the sum of the main network path and the residual path, as shown in formula (14): (14) in, That is, the first The table encoding results of each sample, which is also the table modal feature representation used in the subsequent multimodal fusion stage; it represents the high-dimensional embedding features of the structured table information related to the user's health status or fatigue status after encoding.

[0012] Furthermore, the specific implementation method of the spectral rank-guided attention module in step 2 of the present invention is as follows: For facial image modalities and audio modalities, after modality-specific feature extraction and unified projection, dimension-aligned feature representations are obtained respectively. and Since the spectral rank-guided attention module performs attention calculations in a sequential manner, the facial image features are... and audio features Expand them into sequence features along their spatial or temporal dimensions respectively. N is the sequence length, D is the feature dimension, and the sequence is passed through a linear layer. Linear Projecting the features into query Q, key K, and value V is achieved according to formula (15): QKV=Linear ( X (15) Reshape and replace the dimensions of the output features, splitting them into... Q , K , V ,in For the number of attention heads, It is a single-head feature dimension; Next, based on the four-dimensional spatial feature map , For batch size, The number of feature channels, , Given the height and width of the feature map, calculate the local spectral entropy of the feature map to obtain the normalized spectral entropy mapping. The entropy value is normalized to the [0,1] interval, with high values ​​corresponding to noisy regions and low values ​​corresponding to structured regions; the module has built-in learnable scaling parameters. The initial value is set to 5.0, and adaptive optimization is performed during training; the spectral entropy guiding mask M is calculated according to formula (16): M= (16) The negative sign assigns a strong suppression mask to high-entropy regions and a weak suppression mask to low-entropy regions; scaling parameters... Controlling mask strength; attention-guided mask with output size matching input feature map size. This completes the mask generation for noise perception. Flatten the mask as The logarithmic bias is calculated and added to the original attention score to achieve spectral rank guidance and attention score adjustment. The core calculation formula (17) is: . (17) in, The logarithmic bias of the spectral entropy mask is used to achieve adaptive suppression of noisy regions by fusing it with the attention score through a broadcast mechanism. The attention probability distribution is obtained through formula (18). : (18) in, This indicates that normalization is performed on the guided attention score, and introduces... To prevent overfitting, the final features are calculated using formula (19). : (19) in, This represents the relationship between attention probability and value. Weighted summation, followed by dimensional reshaping, outputs the final features through a linear projection layer. .

[0013] Furthermore, the multimodal feature fusion and classification module of the present invention includes a fusion module, and the specific implementation method is as follows: The input to the fusion module comes from the output results of the aforementioned steps 2 and table encoding. After processing by the spectral rank guided attention module, the facial image features and audio features are respectively enhanced output features, denoted as... and After the table data is processed by the table encoding module, the table encoding characteristics are obtained, denoted as... ; wherein, the and This is the final output feature obtained in step 2. Next, the three features are globally pooled to obtain a compact global semantic representation; let the image global feature, audio global feature, and table global encoding feature be denoted as follows: and To achieve multimodal joint representation, the global features are element-wise additively fused after dimensional alignment to obtain a trimodal joint representation. Calculated by formula (20): (20) in, , , These represent the image features, audio features, and tabular encoded features after global average pooling, respectively. This represents the final feature representation after the three-modal fusion.

[0014] Furthermore, the multimodal feature fusion and classification module of the present invention includes a classification module, and the specific implementation method is as follows: The classification module includes two fully connected linear layers, a ReLU activation function, and a Dropout regularization layer, used to classify the health status of the fused multimodal global features; the first fully connected layer maps the input features to a 128-dimensional hidden space, and the calculation formula (21) is as follows: (twenty one) in, ,2,..., Indicates the input feature dimension. =1,2,...,128 represents hidden layer nodes. For the weights of the fully connected layer, For the corresponding bias; When outputting results, use The function transforms the unnormalized predictions of the fully connected layer into a non-negative probability distribution that sums to 1. The output value range is constrained to [0,1], as shown in formula (22): in (twenty two) in, This represents the number of feature nodes in the input. This represents the number of category nodes in the output.

[0015] Furthermore, the method for outputting the user's fatigue detection results according to the present invention is specifically as follows: Based on the obtained predicted probability distributions for each category, the category with the highest probability value is selected as the final fatigue detection result for the user; assuming... Softmax The probability distribution obtained after function processing is ,in This represents the predicted probability that the user belongs to the i-th type of fatigue state. The number of fatigue detection task categories is represented by , and the final fatigue detection result for the user is determined by formula (23). (twenty three) in, This indicates the output fatigue detection results.

[0016] This invention provides a health detection deep learning system based on multimodal spectral rank guided attention, characterized in that it includes: Memory, used to store executable computer programs; A processor, when executing an executable computer program stored in memory, implements a deep learning method for health detection based on multimodal spectral rank-guided attention.

[0017] The beneficial effects of this invention are: 1. This invention proposes a deep learning method for health detection based on multimodal data, using multimodal spectral rank guided attention. It consists of three modules: a multimodal feature encoding and unified projection module, a spectral rank guided spatial relative position attention (SRGA) module, and a multimodal feature fusion and classification module. The multimodal data feature extraction and fusion module effectively improves feature utilization and representation capabilities.

[0018] 2. This invention designs a spectral rank-guided spatial relative position attention mechanism module, which accurately identifies noise from the perspective of information entropy and matrix rank, dynamically suppresses interference regions, and fundamentally solves the problem that traditional attention is easily misled by noise.

[0019] 3. Experiments were conducted on a dataset consisting of 908 user samples. The results showed that the detection accuracy of this method on the dataset was as high as 96.37%, proving the effectiveness of the proposed method. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is the overall framework of the health detection deep learning method based on multimodal spectral rank guided attention in this embodiment of the invention; Figure 2 This is the overall framework of the multi-scale orthogonal feature extraction module in this embodiment of the invention; Figure 3 This is the overall framework of the audio feature downsampling module based on the adaptive spectrum anti-aliasing mechanism in this embodiment of the invention; Figure 4 This is the overall framework of the tabular data encoder in this embodiment of the invention; Figure 5 This is the overall framework of the SRGA module in this embodiment of the invention; Figure 6 It is the performance metric of the model in this embodiment of the invention on the training set. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] Example 1 The health detection deep learning method based on multimodal spectral rank-guided attention in this invention completes feature encoding, spatial structure preservation, and noise-robust feature learning from three types of multimodal health data: images, audio, and tables. It also improves the model's anti-interference ability and generalization performance in complex scenarios through a spectral entropy-guided attention mechanism, thereby achieving high-precision and robust multimodal health status detection.

[0023] Its overall framework is as follows Figure 1 As shown, it consists of three main modules: A. Multimodal feature encoding and unified projection module; B. Spectral Rank Guided Spatial Relative Position Attention (SRGA) module; C. Multimodal feature fusion and classification module. Among them: Step 1 involves feeding the multimodal health monitoring data, consisting of images, audio, and tables, into the multimodal feature encoding and unified projection modules to complete modality-specific feature extraction and dimension alignment. Specifically, the images are extracted using a multi-scale orthogonal feature extraction (MSO) module, the audio data is sampled using an audio feature downsampling module based on an adaptive spectral anti-aliasing mechanism (ASAD), and the table data is encoded using a residual table data encoding module based on identity mapping initialization (ZIR-Tab) to map structured table features.

[0024] The multimodal feature encoding and unified projection module of this invention designs a multi-scale orthogonal feature extraction (MSO) module to extract spatial structural features of facial images, an audio feature downsampling (ASAD) module based on an adaptive spectral anti-aliasing mechanism to extract time-frequency features of speech Mel spectrum, and a residual tabular data encoding (ZIR-Tab) based on identity mapping initialization to map scalar features of human physical characteristics, preserving the original structural information of each modality. The image and audio features are mapped to a unified fusion dimension through 1×1 pointwise convolution, achieving dimensional alignment with the tabular features and eliminating heterogeneous modal distribution differences. The image and audio 2D features are flattened into a sequence form and fed into the SRGA module to preserve the spatiotemporal topological relationship. The enhanced sequence is subjected to global average pooling to obtain a global feature vector. An element-wise summation strategy is used to complete the deep fusion of the three modalities of image, audio, and tabular, outputting a unified multimodal representation.

[0025] To address the challenge of deep fusion of multimodal heterogeneous data, a multimodal feature encoding and unified projection module is designed. Based on the original multimodal data, the module decodes core information from the intramodal spatiotemporal dimension and the intermodal correlation dimension. This module focuses on how to effectively extract fusion features with cross-modal complementarity and spatiotemporal correlation, as well as the interaction correlation between different modal channels.

[0026] Step 2: The extracted feature map is fed into the spectral rank guided attention (SRGA) module. The noise perception bias is calculated based on the local low-rank spectral entropy, and attention calculation is dynamically injected to automatically suppress high-entropy noise regions, enhance low-entropy effective features, and preserve spatial topological relationships.

[0027] The Spectral Rank Guided Attention (SRGA) module in this embodiment of the invention constructs a local spectral entropy generator, performs local neighborhood decomposition on the feature map, calculates spectral entropy to characterize noise intensity, maps high-entropy noise regions to negative bias, and maintains the original weights of low-entropy effective regions, thereby achieving adaptive noise recognition.

[0028] The generated spectral entropy bias is dynamically injected into the multi-head self-attention score to correct the original attention weight distribution, automatically suppress the attention allocation in high-entropy noise regions, enhance the weight of low-entropy key features, guide the model to accurately focus on effective information, and combine spatial relative position bias to preserve the spatial structural dependencies of features, and finally output high-quality features after noise resistance and purification.

[0029] To address noise interference in complex environments, a dynamic gating noise reduction mechanism based on local low-rank spectral entropy is designed to achieve differentiated analysis of fused features and acquisition of key discriminative features. This module focuses on how to segment cross-modal fused features into different sub-feature spaces and extract discriminative local features from each sub-feature space through multi-scale convolution, thereby improving the generalization ability and classification accuracy of multimodal health data classification tasks.

[0030] Step 3 involves global pooling and deep fusion of the enhanced features from the image, audio, and table, with the final classification decision implemented by a fully connected layer. The three main modules of this invention will be described in detail in the following sections.

[0031] Example 2 Based on Embodiment 1, this invention introduces the design methods for each sub-module in the multimodal feature encoding and unified projection module. This module adopts an architecture of modality-specific encoding and unified dimensional projection, designing encoders for images, audio, and tables respectively to achieve structured extraction and alignment of heterogeneous features. Specifically: 1. The overall framework of the multi-scale orthogonal feature extraction module (MSO) of the image encoder is as follows: Figure 2 As shown.

[0032] The method first employs parallel multi-scale orthogonal convolutional branches, using horizontal (1×U) and vertical (U×1) convolutional kernels of different sizes to extract local and contextual features of the image in the horizontal and vertical directions, respectively, constructing a feature set sensitive to texture orientation. Subsequently, the extracted multi-scale, multi-directional features are concatenated along the channel dimension, and spatial and channel adaptive weights are calculated using a scale-orientation attention mechanism to achieve feature selection and weighted fusion. Next, 1×1 convolutions are used to adjust the number of channels and perform non-linear mapping on the weighted fused features. Finally, residual connections are used to add the output of this module to the original input features after appropriate projection, forming the final output. This structure effectively captures the anisotropic information of the image while ensuring stable gradient propagation, significantly improving the model's ability to model directional textures in complex visual tasks.

[0033] 2. This invention designs an audio feature downsampling module based on an adaptive spectrum anti-aliasing mechanism to achieve audio sampling. The overall framework of the audio encoder is as follows: Figure 3 As shown.

[0034] The original audio signal is converted into time-frequency domain spectral features by the spectrum analysis module and then fed into an adaptive anti-aliasing filter. This filter dynamically adjusts the convolutional filter weights based on the energy distribution and frequency structure characteristics of the current frequency band, thereby actively suppressing high-frequency aliasing components while preserving key harmonics and details during spatial downsampling. The filtered features then enter the feature extractor, where multiple convolutions and activation mappings further extract time-frequency structure features. Finally, the dimensionality-reduced features enter the downsampling output module to complete efficient compression and encoding of the audio features. This achieves synergy between spectrum sensing, anti-aliasing optimization, and feature compression, significantly improving the audio encoder's ability to preserve time-frequency structure and information density at low resolution.

[0035] 3. This invention proposes a residual table data encoding module based on identity mapping initialization for efficient mapping of structured table features. The overall framework of the table data encoder is as follows: Figure 4 As shown.

[0036] The original high-dimensional tabular data first enters the main processing path through a linear projection layer. This main processing path involves a multi-layer feedforward network with batch normalization, activation functions, and random deactivation operations to perform nonlinear transformations and dimensionality adjustments on the features, ultimately outputting transformed features. Simultaneously, a residual mapping path is used to adapt the original input to dimensions. When the input and output dimensions are consistent, this path represents an identity mapping; otherwise, an additional linear layer aligns the dimensions. The core innovation lies in initializing the weights and biases of the last layer of the main processing path network to zero, ensuring that the initial model state satisfies the identity mapping relationship, i.e., the output approaches zero. Finally, this module combines the transformed features with the dimension-adapted original features through additive fusion, forming the output of the residual connection. This design enables the model to possess near-identical mapping characteristics from the early stages of training, effectively stabilizing gradient propagation and accelerating model convergence. Furthermore, the residual mechanism preserves original feature information, effectively improving the robustness of feature extraction from sparse tabular data.

[0037] Example 3 Based on embodiments 1 and 2, this invention describes the implementation methods of each module by combining specific parameters and formulas. Specifically, it includes: 1. Construction of Multimodal Feature Encoding and Unified Projection Module Multimodal health data comprises three heterogeneous types of information: facial images, audio recordings, and biometric tables. These data exhibit significant differences in data structure, dimensionality distribution, and feature attributes. To achieve effective representation and alignment of these three modalities, this invention constructs a multimodal feature encoding and unified projection module. This module implements dedicated encoding and dimensionality normalization for different modalities, thus laying the foundation for subsequent deep fusion and attention refinement.

[0038] For facial image data, this invention designs a multi-scale orthogonal feature extraction (MSO) module, assuming the input feature map is... Define the scale set as , direction set These represent the horizontal and vertical directions, respectively. First, for each scale... The module executes two asymmetric convolutional branches in parallel to decouple the directional features of facial texture, thus enabling the lateral branch features to be processed. Calculate according to formula (1): (1) in This indicates that a core size of [size] is used. Convolutional layers capture horizontal features such as forehead lines and lip lines. Vertical branch features... Calculate according to formula (2): (2) in This indicates that a core size of [size] is used. The convolutions capture vertical features, such as nasolabial folds and sagging facial contours. The outputs of all branches are concatenated along the channel dimension to form an anisotropic feature map according to formula (3). : (3) in This indicates channel concatenation, and then features are processed through... Convolution compresses back to the target number of channels And according to formula (4), residual connections are added to retain the original information to obtain the final output. : (4) in This represents a mapping function that ensures the input channel equals the output channel.

[0039] To address the frequency sensitivity of audio data, this invention designs an audio downsampling method based on a learnable dynamic filter kernel. Let the input audio Mel-spectral characteristics be... , For the input tensor, the first The sample, the first Channel, frequency index Time Index The core of this module is a 2D convolution operation with a stride of 2. Unlike traditional fixed-weight average pooling, the convolution kernel here is learned end-to-end through backpropagation. Let the kernel size be... The convolution kernel weights are , bias is Output feature map The calculation formula (5) is: (5) in, For output channel index, , For the frequency and time index of the output space, These are learnable parameters, representing the anti-aliasing filter coefficients that the network automatically learns. To pad the offset and ensure center alignment, batch normalization is performed immediately after the convolution output to accelerate convergence and stabilize training, for each channel. Calculate the mean value within this batch according to formula (6). and variance : (6) Calculate the normalized output according to formula (7). for: (7) in, , For learnable scaling and translation parameters, To prevent small constants from being divided by zero, this step ensures that the spectral energy differences caused by different human voices will not interfere with the downsampling filter. Finally, nonlinearity is introduced by correcting the linear unit ReLU to enhance the model's ability to express complex audio patterns, resulting in the final output Y, which is the high-fidelity feature map after anti-aliasing downsampling.

[0040] For tabular data, a residual tabular data encoding method based on identity mapping initialization is proposed. This method, building upon traditional MLP encoders, introduces an adaptive residual connection mechanism and employs a specific weight initialization strategy to ensure that the network strictly maintains the identity mapping of input features during the initial training phase. Subsequently, iterative learning is used to progressively optimize the feature representation. The key difference between this method and existing techniques lies in the parameter set... Initialization constraints for a specific subset. At the initial training time t=0, the parameters of the last layer of the main processing path are initialized according to formula (8). , Apply the following zero-value constraint: (8) in, To determine the output embedding dimension, during backpropagation, the gradient of the loss function L with respect to the input x can be expressed as follows according to formula (9): (9) Among them, due to the existence of residual paths, even the main processing path In the early stages of training, the gradient is small, and the gradient can still pass through... The path is returned to the input layer without loss or with low loss, which effectively alleviates the gradient vanishing problem in deep table networks.

[0041] 2. Construction of the Spectral Rank Guided Attention (SRGA) module In this method, the low-rank characteristic of the local feature matrix represents the structured effective information, and spectral entropy is used to quantify noise intensity: the lower the entropy value, the stronger the feature structure and the less noise; the higher the entropy value, the more significant the redundant noise. Therefore, this module, based on traditional spatial attention, injects a learnable spectral entropy bias term to dynamically adjust the attention weight distribution, automatically suppressing high-entropy noise regions and enhancing low-entropy key features. The module framework diagram is shown below. Figure 5 As shown.

[0042] The input is sequence features. (N is the sequence length, D is the feature dimension), first through a linear layer Linear Projecting the features into query Q, key K, and value V is achieved according to formula (10): QKV=Linear ( X (10) Next, the output features are reshaped and their dimensions are replaced, splitting them into... Q , K , V ,in For the number of attention heads, It is a single-head feature dimension.

[0043] Next, based on the four-dimensional spatial feature map ( For batch size, The number of feature channels, , Calculate the local spectral entropy of the feature map (height and width) to obtain the normalized spectral entropy map. The entropy value is normalized to the [0,1] interval, with high values ​​corresponding to noisy regions and low values ​​corresponding to structured regions. The module has built-in learnable scaling parameters. The initial value is set to 5.0, and adaptive optimization is performed during training. The spectral entropy guiding mask M is calculated according to formula (11): M= (11) The negative sign assigns a strong suppression mask to high-entropy regions and a weak suppression mask to low-entropy regions; scaling parameters... Control the mask strength. The final output is an attention-guided mask that matches the size of the input feature map. This completes the mask generation for noise perception.

[0044] Then flatten the mask to The logarithmic bias is calculated and added to the original attention score to achieve spectral rank guidance and attention score adjustment. The core calculation formula (12) is: . (12) in, The logarithmic bias of the spectral entropy mask is used to achieve adaptive suppression of noisy regions by fusing it with the attention score through a broadcast mechanism.

[0045] Then, the attention probability distribution is obtained using formula (13). : (13) in, This indicates that normalization is performed on the guided attention score, and introduces... To prevent overfitting, the final features are calculated using formula (14). : (14) in, This represents the relationship between attention probability and value. Weighted summation, followed by dimensional reshaping, outputs the final features through a linear projection layer. .

[0046] 3. Construction of Multimodal Feature Fusion and Classification Module The multimodal feature fusion and classification module of this method is simple and efficient. First, the global image features, global audio features, and tabular encoded features processed by spectral rank-guided attention are element-wise additively fused to obtain a unified multimodal joint representation. The three-modal feature fusion result... Calculated using formula (15): (15) in, , , These represent the image features, audio features, and tabular encoded features after global average pooling, respectively. This represents the final feature representation after the three-modal fusion.

[0047] The classification module of this method consists of two fully connected linear layers, a ReLU activation function, and a Dropout regularization layer, used to classify the health status of the fused multimodal global features. The first fully connected layer maps the input features to a 128-dimensional hidden space, and the calculation formula (16) is as follows: (16) in, ,2,..., Indicates the input feature dimension. =1,2,...,128 represents hidden layer nodes. For the weights of the fully connected layer, This is the corresponding bias.

[0048] When outputting results, use The function transforms the unnormalized predictions of the fully connected layer into a non-negative probability distribution that sums to 1, with the output value constrained to the range [0,1]. Formula (17) is as follows: in (17) in, This represents the number of feature nodes in the input. This represents the number of category nodes in the output.

[0049] The three modules are combined to form the complete multimodal fusion model of this invention. This model can fully explore the spatial feature correlations of different modal data and improve the model's noise robustness through a spectral rank guidance mechanism. This invention will experimentally demonstrate the effectiveness of this method in cross-subject fatigue detection.

[0050] Example 4: Experimental Analysis To demonstrate the performance of the model in this invention and to provide an overall evaluation, this invention uses precision, recall, accuracy, and F1 score to assess the model's effectiveness in facial emotion recognition. The formulas for calculating these metrics are as follows: (18) (19) (20) (twenty one) in, This represents the number of samples that were correctly classified into the corresponding category. This represents the number of samples that do not belong to this category but have been incorrectly classified into it. This represents the number of samples that belong to this category but were incorrectly classified into other categories. This represents the number of samples that do not belong to this class but are correctly identified. Precision is the proportion of true positives predicted by the model; recall is the proportion of true positives predicted by the model; accuracy (ACC) is the proportion of correctly classified samples out of the total number of samples; the F1 score is a weighted average of precision and recall, and a higher F1 score indicates better model performance. This invention was tested on a training set containing 726 samples. Experimental results are as follows... Figure 6 As shown.

[0051] The training curves show that the model converged rapidly in the early training phase (1–25 epochs), with accuracy increasing from 64.19% to 97.66% and F1-Score from 0.4718 to 0.9732, indicating that the model can quickly learn the joint feature distribution of the data. In the mid-training phase (26–45 epochs), the model performance steadily improved amidst fluctuations, with accuracy repeatedly exceeding 99% and F1-Score reaching above 0.9905. Although various metrics fluctuated slightly due to gradient updates, they remained at a high level overall, demonstrating good stability and strong fitting ability. In the later training phase (46–70 epochs), the model gradually converged to its optimal state, with accuracy and multiple metrics reaching a maximum of 100%, and the F1-Score stabilizing above 0.98. Ultimately, it achieved excellent performance on the training set with Accuracy=99.86%, Precision=0.9990, Recall=0.9979, and F1-Score=0.9984. The model demonstrates strong feature representation ability, high recognition accuracy, and good generalization potential in relevant detection tasks. Furthermore, the optimal training weights were selected and tested on a full dataset containing 908 samples. The experimental results are shown in Table 1.

[0052] Table 1. Performance metrics of the model on the test set Overall, the model maintains a performance of over 95% across all metrics on the full dataset, achieving an accuracy of 96.37%. It also demonstrates strong feature extraction capabilities and good generalization performance, meeting the high-precision recognition and health detection requirements across three modalities: image, audio, and structured data.

[0053] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0054] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A health detection deep learning method based on multi-modal spectrum rank guiding attention, characterized in that, The method includes the following steps: Step 1: Collect users' facial images, audio, and tabular data to form multimodal health monitoring data, and send them to the multimodal feature encoding and unified projection module corresponding to each type of data to complete modality-specific feature extraction and dimension alignment. Step 2: Feed the facial image features and audio features into the spectral rank guided attention module, calculate the noise perception bias based on the local low-rank spectral entropy, and dynamically inject it into the attention calculation process to automatically suppress high-entropy noise regions, enhance low-entropy effective features, and preserve spatial topological relationships to obtain enhanced facial image features and enhanced audio features. Step 3: The enhanced facial image features, enhanced audio features, and table features are fed into the multimodal feature fusion and classification module for global pooling and deep fusion. Then, they are fed into the fully connected layer for classification decision and output the user's fatigue detection results.

2. The health detection deep learning method based on multi-modal spectrum rank guiding attention according to claim 1, characterized in that, The multimodal feature encoding and unified projection module in step 1 includes: For the multi-scale orthogonal feature extraction module of facial image data, a multi-scale orthogonal convolution branch is deployed in parallel. Different sizes of horizontal and vertical convolution kernels are used to extract local and contextual features in the horizontal and vertical directions of the image, respectively, constructing a feature set sensitive to texture direction. Subsequently, the extracted multi-scale and multi-directional features are concatenated along the channel dimension, and a scale-direction attention mechanism is used to perform spatial and channel-adaptive weight calculations to achieve feature selection and weighted fusion. Next, convolution is used to adjust the number of channels and perform non-linear mapping on the weighted fused features. Finally, residual connections are used to add the output of this module to the original input features after appropriate projection to form the final output. For audio data, an audio feature downsampling module based on an adaptive spectrum anti-aliasing mechanism is used. The original audio data is converted into time-frequency domain spectral features by a spectrum analysis module and then fed into an adaptive anti-aliasing filter. This filter dynamically adjusts the convolutional filter weights according to the energy distribution and frequency structure characteristics of the current frequency band, actively suppressing high-frequency aliasing components while spatially downsampling and preserving key harmonics and details. The filtered features enter the feature extractor, which further extracts time-frequency structure features through multi-layer convolution and activation mapping. The features after dimensionality reduction are then fed into the downsampling output module to complete the efficient compression and encoding of audio features. For the residual table data encoding module based on identity mapping initialization, the original high-dimensional table data first enters the main processing path through a linear projection layer. A multi-layer feedforward network, including batch normalization, activation functions, and random deactivation operations, performs nonlinear transformation and dimensional adjustment on the features, outputting transformed features. Simultaneously, the original input table data is dimension-adapted through a residual mapping path. When the input and output dimensions are consistent, the path is an identity mapping; otherwise, an additional linear layer is used for dimension alignment. The weights and biases of the last layer of the main processing path network are initialized to zero to ensure that the initial state satisfies the identity mapping relationship, i.e., the output approaches zero. This module combines the transformed features with the dimension-adapted original features through additive fusion to form the output result of the residual connection.

3. The health detection deep learning method based on multimodal spectral rank guided attention as described in claim 2, characterized in that, The specific implementation method of the multi-scale orthogonal feature extraction module is as follows: Let the feature map of the input facial image be... Define the scale set as , direction set Representing the horizontal and vertical directions respectively; firstly, for each scale The module executes two asymmetric convolutional branches in parallel to decouple the directional features of facial texture, thus enabling the lateral branch features to be processed. Calculate according to formula (1): .（1） in, This indicates that a core size of [size] is used. Convolutional processing captures horizontal features such as forehead lines and lip lines; vertical branch features... Calculate according to formula (2): .（2） in, This indicates that a core size of [size] is used. The convolution captures vertical features; the outputs of all branches are concatenated along the channel dimension to form an anisotropic feature map according to formula (3). : .（3） in, This indicates channel concatenation, and then features are processed through... Convolution compresses back to the target number of channels And according to formula (4), residual connections are added to retain the original information to obtain the final output. : .（4） in, This represents a mapping function that ensures the input channel equals the output channel.

4. The health detection deep learning method based on multimodal spectral rank guided attention as described in claim 2, characterized in that, The specific implementation method of the audio feature downsampling module based on the adaptive spectrum anti-aliasing mechanism is as follows: Let the Mel spectrogram features of the input audio data be... , For the input tensor, the first The sample, the first Channel, frequency index Time Index The core setting of the audio feature downsampling module based on adaptive spectrum anti-aliasing mechanism includes a two-dimensional convolution operation with a stride of 2. The convolution kernel is learned end-to-end through the backpropagation algorithm. Let the kernel size be... The convolution kernel weights are , bias is Output feature map The calculation formula (5) is: .（5） in, For output channel index, , For the frequency and time index of the output space, These are learnable parameters, representing the anti-aliasing filter coefficients that the network automatically learns. To pad the offset and ensure center alignment; to accelerate convergence and stabilize training, batch normalization is performed immediately after the convolution output, for each channel. Calculate the mean value within this batch according to formula (6). and variance : .（6） Calculate the normalized output according to formula (7). for: .（7） in, , For learnable scaling and translation parameters, To prevent the small constant from dividing by zero, nonlinearity is introduced by modifying the ReLU linear unit, thus enhancing the model's ability to express complex audio patterns and obtaining the final output. This is the high-fidelity feature map after anti-aliasing downsampling.

5. The health detection deep learning method based on multimodal spectral rank guided attention as described in claim 2, characterized in that, The specific implementation method of the residual table data encoding module based on identity mapping initialization is as follows: Let the input table data be ,in, For batch size, Enter dimensions for the table data. Indicates the first The tabular feature vector of each sample; for each sample First, a nonlinear mapping is performed through the main network path; let the weights and biases of the first fully connected layer be respectively... and The weights and biases of the second fully connected layer are respectively and The hidden layer dimension is The output embedding dimension is The calculation process of the main network path is shown in formulas (8)-(12): .（8） .（9） .（10） .（11） .（12） in, This indicates a batch normalization operation. This indicates a modified linear activation function. This indicates a random deactivation operation, output by the main network path. This represents the amount of compensation for deep nonlinear features learned from the original tabular data; Simultaneously, residual paths are set to preserve the basic health semantic information in the original table data; when the input dimension and the output embedding dimension are consistent, the residual path directly adopts the identity mapping; when the input dimension and the output embedding dimension are inconsistent, the linear projection matrix PPP is used for dimension alignment, as shown in formula (13): .（13） in, The residual projection matrix is ​​used; finally, the output of the table data encoding module is the sum of the main network path and the residual path, as shown in formula (14): .（14） in, That is, the first The table encoding results of each sample, which is also the table modal feature representation used in the subsequent multimodal fusion stage; it represents the high-dimensional embedding features of the structured table information related to the user's health status or fatigue status after encoding.

6. The health detection deep learning method based on multimodal spectral rank guided attention as described in claim 1, characterized in that, The specific implementation method of the spectral rank-guided attention module in step 2 is as follows: For facial image modalities and audio modalities, after modality-specific feature extraction and unified projection, dimension-aligned feature representations are obtained respectively. and Since the spectral rank-guided attention module performs attention calculations in a sequential manner, the facial image features are... and audio features Expand them into sequence features along their spatial or temporal dimensions respectively. N is the sequence length, D is the feature dimension, and the sequence is passed through a linear layer. Linear Projecting the features into query Q, key K, and value V is achieved according to formula (15): QKV=Linear （ X ）.（15） Reshape and replace the dimensions of the output features, splitting them into... Q , K , V ,in For the number of attention heads, It is a single-head feature dimension; Next, based on the four-dimensional spatial feature map , For batch size, The number of feature channels, , Given the height and width of the feature map, calculate the local spectral entropy of the feature map to obtain the normalized spectral entropy mapping. The entropy value is normalized to the [0,1] interval, with high values ​​corresponding to noisy regions and low values ​​corresponding to structured regions; the module has built-in learnable scaling parameters. The initial value was set to 5.0, and adaptive optimization was performed during training. Calculate the spectral entropy guiding mask M according to formula (16): M= .（16） The negative sign assigns a strong suppression mask to high-entropy regions and a weak suppression mask to low-entropy regions; scaling parameters... Controlling mask strength; attention-guided mask with output size matching input feature map size. This completes the mask generation for noise perception. Flatten the mask as The logarithmic bias is calculated and added to the original attention score to achieve spectral rank guidance and attention score adjustment. The core calculation formula (17) is: . （17） in, The logarithmic bias of the spectral entropy mask is used to achieve adaptive suppression of noisy regions by fusing it with the attention score through a broadcast mechanism. The attention probability distribution is obtained through formula (18). : （18） in, This indicates that normalization is performed on the guided attention score, and introduces... To prevent overfitting, the final features are calculated using formula (19). : （19） in, This represents the relationship between attention probability and value. Weighted summation, followed by dimensional reshaping, outputs the final features through a linear projection layer. .

7. The health detection deep learning method based on multimodal spectral rank guided attention as described in claim 6, characterized in that, The multimodal feature fusion and classification module includes a fusion module, which is implemented as follows: The input to the fusion module comes from the output results of the aforementioned steps 2 and table encoding. After processing by the spectral rank guided attention module, the facial image features and audio features are respectively enhanced output features, denoted as... and After the table data is processed by the table encoding module, the table encoding characteristics are obtained, denoted as... ; wherein, the and This is the final output feature obtained in step 2. Next, the three features are globally pooled to obtain a compact global semantic representation; let the image global feature, audio global feature, and table global encoding feature be denoted as follows: and To achieve multimodal joint representation, the global features are element-wise additively fused after dimensional alignment to obtain a trimodal joint representation. Calculated by formula (20): （20） in, , , These represent the image features, audio features, and tabular encoded features after global average pooling, respectively. This represents the final feature representation after the three-modal fusion.

8. The health detection deep learning method based on multimodal spectral rank guided attention as described in claim 7, characterized in that, The multimodal feature fusion and classification module includes a classification module, which is implemented as follows: The classification module includes two fully connected linear layers, a ReLU activation function, and a Dropout regularization layer, used to classify the health status of the fused multimodal global features; the first fully connected layer maps the input features to a 128-dimensional hidden space, and the calculation formula (21) is as follows: （21） in, ,2,..., Indicates the input feature dimension. =1,2,...,128 represents hidden layer nodes. For the weights of the fully connected layer, For the corresponding bias; When outputting results, use The function transforms the unnormalized predictions of the fully connected layer into a non-negative probability distribution that sums to 1. The output value range is constrained to [0,1], as shown in formula (22): in (twenty two) in, This represents the number of feature nodes in the input. This represents the number of category nodes in the output.

9. The health detection deep learning method based on multimodal spectral rank guided attention as described in claim 8, characterized in that, The method for outputting the user's fatigue detection results is as follows: Based on the obtained predicted probability distributions for each category, the category with the highest probability value is selected as the final fatigue detection result for the user; assuming... Softmax The probability distribution obtained after function processing is ,in This represents the predicted probability that the user belongs to the i-th type of fatigue state. The number of fatigue detection task categories is represented by , and the final fatigue detection result for the user is determined by formula (23). （23） in, This indicates the output fatigue detection results.

10. A health detection deep learning system based on multimodal spectral rank guided attention, characterized in that, include: Memory, used to store executable computer programs; A processor, when executing an executable computer program stored in memory, implements the health detection deep learning method based on multimodal spectral rank guided attention as described in any one of claims 1 to 9.

Images