Intelligent mental illness recognition method and device based on multi-modal data and medium

Abstract

The present application provides a kind of mental illness intelligent identification method, equipment and medium based on multi-modal data, the method comprises: obtaining the illness self-report text information of the person to be measured, ear image data, impedance and / or temperature information of each specified acupoint area in the conchal area and the helix area of the person to be measured;Local feature extraction is carried out to ear image data to obtain image feature vector;The impedance and / or temperature information between different acupoint areas is captured using multi-head attention mechanism Measurement value feature vector;Extract the semantic feature vector of illness self-report text information;Image feature vector, measurement value feature vector, semantic feature vector form multi-modal fusion feature vector and input into disease intelligent identification model for classification, output mental illness prediction result.The present application combines ear image, impedance and / or temperature information of each acupoint, patient self-report text information multi-modal data and combines with deep learning model to realize disease classification, improve recognition efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, and in particular to a method, device and medium for intelligent identification of mental illness based on multimodal data. Background Technology

[0002] In the field of medical artificial intelligence, the accurate diagnosis of mental illnesses (such as depression and cognitive impairment) is of great significance. Clinical diagnosis relies heavily on subjective scale assessments and physician experience, resulting in high misdiagnosis rates and low efficiency. To address these issues, current technologies primarily employ assisted diagnostic methods based on single-modal data (such as electroencephalography or MRI). However, existing automated analysis models for single-modal data lack generalization capabilities and the ability to jointly model multimodal data (such as physiological signals, images, and text). While existing multimodal pre-trained models such as MedViLL have demonstrated the potential for cross-modal representation learning in the medical field, current technologies do not provide effective multimodal data fusion analysis for mental illnesses.

[0003] It is evident that traditional diagnostic methods suffer from problems such as reliance on subjective judgment, high misdiagnosis rates, and low efficiency. Existing automated analysis models also face challenges such as insufficient generalization ability and difficulties in joint modeling of multimodal data. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a method, device and medium for intelligent identification of mental illness based on multimodal data that solves or at least partially solves the above technical problems.

[0005] One aspect of the present invention provides a method for intelligent identification of mental illness based on multimodal data, the method comprising:

[0006] Acquire the subject's self-reported medical condition text information, ear image data, and impedance and / or temperature information of each designated acupoint in the concha and helix regions of the subject;

[0007] Local feature extraction is performed on the ear image data to obtain the image feature vector;

[0008] The impedance and / or temperature information of each acupoint is converted into vector form, and the multi-head attention mechanism in the Transformer architecture is used to capture the correlation features between the impedance and / or temperature information of different acupoints to obtain the measured value feature vector.

[0009] Extract semantic feature vectors from the patient's self-reported medical information;

[0010] Image feature vectors, measurement feature vectors, and semantic feature vectors are concatenated and fused to form a multimodal fusion feature vector;

[0011] The multimodal fusion feature vector is input into a pre-trained intelligent disease identification model. The model then classifies the input multimodal fusion feature vector based on the relationship between the multimodal fusion feature vector learned during the model training process and various categories of mental illnesses of different severity. The model outputs a prediction result, which is a probability vector representing the probability that the sample belongs to various categories of mental illnesses of different severity and the normal state.

[0012] Optionally, the method further includes:

[0013] Before performing local feature extraction on the ear image data, the ear image data is subjected to size unification and pixel value normalization operations;

[0014] Before converting the impedance and / or temperature information of each cavity region into vector form, Z-score normalization is performed on the impedance and / or temperature information of each cavity region; and

[0015] Before extracting the semantic feature vector of the self-reported medical condition text, the self-reported medical condition text is segmented to be truncated to the preset maximum length.

[0016] Optionally, after forming the multimodal fusion feature vector, the method further includes:

[0017] A multimodal joint modeling network composed of serially configured multi-layer Transformer blocks is used to iteratively transform the multimodal fusion feature vector multiple times to obtain a multimodal fusion feature vector with deep fusion and feature enhancement.

[0018] Each Transformer block contains a multi-head attention mechanism and a feedforward neural network. The multi-head attention mechanism performs attention calculations on the feature vector input to the current Transformer block to capture the correlation between features of different modalities. The feedforward neural network performs feature transformation and enhancement, and passes the processed feature vector to the next Transformer block. After multiple iterations, the output is a multi-modal fused feature vector that has undergone deep fusion and feature enhancement.

[0019] Optionally, obtaining impedance and / or temperature information for each designated acupoint within the concha and helix regions of the subject includes:

[0020] A pre-trained acupoint detection model was used to segment and detect acupoints in the ear image data of the test subject, so as to obtain the acupoint distribution information of the concha and helix regions of the current test subject;

[0021] A measurement and positioning template for the acupoints of the test subject is generated based on the acupoint distribution information, and the acupoint measurement and positioning template is sent to the ear image data acquisition device so that the acupoint measurement and positioning template is displayed in the video guide box of the image acquisition interface of the acquisition device. By matching and aligning the acupoint measurement and positioning template with the ear image of the test subject, the sensor device is guided to acquire the impedance value and / or temperature value of each designated acupoint in the concha and helix regions of the test subject.

[0022] The impedance and / or temperature values ​​of each designated acupoint in the concha and helix regions of the subject are uploaded by the receiving sensor device.

[0023] Optionally, the impedance values ​​and / or temperature values ​​of each designated acupoint area are sorted according to the collection time and / or acupoint number.

[0024] Optionally, the hole detection model includes a convolutional neural network backbone network, a Neck network, and a Head network;

[0025] The convolutional neural network backbone is used to extract features at different scales and levels from the ear image data of the test subject in order to obtain multi-scale image feature information;

[0026] Neck network is used to perform feature fusion on multi-scale image feature information extracted by the backbone network of convolutional neural network;

[0027] The Head network is used to predict the bounding boxes and key point coordinates of acupoint regions from the fused features using a decoupled head structure, thereby obtaining the acupoint distribution information of the concha and helix regions of the current subject.

[0028] Optionally, the training steps of the disease intelligent recognition model include:

[0029] Obtain labeled sample data. Each sample data includes the patient's self-reported medical condition text information, ear image data, impedance and / or temperature information of each designated acupoint in the patient's concha and helix regions, and the patient's disease label.

[0030] The ear image data in each sample data is subjected to size unification and pixel value normalization to obtain preprocessed image data samples. The impedance and / or temperature information in each sample data is Z-score normalized to obtain preprocessed measurement data samples. The self-reported medical condition text information in each sample data is segmented to be truncated to the preset maximum length to obtain preprocessed text data samples.

[0031] A convolutional neural network (CNN) is used to extract local features from each image data sample, and pooling layers are used to reduce the feature dimensionality to obtain image feature vector samples. The impedance and / or temperature information of each acupoint in each measurement data sample is converted into vector form, and the multi-head attention mechanism in the Transformer architecture is used to capture the correlation features between the impedance and / or temperature information of different acupoints to obtain measurement value feature vector samples. A BERT language model is used to encode each text data sample to extract semantic feature vector samples. The image feature vector samples, measurement value feature vector samples, and semantic feature vector samples corresponding to each sample data are aligned in feature dimension, and feature vector samples from different modalities are concatenated and fused through feature concatenation to form multimodal fused feature vector samples of each sample data.

[0032] The multimodal fusion feature vector samples of each sample data and the disease labels corresponding to each sample are input into a preset fully connected neural network classifier for training of the intelligent disease identification model.

[0033] During model training, the cross-entropy loss function is used to measure the loss between the predicted result of each sample data and the real disease label, and the Adam optimizer is used to optimize the network parameters of the intelligent disease identification model to minimize the loss function and achieve model performance optimization.

[0034] Optionally, the impedance and / or temperature information of the specified acupoints includes the impedance and / or temperature values ​​of 18 acupoints in the concha and 13 acupoints in the antihelix.

[0035] In another aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor; when executed by the processor, the computer program implements the steps of the intelligent identification method for mental illness based on multimodal data as described in any of the preceding claims.

[0036] In another aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the intelligent identification method for mental illness based on multimodal data as described in any of the preceding claims.

[0037] The present invention provides a method, device, and medium for intelligent identification of mental illness based on multimodal data. By innovatively integrating three modalities of data—ear image features, impedance and temperature information of various acupoints, and patient self-report information—it offers a richer and more comprehensive information dimension for the auxiliary identification of mental illness. Furthermore, it utilizes an advanced deep learning model architecture to fuse and process the multimodal data. The model can automatically learn the correlations and complementary information between different modalities, thereby achieving refined classification and identification of mental illnesses such as depression (mild, moderate, and severe) and cognitive impairment (mild, moderate, and severe). This invention not only assists in identifying the presence or absence of mental illness but also distinguishes different degrees of severity, providing a more targeted basis for the formulation of clinical treatment plans and possessing significant clinical application value.

[0038] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings:

[0040] Figure 1 This is a flowchart of the intelligent identification method for mental illness based on multimodal data according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram showing the mapping of the acupoint measurement and positioning template to the video guide frame in an embodiment of the present invention;

[0042] Figure 3 This is a flowchart illustrating the standardization process for impedance and temperature information in an embodiment of the present invention.

[0043] Figure 4 This is a flowchart illustrating the processing of self-reported medical condition text information in an embodiment of the present invention;

[0044] Figure 5 This is a flowchart illustrating the feature extraction process for impedance and temperature information in an embodiment of the present invention.

[0045] Figure 6 This is a flowchart illustrating the feature extraction process for self-reported medical condition text information in an embodiment of the present invention. Detailed Implementation

[0046] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0047] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0048] Example 1

[0049] This invention provides a method for intelligent identification of mental illness based on multimodal data, such as... Figure 1 As shown, the intelligent identification method for mental illness based on multimodal data proposed in this invention includes the following steps:

[0050] S1. Obtain the subject's self-reported medical condition text information, ear image data, and impedance and / or temperature information of each designated acupoint in the concha and helix regions of the subject.

[0051] The ear has certain correspondences with other parts of the body. Observing the shape, color, and luster of the ear can effectively assist in the diagnosis of mental illnesses, such as cognitive decline associated with depression and the continuous spectrum of dementia (subjective cognitive decline, mild cognitive impairment, mild to moderate dementia). This invention proposes a multimodal data-based intelligent identification method for mental illnesses. By acquiring physiological characteristics of the concha region and multimodal data from the subject's self-report, combined with a deep learning model, it assists in the intelligent identification of mental illnesses such as depression and cognitive impairment, thereby effectively improving identification efficiency and accuracy. The multimodal data includes ear image data, impedance and / or temperature values ​​of 18 acupoints in the concha region and 13 acupoints in the antihelix region, and patient self-reported text information.

[0052] S2. Local feature extraction is performed on the ear image data to obtain an image feature vector. Specifically, a convolutional neural network (CNN) can be used to extract local features from the ear image data, and a pooling layer network can be used to reduce the feature dimension to obtain the image feature vector.

[0053] This embodiment employs a feature extraction network based on Convolutional Neural Networks (CNNs) and Transformers to extract features from multimodal data. The feature extraction network consists of three parts to extract features from different modalities of data respectively. The feature extraction network requires powerful computing capabilities, such as being equipped with a high-performance GPU, to accelerate the convolution and Transformer computation processes.

[0054] Specifically, for image data, a convolutional neural network (CNN) is used to perform convolution operations on images of blood vessels and dandruff in the ear to extract local features of the image, such as edges and textures. Pooling operations are then used to reduce the feature dimension to obtain the image feature vector.

[0055] S3. Convert the impedance and / or temperature information of each acupoint into vector form, and use the multi-head attention mechanism in the Transformer architecture to capture the correlation features between the impedance and / or temperature information of different acupoints to obtain the measured value feature vector.

[0056] Specifically, the method for converting the impedance and / or temperature information of each acupoint region into vector form is as follows: Each acupoint region corresponds to an impedance value Z and a temperature value T. Z and T are standardized (Z-score and normalization), and the Z and T of each acupoint region are combined into a two-dimensional vector [Z, T}, which is then arranged into a sequence according to the acupoint region order. Further, after converting the impedance and / or temperature information into vector form, the multi-head attention mechanism in Transformer is used to capture the correlation features between the impedance and / or temperature information of different partitions by calculating the attention weights between features at different locations, thus obtaining the corresponding feature vectors.

[0057] S4. Extract the semantic feature vector of the self-reported medical condition text. Specifically, the BERT language model can be used to encode the self-reported medical condition text and extract its semantic feature vector.

[0058] Specifically, for self-reported medical information, a pre-trained BERT language model based on Transformer is used to perform semantic understanding and feature extraction on the text, resulting in a semantic feature vector of the text.

[0059] S5. The image feature vector, measurement feature vector, and semantic feature vector are concatenated and fused to form a multimodal fusion feature vector. Specifically, the image feature vector, measurement feature vector, and semantic feature vector are aligned in feature dimensions, and then features from different modalities are concatenated and fused to form a multimodal fusion feature vector.

[0060] Specifically, the image branch uses a CNN to extract image features (output dimension 2048) and can randomly sample 180 region features; the measurement branch uses a fully connected layer to map impedance and temperature into a 768-dimensional embedding vector; and the text branch uses BERT to encode the patient's self-reported information into a 768-dimensional feature vector. By extracting features from multimodal data, representative features are extracted from different modalities, transforming multimodal data into a unified feature representation, facilitating subsequent model learning and classification. After completing feature extraction for all input data, each modal feature is embedded into a unified dimensional space (selectable as 768 dimensions), achieving feature dimension alignment, and successfully concatenating them into a multimodal fusion feature vector. This multimodal fusion feature vector serves as input to the subsequent classifier, effectively extracting key features from the multimodal data, enhancing the model's understanding and learning ability, and providing strong support for accurate diagnosis.

[0061] S6. Input the multimodal fusion feature vector into the pre-trained disease intelligent recognition model so that the disease intelligent recognition model can classify the input multimodal fusion feature vector according to the relationship between the multimodal fusion feature vector learned during the model training process and various categories of mental illnesses of different severity, and output the prediction result. The prediction result is a probability vector, which represents the probability that the sample belongs to various categories of mental illnesses of different severity and the normal state.

[0062] The intelligent identification method for mental illness based on multimodal data provided in this invention innovatively integrates three modalities of data: ear image features, impedance and temperature information of various acupoints, and patient self-report information. This provides a richer and more comprehensive information dimension for the auxiliary identification of mental illnesses. Furthermore, it utilizes an advanced deep learning model architecture to fuse and process the multimodal data. The model can automatically learn the correlations and complementary information between different modalities, thereby achieving refined classification and identification of mental illnesses such as depression (mild, moderate, and severe) and cognitive impairment (mild, moderate, and severe). This invention not only assists in identifying the presence or absence of mental illnesses but also distinguishes different degrees of severity, providing a more targeted basis for the formulation of clinical treatment plans and possessing significant clinical application value.

[0063] In this embodiment of the invention, the steps for obtaining impedance and / or temperature information of each designated acupoint in the concha and helix regions of the subject specifically include: using a pre-trained acupoint detection model to segment and detect acupoints in the ear image data of the subject to obtain the acupoint distribution information of the current subject's concha and helix regions; generating an acupoint measurement and positioning template for the subject based on the acupoint distribution information, and sending the acupoint measurement and positioning template to the ear image data acquisition device to display the acupoint measurement and positioning template in the video guidance box of the image acquisition interface of the acquisition device; guiding the sensor device to acquire the impedance and / or temperature values ​​of each designated acupoint in the concha and helix regions of the subject by matching and aligning the acupoint measurement and positioning template with the subject's ear image; and receiving the impedance and / or temperature values ​​of each designated acupoint in the concha and helix regions of the subject uploaded by the sensor device. The acupoint measurement and positioning template is an acupoint distribution image that combines 18 acupoints and key acupoints in the concha region and 13 acupoints and key acupoints in the antihelix region. The boundaries of the acupoints and the key feature points of the acupoints are marked with lines.

[0064] The acupoint detection model in this embodiment is trained using the YOLOv8 model. This model has been trained with a large amount of ear image data and can adapt to the differences in ear features of different individuals. It can accurately segment the boundary of each acupoint and the key feature points of the acupoint, so as to provide a precise acupoint location basis for subsequent impedance value and temperature value acquisition.

[0065] Specifically, an acquisition device equipped with an acupoint detection model is used to acquire images of the patient's ear, obtaining ear image data which is then input into the acupoint detection model. The acupoint detection model analyzes the acquired ear images, quickly and accurately identifying the acupoint distribution information in the concha and helix regions. Ear acupoints exist in two forms: acupoint sites and acupoint regions. The acupoint distribution information in this application includes 18 acupoints and key acupoint sites in the concha and 13 acupoints and key acupoint sites in the helix region. After accurate segmentation and detection of the acupoint distribution information are completed and the acquisition device is ready, the acquisition device maps the acupoint measurement and positioning template onto the video guidance frame, such as... Figure 2As shown, an intuitive guided interface is displayed on the acquisition device screen. Medical staff or patients, following the guidance, accurately place the impedance sensor and / or temperature sensor in the corresponding acupoint area and key acupoint location. The sensors are connected to the acquisition device, and once in place, the device begins to collect the impedance and / or temperature values ​​of each acupoint in real time. The data is transmitted in real time to the device's internal storage module. The impedance and / or temperature values ​​of each acupoint area are sorted and stored according to the collection time and / or acupoint number and displayed on the screen. The video guidance interface of the acquisition device is clear and easy to operate. The impedance and temperature sensors are accurate and sensitive, possessing stable data transmission and storage capabilities. This invention combines acupoint information with video guidance, using video guidance to accurately place sensors in the corresponding acupoints to precisely acquire data and achieve real-time transmission, storage, and display. This operation improves data acquisition accuracy, reduces operational errors, provides real-time data feedback for easy adjustment, ensures data quality, and is simple to operate and easy to promote, providing convenience for large-scale data acquisition and clinical diagnosis. After completing the acquisition of impedance and temperature values ​​for all target acupoints and verifying that the data is normal, the process ends. Real-time impedance and / or temperature data for 18 acupoints in the concha and 13 acupoints in the antihelix are obtained, enabling real-time and accurate acquisition of impedance and / or temperature, thus improving data validity and diagnostic accuracy.

[0066] This invention provides a video-guided data acquisition method that maps 18 acupoints in the concha, 13 acupoints in the antihelix, and key acupoints onto a video guidance frame, enabling real-time acquisition of impedance and / or temperature. In traditional data acquisition processes, impedance and / or temperature acquisition is often disconnected from acupoint location, resulting in inaccurate data mapping to specific acupoints and affecting data validity and diagnostic accuracy. This invention combines acupoint information with video guidance, providing intuitive and accurate guidance for data acquisition. During the acquisition process, medical personnel or the patient simply need to accurately place the impedance and temperature sensors in the corresponding acupoint areas or points according to the instructions on the video guidance frame to achieve real-time data acquisition. This method not only improves the accuracy of data acquisition but also effectively reduces data errors caused by improper operation. Simultaneously, the real-time acquired data can be promptly fed back to the operator, facilitating timely detection and adjustment of abnormal data, further ensuring data quality. Furthermore, the video-guided data acquisition method is simple to operate and easy to promote, enabling application in various medical environments and facilitating large-scale data acquisition and clinical diagnosis.

[0067] In one specific embodiment, the acquisition device can be a multimodal data acquisition device, which integrates video guidance function, impedance sensor, temperature sensor, etc.

[0068] In this embodiment of the invention, the acupoint detection model includes a convolutional neural network backbone, a Neck network, and a Head network. The convolutional neural network backbone is used to extract features at different scales and levels from the ear image data of the test subject to obtain multi-scale image feature information. The Neck network is used to fuse the multi-scale image feature information extracted by the convolutional neural network backbone. The Head network is used to predict the acupoint region bounding box and key point coordinates of the fused features using a decoupled head structure to obtain the acupoint distribution information of the concha and helix regions of the current test subject.

[0069] Specifically, the convolutional neural network backbone of the acupoint detection model borrows from the CSPDarkNet structure. Through a series of convolutional operations, it performs preliminary feature extraction on the input ear image data, providing a foundation for subsequent more refined feature processing and analysis. The convolutional neural network backbone includes the C2f module and the SPPF module. In the preliminary feature extraction process, components such as the C2f module and the SPPF module are used to capture feature maps at various levels to obtain rich image feature information. The main function of the C2f module is to capture feature maps at various levels, enabling further processing and integration of the input features. It can effectively extract feature information at different scales and levels, making the model's feature representation of ear images more comprehensive and in-depth, and helping to improve the model's ability to identify acupoint regions and key points. The C2f module includes the Bottleneck module. As a component of the C2f module, the Bottleneck module plays a role in optimizing feature extraction in the entire network structure. Through specific convolutional kernel settings and connection methods, it processes the input features, which can reduce the amount of computation while retaining important feature information as much as possible, avoiding the loss of key information during feature extraction, thereby improving the model's running efficiency and the accuracy of feature extraction. The SPPF module is used for feature extraction and enhanced expressive power. It further processes and enhances the input features, effectively expanding the receptive field and quickly gathering contextual information, thereby enhancing the model's ability to express features of acupoint regions and key points in ear images, making the model perform better in detection and segmentation tasks. After multi-scale features are extracted by the backbone network, these features are refined and fused by the Neck network using an optimized path aggregation network (PANet). Effective integration and transmission of features at different scales allows the model to fully utilize feature information at different levels, improving feature quality and usability, and providing a more accurate feature foundation for subsequent predictions. The Head network is responsible for generating the final prediction. Changing the coupled head to a decoupled head structure and employing a specific strategy allows the Head module to process and analyze input features more flexibly, thus more accurately predicting the bounding boxes of acupoint regions and the coordinates of key points. Through the decoupled head structure, the model can perform more refined processing on different tasks, improving the accuracy and reliability of predictions. The input image is a patient's ear. Through a series of operations including convolution, C2f modules, upsampling, and concat, different layers output feature maps of different sizes. Finally, the Detect layer performs detection on the output of specific layers to reduce systematic errors and detect all objects in the image. The model treats object detection as a regression problem, directly predicting the bounding box and keypoint coordinates of acupoint regions through a neural network, and extracting image features using operations such as convolution, pooling, and feature fusion.This operation provides a precise acupoint location foundation for subsequent data collection, ensuring that the data corresponds to the accurate acupoint regions, providing a reliable basis for extracting relevant features, and improving the data quality and performance of the diagnostic model. The process ends after successfully outputting the segmentation results and key point location information for 18 acupoint areas in the concha and 13 acupoint areas in the helix, achieving automated and high-precision acupoint segmentation and key point detection, improving data collection efficiency and accuracy, reducing manual positioning errors, and adapting to differences in ear characteristics among different individuals.

[0070] This invention introduces the YOLOv8 model for segmentation and detection of 18 acupoints in the concha and 13 acupoints in the antihelix. The YOLOv8 model possesses highly efficient target detection and segmentation capabilities, enabling rapid and accurate identification of acupoint regions and locations. Accurate acupoint localization is crucial in medical data acquisition. Traditional manual localization methods are not only inefficient but also susceptible to subjective factors, leading to inaccurate positioning. The application of the YOLOv8 model achieves automated, high-precision acupoint segmentation and key point detection, significantly improving the efficiency and accuracy of data acquisition. Through training on a large amount of ear image data, the YOLOv8 model can adapt to differences in ear features among individuals, accurately segmenting the boundaries of each acupoint and marking key feature points for acupoint locations. These precise segmentation and detection results provide a reliable foundation for subsequent extraction of features such as blood vessels and skin flakes from ear images, ensuring data consistency and accuracy, thereby improving the performance of the entire diagnostic model.

[0071] In this embodiment of the invention, before extracting local features from the ear image data, the ear image data is subjected to size unification and pixel value normalization; before converting the impedance and / or temperature information of each acupoint area into vector form, the impedance and / or temperature information of each acupoint area is Z-score normalized; and before extracting the semantic feature vector of the self-reported medical condition text information, the self-reported medical condition text information is segmented to be truncated to a preset maximum length.

[0072] Specifically, in one embodiment, when new test data needs to be input into the disease intelligent recognition model for identification, the data input interface program needs to preprocess the acquired multimodal data, including ear image data, impedance and temperature information of 18 acupoints in the concha and 13 acupoints in the antihelix, and patient self-report text information. The ear image data is preprocessed according to a predefined size and format, including adjusting the image size to a uniform size and normalizing pixel values. The impedance and temperature information are numerically standardized, such as normalizing to a specific range. Figure 3As shown, the acquired impedance and temperature information is previewed and checked to determine if there are missing or outlier values. If so, missing values ​​are imputed with the mean, and outliers are filtered using the 3σ principle (three sigma criterion). The feature mean u and standard deviation σ are calculated for the processed impedance and temperature information. The standardization calculation method is: (feature value - feature mean u) / standard deviation σ. The standardized impedance and temperature data are output, converted into fixed-dimensional impedance and temperature data vectors. The patient's self-reported text information is processed through word segmentation, encoding, etc., such as... Figure 3 As shown, the acquired text information undergoes text preprocessing, including removing special characters and converting to lowercase. A word segmentation tool like jieba can be used for word segmentation. Then, the BERT language model is used for word vector encoding, including word embedding, positional embedding, and segmentation embedding, to obtain the encoded self-narrative text data. The processed data is then input into the corresponding feature extraction network. Further, for ear image data, a convolutional neural network (CNN) can be used to extract local features from the ear image data. For impedance and temperature information converted to vector form, the multi-head attention mechanism in the Transformer can be used to capture the correlation features between impedance and temperature information in different regions by calculating the attention weights between features at different locations. Figure 5 As shown, impedance and temperature data vectors are input into the multi-head attention mechanism input layer to generate a query vector Q, a key vector K, and a value vector V. Attention weights between features at different locations are calculated using a self-attention mechanism, and combined with the value vector V to obtain a weighted value vector. The multi-head attention results are then concatenated and fused through a fully connected layer to output a correlated feature vector. For self-reported medical information, such as… Figure 6 As shown, the encoded self-reported text data (word embedding, position embedding, segmentation embedding) is input into the BERT language model to obtain the semantic feature vector of the self-reported medical condition text information.

[0073] The data input interface program needs to adapt to the format requirements of different modal data and possess the computational resources required for data preprocessing. Based on the characteristics of different modal data, appropriate preprocessing algorithms are employed to convert the data into a format suitable for model processing, ensuring data consistency and compatibility. This operation preprocesses the raw multimodal data, enabling it to be effectively received and processed by the model, providing a foundation for subsequent feature extraction and model training. The process ends after all input data has completed preprocessing and successfully transmitted to the feature extraction module, resulting in preprocessed multimodal data, which is then output to the next module in a unified format, improving data usability, reducing model training complexity, and enhancing model training efficiency.

[0074] In this embodiment of the invention, after forming the multimodal fusion feature vector, the method further includes: using a multimodal joint modeling network composed of serially configured multi-layer Transformer blocks to iteratively transform the multimodal fusion feature vector multiple times to obtain a deeply fused and feature-enhanced multimodal fusion feature vector; wherein each Transformer block contains a multi-head attention mechanism and a feedforward neural network, the multi-head attention mechanism performs attention calculation on the feature vector input to the current Transformer block to capture the correlation between different modal features, and the feedforward neural network performs feature transformation and enhancement, and the processed feature vector is passed to the next Transformer block. After multi-layer iterative processing, a multimodal fusion feature vector with deep fusion and feature enhancement is output.

[0075] Specifically, after the feature extraction network outputs a multimodal fused feature vector, a multi-layer Transformer block begins feature enhancement processing. The multimodal fused feature vector is input into the multi-layer Transformer block, each layer containing a multi-head attention mechanism, a feedforward neural network, residual connections, and a layer normalization network. Optionally, a 12-layer Transformer block with 12 layers of multi-head self-attention mechanisms is used, allowing cross-modal interaction between signals and images, with text generated autoregressively. In the multi-head attention mechanism, multiple attention calculations are performed on the input feature vector to capture the complex relationships between features of different modalities. Then, feature transformation and enhancement are performed through a feedforward neural network, residual connections, and a layer normalization network. The processed feature vector is then passed to the next Transformer block. After multi-layer processing, a vector that has undergone deep fusion and feature enhancement is output. The multi-layer Transformer block has a large computational load, requiring high-performance computing devices, such as multi-GPU parallel computing, to support fast computation. The multi-head attention mechanism in the Transformer block can focus on different aspects of the input features in parallel. The feedforward neural network further transforms and processes the attention outputs using non-linear methods, continuously uncovering deep connections between features through multi-layer stacking. This operation performs deep fusion and feature enhancement on multimodal fusion features, further exploring the complex relationships between different modal features and improving the model's expressive power and classification performance. The multimodal fusion feature vector is processed by all preset Transformer layers, outputting a deeply processed and fused feature vector for subsequent classification operations. This enhances the model's understanding and processing capabilities of multimodal data, strengthens its robustness, and helps improve diagnostic accuracy.

[0076] In this embodiment of the invention, after the multi-layer Transformer blocks complete feature processing and output a multimodal fusion feature vector that has undergone deep fusion and feature enhancement, the pre-built intelligent disease identification model, i.e., the fully connected neural network classifier, begins to work. The feature vector processed by the multi-layer Transformer blocks is input into the intelligent disease identification model. The model performs nonlinear transformations on the input features through multiple hidden layers and finally outputs the prediction result through the output layer. This embodiment uses depression and cognitive impairment as examples of diseases to be predicted; that is, the prediction result is a probability vector, representing the probability that the sample belongs to depression (mild), depression (moderate), depression (severe), cognitive impairment (mild), cognitive impairment (moderate), cognitive impairment (severe), and normal state, respectively. The fully connected neural network performs linear transformations on the input features through a weight matrix, and then introduces nonlinearity through an activation function, continuously combining and transforming the features, and finally outputting the classification result. This operation classifies the input multimodal data based on the relationship between the multimodal features learned by the model and the disease, outputs diagnostic results, and realizes the judgment of depression and cognitive impairment. The system completes the classification calculation of the input feature vector and outputs the predicted probability vector, thus ending the process. This yields probability vectors indicating whether the predicted sample belongs to different categories (mild depression, moderate depression, severe depression, mild cognitive impairment, moderate cognitive impairment, severe cognitive impairment, and normal state), enabling classification and diagnosis of multimodal data and providing diagnostic reference for doctors. During model training, the cross-entropy loss function is used to measure the difference between the predicted result and the true label. By minimizing this loss function, the model parameters are continuously adjusted. The Adam optimizer is selected, combining the advantages of Adagrad and RMSProp optimizers to adaptively adjust the learning rate of each parameter. Based on the calculated cross-entropy loss, the gradient is calculated, and the update step size of the model parameters is adjusted according to the first and second moment estimates of the gradient, ensuring that the model parameters are updated in the direction of decreasing loss function, thereby gradually optimizing the model's performance.

[0077] In this embodiment of the invention, the training steps of the intelligent disease recognition model include:

[0078] First, obtain labeled sample data. Each sample data includes the patient's self-reported medical condition text information, ear image data, impedance and / or temperature information of each designated acupoint in the patient's concha and helix regions, and the patient's disease label.

[0079] Second, the ear image data in each sample data is subjected to size unification and pixel value normalization to obtain preprocessed image data samples. The impedance and / or temperature information in each sample data is Z-score normalized to obtain preprocessed measurement data samples. The self-reported medical condition text information in each sample data is segmented to be truncated to the preset maximum length to obtain preprocessed text data samples.

[0080] In this embodiment, multimodal data acquisition is required beforehand, with the patient authorizing data use, to obtain data for model training. After the patient completes the clinical examination and authorizes data use, the acquisition system (ear acquisition device, impedance sensor, electronic medical record system) begins operation. First, the information collector's self-report is converted into text content. For the text content, annotators, based on descriptions of symptoms related to depression and cognitive impairment, diagnostic results, and other information, mark key statements related to the disease in the information collector's self-report text to obtain the condition self-report text information and label the corresponding disease type. For ear image data, annotators, based on the morphological and texture features in the ear images, combined with medical knowledge, mark the disease-related features and corresponding disease categories for each region. For impedance and / or temperature information table data, annotators, based on the data's numerical range and trends, combined with clinical experience, mark whether the data is abnormal and its association with depression or cognitive impairment. Annotators must possess professional medical knowledge and use pre-set auxiliary annotation software to perform sample annotation. The auxiliary annotation software has a user-friendly annotation interface. Based on medical knowledge and experience, annotators manually judge and label data from different modalities. The annotation software provides functions such as data display, annotation recording, and storage, providing labeled training data for subsequent model training. This enables the model to learn the correspondence between different modalities and diseases, a crucial step for accurate model classification and diagnosis. The process ends after all cleaned data has been annotated and passed quality review, resulting in labeled multimodal medical data samples. Each data sample contains data from different modalities and corresponding disease labels, providing high-quality supervised data for model training and helping to improve the model's diagnostic accuracy for depression and cognitive impairment. Simultaneously, grayscale normalization is performed on the 18-segment images of the concha region (selectable from 0-255), and the size is uniform (selectable from 512×512). Impedance values ​​(Ω) and temperature values ​​(°C) are standardized using Z-scores. Patient self-reported text is segmented and truncated to the maximum length (selectable from 256 words) to eliminate data heterogeneity and ensure input consistency.

[0081] Third, a convolutional neural network (CNN) is used to extract local features from each image data sample, and a pooling layer network is used to reduce the feature dimension to obtain image feature vector samples. The impedance and / or temperature information of each acupoint in each measurement data sample is converted into vector form, and the multi-head attention mechanism in the Transformer architecture is used to capture the correlation features between the impedance and / or temperature information of different acupoints to obtain measurement value feature vector samples. The BERT language model is used to encode each text data sample to extract semantic feature vector samples. The image feature vector samples, measurement value feature vector samples, and semantic feature vector samples corresponding to each sample data are aligned in feature dimension, and feature vector samples from different modalities are concatenated and fused through feature concatenation to form multimodal fused feature vector samples of each sample data.

[0082] Fourth, the multimodal fusion feature vector samples of each sample data and the disease labels corresponding to each sample are input into a preset fully connected neural network classifier for training of the intelligent disease recognition model;

[0083] Fifth, during model training, the cross-entropy loss function is used to measure the loss between the predicted result of each sample data and the real disease label, and the Adam optimizer is used to optimize the network parameters of the intelligent disease identification model to minimize the loss function and achieve model performance optimization.

[0084] In this embodiment, after the model is built, the training task is initiated, and the model training program begins processing the labeled multimodal medical data. The labeled multimodal medical data is divided into training, validation, and test sets according to a certain ratio, such as 70% for training, 15% for validation, and 15% for test. The training data is sequentially input into each module of the model, and forward propagation is performed according to the computational flow set in the model building process to obtain prediction results. Then, the cross-entropy loss is calculated based on the prediction results and the true labels, and the Adam optimizer is used to calculate the gradient based on the loss value and update the model parameters. During training, the model is periodically validated using validation set data, and metrics such as loss and accuracy on the validation set are calculated to observe the model's training effect. Model training is performed on a server with powerful computing capabilities, equipped with multiple high-performance GPUs to accelerate the training process, and sufficient memory to store training data and model parameters. Based on the training principles of deep learning, the model learns the relationship between multimodal data and diseases through forward propagation to calculate prediction results, uses a loss function to measure prediction error, adjusts model parameters through the optimizer, and iterates through continuous training. This operation enables the model to learn effective feature representations and classification decision boundaries from a large amount of labeled data, continuously optimizing the model's performance and improving its diagnostic capabilities for depression and cognitive impairment. Training stops when the model's loss on the validation set no longer decreases or decreases very slowly (e.g., the loss decrease is less than a set threshold, such as 0.001, over multiple training epochs), or when the model's accuracy on the validation set reaches a stable level (e.g., accuracy fluctuations are less than a set threshold, such as 0.01, over multiple training epochs). This results in a well-trained intelligent disease recognition model, also known as the Earvill model. This model possesses the ability to assist in the diagnosis of depression and cognitive impairment using multimodal data from the concha of the ear. Training a model that can effectively process multimodal data and accurately identify diseases provides a powerful auxiliary tool for actual clinical diagnosis.

[0085] Furthermore, after model training is complete, or if it is found in practical applications that the model's diagnostic accuracy needs further improvement, the trained EarViLL model can be optimized using model optimization algorithms. First, cross-validation is performed on the model. The training set data is further divided into multiple subsets, and 5-fold cross-validation is performed, dividing the training set into 5 subsets. Four subsets are used for training each time, and one subset is used for validation, repeated multiple times to comprehensively evaluate the model's performance. Based on the cross-validation results, the model's hyperparameters are adjusted, such as the number of Transformer layers, the number of neurons in the hidden layers of fully connected neural networks, and the learning rate. Then, the model is retrained and validated, and the performance of the model under different hyperparameter settings is compared to select the optimal hyperparameter combination. Furthermore, the amount of training data can be increased by collecting more multimodal medical data from the concha region of the ear to update the sample data. Data labeling, model training, and validation are then performed again, and changes in model performance are observed. The model optimization process is conducted in an environment with sufficient computing power and storage space to record and analyze a large amount of experimental data. Cross-validation, by repeatedly splitting the training and validation sets, more comprehensively evaluates the model's generalization ability; hyperparameter tuning explores the optimal model configuration by changing the model's structure and training parameters; increasing the amount of training data utilizes more sample information, enabling the model to learn more comprehensive features and relationships. Model optimization operations can further improve the model's classification accuracy and generalization ability, allowing the model to better adapt to different application scenarios.

[0086] The present invention provides a method, device, and medium for intelligent identification of mental illness based on multimodal data. By innovatively integrating three modalities of data—ear image features, impedance and temperature information of various acupoints, and patient self-report information—it offers a richer and more comprehensive information dimension for the auxiliary identification of mental illness. Furthermore, it utilizes an advanced deep learning model architecture to fuse and process the multimodal data. The model can automatically learn the correlations and complementary information between different modalities, thereby achieving refined classification and identification of mental illnesses such as depression (mild, moderate, and severe) and cognitive impairment (mild, moderate, and severe). This invention not only assists in identifying the presence or absence of mental illness but also distinguishes different degrees of severity, providing a more targeted basis for the formulation of clinical treatment plans and possessing significant clinical application value.

[0087] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0088] Example 2

[0089] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described intelligent identification method for mental illness based on multimodal data. For example... Figure 1 Steps S1-S6 are shown.

[0090] In the specific implementation process of Embodiment 2, you can refer to Embodiment 1, and it has the corresponding technical effects.

[0091] Example 3

[0092] This invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps described in the above-described embodiments of the intelligent identification method for mental illness based on multimodal data. For example... Figure 1 Steps S1-S6 are shown.

[0093] In the specific implementation process of Example 3, reference can be made to Example 1, and it has the corresponding technical effects.

[0094] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, any of the claimed embodiments can be used in any combination.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent identification of mental illness based on multimodal data, characterized in that, The method includes: The process involves acquiring the subject's self-reported medical condition text information, ear image data, and impedance and / or temperature information of various designated acupoints within the concha and helix regions. Acquiring the impedance and / or temperature information of these acupoints includes: using a pre-trained acupoint detection model to segment and detect acupoints in the subject's ear image data, obtaining the acupoint distribution information in the concha and helix regions; generating an acupoint measurement and positioning template based on the acupoint distribution information, and sending the template to the ear image data acquisition device to display it within the video guidance frame of the acquisition device's image acquisition interface; and guiding the sensor device to acquire the impedance and / or temperature values ​​of various designated acupoints within the concha and helix regions by matching and aligning the template with the subject's ear image; and receiving the impedance and / or temperature values ​​of these designated acupoints uploaded by the sensor device. The acupoint detection model is trained using the YOLOv8 model. Local feature extraction is performed on the ear image data to obtain the image feature vector; The impedance and / or temperature information of each acupoint is converted into vector form, and the multi-head attention mechanism in the Transformer architecture is used to capture the correlation features between the impedance and / or temperature information of different acupoints to obtain the measured value feature vector. Extract semantic feature vectors from the patient's self-reported medical information; Image feature vectors, measurement feature vectors, and semantic feature vectors are concatenated and fused to form a multimodal fusion feature vector; The multimodal fusion feature vector is input into a pre-trained intelligent disease identification model. The model then classifies the input multimodal fusion feature vector based on the relationship between the multimodal fusion feature vector learned during the model training process and various categories of mental illnesses of different severity. The model outputs a prediction result, which is a probability vector representing the probability that the sample belongs to various categories of mental illnesses of different severity and the normal state. The training steps of the intelligent disease recognition model include: acquiring labeled sample data, each sample data including patient's self-reported medical condition text information, ear image data, impedance and / or temperature information of each designated acupoint in the patient's concha and helix regions, and the patient's disease label; preprocessing each sample data to obtain preprocessed data samples; aligning the feature dimensions of the image feature vector samples, measurement value feature vector samples, and semantic feature vector samples corresponding to each preprocessed sample data, and concatenating and fusing feature vector samples from different modalities through feature concatenation to form multimodal fused feature vector samples of each sample data; inputting the multimodal fused feature vector samples of each sample data and the corresponding disease labels of each sample into a preset fully connected neural network classifier for training the intelligent disease recognition model; during the model training process, the cross-entropy loss function is used to measure the loss value between the prediction result of each sample data and the real disease label, and the Adam optimizer is used to optimize the network parameters of the intelligent disease recognition model to minimize the loss function and achieve model performance optimization.

2. The method according to claim 1, characterized in that, The method further includes: Before performing local feature extraction on the ear image data, the ear image data is subjected to size unification and pixel value normalization operations; Before converting the impedance and / or temperature information of each cavity region into vector form, Z-score normalization is performed on the impedance and / or temperature information of each cavity region; and Before extracting the semantic feature vector of the self-reported medical condition text, the self-reported medical condition text is segmented to be truncated to the preset maximum length.

3. The method according to claim 1, characterized in that, After forming the multimodal fusion feature vector, the method further includes: A multimodal joint modeling network composed of serially configured multi-layer Transformer blocks is used to iteratively transform the multimodal fusion feature vector multiple times to obtain a multimodal fusion feature vector with deep fusion and feature enhancement. Each Transformer block contains a multi-head attention mechanism and a feedforward neural network. The multi-head attention mechanism performs attention calculations on the feature vector input to the current Transformer block to capture the correlation between features of different modalities. The feedforward neural network performs feature transformation and enhancement, and passes the processed feature vector to the next Transformer block. After multiple iterations, the output is a multi-modal fused feature vector that has undergone deep fusion and feature enhancement.

4. The method according to claim 1, characterized in that, The impedance and / or temperature values ​​of each designated acupoint area are sorted according to the collection time and / or acupoint number.

5. The method according to claim 1, characterized in that, The acupoint detection model includes a convolutional neural network backbone, a Neck network, and a Head network; The convolutional neural network backbone is used to extract features at different scales and levels from the ear image data of the test subject in order to obtain multi-scale image feature information; Neck network is used to perform feature fusion on multi-scale image feature information extracted by the backbone network of convolutional neural network; The Head network is used to predict the bounding boxes and key point coordinates of acupoint regions from the fused features using a decoupled head structure, thereby obtaining the acupoint distribution information of the concha and helix regions of the current subject.

6. The method according to claim 1, characterized in that, The process of preprocessing each sample data to obtain preprocessed data samples includes performing size unification and pixel value normalization operations on the ear image data in each sample data to obtain preprocessed image data samples, performing Z-score normalization on the impedance and / or temperature information in each sample data to obtain preprocessed measurement data samples, and performing word segmentation processing on the self-reported medical condition text information in each sample data to truncate it to a preset maximum length to obtain preprocessed text data samples. Image feature vector samples are obtained as follows: local features are extracted from each image data sample using a convolutional neural network (CNN), and the feature dimension is reduced by a pooling layer network to obtain image feature vector samples. The measured value feature vector samples are obtained by converting the impedance and / or temperature information of each cavity region in each measured data sample into vector form, and using the multi-head attention mechanism in the Transformer architecture to capture the correlation features between the impedance and / or temperature information of different cavity regions, thus obtaining the measured value feature vector samples. The semantic feature vector samples are obtained by encoding each text data sample using the BERT language model and extracting the semantic feature vector samples from the text data samples.

7. The method according to claim 4, characterized in that, The impedance and / or temperature information for the specified acupoints includes the impedance and / or temperature values ​​for 18 acupoints in the concha and 13 acupoints in the antihelix.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor; when executed by the processor, the computer program implements the steps of the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1-7.

Images