Health state classification method, system and equipment based on multi-modal data and medium

By constructing a classification network model of multimodal feature extraction and fusion encoder, the existing models have solved the problem of large amount of parameters and insufficient performance capabilities in physical health recognition and classification, and efficient health status classification of multimodal data is achieved.

CN119046766AInactive Publication Date: 2024-11-29HANGZHOU HUAXI INTELLIGENT TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411483716.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-23
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When existing models identify and classify physical health, the number of parameters is large and the model performance ability is insufficient, making it difficult to effectively process multimodal data, and cannot meet the needs of health status classification.

Method used

Using a health state classification method based on multimodal data, a classification network model including a multimodal feature extraction module, a multimodal fusion encoder and a fully connected layer is constructed, and a convolutional neural network, Mel spectrogram and long and short-term memory are used. The network extracts features and fusions of features through multimodal attention submodules to reduce the amount of model parameters and improve feature learning ability.

Benefits of technology

It significantly improves the model's performance ability in complex tasks, improves the accuracy and efficiency of health status detection, identification and classification, and can effectively process multimodal data to meet the needs of health status classification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046766A_ABST
    Figure CN119046766A_ABST
Patent Text Reader

Abstract

The invention discloses a health state classification method, system and equipment based on multi-modal data and a medium, relates to the technical field of health state evaluation and classification, and aims to solve the problems that an existing model is large in parameter quantity during body health identification and classification, the expression ability of the model needs to be improved in a complex task, and the classification efficiency is poor. And the requirement of health state classification on the multi-modal data is difficult to meet is solved. According to a multi-modal fusion encoder of a classification network model, on the basis of a classic transformer model, an innovative multi-modal attention module is improved into multi-head attention of the transformer model to obtain a multi-modal attention sub-module, and a convolution sub-module is introduced between the multi-modal attention sub-module and a feedforward network. Therefore, the feature learning capability of a single encoder in the transformer can be remarkably improved, so that the number of the encoders in the transformer is reduced, and the model parameter quantity is reduced to the greatest extent on the premise of ensuring the model performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of health status assessment and classification, and relates to a health status classification method, system, device and medium based on multimodal data, and in particular to a health status classification method, system, device and medium based on multimodal self-attention feature fusion. Background Art

[0002] In recent years, with the improvement of living standards, people have paid more and more attention to their own health status. As for health status, most people now know their health status through physical examinations. To know the health status through physical examinations, first, professional doctors are needed to interpret the physical examination reports, which will greatly increase the workload of medical staff; second, the interpretation of physical examination reports requires high professionalism and rich medical knowledge reserves. With the rapid development of artificial intelligence technology in recent years, large models have demonstrated remarkable processing capabilities in many fields such as natural language processing, image recognition and data analysis, so more and more artificial intelligence technologies are applied to the intelligent interpretation of physical examination reports. These models usually have billions or even hundreds of billions of parameters and can handle and solve highly complex tasks.

[0003] The invention patent application with application number 202311080653.0 discloses a multimodal mental health assessment system and method based on interface-based emotional interaction, which uses the image features extracted by the convolutional neural network as the input of the long short-term memory neural network (Long short-term memory, abbreviated as LSTM) to capture the timing information of the entire image sequence, thereby obtaining the first representation corresponding to each time series image; repeating the above steps to obtain the second representation corresponding to the heart rate variability data and the third representation corresponding to the voice data through the convolutional neural network and the long short-term memory neural network; using the Transformer neural network model to fuse the first representation, the second representation and the third representation; classifying the fusion results to obtain the patient's anxiety probability, depression probability and stress probability in the time window t.

[0004] The invention patent application with application number 202310937135.X discloses a mental health status assessment system and assessment method, which uses the deep learning architecture of convolutional neural networks to perform emotion classification and recognition on user facial expression images; by extracting the sound features of speech signals and combining machine learning models, three types of emotional states in user voice are identified. The sound features include fundamental frequency, energy, and spectrum. The three types of emotional states include happiness, anger, and sadness; a deep neural network model of a recurrent neural network is used to perform sentiment classification on text and capture semantic and contextual information.

[0005] The above invention patent applications all use materials such as images, texts, and audios to achieve the assessment of mental health. However, in daily life, in addition to the need to assess mental health, it is also particularly important to detect and classify the health status of the body. If the above-mentioned assessment method is applied to the classification of physical health status, there is a lack of effective and unified feature representation when processing multimodal data such as images, audio, and text, and it is difficult for the model to learn high-level feature representations, resulting in a large number of parameters in the existing models when performing physical health identification and classification, and the model's performance in complex tasks needs to be improved, making it difficult to meet the needs of health status classification for multimodal data. Summary of the invention

[0006] The purpose of the present invention is to provide a health status classification method, system, device and medium based on multimodal data in order to solve the technical problems that the existing models have a large number of parameters when performing physical health identification and classification, the performance of the models in complex tasks needs to be improved, and it is difficult to meet the needs of health status classification of multimodal data.

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions: A health status classification method based on multimodal data comprises the following steps: Step S1, obtaining sample data; Acquire health status sample data, the health status sample data including image samples, audio samples and text samples; annotate the health status of the health status sample data to form label data; Step S2, constructing a classification network model; Construct a classification network model, which includes a multimodal feature extraction module, a multimodal fusion encoder, and a fully connected layer; the multimodal feature extraction module includes a convolutional neural network, a Mel spectrum map, and a long short-term memory network LSTM; the multimodal fusion encoder includes a multimodal attention submodule, a convolution submodule, a feedforward network, and a normalization layer; Step S3, training a classification network model; The health status sample data and label data in step S1 are input into the classification network model constructed in step S2. The image samples, audio samples and text samples are respectively extracted through convolutional neural networks, Mel spectrograms and long short-term memory networks (LSTM) to obtain image features, audio features and text features. The image features, audio features and text features are input into the multimodal attention submodule for fusion to obtain fusion features. The fusion features are sequentially passed through the convolution submodule, the feedforward network and the normalization layer to output unified features. The unified features are input into the fully connected layer, and the fully connected layer outputs the health warning level. Step S4, real-time health warning level; Obtain the health status data to be tested, the health status data including the image data to be tested, the audio data to be tested and the text data to be tested; input the health status data into the classification network model trained in step S3, and the classification network model outputs the health warning level.

[0008] Furthermore, in step S1, the acquired health status sample data is normalized, and the normalization calculation method is:

[0009] in, Represents the original data, represents the mean of the data, represents the standard deviation of the data, Represents normalized data.

[0010] Furthermore, in step S2, the multimodal attention submodule adjusts the weights of different modal features and obtains fused features, which are expressed as:

[0011] in, represents the dimension of the feature, , , Represents the self-attention mechanism corresponding to image, audio and text respectively , , Component value.

[0012] Further, in step S2, the convolution submodule includes a 1*1 convolution layer, a 3*3 depth convolution layer and a 1*1 convolution layer arranged in sequence.

[0013] Furthermore, in step S3, when training the classification network model, the cross entropy loss function is used to train the model, and the loss function is specifically:

[0014] in, Indicates samples, represents the total number of samples, Representation sample The predicted probability of Representation sample The real label.

[0015] A health status classification system based on multimodal data, comprising: The sample data acquisition module is used to acquire health status sample data, which includes image samples, audio samples and text samples; annotate the health status of the health status sample data to form label data; The classification network model building module is used to build a classification network model. The classification network model includes a multimodal feature extraction module, a multimodal fusion encoder and a fully connected layer; the multimodal feature extraction module includes a convolutional neural network, a Mel spectrum map and a long short-term memory network LSTM; the multimodal fusion encoder includes a multimodal attention submodule, a convolution submodule, a feedforward network and a normalization layer; The classification network model training module is used to input the health status sample data and label data in the sample data acquisition module into the classification network model constructed in the classification network model construction module. The image samples, audio samples and text samples are respectively extracted through the convolutional neural network, Mel spectrum map and long short-term memory network LSTM to obtain image features, audio features and text features. The image features, audio features and text features are input into the multimodal attention submodule for fusion to obtain fusion features. The fusion features are sequentially passed through the convolution submodule, feedforward network and normalization layer to output unified features; the unified features are input into the fully connected layer, and the fully connected layer outputs the health warning level; The real-time health warning level module is used to obtain the health status data to be tested, which includes the image data to be tested, the audio data to be tested and the text data to be tested; the health status data is input into the classification network model trained by the classification network model training module, and the classification network model outputs the health warning level.

[0016] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0017] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the above method.

[0018] The beneficial effects of the present invention are as follows: In the present invention, the multimodal fusion encoder of the classification network model includes a multimodal attention submodule, a convolution submodule, a feedforward network and a normalization layer. The multimodal fusion encoder is based on the classic transformer model, and the innovative multimodal attention submodule (muti-modal-self-attetion) is improved in the multi-head attention (Multi-Head Attention) of the transformer model to obtain a multimodal attention submodule (muti-modal-cross-transformer), and a convolution submodule is introduced between the multimodal attention submodule and the feedforward network, which can significantly improve the feature learning ability of a single encoder in the transformer, thereby reducing the number of encoders in the transformer, so as to minimize the model parameters while ensuring the model performance; in addition, the innovative multimodal fusion encoder can realize feature fusion internally, and can automatically learn high-level feature representations, can effectively process complex multimodal data, so that the performance of the model in complex tasks is greatly improved, and the accuracy and efficiency of health status detection, recognition and classification are improved, which can effectively meet the needs of health status classification of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 It is a structural diagram of the classification network model in the present invention; Figure 3 It is a structural schematic diagram of the multimodal feature extraction module in the present invention; Figure 4 It is a schematic diagram of the structure of the multimodal fusion encoder in the present invention; Figure 5 It is a schematic diagram of the multimodal attention submodule in the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0021] Therefore, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0022] Example 1 This embodiment provides a health status classification method based on multimodal data, which is used to detect and identify images (including patient facial images, medical test images (such as CT images, electrocardiograms, etc.)), audio (mainly patient self-report recordings of physical conditions, medical test recordings) and text (including patient self-report texts of physical conditions, test reports, etc.) to obtain the final health status classification results. Specific classification methods, such as Figure 1 As shown, the following steps are included: Step S1, obtaining sample data; Acquire health status sample data, which includes image samples, audio samples, and text samples; and annotate the health status of the health status sample data to form label data.

[0023] Among them, image samples include patient facial images, patient medical images (such as CT images, electrocardiograms and other medical images); audio samples include patients' self-reported health status recordings, medical test recordings (such as heartbeats, breathing and other medical auscultation sounds); text samples include patients' self-written texts on their health status, test reports and other issues. The above samples come from various tertiary-level A hospitals, physical examination centers and their corresponding patients.

[0024] After obtaining the above samples, professional doctors conducted multiple rounds of labeling to obtain label data. The label contains five levels of health status, namely health, sub-health, false health, illness, and serious illness. Among them, health: refers to the physical and mental state is in good condition, without disease, and can effectively cope with various needs of daily life; sub-health: a state between health and disease, manifested as some non-specific symptoms, such as fatigue, insomnia, etc., but has not yet reached the diagnostic criteria for the disease; false health: although there are no obvious symptoms, there may be problems such as abnormal blood lipids, high blood sugar, and blood pressure fluctuations. The psychological state is under high pressure for a long time, which may lead to psychological problems such as anxiety and depression; illness: including symptoms (such as fever, cough, pain) and signs (such as jaundice, increased heart rate); serious illness: a serious disease state that poses a greater threat to life, including serious abnormalities in body temperature, blood pressure, pulse, respiratory rate, etc., and dysfunction of multiple important organs such as the heart, lungs, liver, and kidneys.

[0025] After obtaining the above sample data, the obtained health status sample data is normalized. The normalization calculation method is:

[0026] in, Represents the original data, represents the mean of the data, represents the standard deviation of the data, Represents normalized data.

[0027] Step S2, constructing a classification network model; Construct a classification network model, which includes a multimodal feature extraction module, a multimodal fusion encoder and a fully connected layer; the multimodal feature extraction module includes a convolutional neural network, a Mel spectrum map and a long short-term memory network LSTM; the multimodal fusion encoder includes a multimodal attention submodule, a convolution submodule, a feedforward network and a normalization layer; specifically Figure 2 shown.

[0028] The specific structure of the multimodal feature extraction module is as follows: Figure 3 As shown in the figure, the convolutional neural network can use the existing convolutional network, and the high-dimensional image features in the image can be extracted through the convolutional neural network. The Mel spectrum can use the existing spectrum, and the audio features in the audio can be extracted through the spectrum. The long short-term memory network LSTM can also use the existing LSTM network, and the text features in the text can be extracted through the LSTM network.

[0029] The specific structure of the multimodal fusion encoder is as follows: Figure 4 As shown in the figure, the innovative multi-modal-self-attention submodule is improved into the multi-head attention of the transformer model to obtain a new multi-modal attention submodule, and a convolution submodule is introduced between the multi-modal attention submodule and the feedforward network. The convolution submodule includes a 1*1 convolution layer, a 3*3 deep convolution layer and a 1*1 convolution layer arranged in sequence. The feedforward network and normalization layer both use the feedforward network and normalization layer in the existing transformer model.

[0030] Image features, audio features and text features are used as the input of the multimodal attention sub-module. The output of the multimodal attention sub-module is jump-connected with the image features, audio features and text features and then used as the input of the convolution sub-module. The output of the convolution sub-module is jump-connected with the output of the multimodal attention sub-module and then input into the feedforward network. The output of the feedforward network is jump-connected with the output of the convolution sub-module and then input into the normalization layer. The normalization layer outputs unified features and then inputs into the fully connected layer. The fully connected layer outputs the health warning level.

[0031] like Figure 5As shown, this embodiment adopts a multi-modal attention submodule (muti-modal-self-attetion) to improve the input of the traditional attention method of the multi-head attention (Multi-Head Attention) module in the transformer model, and obtains a new multi-modal attention submodule (muti-modal-cross-transformer for fusion of high-dimensional features of multi-modal data). The multi-modal attention submodule adjusts the weights of different modal features and obtains fusion features. The fusion features are expressed as:

[0032] in, represents the dimension of the feature, , , Represents the self-attention mechanism corresponding to image, audio and text respectively , , Component value.

[0033] Step S3, training a classification network model; The health status sample data and label data in step S1 are input into the classification network model constructed in step S2. The image samples, audio samples and text samples are respectively subjected to feature extraction through convolutional neural networks, Mel spectrograms and long short-term memory networks (LSTM) to obtain image features, audio features and text features. The image features, audio features and text features are input into the multimodal attention submodule for fusion to obtain fusion features. The fusion features are sequentially passed through the convolution submodule, the feedforward network and the normalization layer to output unified features. The unified features are input into the fully connected layer, and the fully connected layer outputs the health warning level.

[0034] When training the classification network model, the cross entropy loss function is used to train the model. The specific loss function is:

[0035] in, Indicates samples, represents the total number of samples, Representation sample The predicted probability of Representation sample The real label.

[0036] Step S4, real-time health warning level; The health status data to be tested is obtained, and the health status data includes the image data to be tested, the audio data to be tested, and the text data to be tested; the health status data is input into the classification network model trained in step S3, and the classification network model outputs the health warning level.

[0037] Example 2 This embodiment provides a health status classification system based on multimodal data, which is used to detect and identify images (including patient facial images, medical test images (such as CT images, electrocardiograms, etc.)), audio (mainly patient self-report recordings of physical conditions, medical test recordings) and text (including patient self-report texts of physical conditions, test reports, etc.) to obtain the final health status classification results. Specifically including: The sample data acquisition module is used to acquire health status sample data, which includes image samples, audio samples and text samples; and to annotate the health status of the health status sample data to form label data.

[0038] Among them, image samples include patient facial images, patient medical images (such as CT images, electrocardiograms and other medical images); audio samples include patients' self-reported health status recordings, medical test recordings (such as heartbeats, breathing and other medical auscultation sounds); text samples include patients' self-written texts on their health status, test reports and other issues. The above samples come from various tertiary-level A hospitals, physical examination centers and their corresponding patients.

[0039] After obtaining the above samples, professional doctors conducted multiple rounds of labeling to obtain label data. The label contains five levels of health status, namely health, sub-health, false health, illness, and serious illness. Among them, health: refers to the physical and mental state is in good condition, without disease, and can effectively cope with various needs of daily life; sub-health: a state between health and disease, manifested as some non-specific symptoms, such as fatigue, insomnia, etc., but has not yet reached the diagnostic criteria for the disease; false health: although there are no obvious symptoms, there may be problems such as abnormal blood lipids, high blood sugar, and blood pressure fluctuations. The psychological state is under high pressure for a long time, which may lead to psychological problems such as anxiety and depression; illness: including symptoms (such as fever, cough, pain) and signs (such as jaundice, increased heart rate); serious illness: a serious disease state that poses a greater threat to life, including serious abnormalities in body temperature, blood pressure, pulse, respiratory rate, etc., and dysfunction of multiple important organs such as the heart, lungs, liver, and kidneys.

[0040] After obtaining the above sample data, the obtained health status sample data is normalized. The normalization calculation method is:

[0041] in, Represents the original data, represents the mean of the data, represents the standard deviation of the data, Represents normalized data.

[0042] The classification network model building module is used to build a classification network model. The classification network model includes a multimodal feature extraction module, a multimodal fusion encoder and a fully connected layer; the multimodal feature extraction module includes a convolutional neural network, a Mel spectrum map and a long short-term memory network LSTM; the multimodal fusion encoder includes a multimodal attention submodule, a convolution submodule, a feedforward network and a normalization layer; specifically, Figure 2 shown.

[0043] The specific structure of the multimodal feature extraction module is as follows: Figure 3 As shown in the figure, the convolutional neural network can use the existing convolutional network, and the high-dimensional image features in the image can be extracted through the convolutional neural network. The Mel spectrum can use the existing spectrum, and the audio features in the audio can be extracted through the spectrum. The long short-term memory network LSTM can also use the existing LSTM network, and the text features in the text can be extracted through the LSTM network.

[0044] The specific structure of the multimodal fusion encoder is as follows: Figure 4 As shown in the figure, the innovative multi-modal-self-attention submodule is improved into the multi-head attention of the transformer model to obtain a new multi-modal attention submodule, and a convolution submodule is introduced between the multi-modal attention submodule and the feedforward network. The convolution submodule includes a 1*1 convolution layer, a 3*3 deep convolution layer and a 1*1 convolution layer arranged in sequence. The feedforward network and normalization layer both use the feedforward network and normalization layer in the existing transformer model.

[0045] Image features, audio features and text features are used as the input of the multimodal attention sub-module. The output of the multimodal attention sub-module is jump-connected with the image features, audio features and text features and then used as the input of the convolution sub-module. The output of the convolution sub-module is jump-connected with the output of the multimodal attention sub-module and then input into the feedforward network. The output of the feedforward network is jump-connected with the output of the convolution sub-module and then input into the normalization layer. The normalization layer outputs unified features and then inputs into the fully connected layer. The fully connected layer outputs the health warning level.

[0046] like Figure 5As shown, this embodiment adopts a multi-modal attention submodule (muti-modal-self-attetion) to improve the input of the traditional attention method of the multi-head attention (Multi-Head Attention) module in the transformer model, and obtains a new multi-modal attention submodule (muti-modal-cross-transformer for fusion of high-dimensional features of multi-modal data). The multi-modal attention submodule adjusts the weights of different modal features and obtains fusion features. The fusion features are expressed as:

[0047] in, represents the dimension of the feature, , , Represents the self-attention mechanism corresponding to image, audio and text respectively , , Component value.

[0048] Classification network model training module for The health status sample data and label data in the sample data acquisition module are input into the classification network model constructed in the classification network model construction module. The image samples, audio samples and text samples are respectively extracted through the convolutional neural network, Mel spectrum graph and long short-term memory network LSTM to obtain image features, audio features and text features. The image features, audio features and text features are input into the multimodal attention submodule for fusion to obtain fusion features. The fusion features pass through the convolution submodule, feedforward network and normalization layer in turn to output unified features; the unified features are input into the fully connected layer, and the fully connected layer outputs the health warning level.

[0049] When training the classification network model, the cross entropy loss function is used to train the model. The specific loss function is:

[0050] in, Indicates samples, represents the total number of samples, Representation sample The predicted probability of Representation sample The real label.

[0051] The real-time health warning level module is used to obtain the health status data to be tested, which includes the image data to be tested, the audio data to be tested and the text data to be tested; the health status data is input into the classification network model trained by the classification network model training module, and the classification network model outputs the health warning level.

[0052] Example 3 A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of a health status classification method based on multimodal data.

[0053] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0054] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the health status classification method based on multimodal data, etc. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0055] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the health status classification method based on multimodal data.

[0056] Example 4 A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the steps of a health status classification method based on multimodal data.

[0057] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor performs the steps of the health status classification method based on multimodal data as described above.

[0058] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the health status classification method based on multimodal data described in the embodiment of the present application.

Claims

1. A health status classification method based on multimodal data, characterized in that: The following steps are involved: Step S1, obtaining sample data; Acquire health status sample data, the health status sample data including image samples, audio samples and text samples; annotate the health status of the health status sample data to form label data; Step S2, constructing a classification network model; Construct a classification network model, which includes a multimodal feature extraction module, a multimodal fusion encoder, and a fully connected layer; the multimodal feature extraction module includes a convolutional neural network, a Mel spectrum map, and a long short-term memory network LSTM; the multimodal fusion encoder includes a multimodal attention submodule, a convolution submodule, a feedforward network, and a normalization layer; Step S3, training a classification network model; The health status sample data and label data in step S1 are input into the classification network model constructed in step S2. The image samples, audio samples and text samples are respectively extracted through convolutional neural networks, Mel spectrograms and long short-term memory networks (LSTM) to obtain image features, audio features and text features. The image features, audio features and text features are input into the multimodal attention submodule for fusion to obtain fusion features. The fusion features are sequentially passed through the convolution submodule, the feedforward network and the normalization layer to output unified features. The unified features are input into the fully connected layer, and the fully connected layer outputs the health warning level; Step S4, real-time health warning level; The health status data to be tested is obtained, and the health status data includes the image data to be tested, the audio data to be tested, and the text data to be tested; the health status data is input into the classification network model trained in step S3, and the classification network model outputs the health warning level.

2. The health status classification method based on multimodal data according to claim 1, characterized in that: In step S1, the acquired health status sample data is normalized, and the normalization calculation method is: in, Represents the original data, represents the mean of the data, represents the standard deviation of the data, Represents normalized data.

3. The health status classification method based on multimodal data according to claim 1, characterized in that: In step S2, the multimodal attention submodule adjusts the weights of different modal features and obtains fused features. The fused features are expressed as: in, represents the dimension of the feature, , , Represents the self-attention mechanism corresponding to image, audio and text respectively , , Component value.

4. The health status classification method based on multimodal data according to claim 1, characterized in that: In step S2, the convolution submodule includes a 1*1 convolution layer, a 3*3 depth convolution layer and a 1*1 convolution layer arranged in sequence.

5. The health status classification method based on multimodal data according to claim 1, characterized in that: In step S3, when training the classification network model, the cross entropy loss function is used to train the model. The specific loss function is: in, Indicates samples, represents the total number of samples, Representation sample The predicted probability of Representation sample The real label.

6. A health status classification system based on multimodal data, characterized in that: include: The sample data acquisition module is used to acquire health status sample data, which includes image samples, audio samples and text samples; annotate the health status of the health status sample data to form label data; The classification network model building module is used to build the classification network model. The classification network model includes a multimodal feature extraction module, a multimodal fusion encoder and a fully connected layer; the multimodal feature extraction module includes a convolutional neural network, a Mel spectrum map and a long short-term memory network LSTM; the multimodal fusion encoder includes a multimodal attention submodule, a convolution submodule, a feedforward network and a normalization layer; The classification network model training module is used to input the health status sample data and label data in the sample data acquisition module into the classification network model constructed in the classification network model construction module. The image samples, audio samples and text samples are respectively extracted through the convolutional neural network, Mel spectrum map and long short-term memory network LSTM to obtain image features, audio features and text features. The image features, audio features and text features are input into the multimodal attention submodule for fusion to obtain fusion features. The fusion features are sequentially passed through the convolution submodule, feedforward network and normalization layer to output unified features. The unified features are input into the fully connected layer, and the fully connected layer outputs the health warning level; The real-time health warning level module is used to obtain the health status data to be tested, which includes the image data to be tested, the audio data to be tested and the text data to be tested; the health status data is input into the classification network model trained by the classification network model training module, and the classification network model outputs the health warning level.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Multi-modal mental health assessment system and method based on interface type emotional interaction

    CN116807476A

  • Mental health state assessment system and assessment method

    CN117012389A

  • Model-based data processing method and device, computer equipment and storage medium

    CN112908473A

  • Heart failure diagnosis auxiliary method based on multi-modal data fusion

    CN116451068A

  • Bimodal fusion interpretable breast cancer automatic intelligent diagnosis method

    CN117911775A

Cited By

  • Intelligent mental disease identification method and device based on multi-modal data and medium

    CN120496866A