Patient behavior multi-modal perception and analysis system based on deep learning

Abstract

The invention relates to a patient behavior multi-modal perception and analysis system based on deep learning. The patient behavior multi-modal perception and analysis system comprises a data acquisition unit, a patient human body posture recognition unit, a patient physiological signal recognition unit, a patient image information recognition unit, a patient voice information recognition unit anda deep fusion unit, preprocessing and interested area extraction and diagnosis are achieved on collected multi-modal data such as postures, physiology, images and voice of patients, a deep fusion unit adopts a multi-modal two-dimensional feature and three-dimensional feature fusion network structure, and a preliminary segmentation result is obtained through a 2D deep learning network; and a patient behavior detection result is acquired on the basis of the preliminary segmentation result through a 3D deep learning network. Compared with the prior art, the invention has the advantages that thepatient behavior is evaluated more accurately, the focus is accurately positioned, the accuracy of predicting the pathological trend of the patients are remarkably improved, and a powerful basic guarantee is provided for implementation of scientific intervention of the patient behavior and intelligent optimization means of a medical process.

Description

technical field [0001] The invention relates to the field of patient behavior analysis, in particular to a deep learning-based multimodal perception and analysis system for patient behavior. Background technique [0002] With the continuous development of deep learning technology, in dealing with many single-modal perceptual machine learning tasks, deep neural networks have achieved great advantages and information processing effects compared with traditional information processing methods. For example, the proposal of cyclic neural network and recurrent neural network (RNN) has achieved extremely successful engineering promotion and application of medical diagnosis for the processing of sequence problems of patient medical record text information and voice information; the proposal of models such as AlexNet and ResNet, It even surpasses human performance in task processing in the field of patient behavior video information. [0003] Applying deep learning technology to the...

AI Technical Summary

Problems solved by technology

[0004] For complex medical scenarios such as emergency department, ICU, nursing, isolation ward or metabolic cabin, the traditional disadvantage of the multi-dimensional perception algorithm of patient behavior based on deep learning is that it cannot effectively perceive the fine behavior and fine-grainedness of patients. The compliance of behavior implementation cannot be accurately judged. At the same time, most hospitals and medical data centers are still in the stage of manual sample collection and analysis and automated single-mode analysis for the analysis and research of patient behavior, although some institutions have launched Perceptual analysis of multimodal data, but the lack of compatibility processing and consideration of multimodal data greatly restricts data analysis and research on patient behavior and next-step medical outcomes

[0005] Most of the existing deep learning methods are applied to image information processing of patients. The processing of multi-modal data still has defects such as complex network design, slow training speed, and unsolvable gradient problems, which cannot make good use of multi-modal information. Fusion can make single-modal heterogeneous features and multi-modal heterogeneous features complement each other in the dimension of temporal and spatial cues

For example, in the feature extraction based on image information, RGB expresses rich texture information, but it is easily disturbed by light intensity, and it is difficult to express timing clues; although the timing clues of human body pose inertial data obtained based on the 3D human motion capture system are relatively easy Extraction and expression, but it is difficult to express texture information, etc.

[0006] The above issues result in inaccurate placement of patient behavior samples, delays in assessment and treatment of patient behavior, or overdiagnosis and overtreatment

Images