A multimodal perception and analysis system of patient behavior based on deep learning

Abstract

The present invention relates to a patient behavior multimodal perception and analysis system based on deep learning, including a data acquisition unit, a patient body posture recognition unit, a patient physiological signal recognition unit, a patient image information recognition unit, a patient voice information recognition unit and deep fusion unit; through the preprocessing and region-of-interest extraction and diagnosis of the collected multi-modal data such as patient posture, physiology, image and voice, the deep fusion unit adopts a multi-modal two-dimensional feature and three-dimensional feature fusion network structure, composed of 2D The deep learning network obtains the preliminary segmentation results, and obtains the patient behavior detection results based on the preliminary segmentation results through the 3D deep learning network. Compared with the prior art, the present invention achieves a more accurate assessment of patient behavior, accurately locates the lesion, significantly improves the prediction accuracy of the patient's pathological trend, and provides a strong basic guarantee for the implementation of scientific intervention in patient behavior and intelligent optimization of medical procedures. .

Description

technical field [0001] The invention relates to the field of patient behavior analysis, in particular to a deep learning-based multimodal perception and analysis system for patient behavior. Background technique [0002] With the continuous development of deep learning technology, in dealing with many single-modal perceptual machine learning tasks, deep neural networks have achieved great advantages and information processing effects compared with traditional information processing methods. For example, the proposal of cyclic neural network and recurrent neural network (RNN) has achieved extremely successful engineering promotion and application of medical diagnosis for the processing of sequence problems of patient medical record text information and voice information; the proposal of models such as AlexNet and ResNet, It even surpasses human performance in task processing in the field of patient behavior video information. [0003] Applying deep learning technology to the...

AI Technical Summary

Problems solved by technology

[0004] For complex medical scenarios such as emergency department, ICU, nursing, isolation ward or metabolic cabin, the traditional disadvantage of the multi-dimensional perception algorithm of patient behavior based on deep learning is that it cannot effectively perceive the fine behavior and fine-grainedness of patients. The compliance of behavior implementation cannot be accurately judged. At the same time, most hospitals and medical data centers are still in the stage of manual sample collection and analysis and automated single-mode analysis for the analysis and research of patient behavior, although some institutions have launched Perceptual analysis of multimodal data, but the lack of compatibility processing and consideration of multimodal data greatly restricts data analysis and research on patient behavior and next-step medical outcomes

[0005] Most of the existing deep learning methods are applied to image information processing of patients. The processing of multi-modal data still has defects such as complex network design, slow training speed, and unsolvable gradient problems, which cannot make good use of multi-modal information. Fusion can make single-modal heterogeneous features and multi-modal heterogeneous features complement each other in the dimension of temporal and spatial cues

For example, in the feature extraction based on image information, RGB expresses rich texture information, but it is easily disturbed by light intensity, and it is difficult to express timing clues; although the timing clues of human body pose inertial data obtained based on the 3D human motion capture system are relatively easy Extraction and expression, but it is difficult to express texture information, etc.

[0006] The above issues result in inaccurate placement of patient behavior samples, delays in assessment and treatment of patient behavior, or overdiagnosis and overtreatment

Images