Multi-mode interactive intelligent ward system

Abstract

A multi-mode interaction intelligent ward system comprises a voice recognition module, a touch screen module, a gesture recognition module, a face recognition module, a natural language processing module, a computer vision module, a sensor fusion module, a data encryption and access control module and an emotion recognition module. The system also supports a plurality of input modes such as touch, gesture and face recognition, so that the patients with language barrier or mobility difficulty can conveniently interact with the system, the user groups are greatly expanded, and the universality and usability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of smart wards, and in particular to a multimodal interactive smart ward system. Background Art

[0002] With the continuous advancement of medical informatization, the traditional medical service model can no longer meet the needs of modern medicine. As a new medical service model, the multimodal interactive smart ward system realizes the natural interaction between patients and the medical system through natural language processing, computer vision, sensor fusion and other technologies, and improves the efficiency and quality of medical services.

[0003] The existing multimodal interactive smart ward system mainly includes input modes such as voice recognition and touch screen, and output modes such as voice feedback, visual display, and vibration feedback. These technologies can realize the interaction between patients and medical systems. For example, patients can query medical information, make appointments for examinations, and obtain health advice through voice commands, and medical staff can quickly access patient information, record medical records, and obtain real-time monitoring data through voice commands.

[0004] Although the existing multimodal interactive smart ward system has improved the efficiency and quality of medical services to a certain extent, there are still some problems and limitations. First, the existing system mainly relies on voice recognition technology, and cannot provide effective interaction methods for some patients with language barriers or mobility difficulties. Secondly, the existing system has certain limitations in understanding user intentions and may not be able to accurately understand the patient's complex instructions or questions. Finally, the existing system has certain problems in data privacy and security, and patients' personal information and health data may be accessed or leaked without authorization. Summary of the invention

[0005] 1. Technical issues to be resolved In view of the deficiencies in the prior art, the present invention provides a multimodal interactive smart ward system.

[0006] (II) Technical solution To achieve the above object, the present invention provides the following technical solution: a multimodal interactive smart ward system, characterized in that it includes a speech recognition module, a touch screen module, a gesture recognition module, a facial recognition module, a natural language processing module, a computer vision module, a sensor fusion module, a data encryption and access control module, and an emotion recognition module, and specifically also includes the following steps: Step 1: The speech recognition module collects the patient's speech input through a microphone and converts it into text data; Step 2: The natural language processing module performs semantic analysis on the text data and extracts the user's intention; Step 3: The computer vision module collects the patient's gestures and facial images through a camera and converts them into digital image data; Step 4: The gesture recognition module and the facial recognition module process the digital image data to extract the user's gestures and facial features; Step 5: The sensor fusion module fuses the output data of the voice recognition module, touch screen module, gesture recognition module and facial recognition module to extract a more comprehensive user intention; Step 6: The data encryption and access control module encrypts user data to ensure data security. This module uses the AES-256 encryption algorithm; Step 7: The emotion recognition module analyzes user data, identifies the user's emotional state, identifies and analyzes the patient's emotional state in real time, and provides corresponding mental health support; Step 8: Based on the user's intention and emotional state, the system interacts with the user through voice feedback, visual display, and vibration feedback.

[0007] Furthermore, the present invention is improved in that the sampling rate of the speech recognition module is 8kHz and the sampling accuracy is 16 bits.

[0008] Furthermore, the present invention is improved in that the natural language processing module adopts a deep learning algorithm and can use a pre-trained model BERT-Base and a fine-tuned model BiLSTM-CRF.

[0009] Furthermore, the present invention is improved in that the resolution of the computer vision module is 1080p and the frame rate is 30fps.

[0010] Furthermore, the present invention is improved in that the gesture recognition module and the facial recognition module adopt a convolutional neural network algorithm and can use a pre-trained model ResNet-50.

[0011] Furthermore, the present invention is improved in that the emotion recognition module adopts a deep learning algorithm and can use a pre-trained model VGG-Face.

[0012] (III) Beneficial effects Compared with the prior art, the present invention provides a multimodal interactive smart ward system, which has the following beneficial effects: 1. Multimodal interaction: This system supports not only voice input, but also multiple input methods such as touch, gesture and facial recognition, so that patients with language barriers or limited mobility can interact with the system conveniently, greatly expanding the user group and improving the universality and usability of the system; 2. More efficient natural language processing: This system uses advanced natural language processing technology to more accurately understand patients' complex instructions and questions, reduce the risk of misunderstanding and misoperation, improve the system's response speed and accuracy, and enhance the user experience; 3. Enhanced data privacy and security: This system uses encryption and permission control technologies during data transmission and storage to ensure that patients' personal information and health data will not be accessed or leaked without authorization, thus enhancing the data privacy and security of the system; 4. Personalized service: This system can identify the patient's identity through facial recognition technology, provide personalized health management and medical services, and improve the pertinence and effectiveness of medical services; 5. Emotional support: This system uses emotion recognition technology to analyze the patient's emotional state and provide corresponding mental health support, increasing emotional care and improving the patient's mental health level; 6. Improve medical care efficiency: This system provides medical staff with convenient operation methods, such as voice commands to access patient information, record medical records, obtain real-time monitoring data, etc., which greatly reduces the workload of medical staff, allowing them to focus more on patient care and improve work efficiency; 7. Promote health management: This system helps patients better manage their own health through intuitive interactive methods, improves patients' self-health management capabilities, and helps prevent the occurrence and progression of diseases. In general, compared with existing technologies, this technical solution has higher interactivity, accuracy, security and personalization, greatly improves the efficiency and quality of medical services, and has significant advantages. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] First embodiment: A multimodal interactive smart ward system of the present invention is characterized by comprising a speech recognition module, a touch screen module, a gesture recognition module, a facial recognition module, a natural language processing module, a computer vision module, a sensor fusion module, a data encryption and access control module and an emotion recognition module, and specifically also comprising the following steps: Step 1: The speech recognition module collects the patient's speech input through a microphone and converts it into text data; Step 2: The natural language processing module performs semantic analysis on the text data and extracts the user's intention; Step 3: The computer vision module collects the patient's gestures and facial images through a camera and converts them into digital image data; Step 4: The gesture recognition module and the facial recognition module process the digital image data to extract the user's gestures and facial features; Step 5: The sensor fusion module fuses the output data of the voice recognition module, touch screen module, gesture recognition module and facial recognition module to extract a more comprehensive user intention; Step 6: The data encryption and access control module encrypts user data to ensure data security. This module uses the AES-256 encryption algorithm; Step 7: The emotion recognition module analyzes user data, identifies the user's emotional state, identifies and analyzes the patient's emotional state in real time, and provides corresponding mental health support; Step 8: Based on the user's intention and emotional state, the system interacts with the user through voice feedback, visual display, vibration feedback, etc. The above steps are run on the server. Through these steps, a multimodal interactive smart ward system can be realized to improve the patient experience and the work efficiency of medical staff.

[0015] Furthermore, the sampling rate of the speech recognition module is 8kHz and the sampling accuracy is 16 bits.

[0016] Furthermore, the natural language processing module adopts a deep learning algorithm and can use the pre-trained model BERT-Base and the fine-tuned model BiLSTM-CRF.

[0017] Furthermore, the resolution of the computer vision module is 1080p and the frame rate is 30fps.

[0018] Furthermore, the gesture recognition module and the facial recognition module adopt a convolutional neural network algorithm and can use a pre-trained model ResNet-50.

[0019] Furthermore, the emotion recognition module adopts a deep learning algorithm and can use the pre-trained model VGG-Face.

[0020] This embodiment has the following advantages: 1. Multimodal interaction: This system supports not only voice input, but also multiple input methods such as touch, gesture and facial recognition, so that patients with language barriers or limited mobility can interact with the system conveniently, greatly expanding the user group and improving the universality and usability of the system; 2. More efficient natural language processing: This system uses advanced natural language processing technology to more accurately understand patients' complex instructions and questions, reduce the risk of misunderstanding and misoperation, improve the system's response speed and accuracy, and enhance the user experience; 3. Enhanced data privacy and security: This system uses encryption and permission control technologies during data transmission and storage to ensure that patients' personal information and health data will not be accessed or leaked without authorization, thus enhancing the data privacy and security of the system; 4. Personalized service: This system can identify the patient's identity through facial recognition technology, provide personalized health management and medical services, and improve the pertinence and effectiveness of medical services; 5. Emotional support: This system uses emotion recognition technology to analyze the patient's emotional state and provide corresponding mental health support, increasing emotional care and improving the patient's mental health level; 6. Improve medical care efficiency: This system provides medical staff with convenient operation methods, such as voice commands to access patient information, record medical records, obtain real-time monitoring data, etc., which greatly reduces the workload of medical staff, allowing them to focus more on patient care and improve work efficiency; 7. Promote health management: This system helps patients better manage their own health through intuitive interactive methods, improves patients' self-health management capabilities, and helps prevent the occurrence and progression of diseases. In general, compared with existing technologies, this technical solution has higher interactivity, accuracy, security and personalization, greatly improves the efficiency and quality of medical services, and has significant advantages.

[0021] Second embodiment: 1. Multimodal interaction technology: By introducing multiple input and output modes such as voice recognition, touch screen, gesture recognition and facial recognition, patients and medical staff can interact with the system in different ways, improving the universality and accessibility of the system.

[0022] 2. Natural language processing technology: Through natural language processing technology, the system can understand and process the patient's voice input, provide accurate feedback, improve the system's ability to understand user intentions, and can more accurately understand and respond to patients' needs.

[0023] 3. Computer vision technology: Through computer vision technology, the system can perform gesture recognition and facial recognition, so that patients with language barriers or limited mobility can also interact with the system conveniently, improving the universality and accessibility of the system. Using this technology in the medical field can significantly improve the quality of medical care services.

[0024] 4. Sensor fusion technology: Through sensor fusion technology, combining multiple sensor data, the system's ability to understand user intentions is improved, and it can more accurately understand and respond to patients' needs.

[0025] 5. Emotion recognition technology: Through emotion recognition technology, the system can analyze the patient's emotional state and provide corresponding mental health support, thereby improving the patient's experience and satisfaction.

[0026] In the description of this article, it should be noted that relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0027] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. A multimodal interactive smart ward system, characterized by: It includes a speech recognition module, a touch screen module, a gesture recognition module, a facial recognition module, a natural language processing module, a computer vision module, a sensor fusion module, a data encryption and access control module, and an emotion recognition module, and specifically includes the following steps: Step 1: The speech recognition module collects the patient's speech input through a microphone and converts it into text data; Step 2: The natural language processing module performs semantic analysis on the text data and extracts the user's intention; Step 3: The computer vision module collects the patient's gestures and facial images through a camera and converts them into digital image data; Step 4: The gesture recognition module and the facial recognition module process the digital image data to extract the user's gestures and facial features; Step 5: The sensor fusion module fuses the output data of the voice recognition module, touch screen module, gesture recognition module and facial recognition module to extract a more comprehensive user intention; Step 6: The data encryption and access control module encrypts user data to ensure data security. This module uses the AES-256 encryption algorithm; Step 7: The emotion recognition module analyzes user data, identifies the user's emotional state, identifies and analyzes the patient's emotional state in real time, and provides corresponding mental health support; Step 8: Based on the user's intention and emotional state, the system interacts with the user through voice feedback, visual display, and vibration feedback.

2. A multimodal interactive smart ward system according to claim 1, characterized in that: The sampling rate of the speech recognition module is 8kHz and the sampling accuracy is 16 bits.

3. A multimodal interactive smart ward system according to claim 2, characterized in that: The natural language processing module adopts a deep learning algorithm and can use the pre-trained model BERT-Base and the fine-tuned model BiLSTM-CRF.

4. A multimodal interactive smart ward system according to claim 3, characterized in that: The resolution of the computer vision module is 1080p and the frame rate is 30fps.

5. A multimodal interactive smart ward system according to claim 4, characterized in that: The gesture recognition module and the facial recognition module adopt a convolutional neural network algorithm and can use the pre-trained model ResNet-50.

6. A multimodal interactive smart ward system according to claim 5, characterized in that: The emotion recognition module adopts a deep learning algorithm and can use the pre-trained model VGG-Face.